mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
75973b3f0c
The e2e coverage denominator was a hand-written set of per-prefix YAML rows, one per behavior. That is self-fulfilling: the row and its covering test land in the same PR, so a wanted-but-untested behavior never surfaces as a gap and coverage becomes a number we grant ourselves. Each LLM row also restated its own id, since subject_endpoint, route, capability, and streaming just re-split the cell id that schema.py already validates. The conversational core (chat_completions, messages, responses) is now generated in product_surface.py from the typed vocabularies in schema.py crossed with the supports_* capability metadata in model_prices_and_context_window.json, the same file the proxy ships. A flagged capability is emitted for a route only when a model on that route advertises the matching flag, so support added in that json grows the denominator with no registry edit. The Anthropic-format messages surface is the Claude Code compatibility matrix, so its capabilities are the CLI feature set and are intentionally ungated. The id-duplicating fields are gone from the schema. One Cell dataclass replaces the discriminated union; its module is parsed from the id and the structural facets are recovered on demand by parse_llm_id. The human fields (tier, source, rationale, fail_before_fix, supported) move to a single overlay.yaml keyed by cell id, owner-gated via CODEOWNERS, so ordinary test PRs only add a @pytest.mark.covers marker. A generated cell with no overlay row defaults to P2, so a newly grown surface shows up as an uncovered gap rather than vanishing. The collector keeps its output contracts (text, JSON, Prometheus, Loki); the only addition is a text-only route-table drift warning when the vocabulary enumerates an LLM endpoint the live proxy no longer serves. Generation covers the conversational core; the non-core LLM operations and the behavior modules are carried by the overlay for now, and wiring their own product-surface sources into generation is the documented follow-up. |
||
|---|---|---|
| .. | ||
| actions | ||
| codeql | ||
| ISSUE_TEMPLATE | ||
| observatory | ||
| screenshots | ||
| scripts | ||
| workflows | ||
| CODEOWNERS | ||
| dependabot.yaml | ||
| deploy-on-aws.png | ||
| deploy-on-gcp.png | ||
| deploy-to-aws.png | ||
| FUNDING.yml | ||
| pull_request_template.md | ||
| template.yaml | ||
