mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-20 00:11:50 +00:00
75973b3f0c
The e2e coverage denominator was a hand-written set of per-prefix YAML rows, one per behavior. That is self-fulfilling: the row and its covering test land in the same PR, so a wanted-but-untested behavior never surfaces as a gap and coverage becomes a number we grant ourselves. Each LLM row also restated its own id, since subject_endpoint, route, capability, and streaming just re-split the cell id that schema.py already validates. The conversational core (chat_completions, messages, responses) is now generated in product_surface.py from the typed vocabularies in schema.py crossed with the supports_* capability metadata in model_prices_and_context_window.json, the same file the proxy ships. A flagged capability is emitted for a route only when a model on that route advertises the matching flag, so support added in that json grows the denominator with no registry edit. The Anthropic-format messages surface is the Claude Code compatibility matrix, so its capabilities are the CLI feature set and are intentionally ungated. The id-duplicating fields are gone from the schema. One Cell dataclass replaces the discriminated union; its module is parsed from the id and the structural facets are recovered on demand by parse_llm_id. The human fields (tier, source, rationale, fail_before_fix, supported) move to a single overlay.yaml keyed by cell id, owner-gated via CODEOWNERS, so ordinary test PRs only add a @pytest.mark.covers marker. A generated cell with no overlay row defaults to P2, so a newly grown surface shows up as an uncovered gap rather than vanishing. The collector keeps its output contracts (text, JSON, Prometheus, Loki); the only addition is a text-only route-table drift warning when the vocabulary enumerates an LLM endpoint the live proxy no longer serves. Generation covers the conversational core; the non-core LLM operations and the behavior modules are carried by the overlay for now, and wiring their own product-surface sources into generation is the documented follow-up. |
||
|---|---|---|
| .. | ||
| access_control | ||
| batches | ||
| claude_code | ||
| coverage_registry | ||
| grafana | ||
| llm_translation | ||
| logging | ||
| management | ||
| quota_management | ||
| router | ||
| bob_the_builder.py | ||
| CLAUDE.md | ||
| conftest.py | ||
| CONTRIBUTING.md | ||
| docker-compose.yml | ||
| e2e_config.py | ||
| e2e_gateway.py | ||
| e2e_http.py | ||
| e2e_result_reporter.py | ||
| lifecycle.py | ||
| models.py | ||
| pytest.ini | ||
| transport.py | ||