mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-05 02:41:56 +00:00
10462eddaf
* test(e2e): harness fixes for long_context, complexity router, UI, and unit coverage Point long_context_1m at 1M-capable models, harden complexity-smart-router registration and spend-log assertions, fix key models dropdown selectors, and add gateway/lifecycle/transport and claude_code unit tests * test(e2e): harden remaining stage failures in harness Register complexity-smart-router via create_model + callable probe, fix create-key UI navigation race, retry management writes and budget ALB 502s, mark Vertex count_tokens N/A when unsupported, and tighten tool_search model lists for Azure/Bedrock capability gaps * test(e2e): drop claude_code and harness unit tests from this PR Keep management, router, budget, and shared conftest harness fixes only * test(e2e): restore E2E_RESULT pytest_runtest_makereport hook Accidentally dropped in an earlier harness commit; Grafana status history depends on these structured log lines * test(e2e): drop management control-plane write retries Transient 500 retries do not fix the underlying control plane failures * test(e2e): skip stage-red claude_code cells; fix multi-window budget latency Mark the twelve failing claude_code matrix cells skip until product/config lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of Claude so the reset wait stays under ALB target idle timeout rather than masking awselb 502s * test(e2e): require exactly one LLM-tier spend row for complexity router Keep alias membership for compose vs stage model names, but assert len(served) == 1 so a leaked classifier sub-call cannot pass. Also pin LIT-4521 skip and align LIT-4522/23/24 skip reasons * test(e2e): harden router callable probe and multi-window budget exhaustion _router_is_callable treated any non-success chat whose body lacked "Invalid model name" as callable, so an unpropagated probe key (401), a generic 502, or a connection reset let the session proceed and hit real "Invalid model name" failures inside the tests. Require a Success outcome instead; the reload-race 400 and every infra/auth error now correctly read as not-callable. The multi-window budget test capped the tight window at 3e-6, which gpt-5.5 exhausts on the first call but a cheaper CHEAP_OPENAI_MODEL might not within the 20-call loop, turning a reset test into a spurious "window never enforced" failure. Drop the tight cap to 1e-9 so the first billed call exhausts it regardless of model price; the roomy 1m window stays at 1.0 and never blocks. * test(e2e): use a tradeoff-decision prompt for the complexity router classifier "Is P equal to NP?" reads to the LLM classifier as a short yes/no question, so gpt-5.5 classified it SIMPLE and the request routed to the openai backend, which made the test fail even though the classifier was running. The tier definitions key on what the request demands, not how hard the answer is, and a short direct question maps to SIMPLE regardless of subject. Swap in "Should I pay off my mortgage early or invest the extra money instead?". It carries none of the heuristic scorer's reasoning/technical/code keywords and stays short, so heuristic scoring still lands SIMPLE (openai), but the LLM reads it as a decision that has to weigh tradeoffs and lands it above SIMPLE, which the config routes to anthropic. Any non-SIMPLE tier serves anthropic, so the classifier only has to avoid SIMPLE for the test to distinguish a real classifier run from the heuristic fallback.
82 lines
3.7 KiB
Markdown
82 lines
3.7 KiB
Markdown
# e2e coverage registry
|
|
|
|
This directory is the **denominator** for e2e test coverage: the set of behaviors we
|
|
want covered, one row per behavior, checked into the repo so coverage is a number we
|
|
can track instead of a guess. It implements the plan in the "E2E Coverage Tracking"
|
|
note; the naming grammar lives in `tests/e2e/CLAUDE.md`.
|
|
|
|
## The model
|
|
|
|
A **cell** is one customer-noticeable behavior a single e2e test can assert pass/fail
|
|
on, for example `llm.chat_completions.bedrock_converse.tool_use.stream.works`. Cells are
|
|
grouped `module > feature > test`, with LLM cells split into `Core LLMs` and
|
|
`Non-Core LLMs` for dashboarding. Each cell carries a tier (P0/P1/P2), a source, and a
|
|
`fail_before_fix` flag.
|
|
|
|
The rows live in per-prefix YAML files (`llm_*.yaml`, `mgmt.yaml`, `mcp.yaml`,
|
|
`reliability.yaml`, `quota_management.yaml`, `logging.yaml`, `guardrail.yaml`,
|
|
`other.yaml`) and validate against
|
|
the discriminated union in `schema.py`, so an LLM row cannot carry a guardrail field and
|
|
vice versa. `llm` rows with `subject_endpoint` of `chat_completions`, `messages`, or
|
|
`responses` roll up to `Core LLMs`; all other LLM endpoints roll up to `Non-Core LLMs`.
|
|
LLM endpoint, route, and capability values are typed in `schema.py`, so new taxonomy
|
|
values require an explicit schema change. `logging` and `guardrail` are two id-prefixes
|
|
that roll up into the single `Logging & Guardrails` dashboard module.
|
|
|
|
A test declares what it covers with a marker:
|
|
|
|
```python
|
|
@pytest.mark.covers("llm.chat_completions.openai.tool_use.stream.works")
|
|
def test_openai_streaming_tool_calls(self) -> None:
|
|
...
|
|
```
|
|
|
|
## The number
|
|
|
|
`collector.py` diffs the registry against those markers and reports coverage per module.
|
|
It is static: a collect-only pass reads the markers, so it runs no test and needs no live
|
|
proxy. Whether a covered cell currently passes or fails is a separate, live concern.
|
|
|
|
```
|
|
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector
|
|
```
|
|
|
|
Use `--format loki` after the e2e pytest run in the same Kubernetes job/pod to print
|
|
structured stdout lines for Loki:
|
|
|
|
```
|
|
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --format loki --strict
|
|
```
|
|
|
|
This emits exactly one `COVERAGE_TOTAL` line and one `COVERAGE_MODULE` line per module
|
|
in `MODULE_ORDER`, in that order. Loki uses log-safe `module=` labels from
|
|
`LOKI_MODULE_LABELS` (`core_llms`, `management_ui`, etc.) so existing JSON and
|
|
Prometheus consumers keep their human-readable module names unchanged.
|
|
|
|
The headline is overall coverage. The collector also lists markers that point at ids
|
|
not in the registry, so a typo or an unenumerated behavior surfaces instead of being
|
|
silently dropped.
|
|
|
|
Use strict mode in CI once existing draft markers are reconciled:
|
|
|
|
```
|
|
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --strict
|
|
```
|
|
|
|
Strict mode exits non-zero on `@pytest.mark.covers(...)` ids that are not checked into
|
|
the registry. Add `--fail-on-collection-errors` when the job should also fail on pytest
|
|
collection errors.
|
|
|
|
## Status: this is a draft for review
|
|
|
|
The cells were enumerated from the codebase and the tiers are a first proposal. Known
|
|
things to settle before treating the set as final:
|
|
|
|
- tiers are proposed, not signed off; 125 P0 is a lot to prove fail-before-fix, so P0 may
|
|
want tightening
|
|
- a few cells need a support check or a prune (for example `llm.embeddings.anthropic.*`
|
|
and `reliability.perf.throughput.under_slo`)
|
|
- auth is covered in two places (`other.auth.*` and the mgmt authz assertions); the
|
|
boundary needs a decision, and the auth cluster may deserve promotion to its own module
|
|
- the P2 "niche" cells each stand in for a large tail of integrations/providers by design,
|
|
so the denominator is deliberately P0-weighted rather than a full inventory
|