litellm/tests/e2e/coverage_registry
yujonglee c25c669167
feat(auth): decide key and JWT authentication in a Rust core (#51374)
gateway-auth-core gains plan/ and credential/, which plan and decide key and
JWT authentication on top of its route rules. The Python proxy now runs that
core through python-bridge with a PythonAuthHost, which replaces
handle_jwt.py and the key and JWT paths of user_api_key_auth.py.

The core defines Token, MasterKey and KeyHash, and config gains the JWT auth
settings. Route groups named in allowed_routes match as starlette's
compile_path compiles them, like _allowed_routes_check, and a team's
allowed and denied pass-through routes read as RouteLists. allow_user_auth,
allow_requests_on_db_unavailable and enforce_rbac read through python-compat's
IsTrue, StrBool and Truthy adapters, and StrBool reads text with
flags::str_to_bool.

Fixes that landed on main in the code this moves carry over: every user read
the core makes from a JWT hands the raw email_verified claim to the host,
which reads it with read_email_verified, the impersonation header applies
after a master key, cached admin, virtual key or mapped JWT key is accepted,
and the team member budget step uses the native budget decision.

A JWT email claim that is not a string reads as no email, the same as a
missing claim, instead of being stored as the number's text.

The native extension size cap rises from 55 MB to 65 MB, since the auth core
brings the Linux build to 56.85 MB.
2026-11-17 14:07:47 +09:00
..
__init__.py chore: consolidate CLAUDE.md into AGENTS.md 2026-09-19 02:30:35 +00:00
collector.py fix(e2e-changed): keep the gate off suites the stack cannot run 2026-09-05 21:03:50 -07:00
guardrail.yaml fix(guardrails)!: check Anthropic and Responses request tools in tool_permission pre_call (#41011) 2026-11-04 20:34:34 +00:00
llm_claude_code_compat.yaml test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
llm_conversational.yaml chore: move LiteLLM's configs and e2e tests off gpt-4o and gpt-4.1 before their 2027-01-14 shutdown (#51800) 2026-11-14 10:50:59 -08:00
llm_nonconversational.yaml feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
logging.yaml refactor(s3-log): move the rust-backed s3 logger from s3_v3 to s3_v2 (#51893) 2026-11-14 22:24:23 -08:00
management_cases.py test: enforce isolated actors and stop OIDC process groups 2026-09-12 13:49:49 -07:00
mcp.yaml refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) 2026-10-27 17:47:50 +00:00
mgmt.yaml test(e2e): repair fallback management test for config-owned fallbacks (#47023) 2026-10-17 01:27:39 +00:00
other.yaml feat(auth): decide key and JWT authentication in a Rust core (#51374) 2026-11-17 14:07:47 +09:00
quota_management.yaml chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
README.md chore: consolidate CLAUDE.md into AGENTS.md 2026-09-19 02:30:35 +00:00
registry.py ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
reliability.yaml test(e2e): assert the sibling-replica cooldown through the router (#44706) 2026-10-06 02:12:57 +00:00
schema.py feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00

e2e coverage registry

This directory is the denominator for e2e test coverage: the set of behaviors we want covered, one row per behavior, checked into the repo so coverage is a number we can track instead of a guess. It implements the plan in the "E2E Coverage Tracking" note; the naming grammar lives in tests/e2e/AGENTS.md.

The model

A cell is one customer-noticeable behavior a single e2e test can assert pass/fail on, for example llm.chat_completions.bedrock_converse.tool_use.stream.works. Cells are grouped module > feature > test, with LLM cells split into Core LLMs and Non-Core LLMs for dashboarding. Each cell carries a tier (P0/P1/P2), a source, and a fail_before_fix flag.

The rows live in per-prefix YAML files (llm_*.yaml, mgmt.yaml, mcp.yaml, reliability.yaml, quota_management.yaml, logging.yaml, guardrail.yaml, other.yaml) and validate against the discriminated union in schema.py, so an LLM row cannot carry a guardrail field and vice versa. llm rows with subject_endpoint of chat_completions, messages, or responses roll up to Core LLMs; all other LLM endpoints roll up to Non-Core LLMs. LLM endpoint, route, and capability values are typed in schema.py, so new taxonomy values require an explicit schema change. logging and guardrail are two id-prefixes that roll up into the single Logging & Guardrails dashboard module.

A test declares what it covers with a marker:

@pytest.mark.covers("llm.chat_completions.openai.tool_use.stream.works")
def test_openai_streaming_tool_calls(self) -> None:
    ...

The number

collector.py diffs the registry against those markers and reports coverage per module. It is static: a collect-only pass reads the markers, so it runs no test and needs no live proxy. Whether a covered cell currently passes or fails is a separate, live concern.

A skipped test asserts nothing, so its markers do not count. A cell is covered only when at least one test pytest would actually run declares it; a cell claimed by both a live test and a skipped one stays covered. Skip state comes from pytest's own evaluator, so skip and skipif resolve exactly as they do in the e2e run, which also means a skipif on an absent credential makes that cell uncovered in the environments where the test cannot run. Cells left uncovered this way are listed under the headline (and counted by litellm_e2e_coverage_skipped_markers) so an unskipped-pending gap is visible rather than inflating the number. The one skip the collector cannot see is pytest.skip() called from inside a test body, since it does not exist until the test runs.

cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector

Use --format loki after the e2e pytest run in the same Kubernetes job/pod to print structured stdout lines for Loki:

cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --format loki --strict

This emits exactly one COVERAGE_TOTAL line and one COVERAGE_MODULE line per module in MODULE_ORDER, in that order. Loki uses log-safe module= labels from LOKI_MODULE_LABELS (core_llms, management_ui, etc.) so existing JSON and Prometheus consumers keep their human-readable module names unchanged.

The headline is overall coverage. The collector also lists markers that point at ids not in the registry, so a typo or an unenumerated behavior surfaces instead of being silently dropped.

Use strict mode in CI once existing draft markers are reconciled:

cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --strict

Strict mode exits non-zero on @pytest.mark.covers(...) ids that are not checked into the registry. Add --fail-on-collection-errors when the job should also fail on pytest collection errors.

Provider x feature matrix: customer-run Bedrock combinations

The provider and feature combinations customers actually run get explicit cells, expanded here as incidents surface new ones. The current Bedrock set, seeded from a customer's production shape (regional us.anthropic.* inference-profile ids over both chat routes, provider response headers for AWS-side correlation, and the Test Connection probe for a responses-mode Bedrock Mantle deployment):

Cell Feature Covering test
llm.chat_completions.bedrock_converse.basic.nonstream.works regional us. id, Converse llm_translation/test_chat_completions_regression_e2e.py
llm.chat_completions.bedrock_converse.basic.stream.works regional us. id, Converse stream llm_translation/test_chat_completions_regression_e2e.py
llm.chat_completions.bedrock_invoke.basic.nonstream.works regional us. id, Invoke llm_translation/test_bedrock_provider_matrix_e2e.py
llm.chat_completions.bedrock_invoke.basic.stream.works regional us. id, Invoke stream llm_translation/test_bedrock_provider_matrix_e2e.py
llm.chat_completions.bedrock_converse.response_headers.nonstream.works llm_provider-* headers llm_translation/test_bedrock_provider_matrix_e2e.py
llm.chat_completions.bedrock_converse.response_headers.stream.works llm_provider-* headers, stream llm_translation/test_bedrock_provider_matrix_e2e.py
mgmt.model.test_connection.happy_path Test Connection, Bedrock Mantle management/test_model_test_connection_e2e.py

Status: this is a draft for review

The cells were enumerated from the codebase and the tiers are a first proposal. Known things to settle before treating the set as final:

  • tiers are proposed, not signed off; 125 P0 is a lot to prove fail-before-fix, so P0 may want tightening
  • a few cells need a support check or a prune (for example llm.embeddings.anthropic.* and reliability.perf.throughput.under_slo)
  • auth is covered in two places (other.auth.* and the mgmt authz assertions); the boundary needs a decision, and the auth cluster may deserve promotion to its own module
  • the P2 "niche" cells each stand in for a large tail of integrations/providers by design, so the denominator is deliberately P0-weighted rather than a full inventory