86e00c244c
Guardrails: keep bedrock, OpenAI moderations, the LiteLLM content filter and presidio at P0, promote llm_as_a_judge to P0, and drop the third-party vendor tail (lakera, ibm, semantic_guard, purview, pangea, headroom, generic api) to P2. AIM goes to P2 until its DRI lands a mock AIM server, since CI has no live AIM tenant. Logging: s3, GCS, prometheus, datadog and OTEL stay P0; the eval/analytics/cost destinations move to P2. Auth: virtual key and JWT are the committed paths, so route_group_allowed joins them at P0 and OAuth2 introspection plus the IP allowlist drop to P2. README records the policy: P0 + P1 is the number we drive to 95%, P2 rows stay enumerated so tail gaps remain visible without gating the target. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| collector.py | ||
| guardrail.yaml | ||
| llm_claude_code_compat.yaml | ||
| llm_conversational.yaml | ||
| llm_nonconversational.yaml | ||
| logging.yaml | ||
| mcp.yaml | ||
| mgmt.yaml | ||
| other.yaml | ||
| quota_management.yaml | ||
| README.md | ||
| registry.py | ||
| reliability.yaml | ||
| schema.py | ||
| test_collector.py | ||
e2e coverage registry
This directory is the denominator for e2e test coverage: the set of behaviors we
want covered, one row per behavior, checked into the repo so coverage is a number we
can track instead of a guess. It implements the plan in the "E2E Coverage Tracking"
note; the naming grammar lives in tests/e2e/CLAUDE.md.
The model
A cell is one customer-noticeable behavior a single e2e test can assert pass/fail
on, for example llm.chat_completions.bedrock_converse.tool_use.stream.works. Cells are
grouped module > feature > test, with LLM cells split into Core LLMs and
Non-Core LLMs for dashboarding. Each cell carries a tier (P0/P1/P2), a source, and a
fail_before_fix flag.
The rows live in per-prefix YAML files (llm_*.yaml, mgmt.yaml, mcp.yaml,
reliability.yaml, quota_management.yaml, logging.yaml, guardrail.yaml,
other.yaml) and validate against
the discriminated union in schema.py, so an LLM row cannot carry a guardrail field and
vice versa. llm rows with subject_endpoint of chat_completions, messages, or
responses roll up to Core LLMs; all other LLM endpoints roll up to Non-Core LLMs.
LLM endpoint, route, and capability values are typed in schema.py, so new taxonomy
values require an explicit schema change. logging and guardrail are two id-prefixes
that roll up into the single Logging & Guardrails dashboard module.
A test declares what it covers with a marker:
@pytest.mark.covers("llm.chat_completions.openai.tool_use.stream.works")
def test_openai_streaming_tool_calls(self) -> None:
...
The number
collector.py diffs the registry against those markers and reports coverage per module.
It is static: a collect-only pass reads the markers, so it runs no test and needs no live
proxy. Whether a covered cell currently passes or fails is a separate, live concern.
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector
Use --format loki after the e2e pytest run in the same Kubernetes job/pod to print
structured stdout lines for Loki:
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --format loki --strict
This emits exactly one COVERAGE_TOTAL line and one COVERAGE_MODULE line per module
in MODULE_ORDER, in that order. Loki uses log-safe module= labels from
LOKI_MODULE_LABELS (core_llms, management_ui, etc.) so existing JSON and
Prometheus consumers keep their human-readable module names unchanged.
The headline is overall coverage. The collector also lists markers that point at ids not in the registry, so a typo or an unenumerated behavior surfaces instead of being silently dropped.
Use strict mode in CI once existing draft markers are reconciled:
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --strict
Strict mode exits non-zero on @pytest.mark.covers(...) ids that are not checked into
the registry. Add --fail-on-collection-errors when the job should also fail on pytest
collection errors.
Tiering policy
Tier is what we commit to, not how interesting a behavior is. The signed-off rules:
- guardrails: the surfaces we commit to are bedrock guardrails, OpenAI moderations, the LiteLLM content filter, LLM-as-a-judge and presidio masking. Third-party vendor guardrails are tail and sit at P2
- AIM stays P2 until its DRI lands a mock AIM server; there is no live AIM tenant in CI, so promoting it would only buy a permanently red or permanently skipped cell
- logging: s3, GCS, prometheus, datadog and OTEL are P0 on success, failure and stream where the path exists. The remaining eval/analytics/cost destinations are tail at P2
- auth: virtual keys and JWT are the committed paths at P0. OAuth2 introspection and the IP allowlist are P2
- P0 + P1 is the number the team is driving to 95%; P2 rows stay enumerated so a tail gap is visible, but they do not gate the target
Status: this is a draft for review
The cells were enumerated from the codebase. Known things to settle before treating the set as final:
- a few cells need a support check or a prune (for example
llm.embeddings.anthropic.*andreliability.perf.throughput.under_slo) - auth is covered in two places (
other.auth.*and the mgmt authz assertions); the boundary needs a decision, and the auth cluster may deserve promotion to its own module - the P2 "niche" cells each stand in for a large tail of integrations/providers by design, so the denominator is deliberately P0-weighted rather than a full inventory