fcec1488e2
* feat(proxy): add a generic list contract for management/v1 entity lists Paging, sorting, filtering and search for an entity collection, declared once as a ListSpec and served by handle_list. The route injects a ListExecutor that owns its table, so this module never imports Prisma. The caller's scope is derived from the caller alone and ANDed with whatever they filtered on, so a query parameter can only narrow what they may read. This is the shared half of the budgets list; it lands here so the endpoint has something to register against, and drops out when the framework arrives on its own branch. * feat(proxy): add GET /management/v1/budgets The Budgets page reads /budget/list, which returns the whole table as a bare array with no way to page, sort or filter it. A customer with enough budgets to fill the page has no way to find one. Registers LiteLLM_BudgetTable against the management/v1 list contract: sortable on budget_id, max_budget, tpm_limit, rpm_limit and created_at, default order newest-first with budget_id breaking ties, search on budget_id, and filters for budget_duration, max_budget and created_at. budget_duration is deliberately not sortable; the column holds "7d"/"30d" strings, so a lexicographic ORDER BY puts "30d" ahead of "7d". tpm_limit and rpm_limit are BigInt? in Prisma, so rows validate through a pydantic model on the way out and serialize as JSON numbers. A caller without admin view is refused 403 as a problem document rather than served an empty page. /budget/list is untouched. * fix(proxy): rework the budgets list onto the merged list contract PR #35308 landed a different shape than this branch was written against: `where` is a tuple of frozen predicates rather than a Prisma-shaped mapping, `ListSpec` carries both the row and the wire type, and `where_sql` / `order_by_sql` render for a raw-SQL executor. The budgets executor now queries through `query_raw` the way the spend logs facet does, selecting only the columns it serves. Also casts datetime binds in `where_sql`. They cross into the query engine as JSON, so an uncast placeholder arrives as text and Postgres refuses `timestamp >= text` outright; every `filter[created_at][gte|lte]` was answering 500. The cast reads the bind as an instant and drops it to naive UTC to match Prisma's TIMESTAMP(3) column, the same one /spend/logs/ui applies. * refactor(proxy): fold the predicate renderer instead of recursing recursive_detector flags `_render_all`, and the flag is fair: it recursed once per predicate, so the stack grew with the number of filters on the request for no reason. Walking a predicate list is a running bind index, which is a fold. `_render` still re-enters for `AnyOf`, but its clauses are plain comparisons built by `?q=`, so that nesting is one level deep and no caller can drive it deeper. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| collector.py | ||
| guardrail.yaml | ||
| llm_claude_code_compat.yaml | ||
| llm_conversational.yaml | ||
| llm_nonconversational.yaml | ||
| logging.yaml | ||
| mcp.yaml | ||
| mgmt.yaml | ||
| other.yaml | ||
| quota_management.yaml | ||
| README.md | ||
| registry.py | ||
| reliability.yaml | ||
| schema.py | ||
| test_collector.py | ||
e2e coverage registry
This directory is the denominator for e2e test coverage: the set of behaviors we
want covered, one row per behavior, checked into the repo so coverage is a number we
can track instead of a guess. It implements the plan in the "E2E Coverage Tracking"
note; the naming grammar lives in tests/e2e/CLAUDE.md.
The model
A cell is one customer-noticeable behavior a single e2e test can assert pass/fail
on, for example llm.chat_completions.bedrock_converse.tool_use.stream.works. Cells are
grouped module > feature > test, with LLM cells split into Core LLMs and
Non-Core LLMs for dashboarding. Each cell carries a tier (P0/P1/P2), a source, and a
fail_before_fix flag.
The rows live in per-prefix YAML files (llm_*.yaml, mgmt.yaml, mcp.yaml,
reliability.yaml, quota_management.yaml, logging.yaml, guardrail.yaml,
other.yaml) and validate against
the discriminated union in schema.py, so an LLM row cannot carry a guardrail field and
vice versa. llm rows with subject_endpoint of chat_completions, messages, or
responses roll up to Core LLMs; all other LLM endpoints roll up to Non-Core LLMs.
LLM endpoint, route, and capability values are typed in schema.py, so new taxonomy
values require an explicit schema change. logging and guardrail are two id-prefixes
that roll up into the single Logging & Guardrails dashboard module.
A test declares what it covers with a marker:
@pytest.mark.covers("llm.chat_completions.openai.tool_use.stream.works")
def test_openai_streaming_tool_calls(self) -> None:
...
The number
collector.py diffs the registry against those markers and reports coverage per module.
It is static: a collect-only pass reads the markers, so it runs no test and needs no live
proxy. Whether a covered cell currently passes or fails is a separate, live concern.
A skipped test asserts nothing, so its markers do not count. A cell is covered only when
at least one test pytest would actually run declares it; a cell claimed by both a live
test and a skipped one stays covered. Skip state comes from pytest's own evaluator, so
skip and skipif resolve exactly as they do in the e2e run, which also means a
skipif on an absent credential makes that cell uncovered in the environments where the
test cannot run. Cells left uncovered this way are listed under the headline (and counted
by litellm_e2e_coverage_skipped_markers) so an unskipped-pending gap is visible rather
than inflating the number. The one skip the collector cannot see is pytest.skip()
called from inside a test body, since it does not exist until the test runs.
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector
Use --format loki after the e2e pytest run in the same Kubernetes job/pod to print
structured stdout lines for Loki:
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --format loki --strict
This emits exactly one COVERAGE_TOTAL line and one COVERAGE_MODULE line per module
in MODULE_ORDER, in that order. Loki uses log-safe module= labels from
LOKI_MODULE_LABELS (core_llms, management_ui, etc.) so existing JSON and
Prometheus consumers keep their human-readable module names unchanged.
The headline is overall coverage. The collector also lists markers that point at ids not in the registry, so a typo or an unenumerated behavior surfaces instead of being silently dropped.
Use strict mode in CI once existing draft markers are reconciled:
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --strict
Strict mode exits non-zero on @pytest.mark.covers(...) ids that are not checked into
the registry. Add --fail-on-collection-errors when the job should also fail on pytest
collection errors.
Status: this is a draft for review
The cells were enumerated from the codebase and the tiers are a first proposal. Known things to settle before treating the set as final:
- tiers are proposed, not signed off; 125 P0 is a lot to prove fail-before-fix, so P0 may want tightening
- a few cells need a support check or a prune (for example
llm.embeddings.anthropic.*andreliability.perf.throughput.under_slo) - auth is covered in two places (
other.auth.*and the mgmt authz assertions); the boundary needs a decision, and the auth cluster may deserve promotion to its own module - the P2 "niche" cells each stand in for a large tail of integrations/providers by design, so the denominator is deliberately P0-weighted rather than a full inventory