codecov/patch flagged the diff at 75.4% against the 77.1% target. The
uncovered regions were real gaps, not noise: the list/get/stop endpoints
had no tests at all, and neither did the judge's failure paths.
Lifecycle endpoints: list returns newest-first without results, get
aggregates verdicts for one job (404 on unknown), stop completes an
active job and returns its verdicts, stopping a finished job is a 400
that writes nothing, view-only admins can list but not stop.
Judge failure modes: a provider error or unparseable verdict returns
None, bumps failed_count, and never writes a verdict row.
Local coverage on the two flagged files: 79% -> 88%.
Co-Authored-By: Claude <noreply@anthropic.com>
Zero-estimate cap bypass (cursor): a key quiet during the estimate
lookback gets cost_estimate 0.0, which the spend cap treated the same
as 'no estimate' and left uncapped — a later traffic spike on exactly
that job would bill until ends_at. Only a NULL estimate (rows predating
estimates) is uncapped now; $0 still gets the $1 floor.
Concurrent-start race (cursor): the find_first-then-create check passes
on both sides of a race, giving one key two active jobs and double
judge spend. A partial unique index (api_key_id WHERE status IN
(pending, running)) — raw SQL in the unshipped migration, since
schema.prisma cannot express partial indexes — makes the DB the
arbiter; the losing create surfaces as the same 409 as the advisory
check. The old find_many('desc') + reversed() insertion in the logger
cache already prefers the newest job for any legacy duplicates.
Request-path DB read (greptile): an expired job snapshot awaited
find_many inside the success callback, so every N seconds one request
per pod paid a synchronous Prisma read. The lookup is now sync-only:
it serves the current snapshot and kicks a detached refresh task when
stale. Cost: a cold pod's first ~1 refresh-window of samples are
missed (acceptable for a sampled eval); stale-if-error semantics keep
a DB blip from disabling the feature.
Also from review: a collapsed previous-job row said 'no verdicts' for
jobs with thousands of verdicts, because the list endpoint omits
results by design — it now says 'view results' when completed_count>0.
Co-Authored-By: Claude <noreply@anthropic.com>
The per-key TTL cache did one indexed find_first per distinct key hash
per 30s — fine at small scale, but on a proxy serving 10k active keys
that is hundreds of small reads per second across pods, all to discover
that almost every key has no job.
Cache the entire active-job set instead: one find_many per pod per TTL
(the set is admin-started and capped at one job per key, so it is
single-digit rows), served to every key as an in-memory dict hit. DB
load is now flat and constant in the number of keys, idle or active.
On a DB blip the stale snapshot is kept and the next TTL retries, so a
blip degrades freshness rather than disabling the feature. Concurrent
requests share one refresh behind a lock instead of stampeding.
Adds @@index([status]) for the status-only find_many, folded into the
unshipped shadow eval migration.
Co-Authored-By: Claude <noreply@anthropic.com>
The start endpoint counted the key's trailing 7-day requests directly
against LiteLLM_SpendLogs, which has no api_key index — on a busy proxy
that is a scan over every request in the window (potentially tens of
millions of rows) to answer one count, holding a multi-second query per
'Start shadow eval' click.
Read SUM(api_requests) from LiteLLM_DailyUserSpend instead: one indexed
row per key/day, the same table the usage dashboards already use for
this question. A regression test asserts the estimate queries the
rollup and never touches LiteLLM_SpendLogs.
Co-Authored-By: Claude <noreply@anthropic.com>
The judge model box was a free-text input pre-filled with
anthropic/claude-sonnet-5, implying it was sent as-is when nothing was
actually defaulted client-side (the backend's own default only applies
if the field is omitted entirely). Replace it with a searchable combobox
backed by the model cost map (litellm_provider + mode === "chat"),
starting empty and requiring an explicit pick.
Recommends three models spanning different providers — anthropic/
claude-sonnet-5, openai/gpt-4o, gemini/gemini-2.5-pro — pinned to the
top of the list with a 'Recommended' badge, all backend-verified to
resolve correctly via litellm.get_llm_provider/cost_per_token.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds a "Shadow eval" button next to the Auto-router usage heading that
smooth-scrolls to the shadow eval section, so it's reachable without
scrolling past the benchmarks body first.
Also fixes a PR review comment (veria-ai): shadow and judge calls ran
outside the normal auth path, so they never went through
reserve_budget_for_request and could push an already-exhausted key or
team further over budget before their own spend was even recorded.
_key_or_team_is_over_budget reads the same cross-pod spend counters
that path reserves against (via the existing get_current_spend) and
skips the shadow/judge pair outright when the shadowed key or its team
is already at or over budget. This is a read-time check, not a
reservation — appropriate for a best-effort background measurement
task, not a billed user request — so it narrows the window rather than
closing it against concurrent bursts, which the response comment
explains.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(otel): mark v2 server spans as failed for pre-call errors (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): authenticate malformed-body requests before rejecting them (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): cover malformed-body rejection when auth error is recovered
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): skip authorization for a request whose body never parsed
Deferring the parse failure ran the full auth phase, including budget reservation, whose reserved amount is only released by the endpoint's post call path; the endpoint never runs, so malformed requests leaked reservations and locked a budgeted key out. Authorization now runs only when the body parsed, and a parse failure with a rejected key keeps returning the 400 it returned before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
some users do not use make pre-commit as it is a multi-minute process. I personally use it but I want users themselves to decide whether to pre-commit before each commit or not, based on what works best for them
A shadow eval samples ongoing traffic, so a job without an end date keeps
billing judge calls until someone remembers to stop it — and the upfront
estimate silently priced exactly one week regardless. Jobs now take a
duration_days (1-30, default 7): the start endpoint stamps ends_at, the
estimate scales trailing volume to the requested window, and the logger
completes a job past its window through the same guarded update + cache
eviction path as the spend cap (generalized into _finalize_job). The
existing shadow eval migration is amended in place since it has not
shipped anywhere yet.
The start form no longer asks anyone to paste a key hash: the key is a
type-to-search combobox backed by /key/list alias substring search that
submits the token, the auto-router is a filter-as-you-type combobox fed
by the configured auto-router deployments, and duration is a select.
Active job cards show when the job will end.
The judge model field is now labelled as such, with guidance: judging
two answers blind needs solid comprehension and reliable JSON, not
frontier reasoning — a mid-tier model (Claude Sonnet / GPT-4o class) is
recommended, nano/mini-class judges give unreliable verdicts, and
frontier reasoning models add cost without changing outcomes. Same
guidance mirrored into the API field description.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(ui): show team BYOK models in team fallback settings
Team router settings loaded fallback options from /model_group/info, which resolves models without a team, so a team's own BYOK deployments were never selectable in its own fallback config. Load the team-scoped listing when a team id is present.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): ignore stale team model responses in router settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): use react-query for fallback model listing in router settings accordion
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Shadow and judge calls were fired with no caller identity on their metadata.
The proxy's cost callback requires user_api_key/_team_id/... to log spend and
apply budget checks, and silently drops the entry without them — so an
admin-enabled eval billed real provider spend that landed on no key, no team,
and no budget counter, invisible to every limit the shadowed key is normally
subject to.
Extract the identity-forwarding rules the auto-router classifier already
implements into a shared litellm/litellm_core_utils/internal_call_metadata.py:
forward the caller's identity subset, strip the parent's budget reservation
(top-level and the copy nested in user_api_key_auth) so a sub-call can't
finalize a reservation that belongs to the parent, and stamp the sub-call's
origin. The classifier now uses this module instead of its own copy.
Wire both shadow eval call sites (_call_router_shadow, _call_judge) through
it, and add a per-job spend cap (job cost_actual >= max(3x the quoted
estimate, $0.50)) so a bad estimate or a traffic spike can't turn a quoted
eval into a runaway bill; a capped job self-completes and is evicted from
cache so it can't be resurrected by an in-flight request.
Also surface api_key_id/team_id on GetShadowEvalJobResponse so an admin
running several jobs can tell which key's traffic a given win rate belongs
to, and regenerate the dashboard's OpenAPI types for the new fields.
Co-Authored-By: Claude <noreply@anthropic.com>
Three fixes from the live end-to-end run:
- The judge ran with max_tokens=200, which truncated roughly 12% of
verdicts mid-JSON so they were lost to failed_count. Raise it to a
named JUDGE_MAX_OUTPUT_TOKENS=500 and price the upfront cost estimate
off the same constant, so the estimate can't silently drift from what
the judge is actually allowed to emit.
- The UI only ever rendered the newest job, so starting a new eval hid
the results of a populated older one. Prior jobs are now listed in a
collapsible 'Previous evaluations' card, each expandable to its own
per-tier results.
- Move the shadow eval section above the benchmarks body: pre-adoption
keys have no router sessions, so it was buried under an empty state.
It stays outside BenchmarksBody so it survives that early return.
Co-Authored-By: Claude <noreply@anthropic.com>
Since #35491, every Router joins the module-global _live_routers weak set at
construction, and every model cost map swap replays the deployments of every
member on top of the freshly adopted map. #36039 isolated the register_model
ledger half of that replay but not this half: under pytest-xdist, a Router
created by an earlier test in the same worker that was still referenced (or
simply not yet garbage collected) re-registered its deployments during
TestPriceDataReloadIntegration::test_distributed_reload_check_function, and
register_model hydrated the sparse mocked gpt-3.5-turbo entry into a full
ModelInfo dict, failing the exact-equality assert (reruns cannot help since
the polluting router survives in the worker process)
The autouse isolate_litellm_state fixture now snapshots _live_routers before
each test and restores its membership on teardown, so a test's routers stop
contributing to cost map rebuilds once the test ends. A canary pair in
test_conftest_isolation.py asserts the rollback
The lint job's strict-rule budget flagged 17 new violations. Rather than
raise the ceiling, this types the code properly:
- validate the judge verdict into a PairwiseVerdict pydantic model at the
parse boundary instead of dict[str, Any] + cast, which also removes the
defensive float()/str() coercion downstream
- replace the untyped job dict with a frozen ActiveShadowEvalJob dataclass
- validate the prisma job row into _ShadowEvalJobRow, replacing 11 no-op
'# type: ignore[attr-defined]' comments
- annotate the success hook and drop Any from the remaining signatures
- mark the genuine third-party dict shapes (prisma filters/payloads, SDK
message lists) with '# mutable-ok' reasons per the existing convention
The column was declared Float? with no default, so every row started NULL.
The verdict writer increments it, and NULL + x is NULL in Postgres, meaning
judge spend never accumulated and the UI always showed no spend.
Makes the column non-null with a default of 0 across all three schema copies
and the migration, and tightens the response model to a plain float.
- Remove unused imports (datetime, timezone, ModelResponse) flagged by ruff.
- Update test_cost_tracking_adds_two_callbacks_when_prisma_set to expect 2
callbacks on litellm.callbacks (not 1): ShadowEvalLogger now registers
alongside _ProxyDBLogger in cost_tracking(). Test name already said 'two',
now it actually tests for the correct count.
- Format ShadowEvalSection.tsx/.test.tsx per prettier.
Tests: 94/94 passing (lifecycle, shadow-eval, auto-router endpoints).
Lint: ruff + prettier all clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Typing-only pass over the 21 files with the highest reportAny and
reportExplicitAny density among self-contained modules: management
endpoints, guardrails, streaming internals, response transformations,
MCP server, enterprise managed files, and vector store management.
Whole-tree basedpyright drops from 148,648 to 146,984 errors (-1,664),
with reportAny -1,111 and reportExplicitAny -296. No rule increased
repo-wide and no file regressed on any rule. No cast(), type: ignore,
noqa, suppression comments, or new Any annotations anywhere in the diff,
and no runtime behavior changes.
Budgets ratcheted by make lint-budget-update: basedpyright -1,663 across
48 rules, ruff-strict -86, type-discipline -110.
require_managed_files was only checked on upload, so raw provider ids still
reached the batch, fine-tuning and vector store file routes. Ownership rows
exist only for managed ids, so those requests were forwarded under shared
credentials with no tenant check: knowing another tenant's id was enough to
read, run against, cancel or delete their object.
Generalise the file-id guard to validate_managed_id_requirement(resource_id,
resource_kind) and call it on batch create/retrieve/cancel, fine-tuning
create/retrieve/cancel (training_file and validation_file both) and the shared
vector store file id resolver. Behaviour is unchanged when the setting is off.
A budget rule that graduates into a config's hard-fail select list rightly
leaves the budget file, but the ratchet guard read any disappearance as a
silently raised ceiling. Teach it the pairing between ruff-strict-budget.json
and ruff.toml: a dropped rule is excused only when the paired config's
lint.extend-select (minus lint.ignore) now hard-fails it, so deleting a rule
without graduating it still trips the guard.
Every strict-gate rule whose budget ceiling was already 0 moves into the base
config's lint.extend-select, so editors and ruff check --fix surface the
diagnostics directly and the budget file shrinks to rules with real debt.
Graduates stay in ruff-strict.toml's select so the strict RUF100 pass keeps
policing their stale noqa directives, and base external entries they made
redundant (FURB, I001, RUF010, RUF022, RUF023, RUF051) are dropped so base
RUF100 polices those directly. UP037 had two violations hidden behind a star
import; importing Literal explicitly fixes them so UP037 can graduate too.
New drift tests pin the invariants: every strict-selected rule is budgeted or
hard-failed by base, every base-owned rule stays visible to exactly one
RUF100 pass, and graduated rules fail the normal ruff run.
HTTPHandler.post and AsyncHTTPHandler.post call raise_for_status before returning, so the status_code != 200 branches after the create and cancel POSTs could never run. Non-2xx already surfaces as httpx.HTTPStatusError from inside the client. The checks after GETs stay: the get helpers return without raising. Tests that faked a non-raising POST response are replaced by HTTPStatusError propagation coverage.
Base branch moved again while resolving the prior merge. Same conflict shape
in litellm/types/management_endpoints/auto_router_endpoints.py: kept both
the Mapping and Literal imports. Regenerated schema.d.ts to pick up
upstream's ModelInfo pricing field changes.
Co-Authored-By: Claude <noreply@anthropic.com>
