The cap covers reasoning tokens as well as the verdict, and the models people
pick as judges reason before answering whether the call asks them to or not:
Anthropic's 5 family thinks adaptively and cannot be told not to, so the
reasoning bills against max_tokens with nothing in the request to opt out.
At 1500 the reasoning consumed the budget and the reply arrived empty or cut
off mid-object, which the attempt recorded as an unparseable judge verdict
rather than a result. Headroom costs nothing: max_tokens is a ceiling and only
generated tokens bill, so the only movement is that judge calls which used to
bill their full budget and return nothing now return a verdict.
Deliberately not passing reasoning_effort to bound the reasoning instead:
is_thinking_enabled treats any reasoning_effort as thinking-enabled, which
drops the forced tool_choice that json_mode relies on and turns thinking on
with a 1024-token floor for judges that were not reasoning at all.
A complexity tier can name several model groups, but the pool pick and the session-pin
replay both returned a group without consulting deployment health, so a group whose every
deployment was in cooldown was still routed to and the request died at the router's
zero-deployment check while a healthy peer sat in the same tier.
Gate the decided response at the pre-routing hook's exits, the seam the modality gate
already occupies, so every arm that can place a request is covered by one owner: a fresh
classification, a replayed or escalated pin, a plan-mode floor, a context-window
escalation, an adaptive pick, and whatever arm is added next.
Peers come from the decided tier only. Climbing to a higher tier costs more than the
classifier asked for and is left to a follow-up. The gate fails open on every uncertainty:
an unreadable cooldown view, a decision carrying no tier, a group the router knows no
deployments for, or a tier whose peers are all cooling.
litellm only retries 408, 409, 429 and 5xx, so classifying a dropped connection
as a 400 turned one replica set failover into a permanently failed search. It is
a 503 now, with the message still naming the misconfigurations that also close a
connection.
The client cache skipped insertion once it held 32 entries, so any store added
after that rebuilt its client on every search, paying an SRV lookup, a TLS
handshake and topology discovery each time. It evicts the least recently used
entry instead, which only drops the cache's own reference.
Also trims the explanatory comments to the one-line form the repo asks for.
A row that received both priced and unpriced increments used to collapse
to cost NULL, throwing away the priced subtotal and making every unit on
it read as untracked. The rollup now carries a second column,
untracked_units, that the aggregator increments for units with no known
price while cost keeps accruing for the rest, so cost covers exactly
units - untracked_units. Rows written before the migration keep cost
NULL and still read as untracked in full
The endpoints read untracked units off the column (or the whole row for
a legacy NULL) rather than from a NULL filter, and the policies overview
now fills totalUntrackedUsageUnits, which the previous commit missed
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
Evicting before the mutation commits left a race: a concurrent JWT
request could re-cache the old mapping between the eviction and the
commit, keeping a deleted or renamed claim authorized until the cache
TTL expired. Flagged by review on PR #39808.
The base vector store interface grew an embedding_executor argument, and
litellm.vector_stores.search now always passes one. MongoDB still carried its
own embedding_fn/aembedding_fn constructor seam, so every search through the
public entry point failed with an unexpected keyword argument.
Drop the local seam in favour of the shared executor: one path instead of two,
and the unit tests now drive the same seam production uses.
* fix(health): probe test_connection with the credential the request names
/health/test_connection matches the request's model string against the
configured deployments and merges the match's litellm_params underneath the
request. A request that named a stored credential but no key of its own
still satisfied the "request sets no connection fields" test, so it inherited
the matched deployment's api_key and api_base, and load_credentials_from_list
then skipped the named credential because api_key was already set.
A wildcard route covering the model is enough to match, so the Add Model
page's Test Connect probed with an unrelated deployment's key while echoing
back the credential that was selected.
Naming a credential the configuration does not name now withholds the
configuration's credential fields, the same set already withheld from a
request that supplies its own endpoint. Naming no credential still inherits
them, as documented.
* test(health): drop test docstrings that restate their own names
* test(health): assert the credential probe on the wire, not on the call args
The connection-test regressions patched litellm.ahealth_check and read the
params handed to it. Driving the endpoint through the app with respx faking
the upstream instead lets the real credential resolution run, so the tests
assert the key and host that actually go out, which is what the bug was about.
It also drops three of the five patched proxy internals; the two that are left
are proxy-global wiring with no injection seam, the same ones the image_edit
connection test already has to reach for.
* chore(ui): regenerate schema.d.ts for the test_connection docs change
Adds stall_escalation_enabled to the complexity router: when the assistant's
own recent tool calls look stuck (identical repeats, or repeated tool
errors on a surface that reports one), the request is bumped one
configured tier higher, the automatic counterpart to escalation_keywords.
Detection is stateless: it rereads the last stall_escalation_window tool
calls from that request's own message list on every classified turn, so
the bump lasts only as long as the recent calls still look stuck and
lifts on its own once they don't, and evidence survives a plain follow-up
like "try again" instead of resetting on the newest human ask.
Off by default. Rejected together with session_affinity and
classification_mode='user_turn', which both replay a held routing
decision instead of classifying most turns, and with tier_definitions,
for the same reason escalation_keywords is: both rely on the built-in
tier severity order a custom tier set does not define.
Dashboard controls for this are not included; config.yaml and the
management API accept it today through ComplexityRouterConfig.
/key/regenerate carries the JWT-to-key mapping to the new token via FK
cascade, but the jwt_key_mapping cache entry kept resolving the old
(now invalid) token for up to virtual_key_mapping_cache_ttl. Snapshot
the key's mapping cache keys before the token update and evict them
with evict_and_broadcast so every worker drops the stale entry.
Also share the cache-key format through jwt_key_mapping_cache_key and
upgrade the /jwt/key/mapping CRUD endpoints from local-only deletes to
evict_and_broadcast, closing the same cross-worker staleness there.
A row's cost sums only the daily rows that carry a tracked cost, so it
silently under-reports whenever some rows are NULL (pre-migration days,
old pods mid-rollout, an unpriced counter). Both usage endpoints now
return the per-counter units behind those NULL rows next to the cost
(untrackedUsageUnits / totalUntrackedUsageUnits on the overview,
untracked_usage_units on the detail), so a partial cost is never mistaken
for a complete one and the reader can see exactly what it excludes
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
tlsCAFile and tlsCertificateKeyFile are how a self-managed deployment presents a
private CA, so they are the options on-prem operators actually set. pymongo opens
those files itself during TLS setup and lets OSError out, which is neither a
PyMongoError nor a ValueError, so it missed every branch of the translator and
litellm.exception_type turned it into a 500 with a traceback in the body. A
mistyped path, or one that exists on the host but not inside the container, is a
routine mistake and has to read as a 400 naming the file.
Matched on the exception carrying a filename so a socket-level OSError still falls
through to the branches that handle it. Verified against a self-managed mongod with
a missing CA file, a CA path that is a directory, and a missing client certificate.
Both of these are e2e assumptions that PRs #31731 and #39532 invalidated, not
product regressions. They have been red in litellm-e2e builds 119-123.
Wildcard readiness probe (6 errors in test_model_access_group_e2e.py)
#31731 made _get_wildcard_models drop a wildcard route from /v1/models
unconditionally; before it, a wildcard with a matching router deployment stayed
in the list and only the no-router / no-deployment fallbacks removed it. The
shared readiness helper polls /v1/models for an exact id match, so registering
openai/gpt-5.4* now times out at model_servable_timeout every run and every
test in the class errors in setup.
return_wildcard_routes=True still re-adds the route, so the poll asks for it.
The flag is a no-op for a concrete model name -- it only ever adds wildcard
entries -- so it is set unconditionally rather than sniffing the name.
Semantic auto-router spend assertion
#39532 bills the routing embedding to the caller's key on purpose, so the key's
spend logs now legitimately carry an openai/text-embedding-3-small row and
_assert_served_only_by rejects it.
Widening the allowlist would have weakened the assertion this test exists for --
that the request reached the target deployment. Instead the embedding row is
split off and asserted separately, which turns the break into coverage for
#39532. The poll gains a predicate so it waits for the embedding row rather than
racing whichever row is written first.
mongod serves $vectorSearch identically whether mongot runs under Atlas or beside
a self-managed deployment, so the provider already worked against on-prem. The
guidance did not: a refused connection told the operator to check their project's
IP access list and whether the cluster was paused, neither of which exists outside
Atlas, and the index errors claimed an "Atlas Vector Search index" they do not have.
Every message now names a remedy for both, keeping the Atlas-specific hint labelled
as such.
Also diagnoses unescaped credentials, which self-managed deployments hit more often
because the password is usually generated. pymongo reports those three different
ways and none of them mentions the password: '@', ':' and '%' raise an RFC 3986
complaint, '/' is read as the database separator and surfaces as Bad database name,
and an unescaped ':' looks like a bad port and comes back as a plain ValueError.
All three now point at the credentials. The ValueError branch's comment claimed it
fired on an unescaped '/', which pymongo actually reports as InvalidURI; corrected
to the port parse it really catches.
Verified against a self-managed mongod 8.0 with mongot, reached over plain
mongodb:// with no SRV and no TLS: 13 cases with live OpenAI embeddings, and 4
credential cases against an auth-enabled instance whose password holds % @ / and :.
list_search_indexes returns the same queryable and status fields there as on Atlas,
so the index-readiness check needed no change.
An open classifier circuit routed through the ordinary heuristic or
classifier_fallback path, and both causes are pin-worthy, so a session
whose turn landed on the cooldown fallback held that model for the whole
session_affinity TTL and never reclassified after the breaker closed.
The circuit-open signal now blocks the pin, and _classifier_failure_outcome
tags its outcomes through one helper instead of reassigning a Final.
* feat(team): report per-user spend within a team for JWT traffic
Add GET /team/spend/by_user, which groups raw spend logs by (team_id, user)
so JWT/SSO requests with no virtual key are attributed to the user inside
each selected team. Team admins see every member, plain members see only
their own row. The Team Usage page gets a Spend Per User Within Team card
with CSV export backed by the same endpoint.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(team): cover /team/spend/by_user in behavior suite, tf audit allowlist and EntityUsage unit test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(team): drop explanatory docstrings from /team/spend/by_user and regen schema.d.ts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_numeric_form_type only peeled a single ReadOnly layer, so a field still
wrapped in Required/NotRequired was read as non-numeric and dropped from the
mapping. Which qualifiers survive get_type_hints varies by interpreter version
and by include_extras, so on Python 3.10 NotRequired[ReadOnly[int]] reached the
check intact and the field was silently skipped, which is what turns the mapped
test red on the 3.10 leg only.
Peel Required/NotRequired/ReadOnly/Annotated in any order and nesting instead.
The one production caller feeds a schema with no qualifiers, so the resulting
mapping is unchanged on every interpreter in the matrix, but a field written the
house-convention way stops being dropped.
A counter missing from the cost map entry was priced at 0.0 per unit, so
the rollup recorded it as known-free usage. It now stamps None for that
counter and the rollup writes NULL, while the per-request guardrail_cost
that feeds spend and budgets still sums only the known prices.
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
test_event_loop_stall_timeout_burst_keeps_breaker_closed built its timeout
burst by wrapping a healthy fake call in asyncio.wait_for. Before 3.12,
wait_for returns the inner result when the inner future also completed while
the loop was blocked, so no call timed out, the burst never materialised, and
the test's own liveness guard failed with 0 >= 3.
The fake now checks its own client deadline against the clock, the way a client
library does, so the stall produces a real redis TimeoutError burst on every
interpreter. The breaker itself is unchanged: its duration gate is plain
time.time() bookkeeping and never depended on the version.
Drop the two new docstrings, annotate the new locals Final, and replace the
mutable call recorder with a rebuild stub that fails the test if it is ever
reached.
test_no_linear_scans_in_router: #39468 added config_deployments() and
heuristic_v2_router_limit_violation(), which both scan the whole model_list
from admin-only paths (model add/upsert), so add them to the allowlist. The
allowlist becomes a mapping so each exemption carries its reason as data.
test_missing_model_parameter_curl: a request with no model is rejected by the
proxy when nothing can serve it and by the router when a wildcard or default
deployment exists, and by the upstream provider when a wildcard forwards it,
so the message text is not a stable contract. Assert the contract that holds
in every case: HTTP 400 with a non-empty error message.
test_model_group_info_e2e: /model_group/info resolves wildcards, so it can
never return "anthropic/*" verbatim. cc3f9cd65b rewrote the assertion to
expect the raw pattern after claude-3-5-haiku-20241022 left the price map,
which made it unsatisfiable. Assert the expansion instead.
test_should_derive_ocr_mapping_status_from_live_tests: the audit needs a
native bridge built with the trace-parity feature, which CI never builds, so
skip with the harness's own diagnostic instead of erroring. Extract that
check out of ensure_trace_bridge as trace_bridge_error so a pytest run
reports the state without kicking off a maturin rebuild.
Every model bedrock list-foundation-models and list-inference-profiles
report as live in us-gov-west-1 or us-gov-east-1 now has a priced row:
Claude Fable 5.1 (profile plus in-region), Nemotron Nano 9B (profile plus
in-region), Grok 4.6 (profile plus Mantle in both regions), the us-gov.
Claude 3 Haiku profile in the east, Nova Lite, Micro and the Nova 2
multimodal embeddings in the west, and the Gemma 4 and gpt-oss Mantle
SKUs the GovCloud offer files price. Offer-file rates are used where AWS
publishes them; Claude rows carry the 1.2x GovCloud premium.
