The reset job evicts the cached end-user object only from its own worker's
in-memory cache (plus Redis), so every other uvicorn worker and replica keeps
the pre-reset spend for up to user_api_key_cache_ttl (60s by default). Those
workers pass that stale spend as fallback_spend, and since the authoritative
floor read returned None for spend:end_user: keys, get_current_spend handed
the stale value straight back and the end user kept getting 429 after the
rollover on every worker but the one that ran the reset.
The floor read now consults LiteLLM_EndUserTable.spend for end-user counters,
the same way keys, teams, users, and orgs already read their rows. It runs only
when the shared counter sits below the cached spend (a reset or a Redis
restart) and stays behind the existing 5s in-process marker, so the normal
request path still does no DB read. Cold end-user counters keep seeding from
the cached object rather than the row, so from_db is unchanged for them.
* fix(headroom): bound the /v1/compress and /v1/retrieve calls with a timeout
The headroom guardrail builds its client with get_async_httpx_client(GuardrailCallback)
and no params, and passes no timeout on either outbound call. That client's read, write
and pool legs are 600s (litellm.request_timeout when set explicitly, default 6000s), so
an unreachable or stalled compression service holds the caller's pre-call request open
for the whole window before unreachable_fallback ever runs. Because the client is shared
with every other no-params guardrail, each stalled call also pins a pooled connection for
the same window, so a saturated pool makes unrelated requests block on the pool leg.
Bound both calls at 60s by default, honoring litellm_params.timeout when set (the field
already exists and documents itself as the per-guardrail API timeout; headroom accepted
it and ignored it). The connect leg stays at the http_handler default, or the configured
budget when that is shorter, so a dead host still fails fast.
Live on a proxy against a stalled /v1/compress: 600.4s -> 60.2s before the 502, and 5.2s
with timeout: 5 configured.
* fix(headroom): reject non-finite timeouts and trim the timeout commentary
`timeout: .inf` on a Headroom guardrail reached httpx and the aiohttp transport
raised OverflowError, so every request came back as a raw 500 instead of going
through unreachable_fallback. Reject non-finite values the same way as
non-positive ones, and cut the comments and docstrings back to what the code
does not already say.
* fix(anthropic): never carry cache_control on translated thinking blocks
The /v1/messages adapter built every thinking and redacted_thinking block with
cache_control=content.get("cache_control", {}), so a block the client never
marked still came out carrying an empty cache_control. anthropic_messages_pt
replays thinking blocks verbatim and first, so that value landed at content[0]
of the outbound assistant message and Anthropic rejected the request with
messages.N.content.0.thinking.cache_control: Extra inputs are not permitted.
Anthropic's schema has no cache_control on either block type, so there is
nothing to gate or translate here, only to stop copying. Every sibling block
type already routes through _add_cache_control_if_applicable; these two were
the only ones setting the key unconditionally.
This is reachable from any caller that round-trips Anthropic messages through
the OpenAI shape, which is why shadow eval saw it on a majority of sampled
Claude Code turns while the same traffic served natively was fine.
* test(anthropic): assert the outbound wire body for redacted thinking blocks
* feat(cli): sync OpenCode models from /v1/models in lite opencode
lite opencode now fetches the proxy's /v1/models with the resolved key and
hands OpenCode an OPENCODE_CONFIG_CONTENT declaring a litellm provider
(@ai-sdk/openai-compatible, proxy /v1 base URL, {env:OPENAI_API_KEY}) with one
model entry per listed chat model, so the model picker mirrors the proxy
without a hand-maintained opencode.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): sync OpenCode models only after the key check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The test-quality gate rejects patching litellm.acompletion, and faking the
HTTP boundary is the stronger test anyway: the 503, 500 and 502 responses now
travel through the real OpenAI SDK and exception mapping before the router
decides how many times to retry. Adds a case showing that a 503 key does not
govern a 502.
Two issues Bugbot found on #39809.
A keyword_tier_rule forces its tier and returns before any classification
runs, so stall escalation never reached that path even though keyword
escalation did. That left the one path that can pin a weak model to a
whole conversation as the one path a stall could not lift. Stall
detection now resolves before the override branch and both paths bump.
The dashboard switch disabled itself whenever session pinning or
user-turn classification was on, including for a router that already had
stall escalation enabled. The conflicting keys stayed set, the backend
rejected the save, and the disabled switch was the only way to clear
them. It now disables only the off-to-on direction.
Replace the hand-ordered isinstance ladder in get_num_retries_from_retry_policy
with a class-to-field mapping walked along the exception's MRO, most specific
class first. A RetryPolicy field can no longer go silently dead the way
InternalServerErrorRetries did, and subclasses such as
ContentPolicyViolationError or MidStreamFallbackError pick up their parent's
field when they have none of their own.
Add a DefaultRetries catch-all so errors without a dedicated field
(BadGatewayError, APIConnectionError, NotFoundError, ...) can be governed by the
policy too. Specific fields still win over DefaultRetries.
Wiring the previously dead InternalServerErrorRetries changes one test
expectation: a policy of 2 now overrides a per-deployment num_retries of 5, so
the amplification test sees 3 upstream requests instead of 6.
Expose DefaultRetries as "All other errors" in the Admin UI retry settings tab
and ratchet the lint budgets down by the violations this branch fixed.
The access group detail page rendered MCP servers, agents, attached teams and keys as bare ids, so an admin had to look each one up elsewhere to audit a group
Every access group response now also carries access_mcp_servers, access_agents, assigned_teams and assigned_keys as {id, name} pairs. Names come from the DB rows first and fall back to config-declared MCP servers and agents (including legacy agent ids), resolved with one query per table across all groups in a list call. The existing *_ids columns are unchanged
The UI renders the name with the id in a tooltip, links teams and keys to their detail pages, and shows the raw id only when nothing resolves
Counting whichever pattern was most common across the window escalated a
task that had already recovered: three identical failures stay in the
window for a few turns after the model breaks out of them, and on their
own they met the threshold.
Both tests now anchor on the newest call. The repeat test counts calls
matching the newest one, and the error test only runs while the newest
call is itself an error, so a window whose recent calls are healthy no
longer escalates. The matches still do not have to be adjacent, so a
retry loop broken up by an unrelated lookup keeps counting.
Found by Greptile on #39809.
Adds an "Advanced: Stalled Task Escalation" section to the complexity
router config: a toggle plus the repeat threshold and the window of recent
tool calls to examine. Both knobs are seeded on enable and cleared on
disable, so an off router sends none of the three keys, which is what the
backend requires next to session pinning and a custom tier set.
The toggle locks out with an explanation when "How often to classify" is
set to once-per-session or new-user-message, since both replay a held
routing decision instead of classifying and a stall would never reach the
classifier. The keys join the custom-tier restriction registry, which both
strips them from a custom-tier save and marks the section restricted.
ResponseFormatControls moves into its own file to keep
ComplexityRouterConfig.tsx under the 800-line lint ceiling, matching the
one-file-per-control layout its siblings already use.
Async searches reach the sync client through executor threads, so the LRU cache
is shared state. A key could be evicted between the lookup and the reordering
that followed it, and the reordering then raised KeyError and became a 500.
Reproduced at 15 failures per run with 16 threads over 34 keys and a 1ns switch
interval; the regression test is that workload.
A complexity tier can name several model groups, but the pool pick and the session-pin
replay both returned a group without consulting deployment health, so a group whose every
deployment was in cooldown was still routed to and the request died at the router's
zero-deployment check while a healthy peer sat in the same tier.
Gate the decided response at the pre-routing hook's exits, the seam the modality gate
already occupies, so every arm that can place a request is covered by one owner: a fresh
classification, a replayed or escalated pin, a plan-mode floor, a context-window
escalation, an adaptive pick, and whatever arm is added next.
Peers come from the decided tier only. Climbing to a higher tier costs more than the
classifier asked for and is left to a follow-up. The gate fails open on every uncertainty:
an unreadable cooldown view, a decision carrying no tier, a group the router knows no
deployments for, or a tier whose peers are all cooling.
litellm only retries 408, 409, 429 and 5xx, so classifying a dropped connection
as a 400 turned one replica set failover into a permanently failed search. It is
a 503 now, with the message still naming the misconfigurations that also close a
connection.
The client cache skipped insertion once it held 32 entries, so any store added
after that rebuilt its client on every search, paying an SRV lookup, a TLS
handshake and topology discovery each time. It evicts the least recently used
entry instead, which only drops the cache's own reference.
Also trims the explanatory comments to the one-line form the repo asks for.
A row that received both priced and unpriced increments used to collapse
to cost NULL, throwing away the priced subtotal and making every unit on
it read as untracked. The rollup now carries a second column,
untracked_units, that the aggregator increments for units with no known
price while cost keeps accruing for the rest, so cost covers exactly
units - untracked_units. Rows written before the migration keep cost
NULL and still read as untracked in full
The endpoints read untracked units off the column (or the whole row for
a legacy NULL) rather than from a NULL filter, and the policies overview
now fills totalUntrackedUsageUnits, which the previous commit missed
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
Evicting before the mutation commits left a race: a concurrent JWT
request could re-cache the old mapping between the eviction and the
commit, keeping a deleted or renamed claim authorized until the cache
TTL expired. Flagged by review on PR #39808.
The base vector store interface grew an embedding_executor argument, and
litellm.vector_stores.search now always passes one. MongoDB still carried its
own embedding_fn/aembedding_fn constructor seam, so every search through the
public entry point failed with an unexpected keyword argument.
Drop the local seam in favour of the shared executor: one path instead of two,
and the unit tests now drive the same seam production uses.
* fix(health): probe test_connection with the credential the request names
/health/test_connection matches the request's model string against the
configured deployments and merges the match's litellm_params underneath the
request. A request that named a stored credential but no key of its own
still satisfied the "request sets no connection fields" test, so it inherited
the matched deployment's api_key and api_base, and load_credentials_from_list
then skipped the named credential because api_key was already set.
A wildcard route covering the model is enough to match, so the Add Model
page's Test Connect probed with an unrelated deployment's key while echoing
back the credential that was selected.
Naming a credential the configuration does not name now withholds the
configuration's credential fields, the same set already withheld from a
request that supplies its own endpoint. Naming no credential still inherits
them, as documented.
* test(health): drop test docstrings that restate their own names
* test(health): assert the credential probe on the wire, not on the call args
The connection-test regressions patched litellm.ahealth_check and read the
params handed to it. Driving the endpoint through the app with respx faking
the upstream instead lets the real credential resolution run, so the tests
assert the key and host that actually go out, which is what the bug was about.
It also drops three of the five patched proxy internals; the two that are left
are proxy-global wiring with no injection seam, the same ones the image_edit
connection test already has to reach for.
* chore(ui): regenerate schema.d.ts for the test_connection docs change
Adds stall_escalation_enabled to the complexity router: when the assistant's
own recent tool calls look stuck (identical repeats, or repeated tool
errors on a surface that reports one), the request is bumped one
configured tier higher, the automatic counterpart to escalation_keywords.
Detection is stateless: it rereads the last stall_escalation_window tool
calls from that request's own message list on every classified turn, so
the bump lasts only as long as the recent calls still look stuck and
lifts on its own once they don't, and evidence survives a plain follow-up
like "try again" instead of resetting on the newest human ask.
Off by default. Rejected together with session_affinity and
classification_mode='user_turn', which both replay a held routing
decision instead of classifying most turns, and with tier_definitions,
for the same reason escalation_keywords is: both rely on the built-in
tier severity order a custom tier set does not define.
Dashboard controls for this are not included; config.yaml and the
management API accept it today through ComplexityRouterConfig.
/key/regenerate carries the JWT-to-key mapping to the new token via FK
cascade, but the jwt_key_mapping cache entry kept resolving the old
(now invalid) token for up to virtual_key_mapping_cache_ttl. Snapshot
the key's mapping cache keys before the token update and evict them
with evict_and_broadcast so every worker drops the stale entry.
Also share the cache-key format through jwt_key_mapping_cache_key and
upgrade the /jwt/key/mapping CRUD endpoints from local-only deletes to
evict_and_broadcast, closing the same cross-worker staleness there.
A row's cost sums only the daily rows that carry a tracked cost, so it
silently under-reports whenever some rows are NULL (pre-migration days,
old pods mid-rollout, an unpriced counter). Both usage endpoints now
return the per-counter units behind those NULL rows next to the cost
(untrackedUsageUnits / totalUntrackedUsageUnits on the overview,
untracked_usage_units on the detail), so a partial cost is never mistaken
for a complete one and the reader can see exactly what it excludes
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The create form rejected names with slashes or spaces even though the
proxy stores and routes any non-empty string. Drop the client-only
character pattern and trim the name before the required check so a
whitespace-only name is still refused
Claude-Session: https://claude.ai/code/session_01HkaXiD6gssHnx3kqu1rR8C
