Build one ClientConfig that names ring and loads the native roots once, and hand it to every tokio-tungstenite dial as its connector instead of installing a process-wide default from the dial path. Each of the three dial sites gets a wss:// test that reproduces the panic if its connector is dropped.
basedpyright counts a named capture after an exhaustive match under
reportUnnecessaryComparison, which pushed the merged tree one over the
budget; the wildcard case with assert_never on the bound subject is the
shape the rest of the codebase uses
- ruff format on auto_router_compression.py (a long comprehension wrapped
across three lines instead of one)
- prettier on buildAutoRouterCompression.ts and the two test files it touched
- ComplexityRouterConfig.tsx crossed the 800-line eslint max-lines ceiling
once the compression accordion entry landed. Extracted TierRowSelect into
its own file (already self-contained, used only within this file and
PlanModeOverrideControls) and simplified CompressionControls' props to a
single state/onChange pair instead of six individual callbacks, moving the
per-field derivation into the component that already owns this state shape
* fix(anthropic): never carry cache_control on translated thinking blocks
The /v1/messages adapter built every thinking and redacted_thinking block with
cache_control=content.get("cache_control", {}), so a block the client never
marked still came out carrying an empty cache_control. anthropic_messages_pt
replays thinking blocks verbatim and first, so that value landed at content[0]
of the outbound assistant message and Anthropic rejected the request with
messages.N.content.0.thinking.cache_control: Extra inputs are not permitted.
Anthropic's schema has no cache_control on either block type, so there is
nothing to gate or translate here, only to stop copying. Every sibling block
type already routes through _add_cache_control_if_applicable; these two were
the only ones setting the key unconditionally.
This is reachable from any caller that round-trips Anthropic messages through
the OpenAI shape, which is why shadow eval saw it on a majority of sampled
Claude Code turns while the same traffic served natively was fine.
* test(anthropic): assert the outbound wire body for redacted thinking blocks
Replace test-model-map.yml with a pull_request_target guard that validates the
cost map, its backup, and its generated schema on every PR, and additionally
enforces the sync bot contract on litellm_cost_map_sync_* branches: only the
three cost map files may change, no model or field is removed, and the special
root keys stay untouched.
* feat(cli): sync OpenCode models from /v1/models in lite opencode
lite opencode now fetches the proxy's /v1/models with the resolved key and
hands OpenCode an OPENCODE_CONFIG_CONTENT declaring a litellm provider
(@ai-sdk/openai-compatible, proxy /v1 base URL, {env:OPENAI_API_KEY}) with one
model entry per listed chat model, so the model picker mirrors the proxy
without a hand-maintained opencode.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): sync OpenCode models only after the key check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An untagged marker (no tags key or empty tags list) was matching every request
because requested.issuperset(frozenset()) is always true. When an alias carried
multiple markers, the loop tried tag-matched markers first, but an untagged one
could still match the tag-match query, and then the first one with a policy would
be returned. Now only markers with a non-empty tags list can match via the
tag-specific lookup; untagged markers are tried only after all tag-specific ones.
Regression test added: test_tag_scoped_marker_takes_precedence_over_untagged
fails with the old code.
Also removed unused Any import per greptile's typing note.
The parser message alone cannot separate a judge that answered with nothing
from one truncated mid-object, and the two want opposite fixes. Records the
reply's shape, never its text, since no attempt row carries sampled content.
- Suppression markers now carry the per-process token `_pre_call_marker`
already uses, so a caller cannot switch off an always-on PII, content-filter
or compression guardrail by naming it in its own request metadata.
- Routing set to "none" with the model side compressed now classifies on the
pre-compression snapshot instead of the model-side guardrail's output.
- Both the proxy's pre-call arming and the router's routing hook resolve the
policy through one tag-aware `policy_for_model`, so an alias with several
tag-scoped markers can no longer suppress one marker's guardrail and then
route under another marker's policy.
- The pre-compression snapshot moved from request metadata to a ContextVar:
`refresh_proxy_server_request_body_snapshot` copies metadata into
`proxy_server_request.body`, which deployments persist, and the snapshot
holds the prompt as it was before any masking guardrail rewrote it.
- The compression selector lists Compresr guardrails too, not just Headroom.
The test-quality gate rejects patching litellm.acompletion, and faking the
HTTP boundary is the stronger test anyway: the 503, 500 and 502 responses now
travel through the real OpenAI SDK and exception mapping before the router
decides how many times to retry. Adds a case showing that a 503 key does not
govern a 502.
An auto router marker deployment can now set auto_router_routing_compression
and auto_router_model_compression in its litellm_params, naming the
compression guardrail each hop should use (or "none" for no compression on
that hop). Neither key set means the request's own compression guardrails
keep applying to both hops unchanged.
Backend: Router.async_pre_routing_hook resolves the marker's policy and
compresses a copy of the messages for the routing decision only when the
policy differs from what the model call already got; when both hops share
the same compression, it reuses what the ordinary pre-call guardrail
pipeline already produced instead of compressing twice. The proxy layer
suppresses every other compression guardrail once a policy is engaged and
arms the model-side guardrail even when it is not default_on.
UI: the auto router's Detailed Configuration gains an Advanced: Compression
section with a routing-decision selector and a same/different toggle for
the model call, matching the same/different address pattern.
Two issues Bugbot found on #39809.
A keyword_tier_rule forces its tier and returns before any classification
runs, so stall escalation never reached that path even though keyword
escalation did. That left the one path that can pin a weak model to a
whole conversation as the one path a stall could not lift. Stall
detection now resolves before the override branch and both paths bump.
The dashboard switch disabled itself whenever session pinning or
user-turn classification was on, including for a router that already had
stall escalation enabled. The conflicting keys stayed set, the backend
rejected the save, and the disabled switch was the only way to clear
them. It now disables only the off-to-on direction.
Replace the hand-ordered isinstance ladder in get_num_retries_from_retry_policy
with a class-to-field mapping walked along the exception's MRO, most specific
class first. A RetryPolicy field can no longer go silently dead the way
InternalServerErrorRetries did, and subclasses such as
ContentPolicyViolationError or MidStreamFallbackError pick up their parent's
field when they have none of their own.
Add a DefaultRetries catch-all so errors without a dedicated field
(BadGatewayError, APIConnectionError, NotFoundError, ...) can be governed by the
policy too. Specific fields still win over DefaultRetries.
Wiring the previously dead InternalServerErrorRetries changes one test
expectation: a policy of 2 now overrides a per-deployment num_retries of 5, so
the amplification test sees 3 upstream requests instead of 6.
Expose DefaultRetries as "All other errors" in the Admin UI retry settings tab
and ratchet the lint budgets down by the violations this branch fixed.
The access group detail page rendered MCP servers, agents, attached teams and keys as bare ids, so an admin had to look each one up elsewhere to audit a group
Every access group response now also carries access_mcp_servers, access_agents, assigned_teams and assigned_keys as {id, name} pairs. Names come from the DB rows first and fall back to config-declared MCP servers and agents (including legacy agent ids), resolved with one query per table across all groups in a list call. The existing *_ids columns are unchanged
The UI renders the name with the id in a tooltip, links teams and keys to their detail pages, and shows the raw id only when nothing resolves
Counting whichever pattern was most common across the window escalated a
task that had already recovered: three identical failures stay in the
window for a few turns after the model breaks out of them, and on their
own they met the threshold.
Both tests now anchor on the newest call. The repeat test counts calls
matching the newest one, and the error test only runs while the newest
call is itself an error, so a window whose recent calls are healthy no
longer escalates. The matches still do not have to be adjacent, so a
retry loop broken up by an unrelated lookup keeps counting.
Found by Greptile on #39809.
The prior commit claimed claude-sonnet-5 reasons invisibly by default and eats
the judge's budget regardless of what the call asks for. Verified against a
live proxy: with no thinking param (what _call_judge sends today), forced
tool-choice json_mode, native structured output, and even an explicit
thinking=adaptive, the model returned 0 reasoning tokens and a clean compact
verdict every time, on prompts up to several thousand characters.
The real mechanism only shows up with an elevated reasoning_effort or
output_config.effort on the request, which happens when the judge_model
deployment is configured with one, e.g. an admin pointing the judge at their
best reasoning model. Reproduced directly: reasoning_effort=max, 300-token
cap, real Anthropic reply came back finish_reason=length, content=None, 299
of 300 tokens spent on reasoning. Same request at 4096 returned a valid
verdict. This is a narrower, verified claim than the one it replaces.
Adds an "Advanced: Stalled Task Escalation" section to the complexity
router config: a toggle plus the repeat threshold and the window of recent
tool calls to examine. Both knobs are seeded on enable and cleared on
disable, so an off router sends none of the three keys, which is what the
backend requires next to session pinning and a custom tier set.
The toggle locks out with an explanation when "How often to classify" is
set to once-per-session or new-user-message, since both replay a held
routing decision instead of classifying and a stall would never reach the
classifier. The keys join the custom-tier restriction registry, which both
strips them from a custom-tier save and marks the section restricted.
ResponseFormatControls moves into its own file to keep
ComplexityRouterConfig.tsx under the 800-line lint ceiling, matching the
one-file-per-control layout its siblings already use.
Async searches reach the sync client through executor threads, so the LRU cache
is shared state. A key could be evicted between the lookup and the reordering
that followed it, and the reordering then raised KeyError and became a 500.
Reproduced at 15 failures per run with 16 threads over 34 keys and a 1ns switch
interval; the regression test is that workload.
The cap covers reasoning tokens as well as the verdict, and the models people
pick as judges reason before answering whether the call asks them to or not:
Anthropic's 5 family thinks adaptively and cannot be told not to, so the
reasoning bills against max_tokens with nothing in the request to opt out.
At 1500 the reasoning consumed the budget and the reply arrived empty or cut
off mid-object, which the attempt recorded as an unparseable judge verdict
rather than a result. Headroom costs nothing: max_tokens is a ceiling and only
generated tokens bill, so the only movement is that judge calls which used to
bill their full budget and return nothing now return a verdict.
Deliberately not passing reasoning_effort to bound the reasoning instead:
is_thinking_enabled treats any reasoning_effort as thinking-enabled, which
drops the forced tool_choice that json_mode relies on and turns thinking on
with a 1024-token floor for judges that were not reasoning at all.
A complexity tier can name several model groups, but the pool pick and the session-pin
replay both returned a group without consulting deployment health, so a group whose every
deployment was in cooldown was still routed to and the request died at the router's
zero-deployment check while a healthy peer sat in the same tier.
Gate the decided response at the pre-routing hook's exits, the seam the modality gate
already occupies, so every arm that can place a request is covered by one owner: a fresh
classification, a replayed or escalated pin, a plan-mode floor, a context-window
escalation, an adaptive pick, and whatever arm is added next.
Peers come from the decided tier only. Climbing to a higher tier costs more than the
classifier asked for and is left to a follow-up. The gate fails open on every uncertainty:
an unreadable cooldown view, a decision carrying no tier, a group the router knows no
deployments for, or a tier whose peers are all cooling.
litellm only retries 408, 409, 429 and 5xx, so classifying a dropped connection
as a 400 turned one replica set failover into a permanently failed search. It is
a 503 now, with the message still naming the misconfigurations that also close a
connection.
The client cache skipped insertion once it held 32 entries, so any store added
after that rebuilt its client on every search, paying an SRV lookup, a TLS
handshake and topology discovery each time. It evicts the least recently used
entry instead, which only drops the cache's own reference.
Also trims the explanatory comments to the one-line form the repo asks for.
A row that received both priced and unpriced increments used to collapse
to cost NULL, throwing away the priced subtotal and making every unit on
it read as untracked. The rollup now carries a second column,
untracked_units, that the aggregator increments for units with no known
price while cost keeps accruing for the rest, so cost covers exactly
units - untracked_units. Rows written before the migration keep cost
NULL and still read as untracked in full
The endpoints read untracked units off the column (or the whole row for
a legacy NULL) rather than from a NULL filter, and the policies overview
now fills totalUntrackedUsageUnits, which the previous commit missed
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
