* fix(shadow_eval): tell a tool-call shadow reply apart from an empty one
Both arrive at the attempt row as the same 'shadow router returned an empty
response', because _chat_final_text returns empty for a tool-final turn by
design and for a reply that genuinely carried no text. Those are different
things: an arm that chose a tool where the real model wrote prose is a
divergence a text judge cannot score, and the sampling side already drops the
real arm's tool-final turns for exactly that reason, so the shadow side reads
as a fault where the real side reads as a filter. A job that is almost all
'empty response' gives no way to tell a tool-happy arm from a broken one.
The error now names which of the two happened, and carries the finish_reason
and the routed model so the row says what the arm was doing. Every varying
part sits behind the first semicolon: operators read these by grouping on the
error text, and interpolating the model into the leading sentence would make
each row its own group.
The outcome stays 'error'. Whether a tool-call reply should instead be its own
non-judged outcome, excluded from the loss rate the way the real arm's
tool-final turns already are, needs the four aggregation predicates that spell
judged as outcome != 'error' rewritten, and a decision on how to surface the
new bucket. That is a separate change.
* fix(shadow_eval): read the tool name of a custom tool call
A custom tool call carries its name under custom.name with no function key,
so every one of them reported as tool=unnamed.
* feat(shadow_eval): judge tool calls instead of dropping the turn
A turn where either arm called a tool was discarded before it could be
compared: the real arm's at sampling, the shadow arm's as an error row. On
agentic traffic that is most of the traffic, so a job set to sample 10% was
sampling 10% of the prose-only slice. Tool calls now serialize to text on
every surface and are judged like any other response, and the judge is told
a tool call is not a defect so it scores the choice rather than the shape.
* feat(shadow_eval): show the judge what tools were available
Both arms were offered the same tools, but the judge only ever saw the
chosen call in isolation, with no way to tell whether a better tool existed
or the arguments matched what the tool expects. Threads the request's tool
definitions (name and description only) into the judge prompt, capped and
omitted entirely on turns that offered none.
* fix(shadow_eval): read a custom tool definition's name from custom, not function
A chat-completions custom tool definition nests name and description under
custom, mirroring how a custom tool call nests them (openai.types.chat.
ChatCompletionCustomToolParam). Reading only function rendered every one as
unnamed, telling the judge nothing about what it was.
Bedrock passthrough Converse routes flattened every non-empty string under
toolConfig.tools into the guardrail INPUT texts, so tool names, tool
descriptions and JSON-schema strings (object, property names, titles, type
names, enum values) each arrived as a separate guardrail item. A request whose
only prompt was one benign user message could be blocked outright because a
denied term appeared in an app-authored tool definition.
Tool definitions are now excluded from the extracted texts, matching every
other guardrail translation handler, which carries tool definitions in the
structured tools input rather than in texts. Caller content stays scanned:
message text, toolUse.input, toolResult content and json, and
additionalModelRequestFields are unchanged.
Resolves LIT-5797
Bedrock batch cancel (StopModelInvocationJob) and the managed list view
both work through the proxy since LIT-4774, but the batches e2e still
gated them off and the coverage registry claimed no cell for either.
Flip can_cancel/can_list for the Bedrock provider, assert cancel the
same way the OpenAI leg does, add the two registry cells the gates
select, and update COVERAGE.md
* fix(datadog_llm_obs): keep the guardrail audit record under message redaction
Redaction nulled `guardrail_information` on the span whole, so an operator
running `turn_off_message_logging` (or a caller sending
`x-litellm-enable-message-redaction`) lost the record of which guardrails ran,
what they returned, and what they masked. Four of the record's fields can quote
the prompt; the rest report what the guardrail decided without reproducing it.
Replace only those four, the way
`_sanitize_guardrail_information_for_spend_logs` already does for spend logs,
and declare the field list once in `litellm/types/utils.py` so both readers
share it.
* fix(datadog_llm_obs): keep a lone guardrail record, and test through the span
Review round 1.
A guardrail that writes the metadata key itself leaves a single record where
the type says list, which Prometheus already normalizes at
`_guardrail_overhead_seconds`. Redaction dropped that shape and the latency
extraction raised on it, so the span was lost outright. Normalize once and use
it in both places.
The new tests now drive `create_llm_obs_payload` instead of reading the module's
private helpers and the record's declared field names.
The gate has no headroom, so the new module had to stop introducing mutable
collections rather than spend budget on them:
- the marker lookup falls back to () and drops an `or {}` that isinstance
already covered
- the suppression list is stored as the tuple it was built as; the read side
in custom_guardrail accepts list or tuple, since JSON round-trips it to a list
- the snapshot holds MappingProxyType entries, so it is immutable at rest and
_snapshot_messages can hand back the stored tuple with no defensive copy
- arm_pre_call returns None instead of echoing back the dict it mutates in place
- _suppressed_by_auto_router_compression takes a Mapping, which is all it reads
The four remaining mutable spots are external contracts, each suppressed with
the reason: the pre-routing hook protocol types messages as list[dict], the
metadata["guardrails"] key is extended by litellm_pre_call_utils via an
isinstance(..., list) check, apply_guardrail takes a dict it writes stats into,
and pydantic's model_copy takes a dict.
The reset job evicts the cached end-user object only from its own worker's
in-memory cache (plus Redis), so every other uvicorn worker and replica keeps
the pre-reset spend for up to user_api_key_cache_ttl (60s by default). Those
workers pass that stale spend as fallback_spend, and since the authoritative
floor read returned None for spend:end_user: keys, get_current_spend handed
the stale value straight back and the end user kept getting 429 after the
rollover on every worker but the one that ran the reset.
The floor read now consults LiteLLM_EndUserTable.spend for end-user counters,
the same way keys, teams, users, and orgs already read their rows. It runs only
when the shared counter sits below the cached spend (a reset or a Redis
restart) and stays behind the existing 5s in-process marker, so the normal
request path still does no DB read. Cold end-user counters keep seeding from
the cached object rather than the row, so from_db is unchanged for them.
* fix(headroom): bound the /v1/compress and /v1/retrieve calls with a timeout
The headroom guardrail builds its client with get_async_httpx_client(GuardrailCallback)
and no params, and passes no timeout on either outbound call. That client's read, write
and pool legs are 600s (litellm.request_timeout when set explicitly, default 6000s), so
an unreachable or stalled compression service holds the caller's pre-call request open
for the whole window before unreachable_fallback ever runs. Because the client is shared
with every other no-params guardrail, each stalled call also pins a pooled connection for
the same window, so a saturated pool makes unrelated requests block on the pool leg.
Bound both calls at 60s by default, honoring litellm_params.timeout when set (the field
already exists and documents itself as the per-guardrail API timeout; headroom accepted
it and ignored it). The connect leg stays at the http_handler default, or the configured
budget when that is shorter, so a dead host still fails fast.
Live on a proxy against a stalled /v1/compress: 600.4s -> 60.2s before the 502, and 5.2s
with timeout: 5 configured.
* fix(headroom): reject non-finite timeouts and trim the timeout commentary
`timeout: .inf` on a Headroom guardrail reached httpx and the aiohttp transport
raised OverflowError, so every request came back as a raw 500 instead of going
through unreachable_fallback. Reject non-finite values the same way as
non-positive ones, and cut the comments and docstrings back to what the code
does not already say.
* fix(anthropic): never carry cache_control on translated thinking blocks
The /v1/messages adapter built every thinking and redacted_thinking block with
cache_control=content.get("cache_control", {}), so a block the client never
marked still came out carrying an empty cache_control. anthropic_messages_pt
replays thinking blocks verbatim and first, so that value landed at content[0]
of the outbound assistant message and Anthropic rejected the request with
messages.N.content.0.thinking.cache_control: Extra inputs are not permitted.
Anthropic's schema has no cache_control on either block type, so there is
nothing to gate or translate here, only to stop copying. Every sibling block
type already routes through _add_cache_control_if_applicable; these two were
the only ones setting the key unconditionally.
This is reachable from any caller that round-trips Anthropic messages through
the OpenAI shape, which is why shadow eval saw it on a majority of sampled
Claude Code turns while the same traffic served natively was fine.
* test(anthropic): assert the outbound wire body for redacted thinking blocks
* feat(cli): sync OpenCode models from /v1/models in lite opencode
lite opencode now fetches the proxy's /v1/models with the resolved key and
hands OpenCode an OPENCODE_CONFIG_CONTENT declaring a litellm provider
(@ai-sdk/openai-compatible, proxy /v1 base URL, {env:OPENAI_API_KEY}) with one
model entry per listed chat model, so the model picker mirrors the proxy
without a hand-maintained opencode.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): sync OpenCode models only after the key check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An untagged marker (no tags key or empty tags list) was matching every request
because requested.issuperset(frozenset()) is always true. When an alias carried
multiple markers, the loop tried tag-matched markers first, but an untagged one
could still match the tag-match query, and then the first one with a policy would
be returned. Now only markers with a non-empty tags list can match via the
tag-specific lookup; untagged markers are tried only after all tag-specific ones.
Regression test added: test_tag_scoped_marker_takes_precedence_over_untagged
fails with the old code.
Also removed unused Any import per greptile's typing note.
The parser message alone cannot separate a judge that answered with nothing
from one truncated mid-object, and the two want opposite fixes. Records the
reply's shape, never its text, since no attempt row carries sampled content.
- Suppression markers now carry the per-process token `_pre_call_marker`
already uses, so a caller cannot switch off an always-on PII, content-filter
or compression guardrail by naming it in its own request metadata.
- Routing set to "none" with the model side compressed now classifies on the
pre-compression snapshot instead of the model-side guardrail's output.
- Both the proxy's pre-call arming and the router's routing hook resolve the
policy through one tag-aware `policy_for_model`, so an alias with several
tag-scoped markers can no longer suppress one marker's guardrail and then
route under another marker's policy.
- The pre-compression snapshot moved from request metadata to a ContextVar:
`refresh_proxy_server_request_body_snapshot` copies metadata into
`proxy_server_request.body`, which deployments persist, and the snapshot
holds the prompt as it was before any masking guardrail rewrote it.
- The compression selector lists Compresr guardrails too, not just Headroom.
The test-quality gate rejects patching litellm.acompletion, and faking the
HTTP boundary is the stronger test anyway: the 503, 500 and 502 responses now
travel through the real OpenAI SDK and exception mapping before the router
decides how many times to retry. Adds a case showing that a 503 key does not
govern a 502.
An auto router marker deployment can now set auto_router_routing_compression
and auto_router_model_compression in its litellm_params, naming the
compression guardrail each hop should use (or "none" for no compression on
that hop). Neither key set means the request's own compression guardrails
keep applying to both hops unchanged.
Backend: Router.async_pre_routing_hook resolves the marker's policy and
compresses a copy of the messages for the routing decision only when the
policy differs from what the model call already got; when both hops share
the same compression, it reuses what the ordinary pre-call guardrail
pipeline already produced instead of compressing twice. The proxy layer
suppresses every other compression guardrail once a policy is engaged and
arms the model-side guardrail even when it is not default_on.
UI: the auto router's Detailed Configuration gains an Advanced: Compression
section with a routing-decision selector and a same/different toggle for
the model call, matching the same/different address pattern.
Two issues Bugbot found on #39809.
A keyword_tier_rule forces its tier and returns before any classification
runs, so stall escalation never reached that path even though keyword
escalation did. That left the one path that can pin a weak model to a
whole conversation as the one path a stall could not lift. Stall
detection now resolves before the override branch and both paths bump.
The dashboard switch disabled itself whenever session pinning or
user-turn classification was on, including for a router that already had
stall escalation enabled. The conflicting keys stayed set, the backend
rejected the save, and the disabled switch was the only way to clear
them. It now disables only the off-to-on direction.
Replace the hand-ordered isinstance ladder in get_num_retries_from_retry_policy
with a class-to-field mapping walked along the exception's MRO, most specific
class first. A RetryPolicy field can no longer go silently dead the way
InternalServerErrorRetries did, and subclasses such as
ContentPolicyViolationError or MidStreamFallbackError pick up their parent's
field when they have none of their own.
Add a DefaultRetries catch-all so errors without a dedicated field
(BadGatewayError, APIConnectionError, NotFoundError, ...) can be governed by the
policy too. Specific fields still win over DefaultRetries.
Wiring the previously dead InternalServerErrorRetries changes one test
expectation: a policy of 2 now overrides a per-deployment num_retries of 5, so
the amplification test sees 3 upstream requests instead of 6.
Expose DefaultRetries as "All other errors" in the Admin UI retry settings tab
and ratchet the lint budgets down by the violations this branch fixed.
The access group detail page rendered MCP servers, agents, attached teams and keys as bare ids, so an admin had to look each one up elsewhere to audit a group
Every access group response now also carries access_mcp_servers, access_agents, assigned_teams and assigned_keys as {id, name} pairs. Names come from the DB rows first and fall back to config-declared MCP servers and agents (including legacy agent ids), resolved with one query per table across all groups in a list call. The existing *_ids columns are unchanged
The UI renders the name with the id in a tooltip, links teams and keys to their detail pages, and shows the raw id only when nothing resolves
Counting whichever pattern was most common across the window escalated a
task that had already recovered: three identical failures stay in the
window for a few turns after the model breaks out of them, and on their
own they met the threshold.
Both tests now anchor on the newest call. The repeat test counts calls
matching the newest one, and the error test only runs while the newest
call is itself an error, so a window whose recent calls are healthy no
longer escalates. The matches still do not have to be adjacent, so a
retry loop broken up by an unrelated lookup keeps counting.
Found by Greptile on #39809.
The prior commit claimed claude-sonnet-5 reasons invisibly by default and eats
the judge's budget regardless of what the call asks for. Verified against a
live proxy: with no thinking param (what _call_judge sends today), forced
tool-choice json_mode, native structured output, and even an explicit
thinking=adaptive, the model returned 0 reasoning tokens and a clean compact
verdict every time, on prompts up to several thousand characters.
The real mechanism only shows up with an elevated reasoning_effort or
output_config.effort on the request, which happens when the judge_model
deployment is configured with one, e.g. an admin pointing the judge at their
best reasoning model. Reproduced directly: reasoning_effort=max, 300-token
cap, real Anthropic reply came back finish_reason=length, content=None, 299
of 300 tokens spent on reasoning. Same request at 4096 returned a valid
verdict. This is a narrower, verified claim than the one it replaces.
Async searches reach the sync client through executor threads, so the LRU cache
is shared state. A key could be evicted between the lookup and the reordering
that followed it, and the reordering then raised KeyError and became a 500.
Reproduced at 15 failures per run with 16 threads over 34 keys and a 1ns switch
interval; the regression test is that workload.
