The router reused the model hop's compression for routing whenever both hops named
the same guardrail, on the premise that arm_pre_call had already run it. Only the
proxy calls arm_pre_call, so through the SDK nothing armed the guardrail and nothing
had compressed anything: the shortcut skipped routing compression too and served the
request with no compression on either hop. The reuse is now conditional on the model
hop actually having been armed.
The Admin UI hydrated an absent auto_router_model_compression as same-as-routing,
while the backend reads it as no model-hop compression. Opening a router configured
with only auto_router_routing_compression and saving any unrelated edit wrote the
routing guardrail onto the model hop, silently starting to compress the model call.
Both carry a regression test that fails when the fix is reverted.
A shadow eval job could only be scoped by identity, so "this user's traffic on model X
across every key they own" was not expressible and a models field on the start body was
silently dropped. The job now carries a models list that every target is narrowed to,
matched on the requested model group with model_group_alias resolved on both sides. An
unresolvable name is a 400 at start. Empty means every model, which is what every existing
row reads as. The dashboard start form gains an "Only on models" picker and the job
headline shows the scope.
* feat(otel): stamp litellm.request.route on the LLM call span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): drop redundant comment on REQUEST_ROUTE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): Final-annotate route test locals, drop field comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): read litellm.request.route off the server span
The LLM call span took the auth-normalized literal path from logging
metadata, which disagrees with the SERVER span wherever FastAPI matched a
template: on /engines/{model:path}/chat/completions the LLM span spelled the
model name while http.route carried the template, so the two spans grouped
into different buckets and the PR's premise did not hold.
Read the value off the span that already holds it. The request's root SERVER
span is anchored per request for parenting, and its attributes stay readable
after it ends, so request_root_http_route() answers from the async close
callback with the same http.route the SERVER span exports: the route template
on a normal route, the literal path where the passthrough hook rewrote it, and
the mount point on an MCP call. Nothing has to re-derive any of that, so the
two spans cannot drift apart.
The route the proxy recorded at auth stays as the backstop for a deployment
whose FastAPI instrumentation never mounted, where there is no server span to
disagree with. Off the proxy the attribute is omitted rather than empty.
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng He <yucheng@berri.ai>
Suppression state moves out of request metadata into a request-scoped ContextVar.
refresh_proxy_server_request_body_snapshot copies metadata into
proxy_server_request.body, which deployments persist to spend logs, so the marker
naming each suppressed guardrail was readable by the caller whose request produced
it. Recovering it was enough to replay {token}:{name} for any CustomGuardrail and
switch off a PII or content-filter guardrail, since the check never verified the
named guardrail was a compression one. Nothing is read from metadata now, so there
is no marker to forge and the per-process token is no longer needed.
Routing-side compression reads the live messages instead of a pre-guardrail copy.
arm_pre_call runs before the pre-call hook, so its snapshot held the prompt as it
was before any masking guardrail rewrote it, and messages_for_routing handed that
to a compression guardrail which POSTs it to an external service. Masked content
left the proxy anyway. The cost is one combination: when the model hop compressed
and the hops differ, routing now classifies on the compressed text, since no
uncompressed copy survives that a masking guardrail has already seen.
policy_for_model no longer falls back to a marker scoped to tags the request does
not carry, which applied an 'eu' policy to a 'us' request on config order alone.
Each fix carries a regression test; all three fail when the fix is reverted.
The classifier scores extracted text, so a turn whose complexity lives in
its image is invisible to it: a screenshot of a stack trace classifies on
its caption, and an image-only turn flattens to empty text and never
reaches the classifier at all.
classifier_llm_config.vision opts in, off by default, with max_images
bounding what one turn can add. Images are still dropped when the
classifier model is declared supports_vision false. Anthropic and
Responses image parts are rewritten into chat-completions dialect before
they reach the classifier call, since /v1/messages hands the pre-routing
hook its own dialect untranslated.
The local scorer no longer short-circuits heuristic_first or hybrid on a
turn carrying forwarded images, because it reads text alone and its
confidence describes a request it has only partly seen.
The completed-batch early return skipped both the cancel and the list
assertion while the lifecycle's covers markers still credited both cells.
List does not depend on the batch being cancellable, so it now runs either
way; cancel on a completed batch stays a documented vacuous pass
* fix(shadow_eval): tell a tool-call shadow reply apart from an empty one
Both arrive at the attempt row as the same 'shadow router returned an empty
response', because _chat_final_text returns empty for a tool-final turn by
design and for a reply that genuinely carried no text. Those are different
things: an arm that chose a tool where the real model wrote prose is a
divergence a text judge cannot score, and the sampling side already drops the
real arm's tool-final turns for exactly that reason, so the shadow side reads
as a fault where the real side reads as a filter. A job that is almost all
'empty response' gives no way to tell a tool-happy arm from a broken one.
The error now names which of the two happened, and carries the finish_reason
and the routed model so the row says what the arm was doing. Every varying
part sits behind the first semicolon: operators read these by grouping on the
error text, and interpolating the model into the leading sentence would make
each row its own group.
The outcome stays 'error'. Whether a tool-call reply should instead be its own
non-judged outcome, excluded from the loss rate the way the real arm's
tool-final turns already are, needs the four aggregation predicates that spell
judged as outcome != 'error' rewritten, and a decision on how to surface the
new bucket. That is a separate change.
* fix(shadow_eval): read the tool name of a custom tool call
A custom tool call carries its name under custom.name with no function key,
so every one of them reported as tool=unnamed.
* feat(shadow_eval): judge tool calls instead of dropping the turn
A turn where either arm called a tool was discarded before it could be
compared: the real arm's at sampling, the shadow arm's as an error row. On
agentic traffic that is most of the traffic, so a job set to sample 10% was
sampling 10% of the prose-only slice. Tool calls now serialize to text on
every surface and are judged like any other response, and the judge is told
a tool call is not a defect so it scores the choice rather than the shape.
* feat(shadow_eval): show the judge what tools were available
Both arms were offered the same tools, but the judge only ever saw the
chosen call in isolation, with no way to tell whether a better tool existed
or the arguments matched what the tool expects. Threads the request's tool
definitions (name and description only) into the judge prompt, capped and
omitted entirely on turns that offered none.
* fix(shadow_eval): read a custom tool definition's name from custom, not function
A chat-completions custom tool definition nests name and description under
custom, mirroring how a custom tool call nests them (openai.types.chat.
ChatCompletionCustomToolParam). Reading only function rendered every one as
unnamed, telling the judge nothing about what it was.
Bedrock passthrough Converse routes flattened every non-empty string under
toolConfig.tools into the guardrail INPUT texts, so tool names, tool
descriptions and JSON-schema strings (object, property names, titles, type
names, enum values) each arrived as a separate guardrail item. A request whose
only prompt was one benign user message could be blocked outright because a
denied term appeared in an app-authored tool definition.
Tool definitions are now excluded from the extracted texts, matching every
other guardrail translation handler, which carries tool definitions in the
structured tools input rather than in texts. Caller content stays scanned:
message text, toolUse.input, toolResult content and json, and
additionalModelRequestFields are unchanged.
Resolves LIT-5797
Two fixes for the zero-headroom basedpyright budget:
- arm_pre_call's data parameter is dict[str, object], not MutableMapping: the
latter is itself banned by LIT001 with no benefit, and it mismatched every
dict-typed helper (get_or_create_metadata_bucket, resolve_structured_messages,
_get_tags_from_request_kwargs), which is what the budget was actually flagging.
- Router.async_pre_routing_hook computed pre_routing_hook_response in one shot
instead of reassigning a Final-annotated local.
The remaining two reportArgumentType hits are pre-existing: LiteLLM_Params(**merged)
in _create_deployment_object already fails this check for all ~165 of its other
fields, since the merged dict's value type is partly untyped/float; adding two new
string fields to the model just grows that existing pile by two. Suppressed at the
one call site with a reason, since fixing the root typing is out of scope here.
Bedrock batch cancel (StopModelInvocationJob) and the managed list view
both work through the proxy since LIT-4774, but the batches e2e still
gated them off and the coverage registry claimed no cell for either.
Flip can_cancel/can_list for the Bedrock provider, assert cancel the
same way the OpenAI leg does, add the two registry cells the gates
select, and update COVERAGE.md
* fix(datadog_llm_obs): keep the guardrail audit record under message redaction
Redaction nulled `guardrail_information` on the span whole, so an operator
running `turn_off_message_logging` (or a caller sending
`x-litellm-enable-message-redaction`) lost the record of which guardrails ran,
what they returned, and what they masked. Four of the record's fields can quote
the prompt; the rest report what the guardrail decided without reproducing it.
Replace only those four, the way
`_sanitize_guardrail_information_for_spend_logs` already does for spend logs,
and declare the field list once in `litellm/types/utils.py` so both readers
share it.
* fix(datadog_llm_obs): keep a lone guardrail record, and test through the span
Review round 1.
A guardrail that writes the metadata key itself leaves a single record where
the type says list, which Prometheus already normalizes at
`_guardrail_overhead_seconds`. Redaction dropped that shape and the latency
extraction raised on it, so the span was lost outright. Normalize once and use
it in both places.
The new tests now drive `create_llm_obs_payload` instead of reading the module's
private helpers and the record's declared field names.
The gate has no headroom, so the new module had to stop introducing mutable
collections rather than spend budget on them:
- the marker lookup falls back to () and drops an `or {}` that isinstance
already covered
- the suppression list is stored as the tuple it was built as; the read side
in custom_guardrail accepts list or tuple, since JSON round-trips it to a list
- the snapshot holds MappingProxyType entries, so it is immutable at rest and
_snapshot_messages can hand back the stored tuple with no defensive copy
- arm_pre_call returns None instead of echoing back the dict it mutates in place
- _suppressed_by_auto_router_compression takes a Mapping, which is all it reads
The four remaining mutable spots are external contracts, each suppressed with
the reason: the pre-routing hook protocol types messages as list[dict], the
metadata["guardrails"] key is extended by litellm_pre_call_utils via an
isinstance(..., list) check, apply_guardrail takes a dict it writes stats into,
and pydantic's model_copy takes a dict.
The reset job evicts the cached end-user object only from its own worker's
in-memory cache (plus Redis), so every other uvicorn worker and replica keeps
the pre-reset spend for up to user_api_key_cache_ttl (60s by default). Those
workers pass that stale spend as fallback_spend, and since the authoritative
floor read returned None for spend:end_user: keys, get_current_spend handed
the stale value straight back and the end user kept getting 429 after the
rollover on every worker but the one that ran the reset.
The floor read now consults LiteLLM_EndUserTable.spend for end-user counters,
the same way keys, teams, users, and orgs already read their rows. It runs only
when the shared counter sits below the cached spend (a reset or a Redis
restart) and stays behind the existing 5s in-process marker, so the normal
request path still does no DB read. Cold end-user counters keep seeding from
the cached object rather than the row, so from_db is unchanged for them.
* fix(headroom): bound the /v1/compress and /v1/retrieve calls with a timeout
The headroom guardrail builds its client with get_async_httpx_client(GuardrailCallback)
and no params, and passes no timeout on either outbound call. That client's read, write
and pool legs are 600s (litellm.request_timeout when set explicitly, default 6000s), so
an unreachable or stalled compression service holds the caller's pre-call request open
for the whole window before unreachable_fallback ever runs. Because the client is shared
with every other no-params guardrail, each stalled call also pins a pooled connection for
the same window, so a saturated pool makes unrelated requests block on the pool leg.
Bound both calls at 60s by default, honoring litellm_params.timeout when set (the field
already exists and documents itself as the per-guardrail API timeout; headroom accepted
it and ignored it). The connect leg stays at the http_handler default, or the configured
budget when that is shorter, so a dead host still fails fast.
Live on a proxy against a stalled /v1/compress: 600.4s -> 60.2s before the 502, and 5.2s
with timeout: 5 configured.
* fix(headroom): reject non-finite timeouts and trim the timeout commentary
`timeout: .inf` on a Headroom guardrail reached httpx and the aiohttp transport
raised OverflowError, so every request came back as a raw 500 instead of going
through unreachable_fallback. Reject non-finite values the same way as
non-positive ones, and cut the comments and docstrings back to what the code
does not already say.
