* fix(batches): register ownership for every batch create path
Since the team isolation change, the managed files hook decided whether a
response came from a create by looking for the managed input file id on it,
which only the unified input path sets. Batches created from a model-encoded
input file id, a model param, or a raw provider id with ?provider= never got
an ownership row, so they vanished from GET /v1/batches for the key that
created them.
The create endpoint now stamps a create marker on the response before the
hooks run, and the hook keys ownership registration and the batch-created
metric on that marker instead of on the input id format.
* test(batches): assert ownership registration through the managed files hook
The endpoint tests asserted the private create marker, which is wiring, not
behaviour. They now run the create through the real managed files hook and
assert the ownership row is written for the creating key on every create
path, with the unified path driven by a genuine encoded input file id
instead of patched decoders.
* fix(mcp): let config.yaml MCP servers pin server_id
A config-defined MCP server's id is a hash of server_name|url|transport|
auth_type|alias, recomputed on every config load, so editing any of those
fields mints a new id. Every key and team granted the old id via
object_permission.mcp_servers keeps pointing at an id that no longer exists,
and the server disappears from tools/list for them with nothing logged.
load_servers_from_config now uses an explicit server_id from the server's
config entry when present and falls back to the existing hash otherwise, so
grants survive url/name/alias edits. Rejected at config load: a blank or
non-string server_id, two entries claiming the same id, a pinned id already
held by a database-backed server, and a pinned id that is another entry's
server_name or alias (expand_permission_list matches ids before names, so
that one would capture the other server's grants). Because the database
registry loads after the config on startup, a database row that lands on a
pinned config id is reported as a warning from the database reload instead,
where it is decidable; the warning is latched on the shadowed set so the
config-reload timer does not reprint it every interval.
Deployments that do not set server_id keep the exact id they have today.
* fix(mcp): close two more pinned-id capture paths
A pinned server_id equal to an alias supplied through litellm_settings
mcp_aliases was accepted, because the collision index only held the entry's
own alias field. expand_permission_list matches ids before names, so grants
written for the aliased server resolved to the pinning one. mcp_aliases keys
whose target is a config server are now reserved the same way.
A pinned server_id equal to a database-backed server's name, server_name or
alias had the same effect against the database side, and could not be
rejected at config load because the database registry is not loaded yet. The
database reload now warns about it, latched like the existing shadow warning.
* fix(mcp): reserve only the aliases the loader actually assigns
Reserving every mcp_aliases key targeting a config server was too broad in
two ways: the mapping is ignored when the entry sets its own alias, and only
the first mapping for a server is ever applied. Both cases made a pinned
server_id that could never have collided abort proxy startup. Reserve only
the name load_servers_from_config will really assign.
The database capture warning also fired for a database server whose own id is
the config server_id. There the database row wins the id outright through
get_registry precedence, so the shadow warning above it is the accurate one
and the capture message contradicted it. Skip those rows.
Also mark the two litellm-internal patches in the reload test helper, which
the test-quality gate counts; the database reload has no other seam.
* fix(mcp): match the loader's alias check exactly, is None not falsiness
load_servers_from_config consults mcp_aliases only when the entry has no
alias key at all, so an entry setting alias: "" gets no mapped alias. The
collision index used falsiness and reserved the mapped name anyway, which
failed startup on a pinned server_id that could never have collided with it.
* fix(mcp): skip one identifier, not the whole database row
A database row can shadow one config server_id by id and capture another by
name at the same time. Skipping the entire row when its id shadowed a config
entry dropped the second warning, leaving the operator with half a diagnosis.
Skip only the identifier equal to the row's own id.
* fix(mcp): reject conflicting self-pinned server ids
* fix(mcp): validate config server names before building the identifier index
The collision check reads every entry's body up front, so a malformed
entry under an invalid name surfaced as an AttributeError instead of the
name validation error the loader gave before this change.
* fix(responses): decode JSON-string tool schemas before sending to the provider
A caller that hands a tool schema over already JSON-encoded reached the
Responses API with a string `parameters`, and the provider rejected the
request with a 400 naming the routed model instead of the offending tool.
Decode it at the one place every Responses request converges, and refuse
anything that is neither an object nor a string encoding one.
Collapses the duplicated input/tool sanitization block shared by the
request and compact-request builders into a single owner, so the decode
cannot be wired into one path and not the other.
* test(responses): pin null tool schemas as accepted, and type the parametrized cases
The Responses API serves `parameters: null` and an omitted schema alike, so
neither may raise. Pin both against a future tightening, annotate the
parametrized inputs, and trim the docstrings back to what the code does not
already say.
A shadow eval job could only be scoped by identity, so "this user's traffic on model X
across every key they own" was not expressible and a models field on the start body was
silently dropped. The job now carries a models list that every target is narrowed to,
matched on the requested model group with model_group_alias resolved on both sides. An
unresolvable name is a 400 at start. Empty means every model, which is what every existing
row reads as. The dashboard start form gains an "Only on models" picker and the job
headline shows the scope.
* feat(otel): stamp litellm.request.route on the LLM call span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): drop redundant comment on REQUEST_ROUTE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): Final-annotate route test locals, drop field comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): read litellm.request.route off the server span
The LLM call span took the auth-normalized literal path from logging
metadata, which disagrees with the SERVER span wherever FastAPI matched a
template: on /engines/{model:path}/chat/completions the LLM span spelled the
model name while http.route carried the template, so the two spans grouped
into different buckets and the PR's premise did not hold.
Read the value off the span that already holds it. The request's root SERVER
span is anchored per request for parenting, and its attributes stay readable
after it ends, so request_root_http_route() answers from the async close
callback with the same http.route the SERVER span exports: the route template
on a normal route, the literal path where the passthrough hook rewrote it, and
the mount point on an MCP call. Nothing has to re-derive any of that, so the
two spans cannot drift apart.
The route the proxy recorded at auth stays as the backstop for a deployment
whose FastAPI instrumentation never mounted, where there is no server span to
disagree with. Off the proxy the attribute is omitted rather than empty.
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng He <yucheng@berri.ai>
The classifier scores extracted text, so a turn whose complexity lives in
its image is invisible to it: a screenshot of a stack trace classifies on
its caption, and an image-only turn flattens to empty text and never
reaches the classifier at all.
classifier_llm_config.vision opts in, off by default, with max_images
bounding what one turn can add. Images are still dropped when the
classifier model is declared supports_vision false. Anthropic and
Responses image parts are rewritten into chat-completions dialect before
they reach the classifier call, since /v1/messages hands the pre-routing
hook its own dialect untranslated.
The local scorer no longer short-circuits heuristic_first or hybrid on a
turn carrying forwarded images, because it reads text alone and its
confidence describes a request it has only partly seen.
The completed-batch early return skipped both the cancel and the list
assertion while the lifecycle's covers markers still credited both cells.
List does not depend on the batch being cancellable, so it now runs either
way; cancel on a completed batch stays a documented vacuous pass
* fix(shadow_eval): tell a tool-call shadow reply apart from an empty one
Both arrive at the attempt row as the same 'shadow router returned an empty
response', because _chat_final_text returns empty for a tool-final turn by
design and for a reply that genuinely carried no text. Those are different
things: an arm that chose a tool where the real model wrote prose is a
divergence a text judge cannot score, and the sampling side already drops the
real arm's tool-final turns for exactly that reason, so the shadow side reads
as a fault where the real side reads as a filter. A job that is almost all
'empty response' gives no way to tell a tool-happy arm from a broken one.
The error now names which of the two happened, and carries the finish_reason
and the routed model so the row says what the arm was doing. Every varying
part sits behind the first semicolon: operators read these by grouping on the
error text, and interpolating the model into the leading sentence would make
each row its own group.
The outcome stays 'error'. Whether a tool-call reply should instead be its own
non-judged outcome, excluded from the loss rate the way the real arm's
tool-final turns already are, needs the four aggregation predicates that spell
judged as outcome != 'error' rewritten, and a decision on how to surface the
new bucket. That is a separate change.
* fix(shadow_eval): read the tool name of a custom tool call
A custom tool call carries its name under custom.name with no function key,
so every one of them reported as tool=unnamed.
* feat(shadow_eval): judge tool calls instead of dropping the turn
A turn where either arm called a tool was discarded before it could be
compared: the real arm's at sampling, the shadow arm's as an error row. On
agentic traffic that is most of the traffic, so a job set to sample 10% was
sampling 10% of the prose-only slice. Tool calls now serialize to text on
every surface and are judged like any other response, and the judge is told
a tool call is not a defect so it scores the choice rather than the shape.
* feat(shadow_eval): show the judge what tools were available
Both arms were offered the same tools, but the judge only ever saw the
chosen call in isolation, with no way to tell whether a better tool existed
or the arguments matched what the tool expects. Threads the request's tool
definitions (name and description only) into the judge prompt, capped and
omitted entirely on turns that offered none.
* fix(shadow_eval): read a custom tool definition's name from custom, not function
A chat-completions custom tool definition nests name and description under
custom, mirroring how a custom tool call nests them (openai.types.chat.
ChatCompletionCustomToolParam). Reading only function rendered every one as
unnamed, telling the judge nothing about what it was.
Bedrock passthrough Converse routes flattened every non-empty string under
toolConfig.tools into the guardrail INPUT texts, so tool names, tool
descriptions and JSON-schema strings (object, property names, titles, type
names, enum values) each arrived as a separate guardrail item. A request whose
only prompt was one benign user message could be blocked outright because a
denied term appeared in an app-authored tool definition.
Tool definitions are now excluded from the extracted texts, matching every
other guardrail translation handler, which carries tool definitions in the
structured tools input rather than in texts. Caller content stays scanned:
message text, toolUse.input, toolResult content and json, and
additionalModelRequestFields are unchanged.
Resolves LIT-5797
Bedrock batch cancel (StopModelInvocationJob) and the managed list view
both work through the proxy since LIT-4774, but the batches e2e still
gated them off and the coverage registry claimed no cell for either.
Flip can_cancel/can_list for the Bedrock provider, assert cancel the
same way the OpenAI leg does, add the two registry cells the gates
select, and update COVERAGE.md
* fix(datadog_llm_obs): keep the guardrail audit record under message redaction
Redaction nulled `guardrail_information` on the span whole, so an operator
running `turn_off_message_logging` (or a caller sending
`x-litellm-enable-message-redaction`) lost the record of which guardrails ran,
what they returned, and what they masked. Four of the record's fields can quote
the prompt; the rest report what the guardrail decided without reproducing it.
Replace only those four, the way
`_sanitize_guardrail_information_for_spend_logs` already does for spend logs,
and declare the field list once in `litellm/types/utils.py` so both readers
share it.
* fix(datadog_llm_obs): keep a lone guardrail record, and test through the span
Review round 1.
A guardrail that writes the metadata key itself leaves a single record where
the type says list, which Prometheus already normalizes at
`_guardrail_overhead_seconds`. Redaction dropped that shape and the latency
extraction raised on it, so the span was lost outright. Normalize once and use
it in both places.
The new tests now drive `create_llm_obs_payload` instead of reading the module's
private helpers and the record's declared field names.
The reset job evicts the cached end-user object only from its own worker's
in-memory cache (plus Redis), so every other uvicorn worker and replica keeps
the pre-reset spend for up to user_api_key_cache_ttl (60s by default). Those
workers pass that stale spend as fallback_spend, and since the authoritative
floor read returned None for spend:end_user: keys, get_current_spend handed
the stale value straight back and the end user kept getting 429 after the
rollover on every worker but the one that ran the reset.
The floor read now consults LiteLLM_EndUserTable.spend for end-user counters,
the same way keys, teams, users, and orgs already read their rows. It runs only
when the shared counter sits below the cached spend (a reset or a Redis
restart) and stays behind the existing 5s in-process marker, so the normal
request path still does no DB read. Cold end-user counters keep seeding from
the cached object rather than the row, so from_db is unchanged for them.
* fix(headroom): bound the /v1/compress and /v1/retrieve calls with a timeout
The headroom guardrail builds its client with get_async_httpx_client(GuardrailCallback)
and no params, and passes no timeout on either outbound call. That client's read, write
and pool legs are 600s (litellm.request_timeout when set explicitly, default 6000s), so
an unreachable or stalled compression service holds the caller's pre-call request open
for the whole window before unreachable_fallback ever runs. Because the client is shared
with every other no-params guardrail, each stalled call also pins a pooled connection for
the same window, so a saturated pool makes unrelated requests block on the pool leg.
Bound both calls at 60s by default, honoring litellm_params.timeout when set (the field
already exists and documents itself as the per-guardrail API timeout; headroom accepted
it and ignored it). The connect leg stays at the http_handler default, or the configured
budget when that is shorter, so a dead host still fails fast.
Live on a proxy against a stalled /v1/compress: 600.4s -> 60.2s before the 502, and 5.2s
with timeout: 5 configured.
* fix(headroom): reject non-finite timeouts and trim the timeout commentary
`timeout: .inf` on a Headroom guardrail reached httpx and the aiohttp transport
raised OverflowError, so every request came back as a raw 500 instead of going
through unreachable_fallback. Reject non-finite values the same way as
non-positive ones, and cut the comments and docstrings back to what the code
does not already say.
* fix(anthropic): never carry cache_control on translated thinking blocks
The /v1/messages adapter built every thinking and redacted_thinking block with
cache_control=content.get("cache_control", {}), so a block the client never
marked still came out carrying an empty cache_control. anthropic_messages_pt
replays thinking blocks verbatim and first, so that value landed at content[0]
of the outbound assistant message and Anthropic rejected the request with
messages.N.content.0.thinking.cache_control: Extra inputs are not permitted.
Anthropic's schema has no cache_control on either block type, so there is
nothing to gate or translate here, only to stop copying. Every sibling block
type already routes through _add_cache_control_if_applicable; these two were
the only ones setting the key unconditionally.
This is reachable from any caller that round-trips Anthropic messages through
the OpenAI shape, which is why shadow eval saw it on a majority of sampled
Claude Code turns while the same traffic served natively was fine.
* test(anthropic): assert the outbound wire body for redacted thinking blocks
* feat(cli): sync OpenCode models from /v1/models in lite opencode
lite opencode now fetches the proxy's /v1/models with the resolved key and
hands OpenCode an OPENCODE_CONFIG_CONTENT declaring a litellm provider
(@ai-sdk/openai-compatible, proxy /v1 base URL, {env:OPENAI_API_KEY}) with one
model entry per listed chat model, so the model picker mirrors the proxy
without a hand-maintained opencode.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): sync OpenCode models only after the key check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
