A refused or failed /v1/realtime session never ran the success cost callback
or a failure hook, so its pre-call budget reservation stayed open and kept the
key/team/user spend counters pinned above real spend, 429ing later requests on
the same key until the counter's TTL expired. The endpoint now reconciles the
reservation in a finally, reusing a shared release_or_invalidate_budget_reservation
helper that mirrors the success/failure paths (release to zero, else invalidate
the reserved counters and finalize).
The relayed upstream close message and reason also go through the proxy's
client-facing redaction, so a credential, internal hostname, private IP, or
server path echoed by the upstream never reaches the client verbatim.
* fix(batches): register ownership for every batch create path
Since the team isolation change, the managed files hook decided whether a
response came from a create by looking for the managed input file id on it,
which only the unified input path sets. Batches created from a model-encoded
input file id, a model param, or a raw provider id with ?provider= never got
an ownership row, so they vanished from GET /v1/batches for the key that
created them.
The create endpoint now stamps a create marker on the response before the
hooks run, and the hook keys ownership registration and the batch-created
metric on that marker instead of on the input id format.
* test(batches): assert ownership registration through the managed files hook
The endpoint tests asserted the private create marker, which is wiring, not
behaviour. They now run the create through the real managed files hook and
assert the ownership row is written for the creating key on every create
path, with the unified path driven by a genuine encoded input file id
instead of patched decoders.
* fix(mcp): let config.yaml MCP servers pin server_id
A config-defined MCP server's id is a hash of server_name|url|transport|
auth_type|alias, recomputed on every config load, so editing any of those
fields mints a new id. Every key and team granted the old id via
object_permission.mcp_servers keeps pointing at an id that no longer exists,
and the server disappears from tools/list for them with nothing logged.
load_servers_from_config now uses an explicit server_id from the server's
config entry when present and falls back to the existing hash otherwise, so
grants survive url/name/alias edits. Rejected at config load: a blank or
non-string server_id, two entries claiming the same id, a pinned id already
held by a database-backed server, and a pinned id that is another entry's
server_name or alias (expand_permission_list matches ids before names, so
that one would capture the other server's grants). Because the database
registry loads after the config on startup, a database row that lands on a
pinned config id is reported as a warning from the database reload instead,
where it is decidable; the warning is latched on the shadowed set so the
config-reload timer does not reprint it every interval.
Deployments that do not set server_id keep the exact id they have today.
* fix(mcp): close two more pinned-id capture paths
A pinned server_id equal to an alias supplied through litellm_settings
mcp_aliases was accepted, because the collision index only held the entry's
own alias field. expand_permission_list matches ids before names, so grants
written for the aliased server resolved to the pinning one. mcp_aliases keys
whose target is a config server are now reserved the same way.
A pinned server_id equal to a database-backed server's name, server_name or
alias had the same effect against the database side, and could not be
rejected at config load because the database registry is not loaded yet. The
database reload now warns about it, latched like the existing shadow warning.
* fix(mcp): reserve only the aliases the loader actually assigns
Reserving every mcp_aliases key targeting a config server was too broad in
two ways: the mapping is ignored when the entry sets its own alias, and only
the first mapping for a server is ever applied. Both cases made a pinned
server_id that could never have collided abort proxy startup. Reserve only
the name load_servers_from_config will really assign.
The database capture warning also fired for a database server whose own id is
the config server_id. There the database row wins the id outright through
get_registry precedence, so the shadow warning above it is the accurate one
and the capture message contradicted it. Skip those rows.
Also mark the two litellm-internal patches in the reload test helper, which
the test-quality gate counts; the database reload has no other seam.
* fix(mcp): match the loader's alias check exactly, is None not falsiness
load_servers_from_config consults mcp_aliases only when the entry has no
alias key at all, so an entry setting alias: "" gets no mapped alias. The
collision index used falsiness and reserved the mapped name anyway, which
failed startup on a pinned server_id that could never have collided with it.
* fix(mcp): skip one identifier, not the whole database row
A database row can shadow one config server_id by id and capture another by
name at the same time. Skipping the entire row when its id shadowed a config
entry dropped the second warning, leaving the operator with half a diagnosis.
Skip only the identifier equal to the row's own id.
* fix(mcp): reject conflicting self-pinned server ids
* fix(mcp): validate config server names before building the identifier index
The collision check reads every entry's body up front, so a malformed
entry under an invalid name surfaced as an AttributeError instead of the
name validation error the loader gave before this change.
The handshake error path already runs client-facing error strings through
_redact_string; the relay's _close_client did not, so a secret echoed in an
upstream close reason could reach the client verbatim. Mirror the handshake
path and scrub the close message and reason before relaying them.
Health checks probed every OCR deployment with a PDF, which Cohere Parse
rejects, so /health, background health checks, and the UI Test Connection
button marked Cohere Parse deployments unhealthy. BaseOCRConfig gains a
get_health_check_document hook (PDF by default) that CohereParseConfig
overrides with a 1x1 PNG data URI. cohere also gains ocr in the provider
endpoint matrix
* fix(responses): decode JSON-string tool schemas before sending to the provider
A caller that hands a tool schema over already JSON-encoded reached the
Responses API with a string `parameters`, and the provider rejected the
request with a 400 naming the routed model instead of the offending tool.
Decode it at the one place every Responses request converges, and refuse
anything that is neither an object nor a string encoding one.
Collapses the duplicated input/tool sanitization block shared by the
request and compact-request builders into a single owner, so the decode
cannot be wired into one path and not the other.
* test(responses): pin null tool schemas as accepted, and type the parametrized cases
The Responses API serves `parameters: null` and an omitted schema alike, so
neither may raise. Pin both against a future tightening, annotate the
parametrized inputs, and trim the docstrings back to what the code does not
already say.
The refusal predicate also required the session log to be empty, but that
log is not limited to upstream frames. With gemini_live_defer_setup the
handler stores a synthetic session.created before the relay starts, and
the transcription usage flush appends a usage event before the check
runs, so an upstream policy close with no received frames was still
logged as a $0 success. Key the check off the received-frames flag only
The relay's failure dispatch runs the async handler and then the legacy sync
failure_handler for the proxy's callable callbacks. The realtime logging object
carried no async marker, so failure_handler treated the session as a sync SDK
call and fired every CustomLogger's sync failure hook on top of the async one:
Langfuse recorded two ERROR observations per refused session, and OpenTelemetry,
MLflow, Braintrust, Literal AI, DeepEval and New Relic implement the same sync
hook. Plant the _arealtime marker in litellm_params the way aanthropic_messages
and agenerate_content already do, so both dispatchers classify the session async.
A shadow eval job could only be scoped by identity, so "this user's traffic on model X
across every key they own" was not expressible and a models field on the start body was
silently dropped. The job now carries a models list that every target is narrowed to,
matched on the requested model group with model_group_alias resolved on both sides. An
unresolvable name is a 400 at start. Empty means every model, which is what every existing
row reads as. The dashboard start form gains an "Only on models" picker and the job
headline shows the scope.
* feat(otel): stamp litellm.request.route on the LLM call span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): drop redundant comment on REQUEST_ROUTE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): Final-annotate route test locals, drop field comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): read litellm.request.route off the server span
The LLM call span took the auth-normalized literal path from logging
metadata, which disagrees with the SERVER span wherever FastAPI matched a
template: on /engines/{model:path}/chat/completions the LLM span spelled the
model name while http.route carried the template, so the two spans grouped
into different buckets and the PR's premise did not hold.
Read the value off the span that already holds it. The request's root SERVER
span is anchored per request for parenting, and its attributes stay readable
after it ends, so request_root_http_route() answers from the async close
callback with the same http.route the SERVER span exports: the route template
on a normal route, the literal path where the passthrough hook rewrote it, and
the mount point on an MCP call. Nothing has to re-derive any of that, so the
two spans cannot drift apart.
The route the proxy recorded at auth stays as the backstop for a deployment
whose FastAPI instrumentation never mounted, where there is no server span to
disagree with. Off the proxy the attribute is omitted rather than empty.
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng He <yucheng@berri.ai>
Suppression state moves out of request metadata into a request-scoped ContextVar.
refresh_proxy_server_request_body_snapshot copies metadata into
proxy_server_request.body, which deployments persist to spend logs, so the marker
naming each suppressed guardrail was readable by the caller whose request produced
it. Recovering it was enough to replay {token}:{name} for any CustomGuardrail and
switch off a PII or content-filter guardrail, since the check never verified the
named guardrail was a compression one. Nothing is read from metadata now, so there
is no marker to forge and the per-process token is no longer needed.
Routing-side compression reads the live messages instead of a pre-guardrail copy.
arm_pre_call runs before the pre-call hook, so its snapshot held the prompt as it
was before any masking guardrail rewrote it, and messages_for_routing handed that
to a compression guardrail which POSTs it to an external service. Masked content
left the proxy anyway. The cost is one combination: when the model hop compressed
and the hops differ, routing now classifies on the compressed text, since no
uncompressed copy survives that a masking guardrail has already seen.
policy_for_model no longer falls back to a marker scoped to tags the request does
not carry, which applied an 'eu' policy to a 'us' request on config order alone.
Each fix carries a regression test; all three fail when the fix is reverted.
client_ack_messages classified a websockets ConnectionClosed raised by
the client socket as the backend closing, so bidirectional_forward kept
waiting on the upstream instead of ending the session. Starlette clients
raise WebSocketDisconnect, but the realtime test client in
tests/llm_translation/realtime raises websockets.exceptions.ConnectionClosed,
which hung test_openai_realtime_simple.py until the run was killed.
Only the receive_text call now maps every exception to
CLIENT_DISCONNECTED; the loop body keeps ConnectionClosed as
BACKEND_CLOSED, since the backend socket is the only websockets socket
touched there.
The detail endpoint now returns untracked_usage_units_by_team and
untracked_usage_units_by_key next to the cost breakdowns, and the By team and
By key tables show them in an Unpriced Units column, so a row that pairs its
total units with a partial cost says how many units that cost leaves out.
The overview comparator no longer treats a missing cost as zero: guardrails
with no known cost sort last in both directions instead of mixing in with
genuinely free ones.
Refs LIT-5652
The overview table gains Usage Units and Cost columns plus a Guardrail Cost
card, and the detail page gains a Usage & Cost section that breaks units and
cost down by counter, team and key. Units the cost map could not price are
called out next to the cost they are left out of.
Both pages now read /guardrails/usage/* through $api.useQuery so the rows are
typed from schema.d.ts; the hand-written PerformanceRow and the untyped fetch
helpers are gone. fetchClient resolves fetch per request so integration tests
that stub the global see typed-client calls too.
Refs LIT-5652
When the upstream closes while the proxy is forwarding a client message,
the client loop ends before the backend relay sees the close, and the
relay skipped closing the client because it read the client loop's exit
as the client hanging up. The client loop now reports why it stopped, so
a close observed on the backend send still reaches the client with the
error event and the upstream close code
When the provider closes the realtime websocket (for example Vertex Live
refusing the session with 1008 "Publisher model ... was not found"), the
proxy swallowed the close and kept waiting on the client, so the client
sat on an open socket with nothing coming back and the session was logged
as a $0 success
The backend relay now returns the upstream close, and bidirectional_forward
sends the client an OpenAI-style error event naming the upstream code and
reason, then closes the client socket with the same code (or 1011 when the
upstream code is one a server may not send). A session the upstream refused
before sending any frame is logged through the failure handlers instead of
as a success
