* fix(responses): decode JSON-string tool schemas before sending to the provider
A caller that hands a tool schema over already JSON-encoded reached the
Responses API with a string `parameters`, and the provider rejected the
request with a 400 naming the routed model instead of the offending tool.
Decode it at the one place every Responses request converges, and refuse
anything that is neither an object nor a string encoding one.
Collapses the duplicated input/tool sanitization block shared by the
request and compact-request builders into a single owner, so the decode
cannot be wired into one path and not the other.
* test(responses): pin null tool schemas as accepted, and type the parametrized cases
The Responses API serves `parameters: null` and an omitted schema alike, so
neither may raise. Pin both against a future tightening, annotate the
parametrized inputs, and trim the docstrings back to what the code does not
already say.
Every /azure_ai/<router model>/<native path> relay failed with HTTP 500 because
azure_ai had no passthrough config. The new AzureAIPassthroughConfig strips the
router-model prefix from the relayed path, forwards to the deployment's api_base
with its own credential (api-key on Foundry and Azure OpenAI hosts, Bearer
elsewhere, Entra as the fallback), and delegates chat/completions cost logging
to the Azure passthrough config.
The router's provider inference now receives the deployment's api_base so an
OpenAI-family model on a Foundry resource stays azure_ai instead of flipping to
azure through the AZURE_AI_API_BASE env var.
The refusal predicate also required the session log to be empty, but that
log is not limited to upstream frames. With gemini_live_defer_setup the
handler stores a synthetic session.created before the relay starts, and
the transcription usage flush appends a usage event before the check
runs, so an upstream policy close with no received frames was still
logged as a $0 success. Key the check off the received-frames flag only
The relay's failure dispatch runs the async handler and then the legacy sync
failure_handler for the proxy's callable callbacks. The realtime logging object
carried no async marker, so failure_handler treated the session as a sync SDK
call and fired every CustomLogger's sync failure hook on top of the async one:
Langfuse recorded two ERROR observations per refused session, and OpenTelemetry,
MLflow, Braintrust, Literal AI, DeepEval and New Relic implement the same sync
hook. Plant the _arealtime marker in litellm_params the way aanthropic_messages
and agenerate_content already do, so both dispatchers classify the session async.
The SSRF check in async_safe_get resolved DNS on the event loop and a blocked
address was retried three times; validate_url now runs in a thread and an
SSRFError fails the fetch on the first attempt in both fetchers. The shared
HTTP handler signed the request and ran pre_call logging on the loop after an
async transform; both now run in a thread. Vertex AI Gemini still fetched
http:// images and https images without an inferrable mime type with the sync
converter inside its async body builder; the walker takes a should_inline
predicate and Vertex AI inlines exactly those URLs, leaving https images with a
known mime type and Files API refs to Google. When one download fails the
other in-flight downloads for that request are now cancelled instead of
finishing in the background
A shadow eval job could only be scoped by identity, so "this user's traffic on model X
across every key they own" was not expressible and a models field on the start body was
silently dropped. The job now carries a models list that every target is narrowed to,
matched on the requested model group with model_group_alias resolved on both sides. An
unresolvable name is a 400 at start. Empty means every model, which is what every existing
row reads as. The dashboard start form gains an "Only on models" picker and the job
headline shows the scope.
* feat(otel): stamp litellm.request.route on the LLM call span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): drop redundant comment on REQUEST_ROUTE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): Final-annotate route test locals, drop field comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): read litellm.request.route off the server span
The LLM call span took the auth-normalized literal path from logging
metadata, which disagrees with the SERVER span wherever FastAPI matched a
template: on /engines/{model:path}/chat/completions the LLM span spelled the
model name while http.route carried the template, so the two spans grouped
into different buckets and the PR's premise did not hold.
Read the value off the span that already holds it. The request's root SERVER
span is anchored per request for parenting, and its attributes stay readable
after it ends, so request_root_http_route() answers from the async close
callback with the same http.route the SERVER span exports: the route template
on a normal route, the literal path where the passthrough hook rewrote it, and
the mount point on an MCP call. Nothing has to re-derive any of that, so the
two spans cannot drift apart.
The route the proxy recorded at auth stays as the backstop for a deployment
whose FastAPI instrumentation never mounted, where there is no server span to
disagree with. Off the proxy the attribute is omitted rather than empty.
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng He <yucheng@berri.ai>
Batch creation snapshotted the team's organization with a direct
litellm_teamtable query on every create. Go through get_team_object
instead, which serves the team auth already cached and only falls back
to the database when the team was never cached.
Suppression state moves out of request metadata into a request-scoped ContextVar.
refresh_proxy_server_request_body_snapshot copies metadata into
proxy_server_request.body, which deployments persist to spend logs, so the marker
naming each suppressed guardrail was readable by the caller whose request produced
it. Recovering it was enough to replay {token}:{name} for any CustomGuardrail and
switch off a PII or content-filter guardrail, since the check never verified the
named guardrail was a compression one. Nothing is read from metadata now, so there
is no marker to forge and the per-process token is no longer needed.
Routing-side compression reads the live messages instead of a pre-guardrail copy.
arm_pre_call runs before the pre-call hook, so its snapshot held the prompt as it
was before any masking guardrail rewrote it, and messages_for_routing handed that
to a compression guardrail which POSTs it to an external service. Masked content
left the proxy anyway. The cost is one combination: when the model hop compressed
and the hops differ, routing now classifies on the compressed text, since no
uncompressed copy survives that a masking guardrail has already seen.
policy_for_model no longer falls back to a marker scoped to tags the request does
not carry, which applied an 'eu' policy to a 'us' request on config order alone.
Each fix carries a regression test; all three fail when the fix is reverted.
