Under otel_v2, a client that propagates W3C trace context in params._meta
(SEP-414) pulled the tools/call span out of the gateway's trace:
resolve_mcp_span_context parented the MCP span to the client's remote
context and demoted the gateway's own transport span to a span link. The
gateway's tracing backend only ever receives the gateway's half of such a
trace, so the span was unreachable from the trace view and the POST
transaction showed a dangling link.
Invert the anchoring: the MCP tool-call and tools/list spans now always
nest under the transport span of the request carrying the message, and the
client's propagated context is recorded as the span link instead, so the
correlation survives while every trace stays renderable. With no transport
at all the span roots its own trace and still carries the link, keeping a
single shape for the event. Both returned contexts are built on an
explicitly empty base so ambient session state can never leak in, and the
span inherits the transport's sampling decision like every other
request-level span.
* fix(key_management): allow /key/update to keep or shrink MCP server grants the key already holds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(key_management): reuse key row's included object_permission instead of a second lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): skip guaranteed-miss team lookup for the litellm-dashboard sentinel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: assert builder result instead of swallowing exceptions; drop redundant comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): opt-in budget rollover carrying overage into the next window
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): zero under-cap rows before decrementing over-cap rows in cascade resets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: suppress misleading register_model unresolved-cost warnings for entries without custom pricing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: do not warn about zero cache costs for tiered pricing entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep upstream OAuth Authorization when jwt signer hook injects one on tools/call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): only treat server credential as occupying Authorization when it maps to that header
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(health): opt-in model-group allowlist for background health checks and health-check routing
* fix(health): merge shared health states per writer scope instead of replacing
* refactor(health): drop restating comment and parameterize test scope annotations
* chore: remove stray generated prisma migration file
* fix(health): merge health states against the Redis snapshot, not the pod-local copy
* fix(health): fall back to the pod-local snapshot when the Redis read returns nothing
Gemini Live sends no usageMetadata and no turnComplete for
gemini-3.5-transcribe-live sessions, so realtime spend logged as 0.0.
Attach estimated usage to the input_audio_transcription.completed event
using Google's published billing estimate (25 audio tokens/sec of input,
175 text tokens/min of output) derived from the streamed pcm16 audio
duration, gated to audio_transcription-mode models so conversational
Live models keep billing through usageMetadata. Also capture that usage
in the provider_config backend path so realtime cost calculation sees it.
* fix(ui_sso): resolve highest privilege Entra app role, not first in claim
A user assigned more than one Entra app role — commonly by belonging to
several assigned groups — arrives at the Microsoft SSO callback with every
role in the id_token `roles` claim. LiteLLM stores a single role per user,
and get_microsoft_callback_response collapsed the list by taking the first
value that resolved to a LitellmUserRoles and breaking.
Entra does not guarantee the ordering of the `roles` claim, so which role
won was effectively arbitrary: a user in one group mapped to internal_user
and another mapped to proxy_admin_viewer could be silently demoted to
internal_user, and proxy_admin could lose to either.
The generic/Okta path already resolves this correctly via
determine_role_from_groups, which walks a documented privilege hierarchy.
Hoist that hierarchy into LITELLM_USER_ROLE_HIERARCHY and reuse it, so
app-role logins and group-mapping logins agree.
Extract the selection into MicrosoftSSOHandler.get_user_role_from_app_roles
so it is directly testable — the existing tests re-implemented the loop
inline, which is why the ordering bug was not caught.
Behaviour is unchanged for single-role claims, unrecognised values, and
empty claims. Roles the hierarchy does not rank (org_admin, team, customer)
are resolved deterministically rather than by claim order.
* refactor(ui_sso): trim role selection prose and use immutable annotations
Addresses review feedback on the app role selection helper.
Drop the explanatory comments and the Args/Returns docstring boilerplate that
restated the control flow, keeping only the part a reader cannot infer from the
code: that Entra does not guarantee claim ordering, and how unranked roles
resolve.
Type the parameter as Sequence[str] rather than list[str] and build the resolved
set as a frozenset, so the helper stops adding an LIT001 mutable-collection
annotation. Make LITELLM_USER_ROLE_HIERARCHY a tuple for the same reason.
No behaviour change: the ordering regression tests still fail against the
previous first-match-wins logic and pass here.
Adds a Gemini audio transcription config that maps /v1/audio/transcriptions
onto the Interactions API (speaker attribution and word timestamps land on
the OpenAI verbose_json shape), registers both models with published pricing,
routes text-only Live sessions to TEXT responseModalities so
gemini-3.5-transcribe-live sessions survive, and makes the token-priced
transcription cost path provider-aware instead of hardcoding OpenAI.
* feat(newrelic): per-team cost and usage metrics via team callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(newrelic): retry transient 429/408 metric posts instead of dropping
* fix(newrelic): drop only records queued when the drain began, not mid-drain arrivals
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(complexity_router): heuristic-first classifier chaining
Adds classifier_type 'heuristic_first', which scores locally on every request and
only calls the LLM classifier for traffic the scorer could not place at or below
heuristic_first_max_tier. A request short-circuits when the scorer landed at or
below the threshold and produced at least one signal; everything else escalates.
The signal requirement is load-bearing. A prompt where no dimension fires scores
exactly 0.0, which is under simple_medium, so the score-to-tier mapping calls it
SIMPLE by default rather than by evidence, and that is about half of general
traffic. Gating on the tier alone would route it to the cheapest model without
ever consulting the classifier.
Introduces uses_llm_classifier as the single owner of 'does this router call the
classifier model', replacing the classifier_type == 'llm' comparisons in the
config validator, the prompt prebuild, the health dependency graph, the
routing-test authorizer, and six dashboard sites.
* fix(complexity_router): reuse the heuristic verdict on classifier failure, load the threshold on edit
Three review findings, one push.
The heuristic-first fallback re-scored the prompt after a classifier failure,
which the README already documented as a reuse. The outcome computed before
escalation is now handed to the failure path, so the scorer runs once per request.
The edit modal never hydrated heuristic_first_max_tier, while save rebuilds every
managed key from form state, so opening a heuristic-first router and saving it
dropped a field the proxy requires. The dropdown's display fallback hid it. Both
are fixed, and the hydration is extracted into a pure function so a test can pin
the invariant: every managed key present in a stored config survives an untouched
open-and-save. That test also covers every field added later.
Classifier radio labels lost their em dashes, per the repo writing convention.
* fix(logging): stop billing and logging response reads as LLM calls
Retrieving, deleting or cancelling a stored response, and vector store management calls, run through the same logging lifecycle as inference. A retrieved response replays the usage of the call that created it, so every read priced it again and wrote a second spend log row for the same tokens. Non-inference calls now cost 0, report no usage, log no placeholder chat message, and get a litellm.responses_management operation name instead of reading as chat.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep billing background response jobs after the poll
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): use an empty list for read-call messages
A tuple matches no branch in the loggers that walk this value, so lunary's
parse_messages falls through to clean_message and raises AttributeError on the
success hook. An empty list reads as no messages everywhere: it satisfies the
isinstance(list) checks in newrelic, mlflow and datadog, iterates zero times in
traceloop and helicone, and is what StandardLoggingPayload.messages is typed to
hold. None would be type-legal too but is not iterable, so it trades one crash
for another in mlflow and traceloop.
* fix(otel): stop the legacy emitter reporting replayed tokens on response reads
The zeroing so far lands in the standard logging payload, which the legacy
OpenTelemetry emitter does not read for usage: it takes prompt, completion and
total tokens straight off the response object, so a retrieval span still carried
the token counts of the call that produced the response, and the token usage
histogram still recorded them. That emitter is the default, so the spend row said
zero while the trace said otherwise. The background cost poller keeps its counts,
the same exemption the pricing path already makes.
* fix(logging): keep billing a background response when its retrieval is read
A response created with background=true comes back queued and carries no usage, so
its create bills nothing. The retrieval that first sees the finished job is the only
place that job's tokens are ever visible, and pricing every read at zero therefore
loses the spend outright rather than deduplicating it. On a proxy without the
enterprise cost poller a background job ended up costing $0 end to end.
is_unbilled_non_inference_call now takes the response it is deciding about and treats
a background response the same way it already treats the poller's own read, which is
the same exemption seen from the other side. The legacy OpenTelemetry emitter's time
per output token metric picks up the read gate it was missing, so it stops dividing a
read's latency by the replayed completion token count.
* test(proxy): pass the read response to the non-inference predicate
The poller test called is_unbilled_non_inference_call with the pre-background signature, so it broke when the predicate gained the response it classifies. It now hands the predicate a foreground read, and asserts that the same read is free without the origin stamp, so the stamp is what the test proves.
* fix(otel): stop the v2 metrics recorder reporting replayed tokens on response reads
The v2 span builder sources usage from the standard logging payload, so the
earlier fix already zeroes it there. The metrics recorder reads response_obj
directly, so a responses-management read still recorded the original
generation's tokens into gen_ai.client.token.usage and divided generation time
by them for gen_ai.server.time_per_output_token.
The read still records operation and response duration, under the
litellm.responses_management operation, so it stays observable.
* fix(proxy): keep the response-cost headers on calls priced at zero
Pricing responses reads and vector-store management routes at zero dropped the whole
x-litellm-response-cost family off those replies. The header build reads a falsy zero as
a cost this response never recorded and filters it out, and a call that returns before
pricing stores no cost breakdown for the component headers to read, so a client parsing
the cost off a read got a KeyError where it had previously been handed a number.
Those calls now advertise the family at zero. Retrieving a background response, and the
cost poller's read of one, still report their real cost.
The params-taking form of the predicate moves from opentelemetry into
internal_call_metadata so the proxy header build and the OTEL recorders share one copy.
* fix(proxy): report a zero cost split only under a zero cost total
The component headers were filled from call-type membership alone, while the
total they sit beside keeps its real value when the read priced normally, so a
breakdown that had not landed by the time headers were built could advertise a
real total next to an all-zero split. The split is now reported as zero only
when the total agrees with it, and is otherwise left absent.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
For non-Anthropic models served over /v1/messages, the outer wrapper recomputes
cost over the adapter-translated Anthropic response dict. That dict dropped every
web search usage signal, so the recompute overwrote the correct cost breakdown
with a token-only one: x-litellm-response-cost-tool-usage read 0.0 and
x-litellm-response-cost-original excluded the search cost, while the total kept it.
The adapter now maps web search request counts (from Usage.server_tool_use or
Gemini's prompt_tokens_details) into usage.server_tool_use.web_search_requests,
matching the Anthropic API shape, and the Gemini web search cost calculator falls
back to server_tool_use when prompt_tokens_details carries no count. The shared
get_web_search_requests helper is now public since five modules consume it.
Resolves LIT-6288
Reverts #37725. The field existed so SDK callers that cannot read
`x-litellm-model-id` could tell which tier an auto-router picked, and the
framework that motivated it was LangChain. `@langchain/openai` builds
`additional_kwargs` and `response_metadata` from fixed key allowlists and drops
unknown fields at both the chunk top level and inside `delta`, so no
proxy-side placement of a namespaced key can reach a LangChain caller.
The complexity router's existing `return_raw_model_name` already covers that
case: it puts the resolved model in the standard `model` field, which
LangChain does propagate (`model_name` is on its metadata allowlist), and the
proxy honors it on both the streaming and non-streaming paths.
Keeps the unrelated cleanup from #37725 that dropped the redundant
function-local `ProxyBaseLLMRequestProcessing` import shadowing the
module-level one in `async_data_generator`.
`TestModelGroupAliasReachesPreRoutingStrategies` asserted on the marker as a
proof of strategy dispatch; the surviving `response.model == "gemini-flash"`
assertion already proves it.
