Three findings, one habit: capturing a value that an owner recomputes on the replay
path anyway, and thereby changing what it means.
Caller tags are reverted. add_key_team_project_metadata already merges the key's and
team's tags onto every replay through _merge_tags, so those were never missing;
capture ran at pre-routing, after that merge, so what it stored was the merged
result rather than the caller's own tags. Restoring it made proxy-derived tags look
client-supplied, which _reject_clientside_metadata_tags_check refuses outright when
the operator forbids client tags, so tagged sessions would have stopped warming
while real traffic on the same key kept working. The test now asserts what the owner
puts on a replay instead of what warming remembered.
Tenant rate limits are the mirror case, where an owner exists and warming was not
reaching it. The v3 limiter builds its descriptors from limit fields on the
principal, so a missing field is a ceiling that does not apply rather than one that
refuses. Most need nothing: LiteLLM_VerificationTokenView joins the team and
organization onto the key row, so team, team-member and organization limits arrive
with get_key_object and already bound. User and end-user limits are added during
auth from objects the auth path loads, and warming loads the same objects for the
same gates, so it now applies them there, calling
_apply_budget_limits_to_end_user_params rather than restating its mapping.
Two findings with one shape: the ceilings warming routes through read their inputs
off the request body, so anything the original request carried that a replay does
not is not refused by those ceilings, it is invisible to them.
Caller tags were dropped. Tag budgets, tag budget reservation and the limiter's tag
descriptors all resolve tags through one owner, get_tags_from_request_body, reading
them off the body. Warming already enters through all three entry points, so the
fix is not a tag check of its own; it is capturing the caller's tags and presenting
them again, which is what those owners were always going to read. The warming
marker stays on spend_logs_metadata rather than joining them, because tags feed
deployment selection and operators can forbid requests from carrying them.
The session id is caller-controlled and was embedded verbatim in the record key,
the index member, the touched key and every warmth key, none of which
max_payload_bytes bounds, so a caller could hold far more Redis memory than the
payload bound implies. It is now hashed inside record_key, which is the one place
capture and the warm-aware pick both derive keys from, so they cannot disagree; the
record body keeps the real value because a replay carries it for deployment
affinity to pin on. A session id past a sane ceiling is not captured at all rather
than truncated, since a truncated one would pin against traffic that does not exist.
A record binds a payload to a tenant, and a replay spends that tenant's budget,
consumes its rate limits, reaches the models it grants and fans out to its logging
callbacks. So the only safe question at replay time is whether the tenancy stamped
at capture still holds, and until now only one way of failing that question was
handled.
Keyless callers were skipped because their tenancy cannot be re-read. A virtual key
whose tenancy CHANGED was not: the row was read, the principal took the new tenant
wholesale, and the old payload was replayed under it. A key moved between teams
therefore attributed prompts captured under the previous team to the new one and
fanned them out to the new team's logging callbacks, where its members could read
them. Neither available answer is right on its own, which is the tell that the
record itself is no longer valid: keeping the captured tenant spends a budget for a
key that has left it, and adopting the current tenant discloses the old tenant's
prompts to the new one.
_unverifiable_keyless_tenancy is generalized rather than joined by a second
predicate, since both cases are the same question with different answers. It now
returns why the tenancy failed, or None when it is confirmed to still hold, and the
tick skips anything that is not a confirmed match. The session re-captures under
whatever is true on its next real turn, so the cost of being wrong is one skipped
warm.
A keyless caller (JWT, and anything else the proxy authenticated without a
virtual key) leaves no key row to re-read, so the tenancy the proxy stamped at
capture is the only copy of it and no tick can tell whether the caller still
belongs to that team, organization, project or user. Replaying on it spends
that tenant's budget, consumes its rate limits and reaches the models it grants
for as long as the record lives, with nothing able to notice the association
was withdrawn
A virtual key has the fresh row to answer with, which is why a loaded row wins
wholesale there. This shape has nothing to answer with at all, and authority
that cannot be resolved at use time is not authority warming may spend under,
so the session is skipped instead. A caller with no recorded tenancy is
untouched: there is nothing to go stale and its replay stays unattributed
_replay_principal still restores the recorded fields for that shape rather than
dropping them, so if the skip ever misses a record the replay is
over-constrained by a stale tenant instead of escaping every tenant control on
an empty principal
Anything stored under a partition is also attributed to that caller, so the
partition has to cover every dimension the attribution does. It did not. The
attribution carried seven identity fields while the partition returned the first
of four, and the gap between those two lists was a cross-tenant collision channel:
principals differing only in an omitted dimension shared one record, and whoever
wrote last owned the payload and the attribution while state accumulated under it
stayed.
End users are the reachable case, since they share a virtual key by design and a
key hash cannot tell them apart. One end user could seed a session's touched-model
set, the next end user's turn would overwrite the payload and attribution while
inheriting those models, and the replays then sent that caller's prompt to a model
they had never used and billed it to their end-user budget. Projects were the same
bug, unreported.
The partition is now composed from every dimension rather than the strongest one,
and both it and the attribution read the same list, taken from the type that
declares those fields rather than spelled out in either place, so they cannot
drift apart again. That also removes the need for a rule about state surviving an
attribution change: a different attribution is a different partition, so there is
no shared state to clear.
Values are hashed rather than concatenated. End-user and project identifiers are
customer-chosen strings that would otherwise sit in Redis key names and in every
log line quoting one, and hashing identifiers before they become cache keys is
what DeploymentAffinityCheck already does.
Operational note: this changes partition strings, so existing cache-warming records
and complexity-router session-affinity pins are not read after deploy. Both are
caches with TTLs; pins re-derive on the next turn and records re-capture.
The replay principal merged the freshly loaded key row with the tenancy captured on
the record, taking the record's value whenever the row's was falsy. That cannot
distinguish "this field was never captured" from "this association was deliberately
removed", and those need opposite outcomes.
A key an admin takes out of a team is still valid and still loads; the row correctly
reports no team. The merge restored the captured team instead, so every replay for
the rest of that session ran as if the key were still a member: reaching models the
team grants, spending the team's budget, and consuming the team's rate limits on
behalf of a key no longer in it. The same held for organization, project and user.
Key deletion and revocation were never affected, since those are dropped a step
earlier when the row fails to load at all.
The request path has no equivalent seam to get wrong: it re-derives everything from
the token on each request and a None simply skips that gate. Warming is the only
place holding a snapshot, so it now follows the same rule. A loaded row is used
verbatim, Nones included, and the captured tenancy is restored only when there is no
row to read, which is the keyless case where the record is the sole source.
Fetching the touched models separately added a sequential Redis round trip per live
session on every tick, doubling them against the max_sessions cap for data that
lives in the same slot. One script now returns both, and get_record is expressed in
terms of it so the warm-aware pick keeps its single call too.
The eligibility universe also counts a record's served_model as touched, which it is
by definition. That matters on upgrade: records captured before the touched set
existed would otherwise warm nothing until their next turn rewrote them.
Deletes the package __init__ re-exports. Nothing imported those names from the
package; every consumer imports the defining module directly, so the block was a
second place to list every public name and nothing else.
Warming resolved its target set from configuration, so every active session was
replayed against a representative of every tier on every interval whether or not it
had ever been routed there. Most sessions never leave their starting tier, so that
spent N replays per interval to keep caches warm that nobody would read, and for a
pooled tier it warmed the wrong member entirely.
The set is now per session and comes from what the session actually did. Capture
records each served model into a per-session Redis set inside the same atomic script
that writes the record, sharing the session's hash tag, so the touched set cannot
disagree with the record it belongs to and expires with it. The refresher replays
exactly that set.
This is the intended shape of the feature: keep a session's own caches alive so
returning to a tier it has already used is a read, rather than pre-warming tiers on
speculation. The first switch to a new tier is a normal cache write, and every visit
after it is warm. warm_models changes meaning accordingly, from a pre-warm list to
an allowlist that narrows what a session may be warmed on, and resolve_warm_models
now returns every model across the tier pools since it bounds eligibility rather
than naming the targets.
Tests that expected a replay on a tier the seeded session had never visited now
declare a session that has been to both, which is the case the feature serves.
A tier may be a pool the router picks from at random, but the warm set resolved to
one representative per tier, so for a pooled tier the member holding the session's
cache was usually not warmed at all. That inverted the feature: the session's own
cache expired at the provider TTL and its next turn on the same tier paid a full
cache write, while warming spent on a pool member the session had never touched and
might never be routed to.
The warm set is now per session, leading with the record's served_model and then the
tier representatives, so a session's own cache is always refreshed. Eligibility is
still resolved once per tick over the union, so no extra model-list lookups happen
per session.
Single-model tiers, which every existing test and the live proof used, are
unaffected: the served model is the tier representative there, which is why the gap
did not surface earlier.
The standalone predicate test asserted four return values of
user_is_scim_deactivated and never exercised warming, which is coverage of the
helper rather than of the feature; the helper is already pinned by the auth and MCP
paths that own its behavior. The ceiling matrix is the right home, since it is the
test that answers whether a refusal actually stops a replay, so SCIM deactivation
joins the blocked key, expired key, blocked team, denied model and budget arms and
asserts the session is not warmed. Removing the gate fails that arm.
Every session in a tick is started at once, so anything a session materializes
before acquiring its replay slot scales with max_sessions rather than with the
concurrency setting. The payload was inflated above the semaphore and held for the
whole replay, and capture admits payloads up to eight times the compressed cap, so
one tick could hold a decompressed payload per active session instead of per replay
in flight.
Decompression moves inside the slot, which is now held across the session's whole
due set. The replay ceiling is unchanged, since a session's models are replayed in
sequence inside its slot, and the CPU burst of inflating is now serialized to the
same bound rather than running for every session at once.
The bound had no test at all; the rig tracked peak in-flight replays and nothing
asserted on it. One test now pins both halves through a real tick: replays in
flight and payloads inflated must both respect max_concurrent_replays. It fails on
the previous ordering with the payload count at the session count, not the bound.
should_redact_message_logging takes a model_call_details dict whose required shape
is implicit: the header and global forms are read off litellm_params, but the
per-request form is read from standard_callback_dynamic_params at the top level.
Capture passed only litellm_params, so a caller setting turn_off_message_logging
in the request body had its opt-out silently dropped and its prompts retained in
Redis anyway; the two forms that happened to be tested, headers and the global
setting, both worked, which is why the gap survived. The gate's own docstring
claimed the per-request form was honored.
The value is owned by the logging object, which initializes it from the request in
its constructor, so it is read off the object rather than re-derived, and the key
is spelled exactly as the request path spells it when it builds the same dict for
the same predicate. Where no logging object rides the request (SDK-direct use)
the behavior is unchanged, and those callers consent through cache_warming.enabled
itself.
The matrix test already named a per-request case but exercised the header leg, so
it now drives all three body spellings (root, metadata, litellm_metadata) through
a real Logging object built the way function_setup builds it, plus a
no-opt-out control proving the same path still captures. The three body cases fail
without this change and the control does not.
The SCIM-deactivation check was spelled out inline at five call sites (the standard auth
builder, both MCP admission arms, and both bridge-refresh revalidation paths). Each one
re-derived the same three conditions, and cache warming, which resolves a user through
get_user_object exactly like those five do, shipped without the check at all; a sixth
caller getting it wrong is what a copy-pasted predicate guarantees eventually.
The predicate now lives beside get_user_object in auth_checks.py, since the resolver is
what hands back a live-looking row for a deactivated user and every caller owes the check
afterwards. It returns a bool rather than raising, so each caller keeps its own reaction:
the standard builder raises, the MCP arms return 401, the bridge returns a status, and the
warming refresher raises a ProxyException before it can spend against a deactivated
owner's key. Only an explicit scim_active of False deactivates, so a missing user or
absent metadata still fails open exactly as before at every site.
caller_scope fell back to the literal "unscoped" whenever user_api_key_hash was
absent, which is the normal shape for JWT and other keyless proxy principals.
Distinct tenants that reused a session_id therefore collided on one Redis record,
so the last writer's payload and attribution won and later replays could spend
under the wrong team or user.
Scoping now derives from the strongest identity the request actually carries, in
one shared owner (core_helpers.get_caller_scope) that capture, the affinity-aware
pick and the session-affinity pin all call. This mirrors how the v3 rate limiter
builds an api_key descriptor only when api_key is present and separate descriptors
per user, team and organization (parallel_request_limiter_v3.py:1988), and how
DeploymentAffinityCheck declines to scope rather than sharing a bucket.
"unscoped" now means only what it says: no proxy identity at all, which is direct
SDK use where there is one tenant by construction.
Eligibility, every-member cacheability, deployment affinity and pricing are
properties of the target model group. When a replay failed and the key had not
explicitly disabled fallbacks, the router could send it to a fallback group
carrying none of that validation, spending the customer's money without warming
the tier the session will actually switch to.
Fallbacks are now disabled at the dispatch site rather than in the request body,
so no key-level control or pre-call mutation can re-enable them. router.py:6157
raises before fallbacks, context_window_fallbacks and content_policy_fallbacks
are consulted, so the single flag covers all three kinds; a failed replay simply
retries on the next tick, which was already the documented intent.
Provider prompt caches are per-model, so every mid-session tier switch the
complexity auto-router makes lands on a cold cache and pays the full cache write
again. Opt-in cache_warming captures each session's latest payload at the routing
decision and a leader-elected background refresher replays it with max_tokens=1
against every cacheable tier model just under the provider cache TTL, so the
switch is a pure cache read.
A replay is a request, so it is admitted through the request path's own entry
points rather than beside them. For each replay the refresher assembles a request
body, reserves budget through the same wrapper auth calls right after
common_checks, stamps identity with the proxy's own stamper, applies every
key-level, team-level and project-level control, applies the key and team scoped
dynamic logging settings, runs ProxyLogging.pre_call_hook, applies the
fully-blocked-model check, and hands the dict that hook returns to
Router.acompletion or Router.aanthropic_messages. post_call_failure_hook runs on
every rejection and every dispatch failure, so the parallel request slot, the
reserved TPM tokens and the budget reservation all come back. Warming therefore
inherits both halves of every contract it touches (the limiter's descriptors
across every scope with its own configured window, its RPM and max-parallel
check, its upfront reservation and the stash its success callback reconciles
from; the key, team, user, end-user, organization and tag budget counters; every
configured guardrail and pipeline, including the ones defined on the deployment)
instead of reimplementing them. That deletes nine functions and the admission
block they served.
Warming writes no spend logs of its own, so the replay rows are the only record
of warming cost that will exist. They carry the customer's own tags and
spend_logs_metadata with the litellm_cache_warming tag alongside rather than
instead, and they fan out to the key and team scoped loggers, so warming is
included in per-tag chargeback and filterable out of it. Two Request-free blocks
of add_litellm_data_to_request are extracted verbatim as
LiteLLMProxyRequestSetup.add_key_team_project_metadata and
apply_dynamic_logging_settings so both callers share them; the move is
statement-for-statement identical, with no behavior change on the request path.
Ordering there is load-bearing: add_key_level_controls resets data["cache"] and
refills it from key metadata, so it runs after the body is built and a key's own
cache controls override warming's response-cache bypass exactly as they override
a caller's.
Blocked and expired keys are still checked locally because common_checks
dereferences the FastAPI Request; extracting a Request-free core so its other
gates bind on a replay too is a follow-up. Sessions on a key that declares
max_iterations are skipped entirely, because that limiter counts every request on
a session_id and cannot be consulted without incrementing it.
Metadata precedence (litellm_metadata before metadata, stringified) had three
implementations; core_helpers.iter_request_metadata_dicts and
get_request_metadata_field are now the single owner and DeploymentAffinityCheck
deletes its four private copies to delegate to them.
Resolves LIT-4865
Two defects combined to make a Redis outage take the proxy down rather than
degrade it.
First, connection kwargs were dropped whenever Redis was configured by url.
_get_redis_url_kwargs built its allowlist from
inspect.getfullargspec(redis.Redis.from_url); from_url is declared
(cls, url, **kwargs), so the argspec carried no connection kwargs and the
function returned ['cls', 'url', 'url']. socket_timeout went with the rest,
and socket_connect_timeout falls back to it, so both ended up None and a
Redis host that drops packets rather than refusing them blocked callers
indefinitely. get_redis_connection_pool's url branch lost the same kwargs by
a different route, rebuilding its pool kwargs from scratch.
The allowlist now comes from the connection class redis-py actually forwards
those kwargs to, walking the MRO because redis-py splits them between
AbstractConnection and its subclasses. Deriving it from the client instead
would admit client-only settings such as single_connection_client and the
SSLConnection-only ssl_* family, which reach AbstractConnection and raise
TypeError on first connect.
Second, the circuit breaker could not trip even once calls failed fast.
_redis_circuit_breaker_guard inferred success from the method returning, but
async_get_cache, async_batch_get_cache, async_set_cache, async_set_cache_pipeline,
async_set_cache_sadd and async_get_ttl catch their own connection errors and
return a default so callers degrade. Each failed call therefore reset the
failure streak and the breaker never opened, so an unreachable Redis stayed in
the pool and every request kept paying a full socket timeout on it. Those
methods now mark the failure and the guard records success only when nothing
failed while the method ran. Lua script execution went through none of this,
which mattered most because the rate limiter issues all of its Redis traffic
that way, so the guard is now a small helper shared by both.
The per-call marker is a ContextVar rather than a counter on the breaker.
Breakers are shared by every concurrent caller, so a shared counter cannot
tell "my call failed" from "some other in-flight call failed", and a success
overlapping someone else's failure would be discarded until a Redis that was
still answering got evicted from the pool anyway.
Only connectivity failures feed the breaker. Command and data errors say
nothing about whether Redis is reachable, and counting them would let a caller
provoke evictions on demand (an INCR against a non-numeric value, say),
dropping rate limiting to per-process counters that spreading traffic across
replicas can outrun.
Guardrails could only see the MCP tool call request (pre_mcp_call /
during_mcp_call); the tool result went back to the client unscanned, so a tool
that returns sensitive data bypassed every configured guardrail.
Adds a `post_mcp_call` event hook that runs after the tool executes and routes
the result through the unified apply_guardrail seam, so a text guardrail (e.g.
presidio) can mask sensitive values in the tool output or reject the result
without any MCP-specific code of its own.
- MCPGuardrailTranslationHandler.process_output_response now extracts the tool
result's text content into GenericGuardrailAPIInputs["texts"], calls
apply_guardrail with input_type="response", and writes the returned text back
into the content list in place (the logging payload already references that
object, so a copy would leave the unmasked text in the spend log)
- ProxyLogging.post_mcp_call_hook dispatches guardrails that implement
apply_guardrail, gated on should_run_guardrail(post_mcp_call); guardrails
implementing async_post_mcp_tool_call_hook keep their existing dispatch and
are not run twice
- both MCP tool-call paths (mcp_server and the Responses API handler) now honor
the rewritten result, and the REST path no longer swallows a guardrail
rejection as a logging failure
- shared, duck-typed MCP content helpers live in mcp_server/utils.py next to
extract_mcp_tool_result_error_message
- documents that async_post_mcp_tool_call_hook's return value is discarded by
every call site, so that hook only takes effect by mutating in place
#35271 restored the hierarchy where a team-scoped key is governed by the
team and team-member budgets only; the owner's personal max_budget applies
to their personal keys. Three places in the e2e suite still encoded the
old direction and would fail against a proxy built from staging.
test_user_budget_enforced_across_all_their_keys asserted that the owner's
team-member key is refused once their personal budget is exhausted. It now
asserts only the personal keys are refused, and keeps the team key as the
control that must keep serving, which pins the restored direction instead
of leaving it unasserted. Renamed to match what it now covers.
test_team_member_key_user_budget_resets_after_window drove a team key to a
block off the owner's personal budget, so nothing can block it any more and
_drive_to_block could never succeed. Its premise is gone rather than moved,
so it is removed; the sibling personal-key test still covers
quota_management.budget.internal_user.resets_after_window.
The registry rationale for that row dropped its "and team-member keys"
clause for the same reason.
* fix(otel): label retrieval and agent metrics correctly and emit gen_ai.provider.name
The GenAI metric attribute builder mapped only chat, text completion, embedding,
responses and MCP tool calls to an operation name, so vector-store searches and
A2A agent sends fell through to the "chat" default. Their duration and cost then
landed in the same series a Grafana GenAI dashboard reads chat latency off, with
no way to tell them apart. Both now map to the operation names the convention
defines for them, retrieval and invoke_agent, and an unmapped call type says so
at debug instead of silently becoming chat.
The provider label used gen_ai.system, which the convention deprecated in favor
of gen_ai.provider.name; the dashboards built on that vocabulary find nothing
under the old key. Metrics now carry gen_ai.provider.name with the semconv
provider value (bedrock -> aws.bedrock) via the resolve_provider helper the span
path already uses, and keep dual-emitting gen_ai.system with its raw value so a
dashboard already querying it keeps matching. A request litellm cannot attribute
to a provider gets no provider label at all rather than a placeholder "Unknown"
that minted a permanent series nobody can act on.
Resolves LIT-4954
Resolves LIT-4959
* fix(otel): map the rest of the vector-store call types off the chat default
Mapping only the search left the store lifecycle (create, retrieve, list,
update, delete) and the file operations (create, list, retrieve, content,
update, delete) falling through to chat, so vector-store admin traffic kept
polluting the same series a dashboard reads chat latency off. A live run
confirmed it: all 20 metric datapoints from a create, retrieve, list, file-list
and delete came out labelled chat.
The convention names no operation for vector-store management, so these take
vendor values under the litellm. prefix, litellm.vector_store_management and
litellm.vector_store_file_management, one per REST resource. Its note on
gen_ai.operation.name directs instrumentation to use a system-specific name
when no predefined value applies, which is the same allowance resolve_provider
already relies on for unmapped providers. Excluding them from the GenAI metrics
altogether was the alternative; it deletes series an operator may be watching
today and is far harder to reverse than a rename, so it stays available as a
follow-up rather than being decided here. Mapping them onto the semconv memory
store family was rejected: litellm vector stores hold documents, not agent
memory records, and borrowing those names would put document admin calls into
whatever charts agent-memory operations, which is the bug this fixes.
/rag/query reaches the same recorder and is the same operation as a vector-store
search, so query and aquery map to retrieval too; leaving them would have left
the defect alive on a second retrieval surface. /rag/ingest is a write with no
semconv equivalent and no retrieval or agent confusion, so it is left for the
RAG owners to name.
Resolves LIT-4954
* fix(otel): give the streaming A2A path a call type so it labels as invoke_agent
The streaming logging object is built by hand and never runs through
update_environment_variables, the only place call_type reaches
model_call_details, so every streamed agent turn arrived at the recorder
with no call type and fell back to chat. Stamp it, and map the streaming
spelling alongside the non-streaming ones.
Reverts #32005. Team-scoped keys are governed by the team and team-member
budgets only; the key owner personal max_budget no longer applies to them,
restoring the hierarchy that existed before that PR.
The skip_user_budget_on_team_key opt-out existed solely to turn the new
behavior back off, so it is removed along with the behavior: the
ConfigGeneralSettings field, the /config/list allowed_args entry that
surfaced it as an Admin UI toggle, and the argument threaded through
reserve_budget_for_request and _get_budget_counters.
Regression tests cover both enforcement points in the restored direction:
test_common_checks_personal_user_budget_skipped_for_team_key for the
read-time check and test_should_not_reserve_user_budget_counter_for_team_key
for the optimistic reservation path.
* feat(otel): record the GenAI duration metric on failed requests
`_record_metrics` ran only from `async_log_success_event`, so
`gen_ai.client.operation.duration` counted only the requests that worked.
Latency read off it during an incident was the latency of the surviving
traffic, and with no error dimension anywhere there was no way to build a
failure-rate panel or a success/failure split per model.
A failed call now records the same duration histogram, tagged with the
semconv `error.type` (the mapped provider exception's class name, bounded by
construction; the message stays on the span). Success attributes are
untouched, so an existing query can still isolate the old series with
`error_type=""`. The other five instruments describe a completed generation
and are skipped rather than filled with a fabricated zero: litellm hands the
failure callback no `response_obj`, so there is no usage to split and no
completion-token count, and it zeroes `response_cost` on failure. A
proxy-gate rejection (auth / rate limit) records nothing, for the same
reason it gets no span; no upstream call happened.
`error.type` is stamped after the cardinality filter, like
`gen_ai.token.type`, so an `otel.attributes` include/exclude list cannot
strip the discriminator and silently merge failures into the success series.
Resolves LIT-4955
* fix(otel): bound the failure metric's attribute set
The failure datapoint reused the success path's full attribute set, which
carries client-supplied fields (`metadata.requester_metadata`,
`metadata.spend_logs_metadata`, the end-user id taken from the request's
`user` field) and per-request ones (the `hidden_params` blob holding the
provider's response headers). A failed request needs no provider spend, so
nothing rate-limits a caller who puts a unique value in a field they control
and mints one histogram series per request.
A failure now carries a bounded allowlist: the operation enum, provider,
request model, framework, the key/alias/team/org/user identifiers, and
`error.type`. Every entry is a fixed enum or an operator-provisioned
identifier, so the failure series count is bounded by the deployment's own
key, team and user count while the labels still answer which team on which
model is failing and how. The user email is left out as PII duplicating the
user id already on the series. The operator's `otel.attributes` filter layers
on top, so it narrows the allowlist further and never widens it.
* fix(otel): cap metric attributes so series count does not grow with traffic (#35166)
`GenAIMetricRecorder._common_attributes` dumped the whole `hidden_params` object
onto every metric datapoint as one label value. That object is per-request by
construction: `response_cost`, `litellm_overhead_time_ms`, `cache_key`,
`usage_object` and the provider's `additional_headers` rate-limit counters all
move on every call. A unique label value is a new time series, and all six GenAI
instruments share those attributes, so one request minted up to six series that
would never be written to again
That is the steady-state behavior of the feature rather than an abuse case, and
it is wrong twice over. Hosted backends bill on series count, so recommending
metrics be enabled would have meant a bill proportional to traffic. And a
histogram whose every datapoint sits in its own series cannot be aggregated, so
the dashboards would have looked populated while answering nothing
Both paths now cap their attributes at METRIC_ATTRIBUTE_CEILING, which replaces
the failure-only allowlist so the two paths cannot drift. The cap runs before the
operator's `otel.attributes` filter, so an operator can narrow it and never widen
it back to an unbounded label. Client-supplied and per-request metadata
(`requester_metadata`, `spend_logs_metadata`, `user_api_key_end_user_id`,
`requester_ip_address`) is metric-ineligible and stays on the span, which already
carries it and where cardinality is free. `hidden_params` survives as a label but
carries only `model_id` and `api_base`, which are bounded by the router's own
deployment list and are the part a per-deployment panel reads
Four tests fail against the previous behavior, the load-bearing one being that
two requests differing only in per-request fields must land in one series rather
than two
The MCP gateway resolved a caller's allowed servers and per-server tool
allowlists from the key, the team, the end user and the agent, but never from
the internal user row, so an admin had no way to bound what a person may call
across every key they hold. Anything the key allowed went through
The internal user now carries the same object_permission an admin already
attaches to a key or a team, and the resolver applies it as a ceiling: the
caller ends up with the intersection of what the key allows and what the user
allows, so adding a user entitlement can only narrow, never widen. A level
that names no server and no tool places no ceiling, which keeps every existing
deployment on its current behavior
/user/new and /user/update accept object_permission and reuse the same
create-or-update helper the team endpoints use, so the row is written once and
the three cached views of it (the user row, the object-permission link and the
permission itself) are invalidated on write. Clearing it with an empty object
now really unlinks the permission instead of being swallowed as an empty value
A row that cannot be read at all places no ceiling, but a row that names a
permission the database cannot return denies the call rather than falling
through to the wider set, so a partial outage cannot hand out access the admin
withheld
The users page grows the MCP servers, access groups, toolsets and per-server
tool pickers the key and team pages already have. A save keeps a tool
allowlist whenever an access group or toolset the admin retained could still
supply that server, since an allowlist is what narrows a grant and an absent
one reads as no restriction; it drops the allowlist once nothing indirect
survives to supply the server, so removing a grant really removes it
* fix(otel): cap tool-definition attributes so they cannot evict gen_ai.* from the LLM span
The genai and legacy mappers each spelled out every declared tool as
per-index span attributes. A request declaring hundreds of tools produced
roughly 500 attributes against the OTel SDK's default 128-attribute span
limit, which evicts oldest-first, so the canonical gen_ai.* set written
first was discarded and the span exported with only a tail of tool
schemas. Cap the family at 8 tools, shared by both vocabularies, and
carry the declared total on litellm.request.tools.declared so the
truncation is visible rather than silent.
* fix(otel): apply the tool-definition cap to the OpenInference mapper
The OpenInference vocabulary emits its own unbounded llm.tools.{idx}.*
family, which Arize and Phoenix layer on top of the default two, so those
configurations still overran the span attribute limit and evicted the
core gen_ai.* attributes. Route it through the same shared cap and cover
the layered-mapper path with a test.
* fix(otel): share one span-wide tool-definition budget across vocabularies
Capping the tool-definition family per mapper left each active vocabulary
its own allowance, and several vocabularies write to the same span. With
every vendor vocabulary configured, the three that spell tools out per
index still summed past the SDK's 128-attribute span limit, so the core
gen_ai.* set written first was evicted exactly as before: measured at 128
attributes with 7 dropped and gen_ai.request.model gone.
Reserve a quarter of the span for tool detail and split that ceiling
across the distinct tool-emitting vocabularies at mapper-resolution time,
so the family is bounded span-wide no matter how many are configured. The
same worst case now exports 90 attributes with nothing dropped.
Auto-routed requests were indistinguishable from ordinary ones once logged:
the spend log recorded the requested model group and the resolved deployment,
but nothing about which tier was chosen or what chose it. That information
existed only inside verbose_router_logger f-strings, so answering "why did my
prompt land on the cheap model" required log access and a running proxy.
The complexity, quality, and adaptive pre-routing strategies now return a typed
StandardLoggingRoutingDecision on their PreRoutingHookResponse, and
Router.async_pre_routing_hook records it once for every attempt. Those three
previously side-channelled their own state through three different metadata
keys; the decision now travels on the hook contract itself, so the bucket is
resolved in one place, through get_or_create_metadata_bucket, which already
owns the question of which dict holds proxy-internal metadata and replaces a
non-dict value instead of skipping the write. Recording happens on every
attempt rather than only on a successful route: a fallback from an auto-router
group to a plain group re-enters the hook with the same request kwargs, and a
decision left behind there would attribute the first router's tier to the
deployment that actually served the retry. The log details drawer renders the
result as a Routing card between Request Details and Metrics; the card is
absent on rows that carry no decision, so ordinary and pre-upgrade rows are
unchanged.
Three defects surfaced while making the recorded cause truthful, each of which
would have persisted a wrong answer. The complexity router hardcoded
cause=complexity_scorer even when the LLM classifier decided, and its silent
fallback to the heuristic on classifier failure meant a row could claim an LLM
verdict the LLM never gave; the cause now reports the path that actually ran.
The keyword that triggered a tier rule was discarded before logging, as was
the escalation keyword. The 2-reasoning-marker override returned REASONING with
a score far below the REASONING boundary and no marker saying so, which reads
as a scoring bug to anyone comparing the two; it now emits a reasoning-override
signal, and the card labels those rows as an override instead of claiming the
score met a boundary. The LLM path no longer reports a synthetic score of 1.0,
and heuristic decisions carry a snapshot of the tier boundaries that mapped the
score, so a historical row stays interpretable after the boundaries change.
Signals name a matched term only when the caller's own message contains it.
Scoring still reads the system prompt, but a term matched solely there is
reported as a count, since signals reach a spend row the caller can read and
naming one would disclose a term from a prompt it cannot see.
routing_decision is stripped from caller-supplied metadata at ingress, so a
client cannot forge its own provenance.
The existing dashboards in this folder chart the litellm_* Prometheus metrics.
Nothing charted the gen_ai.* metrics the OpenTelemetry v2 integration emits, and
Grafana's own prebuilt GenAI dashboards cannot: twenty of their twenty-two panels
filter on telemetry_sdk_name="openlit", a label LiteLLM does not carry and has no
setting to add.
Ten panels over the six gen_ai instruments: spend, tokens, request count and p95
duration as stats, then request rate, spend per hour, tokens per minute split by
input and output, and p95 duration, time to first token, and provider generation
time by model. Template variables for data source, service, and model.
Verified against a live Grafana Cloud stack with real traffic across three
models. The readme documents the attribute filter the panels depend on, since the
default attribute set gives nearly every request its own series and makes every
rate-based panel read zero.
Replace `Model(**untyped_dict)` construction with `Model.model_validate(...)` at
the hot Any seams, and give the repository layer a real record type instead of
`Any`.
reportAny 22710 -> 21448, reportExplicitAny 7283 -> 7269, with every other rule
at or below its baseline repo-wide.
The Auto-Routers tab was proxy-admin only, while Add Model on the same page already
admits team admins. The asymmetry was not a policy decision; the auto-router create form
simply never mounted a team selector, so a team admin's submit was unscoped and POST
/model/new rejects an unscoped create from any non-proxy-admin. Mounting the shared
TeamDropdown closes it, and the tab now takes the same audience as its sibling.
Fixing that surfaced a second, larger problem. The dashboard decided who may edit or
delete a deployment with `(userRole === "Admin" || created_by === userID) && db_model`,
but `created_by` is written at creation and never read by any backend auth check. The API
authorizes on team-admin membership of model_info.team_id, so the dashboard was wrong in
both directions: it hid controls from team admins the API accepts, and offered them to
former team admins the API rejects. Verified against a live proxy; a model created by the
proxy admin was PATCHed and DELETEd 200 by a team admin who did not create it, while the
same key got 403 on another team's row and on an unscoped row.
Both questions now have one owner in utils/modelPermissions.ts, deliberately shaped as a
mirror of ModelManagementAuthChecks. Creation returns a tagged union rather than a pair of
booleans, so "may not create" and "may create unscoped" cannot be confused, and the five
places that had each invented their own spelling (the models page, the auto-routers tab
and panel, the auto-router form, and both branches of AddModelForm) call it instead.
Row affordances are now per row rather than per tab, because opening the tab to team
admins puts routers they cannot act on in the same list.
Note for reviewers: collapsing AddModelForm onto the shared owner changes behaviour for
org_admin and Admin Viewer who also admin a team. They previously got the optional team
selector, because all_admin_roles counts them as admins, and could submit an unscoped
create that the API always 403s; they now get the required selector.
Also corrects stale copy left by the auto-router move. The exclude_auto_routers API
description named a dashboard page, which went stale inside a single PR; it now describes
the concept so it cannot drift with the UI again.
The eslint-suppressions prune includes one entry for caching/_components/cache_dashboard.tsx,
which this branch does not touch. Its baseline was already stale; the gate measures the whole
tree, so it could not be left behind.
Completed-batch cost tracking parsed the whole output file into a list of
dicts, pretty-printed it into debug strings even with debug logging off, and
walked the list three times (cost, usage, models), so a large batch output
could pin a worker's memory. The output is now folded line by line into small
per-line stats records via _aggregate_batch_cost_usage_models, the eager
json.dumps debug calls are gone, and the raw-vertex path computes cost and
usage in one call instead of two. _get_batch_output_file_content_as_dictionary
becomes _fetch_batch_output_file_content (returns bytes); the superseded
three-pass helpers are deleted and their tests migrated
