* feat(otel): record the GenAI duration metric on failed requests
`_record_metrics` ran only from `async_log_success_event`, so
`gen_ai.client.operation.duration` counted only the requests that worked.
Latency read off it during an incident was the latency of the surviving
traffic, and with no error dimension anywhere there was no way to build a
failure-rate panel or a success/failure split per model.
A failed call now records the same duration histogram, tagged with the
semconv `error.type` (the mapped provider exception's class name, bounded by
construction; the message stays on the span). Success attributes are
untouched, so an existing query can still isolate the old series with
`error_type=""`. The other five instruments describe a completed generation
and are skipped rather than filled with a fabricated zero: litellm hands the
failure callback no `response_obj`, so there is no usage to split and no
completion-token count, and it zeroes `response_cost` on failure. A
proxy-gate rejection (auth / rate limit) records nothing, for the same
reason it gets no span; no upstream call happened.
`error.type` is stamped after the cardinality filter, like
`gen_ai.token.type`, so an `otel.attributes` include/exclude list cannot
strip the discriminator and silently merge failures into the success series.
Resolves LIT-4955
* fix(otel): bound the failure metric's attribute set
The failure datapoint reused the success path's full attribute set, which
carries client-supplied fields (`metadata.requester_metadata`,
`metadata.spend_logs_metadata`, the end-user id taken from the request's
`user` field) and per-request ones (the `hidden_params` blob holding the
provider's response headers). A failed request needs no provider spend, so
nothing rate-limits a caller who puts a unique value in a field they control
and mints one histogram series per request.
A failure now carries a bounded allowlist: the operation enum, provider,
request model, framework, the key/alias/team/org/user identifiers, and
`error.type`. Every entry is a fixed enum or an operator-provisioned
identifier, so the failure series count is bounded by the deployment's own
key, team and user count while the labels still answer which team on which
model is failing and how. The user email is left out as PII duplicating the
user id already on the series. The operator's `otel.attributes` filter layers
on top, so it narrows the allowlist further and never widens it.
* fix(otel): cap metric attributes so series count does not grow with traffic (#35166)
`GenAIMetricRecorder._common_attributes` dumped the whole `hidden_params` object
onto every metric datapoint as one label value. That object is per-request by
construction: `response_cost`, `litellm_overhead_time_ms`, `cache_key`,
`usage_object` and the provider's `additional_headers` rate-limit counters all
move on every call. A unique label value is a new time series, and all six GenAI
instruments share those attributes, so one request minted up to six series that
would never be written to again
That is the steady-state behavior of the feature rather than an abuse case, and
it is wrong twice over. Hosted backends bill on series count, so recommending
metrics be enabled would have meant a bill proportional to traffic. And a
histogram whose every datapoint sits in its own series cannot be aggregated, so
the dashboards would have looked populated while answering nothing
Both paths now cap their attributes at METRIC_ATTRIBUTE_CEILING, which replaces
the failure-only allowlist so the two paths cannot drift. The cap runs before the
operator's `otel.attributes` filter, so an operator can narrow it and never widen
it back to an unbounded label. Client-supplied and per-request metadata
(`requester_metadata`, `spend_logs_metadata`, `user_api_key_end_user_id`,
`requester_ip_address`) is metric-ineligible and stays on the span, which already
carries it and where cardinality is free. `hidden_params` survives as a label but
carries only `model_id` and `api_base`, which are bounded by the router's own
deployment list and are the part a per-deployment panel reads
Four tests fail against the previous behavior, the load-bearing one being that
two requests differing only in per-request fields must land in one series rather
than two
The MCP gateway resolved a caller's allowed servers and per-server tool
allowlists from the key, the team, the end user and the agent, but never from
the internal user row, so an admin had no way to bound what a person may call
across every key they hold. Anything the key allowed went through
The internal user now carries the same object_permission an admin already
attaches to a key or a team, and the resolver applies it as a ceiling: the
caller ends up with the intersection of what the key allows and what the user
allows, so adding a user entitlement can only narrow, never widen. A level
that names no server and no tool places no ceiling, which keeps every existing
deployment on its current behavior
/user/new and /user/update accept object_permission and reuse the same
create-or-update helper the team endpoints use, so the row is written once and
the three cached views of it (the user row, the object-permission link and the
permission itself) are invalidated on write. Clearing it with an empty object
now really unlinks the permission instead of being swallowed as an empty value
A row that cannot be read at all places no ceiling, but a row that names a
permission the database cannot return denies the call rather than falling
through to the wider set, so a partial outage cannot hand out access the admin
withheld
The users page grows the MCP servers, access groups, toolsets and per-server
tool pickers the key and team pages already have. A save keeps a tool
allowlist whenever an access group or toolset the admin retained could still
supply that server, since an allowlist is what narrows a grant and an absent
one reads as no restriction; it drops the allowlist once nothing indirect
survives to supply the server, so removing a grant really removes it
* fix(otel): cap tool-definition attributes so they cannot evict gen_ai.* from the LLM span
The genai and legacy mappers each spelled out every declared tool as
per-index span attributes. A request declaring hundreds of tools produced
roughly 500 attributes against the OTel SDK's default 128-attribute span
limit, which evicts oldest-first, so the canonical gen_ai.* set written
first was discarded and the span exported with only a tail of tool
schemas. Cap the family at 8 tools, shared by both vocabularies, and
carry the declared total on litellm.request.tools.declared so the
truncation is visible rather than silent.
* fix(otel): apply the tool-definition cap to the OpenInference mapper
The OpenInference vocabulary emits its own unbounded llm.tools.{idx}.*
family, which Arize and Phoenix layer on top of the default two, so those
configurations still overran the span attribute limit and evicted the
core gen_ai.* attributes. Route it through the same shared cap and cover
the layered-mapper path with a test.
* fix(otel): share one span-wide tool-definition budget across vocabularies
Capping the tool-definition family per mapper left each active vocabulary
its own allowance, and several vocabularies write to the same span. With
every vendor vocabulary configured, the three that spell tools out per
index still summed past the SDK's 128-attribute span limit, so the core
gen_ai.* set written first was evicted exactly as before: measured at 128
attributes with 7 dropped and gen_ai.request.model gone.
Reserve a quarter of the span for tool detail and split that ceiling
across the distinct tool-emitting vocabularies at mapper-resolution time,
so the family is bounded span-wide no matter how many are configured. The
same worst case now exports 90 attributes with nothing dropped.
Auto-routed requests were indistinguishable from ordinary ones once logged:
the spend log recorded the requested model group and the resolved deployment,
but nothing about which tier was chosen or what chose it. That information
existed only inside verbose_router_logger f-strings, so answering "why did my
prompt land on the cheap model" required log access and a running proxy.
The complexity, quality, and adaptive pre-routing strategies now return a typed
StandardLoggingRoutingDecision on their PreRoutingHookResponse, and
Router.async_pre_routing_hook records it once for every attempt. Those three
previously side-channelled their own state through three different metadata
keys; the decision now travels on the hook contract itself, so the bucket is
resolved in one place, through get_or_create_metadata_bucket, which already
owns the question of which dict holds proxy-internal metadata and replaces a
non-dict value instead of skipping the write. Recording happens on every
attempt rather than only on a successful route: a fallback from an auto-router
group to a plain group re-enters the hook with the same request kwargs, and a
decision left behind there would attribute the first router's tier to the
deployment that actually served the retry. The log details drawer renders the
result as a Routing card between Request Details and Metrics; the card is
absent on rows that carry no decision, so ordinary and pre-upgrade rows are
unchanged.
Three defects surfaced while making the recorded cause truthful, each of which
would have persisted a wrong answer. The complexity router hardcoded
cause=complexity_scorer even when the LLM classifier decided, and its silent
fallback to the heuristic on classifier failure meant a row could claim an LLM
verdict the LLM never gave; the cause now reports the path that actually ran.
The keyword that triggered a tier rule was discarded before logging, as was
the escalation keyword. The 2-reasoning-marker override returned REASONING with
a score far below the REASONING boundary and no marker saying so, which reads
as a scoring bug to anyone comparing the two; it now emits a reasoning-override
signal, and the card labels those rows as an override instead of claiming the
score met a boundary. The LLM path no longer reports a synthetic score of 1.0,
and heuristic decisions carry a snapshot of the tier boundaries that mapped the
score, so a historical row stays interpretable after the boundaries change.
Signals name a matched term only when the caller's own message contains it.
Scoring still reads the system prompt, but a term matched solely there is
reported as a count, since signals reach a spend row the caller can read and
naming one would disclose a term from a prompt it cannot see.
routing_decision is stripped from caller-supplied metadata at ingress, so a
client cannot forge its own provenance.
Bedrock routes Claude Sonnet 5 through the same Anthropic-compatible
validator as Opus 4.7/4.8 and Sonnet 4, which rejects toolSpec.strict
with 'tools.0.custom.strict: Extra inputs are not permitted'. Set
bedrock_converse_supports_strict_tools: false on all six Sonnet 5
entries so the existing gate strips the field, matching the fix shape
of #31582
Co-authored-by: Yaroslav Budyanskiy <y_budyanskiy@wargaming.net>
Replace `Model(**untyped_dict)` construction with `Model.model_validate(...)` at
the hot Any seams, and give the repository layer a real record type instead of
`Any`.
reportAny 22710 -> 21448, reportExplicitAny 7283 -> 7269, with every other rule
at or below its baseline repo-wide.
Completed-batch cost tracking parsed the whole output file into a list of
dicts, pretty-printed it into debug strings even with debug logging off, and
walked the list three times (cost, usage, models), so a large batch output
could pin a worker's memory. The output is now folded line by line into small
per-line stats records via _aggregate_batch_cost_usage_models, the eager
json.dumps debug calls are gone, and the raw-vertex path computes cost and
usage in one call instead of two. _get_batch_output_file_content_as_dictionary
becomes _fetch_batch_output_file_content (returns bytes); the superseded
three-pass helpers are deleted and their tests migrated
Auto-routers had no home and no list. The create form was mounted in two unrelated places,
inside Models + Endpoints > Add Model and again under Cost Optimization, and neither showed
which auto routers already existed; seeing or editing one meant finding its row in the models
table and drilling in. They now get a dedicated Auto-Routers tab beside All Models, listing
every auto_router/* deployment with create, edit and delete in one place, and both former
entry points are removed.
Creating opens in a shadcn dialog rather than swapping the whole panel out, so the list stays
on screen behind it; the dialog caps its height and scrolls, since the complexity form is long.
The form's own heading goes with it, the dialog header owning that now.
An auto router is a routing construct rather than a deployment, so it also comes off the All
Models table. That table pages server-side off total_count, so a client-side filter would page
over a total including rows it never renders; /v2/model/info therefore gains
exclude_auto_routers (default false, so every existing caller is unaffected) and the filter
runs before the count. /v1/models is untouched, so clients still see auto-routers as models.
Clicking a router opens the same `?model=` drill-in the All Models table uses, so it lands in
ModelInfoView with the full Model Settings, Edit Settings, Edit Auto Router and Delete. An
earlier revision had a bespoke detail page here, which was a partial reimplementation of that
view and showed the router's type twice, once as a Type pill and again as a "Routing strategy"
field saying the same thing. Both are gone.
The auto-router list is keyed under the same `models/list` namespace as the models table
rather than a private one. It reads the same /v2/model/info data, and six call sites across
the app already invalidate ["models","list"] after a write; a separate key meant an edit made
through ModelInfoView left the tab stale until a full reload, and every future writer would
have had to remember a second key.
An auto router has no upstream credential, so its detail header drops Update API Key and
Re-use Credentials, and the destructive action names what it removes rather than saying model.
Test Connection was gated on the editor-aware predicate, which let adaptive and quality routers
through to a check that builds its targets from complexity config they do not have; it now
gates on the deployment predicate.
The edit modal also applies the semantic-matching guard the create form has. It renders those
controls now, and the backend raises on semantic_keyword_matching without an embedding model or
keyword rules, so skipping the shared validator turned an inline message into a raw 400.
Whether a row is writable has two independent axes and the dashboard needs both. STRATEGY:
there are four auto_router/* kinds and only complexity and semantic have a form here, so
adaptive and quality must not be handed an editor that would write auto_router_config onto a
deployment storing its settings elsewhere. ORIGIN: a config.yaml row reports db_model false and
the API refuses it whatever its strategy (PATCH /model/{id}/update 404s, POST /model/delete
400s). Capability is derived per capability rather than as one editable flag, because the
constraints differ: editing needs an editor, deleting removes a row by id and never reads its
config, so a DB-created adaptive router stays deletable. Both axes live in
add_model/auto_router_strategies.ts as a declarative table, one record per strategy, so a fifth
strategy is a table row rather than another branch. That also retired four copies of "is this a
complexity router", one of which was written twice in a row in model_info_view.
Creation narrows to the complexity router, which the UI calls Auto-Router v2; the semantic
option was already badged "to be deprecated" in the picker, so the picker goes away along with
the semantic submit path and its validation helper. Existing semantic routers stay editable.
The edit modal mounted ComplexityRouterConfig without the keyword, escalation and
semantic-matching handlers, so those sections never rendered and could only be set at create
time. It now hydrates them from the stored config, and the five keys become managed only when a
caller supplies that state, so a caller rendering no such control still carries them through. A
component-level round-trip test covers it: a payload-builder test cannot see a hydration bug.
A complexity tier is str | list[str] on the backend, and the UI carried three readers of that
rule, one of which dropped a pinned string. They collapse into one owner,
add_model/complexity_router_tiers.ts.
gpt-5.4-mini and gpt-5.4-nano are 400K-context models (272K input,
128K output), but their cost map entries carried gpt-5.4's 1.05M window.
The router's pre-call context window check therefore admitted prompts far
past what the models accept, so oversized requests were dispatched to the
provider and failed there instead of being caught locally or routed
through context_window_fallbacks.
The azure_ai entries also inherited gpt-5.4's above-272K tiered pricing.
OpenAI applies that surcharge to the 1.05M-window models only, so those
keys are removed.
Limits per OpenAI's model reference and Azure AI Foundry's model table:
gpt-5.4-mini and gpt-5.4-nano are 400,000 context / 272,000 input /
128,000 output
Fireworks publishes a 262144-token context window for the Kimi K2.5, K2.6
and K2.7 models but caps generation well below that. Every fireworks_ai
Kimi K2.5/K2.6/K2.7 alias had max_output_tokens/max_tokens flattened to
262144 (equal to the context window), so the pre-call context-window check
admitted requests asking for a full 262144-token completion that Fireworks
rejects. Correct max_output_tokens/max_tokens to 32768 while keeping
max_input_tokens at 262144, and add a regression test pinning the limits
for all ten aliases.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The classifier and semantic-embedding sub-calls now capture proxy_server_request,
but neither forwarded the caller's turn_off_message_logging opt-out. A caller who
disabled message logging still had their prompt stored in the clear in these
internal sub-calls' spend-log rows, since should_redact_message_logging reads the
flag per-call and this internal call never inherited it.
The classifier read its metadata only from litellm_metadata, which the proxy
populates just for LITELLM_METADATA_ROUTES (/v1/messages, /v1/responses, ...);
/v1/chat/completions puts it under metadata, so the classifier call arrived
unattributed and _should_track_cost_callback dropped it, leaving no spend-log
row at all for the captured request body to show up in.
Also log response_format in the wire shape litellm actually sends
(type_to_response_format_param) instead of the bare pydantic JSON schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cli): read base_url from persistent config file
Adds a lite config command group (set/get/unset) backed by
~/.litellm/config.json so users no longer need to export
LITELLM_PROXY_URL in every shell session. Resolution order is
--base-url flag, then LITELLM_PROXY_URL, then the config file,
then the localhost default. A config-file base_url counts as an
explicit server choice for lite auth print-token, matching the
env var semantics it replaces.
* fix(cli): harden config persistence after review feedback
Rejects base_url values containing a query string or fragment,
including bare trailing ? or # which parse as empty but still
corrupt every joined request URL. Writes config.json and token.json
atomically through a shared write_private_json helper (0600 at
creation, fsync, os.replace) so an interrupted save can no longer
truncate the file or leave it world-readable. Warns on stderr when
an existing config file is invalid instead of silently ignoring it,
including invalid UTF-8. Resolves the eager --version flag through
the same env, config file, default chain as every other command,
and reads the config file once per invocation so base_url and
base_url_explicit always come from the same snapshot.
* fix(cli): resolve --version after option parsing
The eager --version callback ran before --base-url and --api-key were
parsed, so it could not see an explicitly named server. Combined with
the env fallback added for config-file support, that sent the resolved
API key to whichever server the config file pointed at even when the
user named a different one on the command line. Making the flag a
normal option and handling it in the group callback gives the version
request the same flag, env, config, default precedence as every other
command, and lets the stored-token lookup stay origin-checked.
An interactive oauth2 MCP server created with explicit endpoint URLs and no issuer served
400 "authorization url is not configured" from /authorize about a minute after creation,
with the admin's endpoints intact in the row the whole time (#34985). Discovery wrote its
trust-on-first-use issuer into the same column an admin writes, so the next registry build
read the gateway's own output back as an admin pin, anchored the server to RFC 8414
section 3.3, and discarded the stored endpoint columns; one transient metadata fetch
failure then had nothing to serve, and the reload fast path pinned the broken entry until
an unrelated config write
The core of the fix is a deletion. The gateway no longer writes discovery results anywhere:
the OAuth columns and credentials.scopes carry admin intent alone, and everything discovery
learns lives on the in-memory registry entry, as the existing carry-forward already
assumes. With no gateway write there is no value whose provenance a later build can
misread, so the accidental anchoring cannot be expressed
Deleting the write cannot fix a row a released version already stamped, which still reads
as pinned, so a one-time startup heal clears those stamps. The signal is necessarily a
heuristic: updated_by records only the most recent writer and no audit trail says which
field it touched. A row is therefore healed only on the full signature of the defect, which
is discovery as the last writer plus an issuer plus at least one configured endpoint column
that anchoring is actively discarding; rows with an issuer but no configured endpoints are
left alone, since for them both paths resolve from the same upstream document. Every heal
logs the cleared value so an admin who pinned deliberately can re-pin, and the heal records
its own actor, which makes it idempotent
The reload fast path exempts servers missing an endpoint their flow needs, so failed
discovery retries on the normal reload cadence rather than waiting for a config write. Flow
requirements are read through effective_oauth2_flow, the column-first shape-fallback judge
every flow decision uses, so a legacy null-flow M2M row is classified exactly as the
request path classifies it instead of re-discovering forever; a dcr_bridge server with no
configured client needs its registration endpoint for the relay arm, and an entra_obo
server needs a scope, both of which discovery can supply. Retries back off per server,
doubling from one reload cadence to a fifteen-minute cap, so a permanently unresolvable
server cannot re-run the RFC 9728 to 8414 chain and re-log its warning every cycle forever
Deployments with store_model_in_db unset or false loaded MCP servers exactly once at
startup, leaving that retry with no driver, so they now refresh the registry on the same
reload interval. That job deliberately calls a reload-only entry point rather than the
startup composite, keeping the one-time oauth2_flow backfill and issuer heal out of a
recurring path
Losing the persisted trust-on-first-use issuer also means the issuer column no longer
changes underneath the OAuth token identity, so user tokens are purged only when an admin
actually edits the server
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
`test_content_schema_uses_discriminator` fetches Google's live Interactions
OpenAPI document and required an OpenAPI `discriminator` on the `Content` union.
Google has since dropped that keyword and now pins `type` with a `const` on each
variant instead, so the assertion fails on the current spec and the `misc` shard
is red on every open PR against staging
The information the transformation actually needs did not change: a content part
is still routed by reading its `type`, and each variant still declares exactly
one distinct value for it. So the test now asserts that property directly, and
accepts either spelling, a `discriminator` on the union or a `const` (or
single-value `enum`) on each member
It stays a real check rather than a weakened one. Against the live spec it fails
if TextContent loses its type property, if `text` is renamed, if two variants
claim the same type value, if `Content` stops being a union of named variants,
or if a discriminator appears on some property other than `type`
* feat(dashscope): add qwen3.7-plus and qwen3.7-max to the model cost map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: limit backup cost map diff to the new dashscope entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost_calculator): adjust tier-only alias assertion for mapped qwen3.7-plus
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(dashscope): drop redundant cost map pinning tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost_calculator): point tier-only alias check at an unmapped model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Drops tests/e2e/guardrails/test_presidio_guardrail_e2e.py and the
PresidioParamsBody it was the only caller of.
Both cases were red on most stage runs between 07-25 and 07-29: pre_call
failed 6 of 11 runs, post_call 6 of 11, with post_call reporting the raw
address reaching the caller while apply_to_output was set.
The cause was propagation, not masking. GuardrailsClient.register() posts
/guardrails and returns immediately with no readiness wait, unlike
ProxyClient._await_model_servable or GuardrailsClient._await_team, and the
data plane only picks a new guardrail up on its next periodic DB sync. Calls
issued before that sync pass the raw value through. #34833 has since made
both cases poll to the deadline, and on the current build each masks on the
first attempt, so the suite is expected to be green now; it is being removed
because it spends real provider money on every retry and because a pod
replaced mid-poll still reproduces the old failure.
The three guardrail.presidio.* rows stay in coverage_registry/guardrail.yaml
and go uncovered on purpose, so Presidio reads as a tier-P0 gap in Grafana
rather than dropping out of the denominator.
