`test_content_schema_uses_discriminator` fetches Google's live Interactions
OpenAPI document and required an OpenAPI `discriminator` on the `Content` union.
Google has since dropped that keyword and now pins `type` with a `const` on each
variant instead, so the assertion fails on the current spec and the `misc` shard
is red on every open PR against staging
The information the transformation actually needs did not change: a content part
is still routed by reading its `type`, and each variant still declares exactly
one distinct value for it. So the test now asserts that property directly, and
accepts either spelling, a `discriminator` on the union or a `const` (or
single-value `enum`) on each member
It stays a real check rather than a weakened one. Against the live spec it fails
if TextContent loses its type property, if `text` is renamed, if two variants
claim the same type value, if `Content` stops being a union of named variants,
or if a discriminator appears on some property other than `type`
Two issues surfaced in review:
- TenantTracerCache and the fan-out processor cache dropped the LRU entry on eviction
without shutting it down, leaking its BatchSpanProcessor worker thread for the life of
the process (306 distinct destinations -> 306 live threads at a 256 cap). Shut the
evicted provider/processor down on a background daemon thread, so the worker is
reclaimed without blocking the request path.
- The request-time destination resolver ran the per-request org/team lookup
(_effective_org_id) before checking whether any logging destination exists, so a proxy
not using admin-owned destinations still paid a team fetch on the auth path for
team-scoped keys. Short-circuit to () when the registry holds no logging credential,
before the lookup.
The prior commit gated credential management on credential_type at the handler
(trace destinations proxy-admin-only). That is more machinery than the case warrants:
/credentials is already governed by route authorization, so a non-admin reaches it
only when an admin explicitly delegates the route via allowed_routes, which is the
admin's decision to make.
Remove the handler-level authorization entirely (create/update/delete/list/by_name)
so credential access matches base: the proxy admin, or a key delegated /credentials
via allowed_routes, manages any credential; a plain non-admin key is still blocked at
route auth. The destinations feature adds logging-typed credentials to the store
without changing who may manage the store. Keeps the access-shape validation, the
partial-PATCH access merge, and otel_headers masking.
Credential create/update/delete and reads were gated to the proxy admin for every
credential, which tightened provider-credential management for keys an admin had
delegated /credentials to via allowed_routes; a backward-incompatible change beyond
the OTEL trace-destination feature.
Restrict the gate to trace destinations (credential_type="logging"). Those require the
proxy admin regardless of allowed_routes, so a non-admin cannot create a global
destination and receive other tenants' traces, nor convert a provider credential into
one. Provider credentials keep their existing route-level authorization. Reads follow
suit: the list hides destinations from non-admins and by_name returns 403 for a
destination, while provider credentials read as before.
* feat(dashscope): add qwen3.7-plus and qwen3.7-max to the model cost map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: limit backup cost map diff to the new dashscope entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost_calculator): adjust tier-only alias assertion for mapped qwen3.7-plus
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(dashscope): drop redundant cost map pinning tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost_calculator): point tier-only alias check at an unmapped model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Drops tests/e2e/guardrails/test_presidio_guardrail_e2e.py and the
PresidioParamsBody it was the only caller of.
Both cases were red on most stage runs between 07-25 and 07-29: pre_call
failed 6 of 11 runs, post_call 6 of 11, with post_call reporting the raw
address reaching the caller while apply_to_output was set.
The cause was propagation, not masking. GuardrailsClient.register() posts
/guardrails and returns immediately with no readiness wait, unlike
ProxyClient._await_model_servable or GuardrailsClient._await_team, and the
data plane only picks a new guardrail up on its next periodic DB sync. Calls
issued before that sync pass the raw value through. #34833 has since made
both cases poll to the deadline, and on the current build each masks on the
first attempt, so the suite is expected to be green now; it is being removed
because it spends real provider money on every retry and because a pod
replaced mid-poll still reproduces the old failure.
The three guardrail.presidio.* rows stay in coverage_registry/guardrail.yaml
and go uncovered on purpose, so Presidio reads as a tier-P0 gap in Grafana
rather than dropping out of the denominator.
Three defects kept LiteLLM's OTel metrics from reaching an OTLP backend.
OTEL_EXPORTER_OTLP_HEADERS is W3C Baggage encoded per the OTLP spec, so its
values are percent-encoded. litellm split the string on "," and "=" and passed
the raw value straight to the exporter, so a vendor that documents
"Authorization=Basic%20<token>" got a literal "%20" on the wire and the backend
rejected the credential. Grafana Cloud documents exactly that shape, which made
its OTLP gateway unreachable. Header parsing now delegates to the OTel SDK's own
W3C Baggage parser in liberal mode, so percent-encoded values decode and values
that were never encoded keep working. It moves from model/utils.py to
plumbing/providers.py because model/ is deliberately free of opentelemetry
imports; providers.parse_headers was already the entry point every caller used.
The OTLP metric exporters then overrode histogram temporality to delta.
Prometheus and Mimir, which back Grafana Cloud's OTLP gateway, reject delta
histograms outright: the gateway answers 400 "invalid temporality and type
combination" and drops the entire batch, so every GenAI metric was silently lost
while traces kept flowing. Backends that prefer delta still accept cumulative, so
the SDK default is the compatible choice in both directions, and the enterprise
billing exporter already relies on it.
Three GenAI instruments also carried names no convention or backend defines, so
nothing downstream could chart them. Time to first token and time per output
token take their semconv names, gen_ai.server.time_to_first_token and
gen_ai.server.time_per_output_token; the gen_ai.client.response.* spellings
litellm used are not conventions at all. Cost has no semconv instrument, so it
takes gen_ai.usage.cost, the name backends already query for spend. All three are
listed verbatim in Grafana Cloud's AI Observability integration reference, so its
prebuilt panels find them. Both engines now read the names from the shared Metric
constants rather than repeating string literals, so v1 and v2 cannot drift.
The renames are breaking for anyone charting the former names; the docs and the
release changelog carry the migration note.
* feat(ui): shareable log links via log_id query param on the logs page
Clicking a log row now writes ?log_id=<request_id> to the URL, closing the
drawer removes it, and loading the logs page with ?log_id= opens the drawer
for that log. When the log is not in the loaded page, it is fetched by
request_id (the backend already drops the date window for id lookups), so
links keep working for logs of any age. Drawer open state derives from the
URL, mirroring the models page ?model= pattern.
* fix(ui): close the log drawer on browser back after opening via session id
Session opens now write ?session_id= to the URL instead of holding local
state, so back removes both params and the drawer closes (Greptile P1).
Session views become shareable links as a side effect. In-drawer log
switching now replaces the history entry instead of pushing, so back
always closes the drawer in one step rather than replaying every viewed
log.
* fix(proxy): scope /spend/logs/session/ui to the requesting user's visible logs
Non-admin callers now only receive session rows they could already see on
/spend/logs/ui: their own logs plus logs of teams where they hold the
spend-logs permission. Previously any authenticated user could read any
session's log metadata by id, which shareable ?session_id= links made
trivial to trigger. Admin views are unchanged. Also, clicking a log row
now clears a lingering ?session_id= from the URL so the drawer shows the
clicked log instead of a stale session (Greptile P1).
* feat(ui): chart failed requests as their own series on the cache dashboard
Spend logs for failed requests are stored with an empty call_type, so the
Cache Hits vs API Requests chart lumped them into an Unknown bar that read
as normal LLM API traffic. The activity query now also returns a per-group
failed_rows count (status = 'failure') and the dashboard charts it as a
third stacked series, so failures are visibly separate from successful
requests and cache hits. The chart data transform moves into a pure
summarizeCacheActivity helper with unit tests; header stats keep their
existing semantics (cache hit ratio still counts failures in the
denominator).
* refactor(ui): move cache dashboard aggregation server-side with a typed response
The /global/activity/cache_hits endpoint previously returned raw per
(key, call_type, model) spend-log aggregates typed as LiteLLM_SpendLogs
(wrong), and the dashboard reduced them in the browser: grouping by
call_type, relabeling empty call_type as Unknown, and computing the stat
card totals. All of that now happens server-side. The SQL groups per
call_type and splits cache hits vs successful vs failed requests, a new
cache_activity module validates rows into Pydantic models and computes
totals plus the key-alias/model filter options, and the endpoint declares
a real response_model so schema.d.ts types it correctly. The dashboard
consumes it through a typed $api react-query hook (filters ride the
query key and are applied in SQL instead of the browser), the hand-rolled
summarizeCacheActivity transform and the adminGlobalCacheActivity fetch
helper are deleted, and the refresh button now actually refetches.
The endpoint is UI-internal (hidden from the public swagger), so the
response reshape is not a public API break.
* fix(scim): stop provisioning nested group ids as internal users
POST/PUT/PATCH /scim/v2/Groups treated every member.value as a user id, so
with the default scim_upsert_user=true an unknown id was auto-created as an
internal user. Entra sends nested groups as members carrying "type": "Group",
which meant every nested group produced a phantom internal user whose id and
email were the group GUID, and those users counted toward licensed seats.
Group members are now classified before they are used: members typed "Group"
are skipped without a database hit, an id that names an existing team is
skipped too (Okta sends untyped ids through filtered paths, so the type alone
is not enough), and only ids that resolve to a user, or that resolve to
nothing at all, keep today's behavior. The user lookup runs before the team
lookup so a user whose id collides with a team id keeps syncing.
The type was previously dropped at parse time on POST/PUT because SCIMMember
had no such field, and on PATCH because the raw member dicts were reduced to
bare ids; both paths now share one resolver and one parser that preserves it.
Member removals no longer upsert: a remove of an id we do not know is an
idempotent no-op rather than a reason to create a user and immediately drop
it, and strict mode (scim_upsert_user=false) no longer rejects it. Removal of
an id that is on the roster but has no user row still cleans up membership.
Responses now state members are of type "User" instead of emitting a null,
and the advertised Group schema documents the members.type sub-attribute.
* fix(scim): harden group member classification after adversarial review
Removals now bypass classification and drop exactly the ids they name,
restoring cleanup of roster entries the old bug left behind. The
team-id fallback only applies to untyped members, so an explicit User
type always provisions even when the id collides with a team. Member
types are normalized before matching; a type other than User or Group
only skips when the id is not an existing user. Non-string type values
are tolerated as absent on every verb instead of failing validation.
Admitted member ids are deduped order-preserving, which also closes a
pre-existing duplicate-row hazard on group creation.
* fix(scim): only treat scim-managed teams as nested groups
A PR reviewer flagged that an untyped SCIM member whose id collides
with an admin-created team was silently skipped, suppressing that
user's provisioning. SCIM group writes (POST, PUT, and every PATCH)
now stamp the team with scim_managed metadata, and the typeless
team-id skip only applies to teams carrying that marker or the
scim_data blob older PUTs already wrote. Admin-created teams stay
unmarked, so a colliding untyped member provisions the user in
permissive mode and returns the standard unknown-user 400 in strict
mode. Teams SCIM touched before this change adopt the marker on their
next group write.
Add regression tests for the db-fetch paths whose converted construction
lines were uncovered: the auth_checks getters (default end user budget, end
user, team membership, access group, team by alias, org by alias, object
permission, managed vector stores, project), get_all_team_memberships and
list_available_teams in team_endpoints, and the proxy admin user info
helper. Each test feeds a mocked prisma row through the real function and
asserts the validated model's fields, so a bad model_validate conversion on
any of these paths now fails a test instead of only dropping coverage.
Convert pydantic table-model construction from Cls(**row.model_dump())
kwargs-unpacking to Cls.model_validate(...) across the management endpoint
hotspot files (team, key, internal user, scim, model management, spend
tracking, auth checks, proxy_server). Unpacking an untyped dict reports one
Any-typed argument per matched model field, so each converted site clears
10-35 diagnostics while running the exact same pydantic validation.
Conversions were limited to models verified to use pydantic's default
__init__; UserAPIKeyAuth and LiteLLM_VerificationTokenView keep their custom
kwargs-rewriting __init__ and are untouched. Two locally-verified helper
params move from Any to object.
Whole-tree basedpyright, measured against the branch point in the same
environment: reportAny 24,431 -> 22,741 (-1,690), reportArgumentType
2,189 -> 2,136 (-53), reportUnknownArgumentType 34,370 -> 34,067 (-303),
reportExplicitAny 7,285 -> 7,283 (-2); total 154,882 -> 152,834 (-2,048)
with no rule increasing anywhere and no per-file increases. No casts, no
suppressions, no behavior changes. Budgets ratcheted: basedpyright -2,048
across 4 rules, ruff ANN401 -2.
Three review findings on the auth/key hot paths, all resolver/DB redundancy rather
than correctness bugs.
_hoist_request_destinations fires early in the auth builder and again as an outer
catch-all, so _resolve_logging_exporters ran twice per authenticated request. It now
early-returns when request.state already holds the result, keeping the early hoist
(for span timing) and the catch-all (a failed first call leaves state unset and still
retries) while resolving once.
OtelDestinationParams gains the resource_attributes key the resolver populates and the
hoist reads, so the resolver's dict type-checks.
regenerate_key_fn loaded the key's team via get_team_object on every team-key
regeneration, but only uses it when the request sets access_group_ids or
object_permission. The fetch moves back inside that guard, so an ordinary regenerate
does no extra DB round-trip; an access-group/object-permission regenerate still fetches.
Verified live: one chat resolves destinations once (was twice); a plain regenerate does
zero team fetches (was one) while an object_permission regenerate still does one; fan-out
isolation unchanged.
* fix(mcp): resolve call_tool by registry without requiring tool map
Multi-worker reloads put MCP servers in the registry from the DB but do
not re-run tools/list on every process. Gating call_tool on
tool_name_to_mcp_server_name_mapping made cold workers 500 with Tool not
found after another worker had already listed the tool. Treat a registry
match on server id/name/alias as enough; upstream rejects unknown tools
* test(e2e): poll MCP register, tools/list, and tools/call across multi-worker lag
Stage multi-worker gateways only load MCP servers and tool maps on the
process that handled the request. Poll until the server is listed, the
tool appears on tools/list, and tools/call is not a cold-worker 500 so
key-access and Datadog MCP e2e stop racing the LB
* Revert "fix(mcp): resolve call_tool by registry without requiring tool map"
This reverts commit 8b56e51e39.
* test(e2e): tighten MCP multi-worker lag classifier
Only retry tools/call on gateway shapes Tool <name> not found and
server_not_found, not any 500 that mentions tool/server not found, so
upstream failures are not retried until the poll deadline
* test(e2e): drop unit file for MCP lag classifier
The live await_call_tool polls already cover multi-worker lag; a separate
string-match unit module is not worth keeping
The two regression-test docstrings recounted the bug history and repro, which
belongs in the PR description rather than the code; they now state only what the
test checks. Also drops a leftover section comment and updates a few docstring
phrases that still said "assignment" to describe the access grant, since access is
the only routing input now.
A logging destination's credential_info.access (global / teams / orgs) now fully
decides which requests it receives; a destination fires for a request exactly when
its access grants the request's team or org. This removes the second, redundant way
to express the same team-to-destination mapping that the admin-only model left
behind: the auto_enable flag and the per-team/org logging_exporters assignment
column both existed for tenant self-service opt-in, and once assignment became
proxy-admin-only they only duplicated what access already says.
Removed: the auto_enable field on CredentialInfo; the team and organization
logging_exporters columns and their assignment gate (validate_logging_exporter_field
/ validate_logging_exporter_assignment); the request-time naming union in
litellm_pre_call_utils; and the dashboard's per-team/org destination picker and the
"Enable for entire scope" toggle. The access-shape validator stays, the credential's
access fields stay, and /team/info and /organization/info still disclose
resolved_logging_exporters computed from access alone.
This also removes the /v2/organization write that two review bots flagged (there is
no longer a logging_exporters field on that endpoint) and the "(via scope)" UI
ambiguity that came from carrying two representations of the same mapping.
Verified live on a 2-org / 4-team matrix against Langfuse, Arize, Weave, a generic
OTLP collector, and a self-hosted Phoenix: per-team and per-org isolation, empty
access as deny-all, injection defense, admin-only credential management, and
complete trace trees read back from each destination's own API.
An auto-router deployment's litellm_params.model (auto_router/...) is the
discriminator the router loads it by, but the model management endpoints
accepted any client-supplied value verbatim; a doubled or stripped prefix
made router init fail on the next load and ignore_invalid_deployments
silently dropped the deployment. Validate writes that supply
litellm_params.model at all three endpoints against the merged params and
reject incoherent values with an actionable 400. Classification is
extracted to router_utils/auto_router_model_naming.py so the Router
predicates and the validation share one source
Every model-write endpoint returned 200 off the DB write alone; a model the
reload dropped (ignore_invalid_deployments, or a wholesale reload failure)
stayed invisible on every channel at once, which is how the registry-leak
defect went undiagnosed for three weeks. ProxyConfig.add_deployment and
clear_cache now return whether the reload pass completed, and each write
endpoint verifies the rows it wrote are live in this pod's router afterwards,
distinguishing a deliberately environment-inactive model via the same
predicate the Router's own gate uses. The access-group writers return the
mutated id set instead of discarding it
A request rejected before any upstream call (rate limit, budget, pre-call
guardrail) fires the failure callback with a built payload while the
auth-hoisted destinations are still resolved, so _close_llm_call's carrier-None
branch fabricated a gen-AI span for a call that never reached a provider. It now
re-reads call.is_no_upstream_call, the same marker log_pre_api_call already
honors, so a rejected request no longer lands a fake "chat <model>" span in the
tenant's destination. Reproduced live against a real destination (a 429 emitted
a chat span in Arize before the fix, none after) and pinned by a regression test
that fails on the pre-fix code
The pre-call resolver re-run in _apply_admin_logging_exporters was not wrapped,
unlike the auth-time hoist, so a non-HTTPException from the org fallback lookup
could abort a real request; it is now best-effort so admin-owned telemetry setup
can never break request handling
Also drops the explanatory inline comments this feature added across the otel v2
modules, keeping only docstrings and lint suppressions per the repository
convention that new code carries no comments
Destinations now bind at tenancy granularity only. The per-key logging_exporters
surface was half-shipped (edit-form picker but no create flow) and unrequested,
so it goes: the column leaves the key tables and the migration, /key/generate,
/key/update, and /key/regenerate stop accepting the field, the resolver's union
reads team and org columns only, and the key pages drop the picker and exporter
badges. A key's traces route by its team and org, which the live check confirms:
a brand-new team key exports to the team's destinations with no assignment.
Subset targeting below a whole scope remains available at team granularity
(a multi-team scope with enable-for-entire-scope off, named on specific teams).
Re-adding key granularity later is a purely additive column and field.
The team and org pages showed different Logging Exporters lists per role: the
(via scope) badges were derived client-side from GET /credentials, which is
proxy-admin only, so non-admin viewers saw only the identity's own assignments.
Add resolved_logging_exporters to the /team/info and /organization/info
responses: the destination names that will receive the identity's traces,
computed server-side with the same selection the request-time resolver uses
(access grants the identity AND auto_enable or named). Names only; endpoints,
headers, and the access map stay proxy-admin information. The UI renders the
badges from this field, deleting the client-side credentials derivation, so
every role sees the identical list.
* fix(proxy): warm rotate Prisma client for IAM refresh
* fix(proxy): drain Prisma operations during IAM rotation
* fix(proxy): bound the drain wait when retiring a replaced prisma engine
A replaced engine waited indefinitely for its drain tracker to empty.
Hung queries self-release via prisma's 30s default HTTP timeout, but a
transaction whose owner is hard-cancelled before commit/rollback leaks
its drain count forever, keeping the retired engine and its DB
connection pool alive indefinitely; at one rotation per 12 minutes such
engines accumulate. Cap the wait at 90 seconds, which exceeds every
legitimate operation bound (30s HTTP timeout, 60s max interactive
transaction timeout in this codebase), then kill the engine anyway.
Work killed at the deadline degrades to the pre-drain behavior and is
retried by the existing reconnect/backoff layers.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Scrub aliases on delete only when the deleted deployment's model_name no
longer resolves in the router. A legacy load-balanced team model can have
several deployment rows sharing one internal name; deleting one replica
must not remove aliases that still route to the survivors, in any team
