Three defects kept LiteLLM's OTel metrics from reaching an OTLP backend.
OTEL_EXPORTER_OTLP_HEADERS is W3C Baggage encoded per the OTLP spec, so its
values are percent-encoded. litellm split the string on "," and "=" and passed
the raw value straight to the exporter, so a vendor that documents
"Authorization=Basic%20<token>" got a literal "%20" on the wire and the backend
rejected the credential. Grafana Cloud documents exactly that shape, which made
its OTLP gateway unreachable. Header parsing now delegates to the OTel SDK's own
W3C Baggage parser in liberal mode, so percent-encoded values decode and values
that were never encoded keep working. It moves from model/utils.py to
plumbing/providers.py because model/ is deliberately free of opentelemetry
imports; providers.parse_headers was already the entry point every caller used.
The OTLP metric exporters then overrode histogram temporality to delta.
Prometheus and Mimir, which back Grafana Cloud's OTLP gateway, reject delta
histograms outright: the gateway answers 400 "invalid temporality and type
combination" and drops the entire batch, so every GenAI metric was silently lost
while traces kept flowing. Backends that prefer delta still accept cumulative, so
the SDK default is the compatible choice in both directions, and the enterprise
billing exporter already relies on it.
Three GenAI instruments also carried names no convention or backend defines, so
nothing downstream could chart them. Time to first token and time per output
token take their semconv names, gen_ai.server.time_to_first_token and
gen_ai.server.time_per_output_token; the gen_ai.client.response.* spellings
litellm used are not conventions at all. Cost has no semconv instrument, so it
takes gen_ai.usage.cost, the name backends already query for spend. All three are
listed verbatim in Grafana Cloud's AI Observability integration reference, so its
prebuilt panels find them. Both engines now read the names from the shared Metric
constants rather than repeating string literals, so v1 and v2 cannot drift.
The renames are breaking for anyone charting the former names; the docs and the
release changelog carry the migration note.
* feat(ui): shareable log links via log_id query param on the logs page
Clicking a log row now writes ?log_id=<request_id> to the URL, closing the
drawer removes it, and loading the logs page with ?log_id= opens the drawer
for that log. When the log is not in the loaded page, it is fetched by
request_id (the backend already drops the date window for id lookups), so
links keep working for logs of any age. Drawer open state derives from the
URL, mirroring the models page ?model= pattern.
* fix(ui): close the log drawer on browser back after opening via session id
Session opens now write ?session_id= to the URL instead of holding local
state, so back removes both params and the drawer closes (Greptile P1).
Session views become shareable links as a side effect. In-drawer log
switching now replaces the history entry instead of pushing, so back
always closes the drawer in one step rather than replaying every viewed
log.
* fix(proxy): scope /spend/logs/session/ui to the requesting user's visible logs
Non-admin callers now only receive session rows they could already see on
/spend/logs/ui: their own logs plus logs of teams where they hold the
spend-logs permission. Previously any authenticated user could read any
session's log metadata by id, which shareable ?session_id= links made
trivial to trigger. Admin views are unchanged. Also, clicking a log row
now clears a lingering ?session_id= from the URL so the drawer shows the
clicked log instead of a stale session (Greptile P1).
* feat(ui): chart failed requests as their own series on the cache dashboard
Spend logs for failed requests are stored with an empty call_type, so the
Cache Hits vs API Requests chart lumped them into an Unknown bar that read
as normal LLM API traffic. The activity query now also returns a per-group
failed_rows count (status = 'failure') and the dashboard charts it as a
third stacked series, so failures are visibly separate from successful
requests and cache hits. The chart data transform moves into a pure
summarizeCacheActivity helper with unit tests; header stats keep their
existing semantics (cache hit ratio still counts failures in the
denominator).
* refactor(ui): move cache dashboard aggregation server-side with a typed response
The /global/activity/cache_hits endpoint previously returned raw per
(key, call_type, model) spend-log aggregates typed as LiteLLM_SpendLogs
(wrong), and the dashboard reduced them in the browser: grouping by
call_type, relabeling empty call_type as Unknown, and computing the stat
card totals. All of that now happens server-side. The SQL groups per
call_type and splits cache hits vs successful vs failed requests, a new
cache_activity module validates rows into Pydantic models and computes
totals plus the key-alias/model filter options, and the endpoint declares
a real response_model so schema.d.ts types it correctly. The dashboard
consumes it through a typed $api react-query hook (filters ride the
query key and are applied in SQL instead of the browser), the hand-rolled
summarizeCacheActivity transform and the adminGlobalCacheActivity fetch
helper are deleted, and the refresh button now actually refetches.
The endpoint is UI-internal (hidden from the public swagger), so the
response reshape is not a public API break.
* fix(scim): stop provisioning nested group ids as internal users
POST/PUT/PATCH /scim/v2/Groups treated every member.value as a user id, so
with the default scim_upsert_user=true an unknown id was auto-created as an
internal user. Entra sends nested groups as members carrying "type": "Group",
which meant every nested group produced a phantom internal user whose id and
email were the group GUID, and those users counted toward licensed seats.
Group members are now classified before they are used: members typed "Group"
are skipped without a database hit, an id that names an existing team is
skipped too (Okta sends untyped ids through filtered paths, so the type alone
is not enough), and only ids that resolve to a user, or that resolve to
nothing at all, keep today's behavior. The user lookup runs before the team
lookup so a user whose id collides with a team id keeps syncing.
The type was previously dropped at parse time on POST/PUT because SCIMMember
had no such field, and on PATCH because the raw member dicts were reduced to
bare ids; both paths now share one resolver and one parser that preserves it.
Member removals no longer upsert: a remove of an id we do not know is an
idempotent no-op rather than a reason to create a user and immediately drop
it, and strict mode (scim_upsert_user=false) no longer rejects it. Removal of
an id that is on the roster but has no user row still cleans up membership.
Responses now state members are of type "User" instead of emitting a null,
and the advertised Group schema documents the members.type sub-attribute.
* fix(scim): harden group member classification after adversarial review
Removals now bypass classification and drop exactly the ids they name,
restoring cleanup of roster entries the old bug left behind. The
team-id fallback only applies to untyped members, so an explicit User
type always provisions even when the id collides with a team. Member
types are normalized before matching; a type other than User or Group
only skips when the id is not an existing user. Non-string type values
are tolerated as absent on every verb instead of failing validation.
Admitted member ids are deduped order-preserving, which also closes a
pre-existing duplicate-row hazard on group creation.
* fix(scim): only treat scim-managed teams as nested groups
A PR reviewer flagged that an untyped SCIM member whose id collides
with an admin-created team was silently skipped, suppressing that
user's provisioning. SCIM group writes (POST, PUT, and every PATCH)
now stamp the team with scim_managed metadata, and the typeless
team-id skip only applies to teams carrying that marker or the
scim_data blob older PUTs already wrote. Admin-created teams stay
unmarked, so a colliding untyped member provisions the user in
permissive mode and returns the standard unknown-user 400 in strict
mode. Teams SCIM touched before this change adopt the marker on their
next group write.
Add regression tests for the db-fetch paths whose converted construction
lines were uncovered: the auth_checks getters (default end user budget, end
user, team membership, access group, team by alias, org by alias, object
permission, managed vector stores, project), get_all_team_memberships and
list_available_teams in team_endpoints, and the proxy admin user info
helper. Each test feeds a mocked prisma row through the real function and
asserts the validated model's fields, so a bad model_validate conversion on
any of these paths now fails a test instead of only dropping coverage.
Convert pydantic table-model construction from Cls(**row.model_dump())
kwargs-unpacking to Cls.model_validate(...) across the management endpoint
hotspot files (team, key, internal user, scim, model management, spend
tracking, auth checks, proxy_server). Unpacking an untyped dict reports one
Any-typed argument per matched model field, so each converted site clears
10-35 diagnostics while running the exact same pydantic validation.
Conversions were limited to models verified to use pydantic's default
__init__; UserAPIKeyAuth and LiteLLM_VerificationTokenView keep their custom
kwargs-rewriting __init__ and are untouched. Two locally-verified helper
params move from Any to object.
Whole-tree basedpyright, measured against the branch point in the same
environment: reportAny 24,431 -> 22,741 (-1,690), reportArgumentType
2,189 -> 2,136 (-53), reportUnknownArgumentType 34,370 -> 34,067 (-303),
reportExplicitAny 7,285 -> 7,283 (-2); total 154,882 -> 152,834 (-2,048)
with no rule increasing anywhere and no per-file increases. No casts, no
suppressions, no behavior changes. Budgets ratcheted: basedpyright -2,048
across 4 rules, ruff ANN401 -2.
An auto-router deployment's litellm_params.model (auto_router/...) is the
discriminator the router loads it by, but the model management endpoints
accepted any client-supplied value verbatim; a doubled or stripped prefix
made router init fail on the next load and ignore_invalid_deployments
silently dropped the deployment. Validate writes that supply
litellm_params.model at all three endpoints against the merged params and
reject incoherent values with an actionable 400. Classification is
extracted to router_utils/auto_router_model_naming.py so the Router
predicates and the validation share one source
Every model-write endpoint returned 200 off the DB write alone; a model the
reload dropped (ignore_invalid_deployments, or a wholesale reload failure)
stayed invisible on every channel at once, which is how the registry-leak
defect went undiagnosed for three weeks. ProxyConfig.add_deployment and
clear_cache now return whether the reload pass completed, and each write
endpoint verifies the rows it wrote are live in this pod's router afterwards,
distinguishing a deliberately environment-inactive model via the same
predicate the Router's own gate uses. The access-group writers return the
mutated id set instead of discarding it
* fix(proxy): warm rotate Prisma client for IAM refresh
* fix(proxy): drain Prisma operations during IAM rotation
* fix(proxy): bound the drain wait when retiring a replaced prisma engine
A replaced engine waited indefinitely for its drain tracker to empty.
Hung queries self-release via prisma's 30s default HTTP timeout, but a
transaction whose owner is hard-cancelled before commit/rollback leaks
its drain count forever, keeping the retired engine and its DB
connection pool alive indefinitely; at one rotation per 12 minutes such
engines accumulate. Cap the wait at 90 seconds, which exceeds every
legitimate operation bound (30s HTTP timeout, 60s max interactive
transaction timeout in this codebase), then kill the engine anyway.
Work killed at the deadline degrades to the pre-drain behavior and is
retried by the existing reconnect/backoff layers.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Scrub aliases on delete only when the deleted deployment's model_name no
longer resolves in the router. A legacy load-balanced team model can have
several deployment rows sharing one internal name; deleting one replica
must not remove aliases that still route to the survivors, in any team
A team's model_aliases can map a public name like gpt-4 to the internal
routing key (model_name_{team_id}_{uuid}) of a team deployment that has
since been deleted, e.g. after replacing per-team duplicates with one
gateway-level model. The pre-call rewrite then sent every request to a
name the router cannot serve, failing with "no healthy deployments for
model_name_..." even though the requested name still resolves at the
gateway level. The rewrite is now skipped when the alias target has no
live deployment in the router
delete_model also skipped the team alias scan for internal-shaped names
on the assumption they can never be alias values, which is exactly the
shape legacy team model aliases have, so deleting a legacy team model
left the stale alias behind. The scan now always runs, and a public
name that still resolves to a live router deployment (e.g. a shared
gateway-level model group) stays in team.models so the delete does not
revoke the team's access to it
delete_deployment resolved the outgoing deployment through get_deployment before
popping it, and ran the strategy release before repairing the index maps. Both
halves of that ordering could leave the router inconsistent. A resolution failure
meant the entry left the model_list with its registry slots still held, so the
alias stayed routable and the name could not be reused; a failure inside the
release meant the outer handler returned None with the entry already popped and
model_id_to_deployment_index_map never repaired, breaking every later lookup and
delete until a restart.
upsert_deployment already had this right: it pops, repairs the caches and indices,
and only then releases the slot. delete_deployment now follows the same sequence
and resolves the deployment from the item it just popped rather than through a
lookup that can fail. Releasing the slot is secondary to structural integrity, so
it runs last and a failure there is logged instead of abandoning a removal that has
already happened.
* fix(ui): validate default team values in Default User Settings
The Default User Settings form accepted any free-text team id, and the
proxy persisted it without checking the team exists. New users were then
silently never added to the default team because the consume-time 404
from team_member_add was swallowed at debug level.
Backend: PATCH /update/internal_user_settings now rejects unknown and
duplicate team ids with a 400 naming them, before any persistence or
team budget side effects. Team-add failures in _add_user_to_team now log
at ERROR with user and team ids.
UI: DefaultUserSettings rewritten as a shadcn + react-hook-form + zod
form following the org-settings pattern. The team id free-text input is
replaced with a searchable server-backed team picker, so only existing
teams can be selected; zod blocks empty and duplicate rows. The shared
deriveErrorMessage helper now unwraps the HTTPException detail.error
shape so backend validation errors surface readably in toasts.
* fix(ui): restore read-only view with Edit Settings toggle on default user settings
Parity with the pre-migration form: the tab renders a read-only summary
of the saved defaults, Edit Settings opens the RHF form, Cancel discards
pending edits and returns to the summary, and a successful save returns
to the summary showing the new values. Model sentinel labels in the
summary are derived from ModelSelect's now-exported special values
instead of duplicating the strings.
* refactor(ui): rename MODEL_SELECT_SPECIAL_VALUES_ARRAY to MODEL_SENTINEL_OPTIONS
* fix(ui): move Edit Settings into the card header action slot
The finalize re-run in upsert_deployment keyed off the auto_router/adaptive_router
prefix only, so editing a complexity router with adaptive enabled released its
adaptive_routers entry (and post-call hook) without rebuilding it: complexity
routing kept serving while bandit recording, DB persistence and
/adaptive_router/state went silently dark until the next full reload. Gate the
re-run on a participation predicate that mirrors both arms of the finalize pass,
drop the import that pass no longer uses, and pin the registry helpers with
direct contract tests
Auto-router-family deployments live in two structures: the model_list, and a
pre-routing strategy registry keyed by (model_name, tags). Removing a deployment
dropped it from the model_list without releasing its registry slot, so the re-add
that follows hit the "already exists" guard in _register_pre_routing_strategy and
ignore_invalid_deployments swallowed it. The deployment came out and never went
back, while the DB row and the endpoint response both looked fine. Only a restart
healed it, and under multiple replicas each pod diverged into holding a different
subset of routers.
Removal now releases the (model_name, tags) slot from every strategy registry, in
both upsert_deployment and delete_deployment, guarded on the auto_router/ prefix so
removing a regular deployment cannot evict a router that merely shares its
model_name. Releasing from every registry rather than the first match is what makes
this correct for hybrids: registration is one-to-many, since a complexity router
configured with adaptive is also registered in adaptive_routers under the same key
by the deferred finalize pass. Releasing only the first match left that adaptive
strategy live, so a deleted or replaced alias stayed routable through it.
Adaptive post-call hooks are rebuilt whenever the adaptive registry changes, not
only at the end of set_model_list. The hook set is defined as exactly one hook per
registered adaptive router, so a released router stops recording turns instead of
holding a hook bound to a strategy nothing points at any more.
The swallowed upsert failure is logged at warning instead of debug, which is below
the default log level and left this failure with no observable signal anywhere.
delete_deployment resolves the outgoing deployment before popping it, and a
resolution failure no longer aborts the removal; previously an entry that failed
validation would have been left in the model_list permanently.
delete_model drops its blanket pop across all four registries. That predates this
change and over-evicts: it removes every tag variant registered under the name
while only one is being deleted, and nothing reloads on that path to restore the
survivors. delete_deployment now handles it correctly and tag-scoped, so the
endpoint-level eviction and its helper are removed rather than left to mask it.
Exercises the streaming field-fill heuristics, model_construct fallbacks,
and the get/cancel/delete/list request and response transforms that had no
tests.
