The create/edit forms passed several large object literals inline as arguments (persist-state
JSON.stringify, storeMCPOAuthUserCredential, setToken, transport-clear setFieldsValue), tripping
local/no-large-inline-object-arg. Assigned each to a named variable at the call site - a pure,
behavior-preserving refactor verified by the create/edit suites - which lowers the whole-tree count so
the eslint baseline is 512 rather than being raised to accommodate them.
The create form wrote the upstream token obtained by Authorize & Fetch into
form.credentials for every mode, so for true_passthrough / oauth_delegate the
browser-held token leaked into the OAuth flow's getCredentials (preview requests)
and the redirect-persist cache, and was a step away from server-level credential
persistence. onTokenReceived now early-returns for the client-forwarded modes,
holding the token only in local state for preview (mirroring the edit form),
instead of writing it into form.credentials.
The 'auth_type is true_passthrough or oauth_delegate' check was duplicated inline
across both server forms' browser-authorize temp payloads, the edit form's
onTokenReceived and tool-preview gate, PassthroughAuthorizeSection, and mcp_tools'
usesBrowserHeldToken. Extracted a single isClientForwardedTokenMode helper in
types.tsx and routed every site through it so the set of client-forwarded modes
lives in one place and cannot drift. Also replaced a pre-existing nested ternary
in the authorize button label surfaced by touching the file.
The edit form's getTemporaryPayload read the server's stored auth_type instead
of the value the admin selected in the dropdown, so an admin who switched an
existing oauth2 server to true_passthrough (or oauth_delegate) and ran the
browser authorize flow built the temporary OAuth-relay server as oauth2. That
made needs_user_oauth_token true and persisted the token to the DB, contrary to
the mode's browser-held contract, and left it inconsistent with onTokenReceived
and the submit payload, both of which already read the form value. It now reads
values.auth_type, matching the create form.
Two correctness fixes for the client-forwarded token modes.
The preemptive-401 connect gate for true_passthrough and oauth_delegate
only inspected the request-wide Authorization, so a caller who bound the
upstream token via the per-server x-mcp-{alias}-authorization header (the
mandatory shape in a multi-server aggregate, where the request-wide
Authorization is withheld) was spuriously 401'd at connect even though
egress already honors that header. The gate now recognizes the per-server
header for both modes via a shared helper, mode-correctly: true_passthrough
treats any Authorization or the per-server header as the upstream token,
oauth_delegate keeps requiring a distinct x-litellm-api-key so a lone
Authorization consumed for admission is never mistaken for an upstream
token. The preemptive raise is also gated to single-server scopes so a
multi-server aggregate degrades gracefully (the listing absorbs a
per-server failure) instead of one missing token 401-ing the whole connect.
The browser-only Authorize flow was writing the upstream access and refresh
token to LiteLLM_MCPUserCredentials, contradicting the modes' persist-nothing
contract: the temp OAuth-relay server was cached with a hardcoded oauth2
auth_type, so needs_user_oauth_token was true and the token exchange stored
it. The create and edit forms now send the real auth_type for these modes,
so the temp server is not oauth2, needs_user_oauth_token is false, and the
exchange skips storage while still returning the token to the browser
session.
The server detail page's Tool Testing Playground gated its browser-held
token handling on the legacy PKCE-passthrough shape, so a
true_passthrough or oauth_delegate server listed tools unauthenticated
and surfaced 'Failed to fetch MCP tools' with no way to authorize. The
playground now treats both modes as browser-held-token servers: it
reads the sessionStorage token established by the create/edit
browser-only Authorize, forwards it via the x-mcp-{alias}-authorization
header, evicts it on a 401, and shows its own Authorize gate when the
token is absent.
That gate's flow uses the gateway's relayed authorize/register/token
endpoints with the real server id, which previously 400ed for anything
but oauth2. Those endpoints now also accept the client-forwarded token
modes (the minted token is upstream-audienced and browser-held; DCR
persistence stays off on this path), and registry builds run the same
RFC 9728/8414 endpoint discovery for these modes that oauth2 rows get,
since their rows never store an authorization_url.
true_passthrough and oauth_delegate persist no upstream credentials, so
the create/edit forms had no way to preview tools or configure the tool
allowlist: tools/list went upstream unauthenticated and came back 401.
This reuses the existing OAuth authorize machinery in browser-only mode
for those two auth types: the admin authorizes against the upstream
(DCR/PKCE, with optional client credentials for IdPs without dynamic
registration), the token lands in sessionStorage exactly like the
legacy PKCE-passthrough path, and the tools preview forwards it via the
per-server x-mcp-{alias}-authorization header, which the passthrough
resolver arm already accepts. Nothing is written to the server row or
the per-user credential store; the create payload keeps excluding
credentials for these auth types via AUTH_TYPES_REQUIRING_CREDENTIALS.
The tools preview endpoint now also extracts the Authorization header
for the two new auth types so the browser-held token reaches the
passthrough arm during create-time previews.
Adds the two client-forwarded token modes to the MCP server create and
edit form auth dropdowns, and shows a warning when true_passthrough is
selected: the gateway performs no admission auth for that server, so
callers reach the upstream without a LiteLLM key and per-key/per-team
rate limits and spend tracking do not apply. The warning is a shared
component so the two forms cannot drift on the copy.
* refactor(ui): point invitation links at the dedicated /onboarding route
Invitation and reset-password links were built as /ui?invitation_id=..., which lands on the dashboard index and renders the onboarding form inline. They now point at the standalone /ui/onboarding route, so the index no longer has to special-case invitations. Old links keep working unchanged; the index still renders onboarding inline for ?invitation_id until the migration closeout removes that branch.
Updates the three generators (the enterprise email builder, bulk user create, and the invitation/reset-password modal) and extracts the modal's URL building into a pure, unit-tested buildOnboardingUrl
Refs LIT-3687
* refactor(ui): guard buildOnboardingUrl against a missing invitation id
Return "" instead of emitting an invitation_id=undefined link when the id is not yet available, matching the existing empty-baseUrl guard. Placed after the SSO branch so the SSO link, which does not use the id, is unaffected
Refs LIT-3687
* feat(ui): add shared table_cells kit and convert logs columns
DateCell, MoneyCell, IdCell and StatusBadge consolidate the duplicated
per-table cell implementations behind one component each. The logs page
columns are the reference conversion; the dead auditLogColumns export
(superseded by audit_logs.tsx) is removed with it
* refactor(ui): consolidate table cells onto the shared table_cells kit
106 cell sites across 44 table files converge onto DateCell, MoneyCell,
IdCell and StatusBadge, replacing 8 date formats, 6 spend formats, 7 id
truncation strategies and 6 status badge styles with one implementation
each. Badge now forwards refs so Base UI tooltip triggers composed over
it can attach (they previously never opened under React 18). TimeCell
is deleted; its two consumers now render DateCell
* fix(ui): suppress cost tooltip for zero spend and drop dead getStatusBadge param
The logs Cost tooltip showed the raw $0 over a "-" cell for zero or
null spend (pre-existing, surfaced by review); the tooltip now only
renders when there is a real amount. healthCheckColumns no longer takes
the unused getStatusBadge callback and its dead definition is removed
* fix(ui): restyle StatusBadge as tinted pill matching the prior antd Tag look
* fix(ui): keep StatusBadge fully rounded like the other kit pills
Staging recently added the local eslint rules no-large-inline-object-arg and
no-long-condition-chain and tightened no-nested-ternary to an error. After
merging staging, the license-banner code tripped them: the banner's tiered
description was a nested ternary (now an error), and two option objects were
passed inline (adding budget debt). Extract the description into an
early-return helper, and hoist the useQuery options and the date-format options
into named constants. No behavior change; keeps the inline-object-arg count at
the committed baseline rather than bumping it
The layout test renders DashboardShell without a QueryClientProvider and mocks
DebugWarningBanner to null for exactly that reason. The new LicenseExpiryBanner
also uses a React Query hook, so it needs the same treatment; without it the
test threw "No QueryClient set". Runtime is unaffected: the app mounts a
QueryClientProvider above the layout (DebugWarningBanner already relies on it)
* feat(ui): add eslint rules for nested ternaries, large inline object args, and long condition chains
Adds three dashboard lint rules to keep new code readable. Nested ternaries
are banned outright via the built-in no-nested-ternary, with the 265 existing
occurrences grandfathered in eslint-suppressions.json so only new ones fail.
Two custom rules ship as a small local plugin under scripts/eslint-rules:
no-large-inline-object-arg flags object literals with 4+ properties passed
straight into a call, nudging toward a named variable, and no-long-condition-chain
flags boolean expressions that combine 4+ conditions, nudging toward a named
boolean. Both are warnings tracked on the existing budget ratchet
(eslint-budgets.json + eslint-metrics.json) with headroom above the current
counts, so they ratchet down over time rather than freezing a baseline. Both
thresholds are configurable rule options and covered by RuleTester unit tests.
* fix(ui): scope no-long-condition-chain to boolean operators, not nullish
Greptile flagged that the rule counted nullish-coalescing chains the same as
&&/|| chains, so a 4-part `a ?? b ?? c ?? d` fallback surfaced "Boolean
expression combines 4 conditions", which is inaccurate since a `??` fallback
is value defaulting, not a condition. Restrict the visitor to && / || nodes so
`??` chains are treated as leaves, while a boolean chain nested inside a `??`
is still caught. Drops 6 miscounted occurrences (240 -> 234).
* chore(ui): sync lint metrics and suppressions with staging
Merge advanced the base branch, adding one no-large-inline-object-arg
occurrence (508 -> 509) and making one grandfathered react-hooks suppression
stale. Regenerate eslint-metrics.json and prune the suppression so the
budget/drift gate passes.
* chore(ui): sync lint metrics with staging
Merge advanced the base, adding four no-large-inline-object-arg occurrences
(509 -> 513). Regenerate eslint-metrics.json so the drift gate passes.
UsageIndicator was fetching /health/license through its own useEffect while
the new expiry banner fetches the same endpoint via useLicenseInfo, so an admin
with the usage widget open made two identical calls per page load. Point
UsageIndicator at useLicenseInfo too; both callers now share one React Query
cache entry, collapsing it back to a single request. The null/error semantics
are preserved (data ?? null matches the previous catch-to-null), and license
errors never fed the widget's error state before either
* test(ui): characterize DataTable behavior before shadcn reskin
Pins the shared view_logs DataTable contract with library-agnostic
queries ahead of the tremor-to-shadcn table migration: loading and
empty states, TanStack column defs with custom cell renderers,
onRowClick payload, both expansion render paths (colspan sub-component
and sibling child rows), the getRowCanExpand gate, and client-side
sorting on and off. These must pass unchanged after the reskin.
* refactor(ui): reskin shared DataTable from tremor onto shadcn table primitives
Swaps the view_logs DataTable's presentational layer from @tremor/react
to the in-repo components/ui/table primitives and hardens the seam that
every later table migration copies:
- getRowId is injected instead of hardcoded to request_id through an
any cast; identity defaults to the row index and the logs page now
passes request_id explicitly, keeping expansion state attached to the
right row across refetch reorders
- one expansion render path: renderChildRows had zero consumers and is
removed; renderSubComponent (colspan cell) is the single path
- the four consumers passing dead no-op renderSubComponent and
getRowCanExpand boilerplate drop it
- loading and empty defaults become generic (Loading... / No results)
instead of log-specific
The characterization tests from the previous commit pass unchanged
except the dead child-rows path test, replaced by a reorder-stability
test for injected getRowId plus coverage of the new generic defaults.
First tremor removal of the tables track; view_logs/table.tsx no longer
imports @tremor/react.
* test(ui): assert child rows hidden before expansion in DataTable test
* fix(ui): suppress row hover on DataTable placeholder rows
* feat(ui): polish DataTable with skeleton loading, header band, and numeric column alignment
* feat(ui): shape DataTable skeletons per column and keep stale rows during refetch
* revert(ui): drop DataTable skeleton loading, restore text loading row
* fix(ui): clip DataTable to its rounded wrapper and right-align Duration/TTFT values
Surfaces a persistent, tiered banner under the dashboard navbar when an airgapped enterprise license is close to expiring: an amber, session-dismissible warning within 30 days, a non-dismissible red alert within 7 days, and a non-dismissible red banner once the date has passed. It reads the existing /health/license endpoint, so no backend change is needed, and is driven strictly by expiration_date; community and remote-validated instances that report no date show nothing. Shared day-count math is extracted to licenseUtils so the banner and the existing UsageIndicator widget stay in sync
Add a tag_rpm_limit field to virtual keys so each request tag gets its own independent RPM counter on the v3 rate limiter. A key configured with per-tag limits tracks each tag/group separately, and requests whose tag has no configured limit fall back to the key-level limit. Includes the dashboard UI to manage per-tag limits on key create and edit.
Resolves LIT-3147
* feat(ui): expose MCP max_concurrent_requests in server create and edit forms
The proxy has enforced a per-server outbound tool-call concurrency cap
(max_concurrent_requests) across every MCP egress path since #31641, and the
management API has accepted the field on create and update all along, but the
dashboard offered no way to set it. Add an optional Max Concurrent Requests
input to the MCP server create and edit forms; it applies to every auth type
and transport, so it renders unconditionally rather than gated on auth mode.
Clearing the field on edit sends null so the stored limit is unset.
Also rebuild the per-server semaphore when the configured limit changes.
Previously the semaphore was created once per server_id and never resized, so
an edited limit only took effect after a proxy restart even though the new
value was persisted and reloaded into the registry.
* feat(ui): mark MCP max concurrent requests field label as optional
* test(ui): stop OBO create-form tests from timing out on CI
The token-exchange payload test and the Entra scope-required test filled five
text fields with user.type, which dispatches a full keystroke sequence per
character; every input event runs the antd form onValuesChange handler and
re-renders the whole CreateMCPServer tree, roughly 120 renders per test. As
the form grew the two tests reached 8s and 18s locally, which crosses the 30s
vitest timeout on slower CI containers; ui_unit_tests failed twice this way.
Switch the plain text fields to fireEvent.change (one input event per field),
matching the existing stdio test pattern. Both tests assert form output, not
keystroke behavior, and now run in about 3s each.
* fix(ui): scope key models dropdown options to the key's team
A teamless key no longer offers the all-team-models option in the create and
edit forms; the backend expands that sentinel to the full proxy model list when
no team is attached, which is rarely what the user intended. A team key no
longer surfaces the all-proxy-models sentinel that leaks in verbatim when the
team's own model list carries it; the dropdown keeps All Team Models plus the
team's individual models.
Adds browser coverage to the management e2e suite: playwright (an optional
dependency behind importorskip) drives the proxy-served dashboard at /ui,
asserts the dropdown options a real user sees for teamless and team keys on
both create and edit, and walks the create modal end to end, reading the
persisted key back through /key/info.
* fix(ui): offer all-proxy-models on teamless keys in the models dropdown
A teamless key has no team allowlist to inherit, so the dropdown now offers All
Proxy Models in place of All Team Models on both the create and edit forms, with
the same exclusive-selection handling. Component and browser e2e tests updated to
pin the swapped option pair; the teamless create case now also walks the modal end
to end and reads the persisted key back through /key/info.
* test(ui): update no-team key creation spec to pick All Proxy Models
The create modal no longer offers All Team Models without a team; the teamless
path now offers All Proxy Models, which is what this spec exercises
* fix(ui): gate All Team Models on the team object being loaded
When a key has a team_id but the teams prop does not yet include the matching team, availableModels stays empty and the models dropdown rendered All Team Models on its own with nothing to compare against. Gate the option on the team object being present so it only appears once team models are known, and add a regression test for the loading state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): filter all-proxy-models from teamless model fetch in key edit form
The teamless fetch path stored modelAvailableCall results without excludeProxyWideSentinel, so an all-proxy-models entry in the response rendered a second option colliding with the hardcoded All Proxy Models sentinel. Apply the same filter used on the team path and add a regression test asserting the sentinel option is not duplicated
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Introduce two first-class MCP server auth_type values that make LiteLLM's
role in upstream authentication explicit, added alongside the existing
delegate_auth_to_upstream / oauth_passthrough flags without changing their
behavior.
true_passthrough is a transparent proxy: LiteLLM performs no admission auth,
requires no x-litellm-api-key, mints/stores/refreshes nothing, and forwards the
client's Authorization to the upstream exactly as received. oauth_delegate keeps
normal LiteLLM admission (x-litellm-api-key / SSO / JWT) and then forwards the
client's separate upstream Authorization unchanged; the admission credential is
never forwarded upstream.
Both modes forward the caller's token via the existing extra_headers path and
defer egress credential resolution to v1 (the v2 to_server_spec returns None for
them). Upstream 401/403 responses are surfaced rather than swallowed so upstream
OAuth challenges are preserved. Servers in either mode require per-user auth, so
userless health checks are skipped.
* feat(mcp): let users select the entra_obo token_exchange profile in the UI and API
The backend token_exchange arm supports two wire dialects via token_exchange_profile
("rfc8693" default, or "entra_obo" for Microsoft Entra's On-Behalf-Of, the RFC 7523
jwt-bearer grant), but it could only be set through config.yaml. This surfaces it to the
create/update REST API and the dashboard so an admin can create an entra_obo server there,
completing the parity started in the parent PR for the other token-exchange fields.
token_exchange_profile becomes a dedicated column on LiteLLM_MCPServerTable, mirroring the
sibling fields: it is added to the request models, read column-first in
build_mcp_server_from_table with the credentials-blob as a back-compat fallback and a
default of rfc8693, and carried through both runtime-to-table builders so registry
round-trips preserve it. It is a non-secret dialect selector, so it is not scrubbed from
non-admin or virtual-key responses.
In the dashboard a Profile dropdown (RFC 8693 vs Microsoft Entra OBO) is added to the
token-exchange section. Entra OBO carries the target resource in the scope, so selecting it
makes the scope required and hints the api://<app-id>/.default form, while audience and
subject_token_type (which that dialect ignores) are hidden.
* fix(mcp): extend the blob-to-column lift and non-admin scrubbing to token_exchange_profile
token_exchange_profile gets the same storage contract as the other three
token-exchange settings: the column is authoritative, a blob copy is the legacy
shape — lifted into the column on every write and stripped from the stored
blob — and switching auth_type away from token exchange clears it
(_AUTH_FLOW_SCOPED_FIELDS). Both restricted-view sanitizers scrub it for
uniformity, and the edit form's auth-switch payload nulling includes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(mcp): assert every token-exchange setting is configurable via config.yaml
Pins the config surface: token_exchange_endpoint, audience, subject_token_type
and token_exchange_profile load from top-level config keys onto the built
server and through to the resolver spec; omitted keys resolve to their
documented defaults (RFC 8693 subject token type, rfc8693 profile), and
token_exchange servers need no oauth2_flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test(ui): pin that the token-exchange fields render only for the oauth2_token_exchange auth type
No form section asserted the visibility contract: the token-exchange fields
(Token Exchange Endpoint, Audience, Subject Token Type) must appear when
'OAuth Token Exchange (OBO)' is selected and for no other auth type. Assert
hidden under plain OAuth, shown under token exchange, hidden again after
switching to API Key.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(ui): assert the stdio transport switch unmounts the token-exchange fields
The create form gates the whole Authentication section on non-stdio transport,
so selecting OAuth Token Exchange (OBO) and then switching to stdio removes the
token-exchange fields (and their required-credential rules, which antd does not
validate while unmounted). Pin that sequence so the section-level gate cannot
regress silently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard
OAuth 2.0 Token Exchange (RFC 8693, a.k.a. OBO) for MCP servers could previously only be
configured through config.yaml; the create/update REST API and the dashboard had no way to
express it. This wires token_exchange_endpoint, audience, and subject_token_type end to end.
These three are persisted as dedicated columns on LiteLLM_MCPServerTable, mirroring how token_url
and oauth2_flow are stored, so the edit form prefills them and they are unaffected by the
credentials-blob clearing on auth_type change. build_mcp_server_from_table reads the columns first
and falls back to the credentials blob so servers persisted before the columns existed still load.
client_id and client_secret continue to ride the existing encrypted credentials path.
On the dashboard, "OAuth Token Exchange (OBO)" is a distinct auth-type option with its own field
section. The McpOAuthMode classifier gains a token_exchange arm keyed off auth_type; the previous
catch-all oauth2 mode was renamed from "obo" to "authorization_code" so the two on-behalf-of
mechanisms are no longer conflated. The token-exchange IdP endpoint and audience are scrubbed from
non-admin and virtual-key responses, matching how token_url is treated.
* fix(ui): gate the MCP list-401 re-auth on authorization_code, not token_exchange
The renamed 401 gate keyed off isTokenExchange, but that condition exists for authorization_code:
when a stored per-user credential is present yet the tools/list 401s (the backend's refresh could
not mint a token), the user must re-authorize via the browser flow. token_exchange has no
gateway-side authorize step, so the Authorize gate never applied to it. The prior isObo flag was
undefined (a compile error) and, per this file's convention and its tests, meant authorization_code;
renaming it to isTokenExchange changed the behavior and broke the mcp_tools auth-gate test for an
authorization_code server whose token expired with no usable refresh. Gate on isAuthorizationCode
instead and drop the now-unused isTokenExchange
* fix(mcp): clear flow-scoped endpoint config when a server's auth_type changes
Switching an existing oauth2 server to oauth2_token_exchange left the old flow's
token_url on the row. The OBO resolver treats token_exchange_endpoint or token_url
as the configured exchange endpoint, so the stale value both suppressed the RFC
9728/8414 discovery this PR adds and sent the exchange grant (client credentials
plus the user's subject token) to the previous flow's token endpoint
update_mcp_server now mirrors its existing stale-credentials rule for the
flow-scoped columns (authorization_url, token_url, registration_url, oauth2_flow,
token_exchange_endpoint, audience, subject_token_type): when auth_type changes,
each one is cleared unless the same request explicitly provides it, so a
deliberate override in the switch request still wins. Updates that keep the
auth_type never touch these columns, which keeps legacy OBO rows that use
token_url as their exchange endpoint working
The edit form sends explicit nulls for the previous flow's fields on an auth type
switch; antd preserves unmounted field values by default, so without this the old
token_url would be re-sent verbatim and read as an explicit override. Transitions
are detected against the persisted auth_type, so saves that keep the auth type
send nothing extra
Reported by Cursor Bugbot on the PR
* fix(mcp): lift legacy blob token-exchange settings into their columns on every write
The three token-exchange settings live in dedicated columns but also exist on
MCPCredentials as the pre-column REST shape. Writes now lift incoming blob
values into the columns (an explicit top-level value wins, including an
explicit null) and strip them from the stored blob; the same-auth credentials
merge migrates legacy rows the same way. The read-time column-or-blob fallback
then only ever serves rows current code has never written, so clearing a column
to re-enable RFC 9728/8414 discovery can no longer be silently undone by a
stale blob copy.
Also asserts the auth-switch clearing fires on the external fields_set path
(PUT /v1/mcp/server).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(mcp): single source for the RFC 8693 default subject_token_type
The default was applied at four egress build sites plus two model defaults,
each with its own copy of the literal. All sites now share
DEFAULT_SUBJECT_TOKEN_TYPE from litellm.types.mcp. A DB-level DEFAULT is
deliberately not used: Prisma writes explicit values on insert, so a column
default would rarely apply, and NULL-means-RFC-default keeps existing rows
correct.
Also documents two review decisions in place: the audience column keeps the
RFC 8693 parameter name (RFC 8707 resource indicators are already a separate
concept named resource in the v2 egress types), and the migration's
out-of-order timestamp is safe under prisma migrate deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: fix import sort order in outbound_credentials/types.py (I001 strict budget)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): purge legacy blob copies when a token-exchange column is written without credentials
The migrate-on-write in the credentials merge lifts blob values into null
columns, which is correct for legacy rows but could repopulate a column an
admin had cleared in an earlier no-credentials update (that path never touched
the blob, so the stale copy survived to be lifted later). An explicit
token-exchange column write (set or clear) now migrates the row even when the
update carries no credentials: untouched null columns are lifted, every blob
copy is stripped, and unrelated blob keys stay as-is. A cleared column can then
never be resurrected, because no write path leaves a blob copy behind.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(mcp): state the blob-to-column lift contract on the legacy credential keys
The three token-exchange keys on MCPCredentials are the pre-column REST shape
(the only REST shape from 2026-05 until this PR). Document on both the blob
type and the request models that the dedicated columns are authoritative and
that writes lift blob values into them and strip the stored copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): scrub subject_token_type in the non-admin and virtual-key sanitizers
The other two token-exchange fields were cleared while subject_token_type was
left visible. It is a public RFC 8693 URN with no disclosure value, but the
sanitizers' rule is that these views receive no token-exchange config at all —
cleared for uniformity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test(ui): characterize DataTable behavior before shadcn reskin
Pins the shared view_logs DataTable contract with library-agnostic
queries ahead of the tremor-to-shadcn table migration: loading and
empty states, TanStack column defs with custom cell renderers,
onRowClick payload, both expansion render paths (colspan sub-component
and sibling child rows), the getRowCanExpand gate, and client-side
sorting on and off. These must pass unchanged after the reskin.
* test(ui): assert child rows hidden before expansion in DataTable test
* feat(ui): OAuth flow selector on the MCP edit page
The edit form had no flow selector: oauth_flow_type was watched but never registered,
so isM2MFlow was always false in edit mode and the flow could only be changed over
REST. That left the backfill's remediation for ambiguous legacy rows (client creds +
token_url, no interactive signal, left unstamped) without a dashboard path
The oauth2 section now opens with an OAuth Flow Type select. Explicit rows prefill
their stored value and re-persist it on save; legacy null rows show a placeholder
instead of a fake preselection, and an untouched save still writes nothing, so the
form never guesses on the admin's behalf. Choosing Machine-to-Machine (M2M) persists
oauth2_flow=client_credentials, choosing Interactive (PKCE) persists
authorization_code, which is exactly the assertion the backfill warning asks for.
Registering the field also brings the existing isM2MFlow gating in the edit form to
life, so M2M rows stop showing the interactive-only token-validation fields
Tests cover the prefill round-trip for both explicit values, the untouched null row
writing nothing, and both selections persisting on a legacy null-flow row
* fix(mcp): registry-to-table conversions must carry oauth2_flow
_build_mcp_server_table and the health-check table builder dropped oauth2_flow when
converting registry servers for GET /v1/mcp/server (list and by-id), so the dashboard
never received the persisted flow: the edit page could not prefill the selector, M2M
gating never activated, and the tools page classifier saw every oauth2 server as
interactive regardless of the column. Found live while proving the edit-selector
persistence path end to end; the write side was fine (PUT persists and the column
reads back correctly), the read side was dropping the field at the conversion
Both builders now carry oauth2_flow; regression test pins the conversion
* docs(mcp): flag _resolve_oauth2_flow as security-sensitive in its docstring
The prior wording ('not called directly by security sites') could read as if the
function has no security relevance, when it is the shape-inference engine both
request-time security helpers delegate to. Reword to state that plainly: it decides
M2M-vs-interactive for an unstamped row, must always be reached through
effective_oauth2_flow or resolve_oauth2_flow_for_request, and its M2M-shape branch
must not be weakened without accounting for those callers. Docstring-only; no logic
change
Raised by review on the stacked PR
* refactor(ui): extract oauth2FlowToFormValue helper for the MCP OAuth flow prefill
The edit form derived the OAuth Flow Type select value from the stored oauth2_flow
with a nested ternary duplicated at two call sites. Extract the mapping into a named
helper in types.tsx (next to getMcpOAuthMode and the flow constants): client_credentials
-> M2M, authorization_code -> Interactive, null/unset -> undefined so the select shows
its placeholder instead of a guessed default. The tool-config call site keeps its
null -> Interactive display fallback via a trailing ?? OAUTH_FLOW.INTERACTIVE, so
behavior is unchanged. Adds unit tests for the helper; the existing prefill/save tests
already cover the call sites
* feat(ui): surface and warn on an unset MCP oauth2_flow (server card + edit page)
An oauth2 MCP server whose oauth2_flow was never classified (legacy null row the
backfill left ambiguous) now advertises that it needs attention instead of silently
falling back. The server card shows an 'OAuth flow not set' warning tag for any
auth_type=oauth2 server with no oauth2_flow, so admins can spot them in the list
without opening each one. The edit page shows a warning alert directly under the new
OAuth Flow Type selector while the flow is unset, and it clears the moment a flow is
picked.
Delegate (delegate_auth_to_upstream) servers are excluded from both: they authenticate
via upstream PKCE passthrough and route to passthrough regardless of oauth2_flow, so
the M2M-vs-interactive classification does not apply and prompting for it would be a
false alarm. The edit page reads the delegate state from the watched switch when it is
mounted and falls back to the stored value otherwise (useWatch returns undefined for an
unmounted field).
Also adds end-to-end coverage of the null-flow chain the selector depends on:
build_mcp_server_from_table carries oauth2_flow=None verbatim into the GET response,
so the dashboard maps it to undefined and shows the placeholder rather than a guessed
default. Tests: backend null carry, the select prefill display for all three states,
the edit-page warning show/hide/clear-on-select and delegate exclusion, and the card
badge across oauth2/non-oauth2, stamped/unstamped, and delegate
* refactor(ui): switch shadcn primitives from Radix to Base UI
shadcn made Base UI the default primitive library in July 2026 and our
only shadcn component so far is the Button canary, so this is the last
cheap moment to switch before the primitives phase adds the full set.
components.json style moves from new-york (a legacy alias that resolves
to the Radix variant) to base-vega. Button is regenerated from the
base-vega registry with the same local adaptations as before: cva beta
object form via lib/cva.config and a React 18 forwardRef wrapper. The
polymorphic asChild prop becomes Base UI's render prop.
radix-ui is replaced by @base-ui/react 1.6.0. Base UI optionally peers
on date-fns 4 while tremor pins 3, so date-fns is bumped to 4.4.0 with
an npm override; our only usage (add) is API-identical and the override
can go away when tremor does.
* refactor(ui): convert chat UI shadcn components from Radix to Base UI
The chat UI migration landed 11 components/ui files generated against
the old Radix registry config after this branch cut over to Base UI,
which would have left them importing a deleted package. All 11 (dialog,
alert-dialog, select, popover, tooltip, tabs, switch, scroll-area,
collapsible, separator, label) are regenerated from the base-vega
registry, with the repo conventions re-applied where relevant (cva beta
object form from lib/cva.config in tabs; the Button canary keeps its
React 18 forwardRef adaptation).
Chat feature call sites move from the Radix asChild pattern to Base
UI's render prop, and TooltipProvider delayDuration becomes delay.
* fix(ui): restore security override pins clobbered by the date-fns override
The date-fns 4 override was written by replacing the whole overrides
object, dropping the ten security pins (prismjs, js-yaml, glob,
minimatch, lodash, ws, braces, axios, postcss, esbuild) that keep
patched versions in the lockfile; osv-scan caught the vulnerable
versions resurfacing. Restores the pins alongside date-fns and
regenerates the lockfile.
* test(ui): pin tremor DateRangePicker behavior on date-fns 4
The date-fns 4 override forces react-day-picker 8 (authored against v3)
onto v4 at runtime, which a build or lint pass cannot validate. This
renders the shared UsageDatePicker wrapper, opens the calendar, checks
the month grid, and selects a day, so a date-fns API break in the
tremor date path fails tests instead of throwing in production. Delete
alongside the override when tremor is removed.
* fix(ui): close the alert dialog when AlertDialogAction is clicked
The base-vega registry template renders AlertDialogAction as a plain
Button with no Close binding, so confirm buttons fired their onClick
but left the dialog open; both consumers (conversation delete,
MCP credential revoke) were written against the Radix semantics where
Action dismisses on click. Binds Action to AlertDialogPrimitive.Close
via the render prop, mirroring AlertDialogCancel, and pins the
behavior with a test so a future shadcn add --overwrite cannot
silently reintroduce the template's non-closing Action.
* fix(spend): bound the logs-tab pagination count to stop full-window scans
The spend-logs UI list endpoint (/spend/logs/ui, /spend/logs/v2) computed an
exact pagination total over the whole selected time window on every load. That
was a standalone SELECT COUNT(*) FROM (SELECT request_id FROM LiteLLM_SpendLogs
WHERE ...) which, on a multi-TB SpendLogs table, scans a huge number of rows and
spikes Aurora ACU. The later LIT-4027 change folded it into COUNT(*) OVER (), but
a window count still drains every matching row before the LIMIT applies, so the
full-window scan remained.
Compute the total with a bounded SELECT COUNT(*) FROM (SELECT 1 ... LIMIT $cap+1)
that probes at most cap+1 rows, and drop the window count from the page query so
the page query is a plain indexed ORDER BY ... LIMIT. When more than the cap
match, report the cap and set total_is_capped so the UI renders "<cap>+". The
bounded subquery terminates early rather than aggregating across all tablets, so
it stays safe on sharded engines like YugabyteDB too.
Resolves LIT-4119
* test(spend): make zero-total mock reflect real COUNT(*), add capped-total tooltip
Address Greptile review on #31825:
- the empty-result test now returns [{"total_count": 0}] for the bounded
count query (real COUNT(*) always returns one row) instead of [], so the
zero-total path exercises the normal branch rather than the defensive guard
- the logs toolbar shows a tooltip explaining the cap when total_is_capped is
set, so a disabled Next button at the cap boundary reads as intentional
Add an opt-in mode so a key that exceeds its own max_budget is throttled to a
globally configured percentage of its TPM/RPM instead of being blocked entirely.
A new litellm_settings global, budget_exceeded_throttle_percentage, sets the
fraction (e.g. 0.1 = 10%). A per-key throttle_on_budget_exceeded flag (stored in
key metadata via the existing management-endpoint metadata routing) opts the key
in. When both are set and the key is over budget, the budget check records the
percentage on a request-scoped budget_throttle_pct instead of raising, and the
rate limiter scales the key's configured TPM/RPM by it. Keys without the flag
keep hard-blocking; team/user/org budgets are unaffected.
The throttle is recomputed from the key's original limits on every request and
the decision is cleared before the auth object is cached, so it never compounds
across requests. Both the budget read-time check and the budget reservation path
honor the opt-in, and both the v3 and legacy rate limiters apply the scaling.
Enabling throttle_on_budget_exceeded is proxy-admin only. It converts an
admin-imposed hard budget block into a soft throttle that keeps spending past
max_budget, so a non-admin must not be able to self-opt-in and bypass their own
spend cap. Both /key/generate and /key/update reject a non-admin setting it to
true (update only gates the transition to enabled, so a non-admin can still edit
other fields and turn the flag off). This matches the feature being wholly
proxy-admin operated: the global percentage is admin-only too.
A key that opts in but has no TPM or RPM limit has nothing to scale, so it stays
hard-blocked rather than serving unlimited requests past its budget (fail-safe).
The global budget_exceeded_throttle_percentage is configurable from the admin UI
(Settings -> General Settings), persisted through litellm_settings so it survives
a restart, not only from config.yaml.
Resolves LIT-3894. Scope for LIT-3893.
* fix(mcp): apply semantic filter to expanded litellm_proxy tools and show filtered-out count
* fix(mcp): guard expansion-path filtering on enabled flag and isolate metadata emission
* fix(mcp): resolve tool name prefix via known server prefixes, not string match
When an MCP server's alias differs from its server_name, tool names are
listed with the alias prefix but _execute_tool_calls compared that prefix
against the server_name stored in tool_server_map. The mismatch silently
skipped prefix stripping, forwarding the fully-prefixed tool name upstream
and causing "Unknown tool" failures. Resolve the actual MCPServer object
and strip using its known prefix forms (alias, server_name, server_id)
instead.
* fix(mcp): preserve tool overrides and scope REST tool listing
Return saved tool display/description overrides from the server table API
so the edit UI reloads them, resolve display names before prefix stripping
on tool calls, and honor mcp_server_name and toolset_name filters on the
REST tools list endpoint.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): inject BYOK credentials on Playground OpenAPI tool calls
Playground and Responses API route MCP execution through call_tool, which
skipped BYOK lookup and never set the OpenAPI auth ContextVar, so upstream
calls went out unauthenticated despite a stored user credential.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(mcp): cover alias-mismatch prefix stripping and display-name reverse mapping
Regression tests for _execute_tool_calls: an MCP server whose alias differs
from its server_name must still have its tool-name prefix stripped correctly,
and a tool called by its configured display name must resolve back to the
original tool name before dispatch.
* fix(mcp): validate tool display names against Bedrock's tool-name pattern
A display name replaces the tool name sent to the LLM provider, so a value
with spaces or other special characters saves successfully but fails every
subsequent Bedrock tool call. Validate tool_name_to_display_name server-side
(create/update payload) against Bedrock's [a-zA-Z0-9_-]+ constraint, and add
matching inline validation plus a save-blocking guard in the Admin UI's
create and edit MCP server forms.
* style(mcp): fix ruff/prettier formatting on CI
No logic changes; satisfies the format checks flagged on PR #32320.
* fix(mcp): fix CI failures on PR - complexity budget and stale test mock
Extract toolset-scope resolution and query-param normalization out of
list_tool_rest_api into helpers to bring it back under the C901 complexity
budget (was 18, now within the 15 threshold).
Add the missing get_mcp_server_by_name stub to the streaming iterator test's
mock manager; the alias-fallback resolution added for tool-name-prefix
stripping calls it unconditionally when _get_mcp_server_from_tool_name misses.
* test(mcp): cover BYOK OpenAPI auth-header helpers to close codecov patch gap
_format_byok_openapi_auth_header, _openapi_forwarded_extra_headers, and
_resolve_byok_mcp_auth_header were only exercised indirectly via a mocked
call_tool test, leaving their branches (auth-type formatting, header
forwarding/stripping, missing-credential 401) uncovered.
* fix(mcp): resolve BYOK auth before queuing the during-hook task
_resolve_byok_mcp_auth_header can raise a 401 when no credential is stored.
Resolving it after during_hook_task was already queued meant a hook's
side effects (audit logging, rate-limit bookkeeping) could run and record
success for a tool call that then fails on the missing credential.
* fix: correct mcp alias routing regressions
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): wire general_settings SSRF allowlist to litellm globals
general_settings.user_url_allowed_hosts was documented in SSRF errors but
never applied at startup, so internal MCP/OpenAPI URLs stayed blocked.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): regenerate dashboard types and satisfy ruff UP006 budget
Use list[str] in ConfigGeneralSettings and run gen:api so schema.d.ts
matches the new SSRF general_settings fields.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: normalize ssrf general settings
* fix: clear ssrf allowlists from null settings
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The x-litellm-semantic-filter-tools response header was sliced mid-name at
MAX_MCP_SEMANTIC_FILTER_TOOLS_HEADER_LENGTH with a trailing "...", so the
admin UI test panel rendered the last selected tool name chopped. Truncate
the CSV at a tool name boundary instead so the header only ever carries
complete names, and note in the test panel how many selected tools did not
fit in the header
* feat(mcp): persist oauth2_flow explicitly on create instead of inferring it at read time
The UI create payload never carried oauth2_flow, so every UI-created oauth2 server
persisted a null flow and relied on _resolve_oauth2_flow's field-shape inference at
registry build. That inference cannot tell a DCR-registered interactive server
(client creds + token_url, no persisted authorization_url) from an M2M server unless
endpoint discovery succeeds first, and the dashboard cannot reproduce it at all
because credentials are redacted in responses
The create form now persists the selected flow for oauth2 servers: authorization_code
for Interactive (PKCE), client_credentials for M2M. The REST create endpoints stamp an
omitted oauth2_flow server-side with the same discriminator the legacy inference uses,
run at write time where the payload carries plaintext credentials, so the decision is
made once with full information and stored. Applied to the admin create, the BYOM
submission, and the temporary session-server endpoints
The edit form derives its flow display from oauth2_flow instead of token_url presence
(token_url is present on authorization_code servers too, so it cannot distinguish M2M)
and deliberately never writes oauth2_flow: it has no flow selector, so a write from
edit could only erase an explicit value, including the authorization_code stamp the
DCR flow persists. Regression tests pin all of this down
Second step of persisting oauth2_flow at every write site so the legacy inference can
eventually be deleted; the backfill for existing null rows lands next
* refactor(mcp): name the create-time flow stamp for its fallback-only contract
stamp_omitted_oauth2_flow with a dedicated explicit-value early return and the shape
check renamed to has_m2m_shape, so the precedence (caller's oauth2_flow always wins,
inference only fills an omitted field) reads directly off the code
* feat(ui): flag experimental dashboard pages on the draft deprecation list
Add a subtle, dismissible info banner to each dashboard surface named in
the draft deprecation discussion (Workflows, Memory, Prompt Management, the
old Usage page, the API Reference tab, the Playground Agent Builder tab, and
MCP Network Settings). The banner links to discussion #32090 and states the
list is a draft and not final.
* Update ui/litellm-dashboard/src/components/DeprecationBanner.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update ui/litellm-dashboard/src/components/DeprecationBanner.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(ui): use next/link and drop trailing blank line in DeprecationBanner
Switch the discussion link from a raw <a> to next/link's <Link>, and wire the
DEPRECATION_TARGET_DATE constant into the copy so it is no longer unused. Also
removes the trailing blank line that was failing the frontend prettier check.
* fix(ui): render DeprecationBanner intro as one string to preserve spacing
Interpolating featureName and the target date directly in JSX let prettier wrap
an expression onto its own line, which drops the adjacent space in the rendered
output. Build the intro as a single template literal so spacing is stable.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(ui): add cost optimization feedback banner to models page
Surfaces a dismissible banner on Models + Endpoints prompting users to
share cost optimization feedback (routing, budgets, etc) via a GitHub
discussion.
* test(ui): add regression test for cost optimization feedback banner
* test(ui): update Models+Endpoints banner tests for cost optimization banner
Missing Provider banner tests are replaced since that banner was removed
in favor of the new always-on cost optimization feedback banner.
* feat(router): add separate ITPM/OTPM deployment rate limits
Support input/output tokens per minute on deployments via enforce_model_rate_limits, with reservation, reconciliation, refund on failure, and rate-limit headers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(router): keep ITPM/OTPM diff minimal in router.py
Drop unrelated Black reformatting from router.py and types/router.py so the PR only contains functional ITPM/OTPM changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): make ITPM/OTPM limits separate and atomic
Address Greptile review on separate ITPM/OTPM deployment rate limits.
- OTPM is now reserved atomically pre-call with rollback, matching the ITPM
path, so concurrent requests can no longer overshoot the configured output
limit before reconciliation
- ITPM counts input tokens only; it no longer accumulates completion tokens,
so the input-token limit and x-ratelimit-limit-input-tokens header describe
input usage as their names imply
- _read_reservation_from_kwargs only falls back to litellm_params.metadata when
the top-level metadata channel is absent, so production requests carrying a
litellm_params.metadata dict still reconcile and refund their reservation
Adds regression tests for OTPM atomicity under concurrency, input-only ITPM
enforcement, and reservation lookup when litellm_params.metadata is present.
* fix(router): subtract input tokens only from remaining-input-tokens header
The in-flight replay for x-ratelimit-remaining-input-tokens subtracted total
tokens (input + output) instead of input tokens only, so clients saw remaining
input quota understated by the completion token count on every response. Now
consistent with the input-only ITPM counter.
* fix(router): make itpm/otpm vs tpm/rpm precedence explicit
When a deployment configures itpm/otpm alongside tpm/rpm, the io-token path
takes over and the tpm/rpm limits are not enforced. Log a warning the first
time such a conflicting deployment is seen so the supersession is not silent,
and document the mutual exclusivity.
Post-call reconciliation now only trues up a counter that was actually
reserved against, so the itpm/otpm keys are no longer incremented for
deployments that never configured that limit.
* fix(router): track actual io-token usage on the reservation-minute key
Post-call reconciliation now keys off the exact cache key stashed at pre-call
time rather than one recomputed from the response-time minute. This fixes two
issues: a request whose pre-call estimate was 0 now still writes its actual
billable input to the ITPM counter (previously it was skipped, leaving the
limit unenforceable for that request), and a call that finishes in a later
minute reconciles against the minute it reserved against instead of pushing a
negative delta into the next minute. Counters are only touched when their
limit is configured.
* fix(router): run io-token reconciliation before the model_id guard
async_log_success_event gated IO reconciliation behind the model_id guard that
only the TPM tracking path needs. Since reconciliation works entirely from the
cache keys stashed in kwargs, a success event whose standard_logging_object
lacks model_id would skip reconciliation and leave the reservation on the
counter until the TTL expired, wasting quota. Route the IO path first.
* fix(router): don't replay in-flight delta for itpm/otpm headers
For ITPM/OTPM model groups the counter is incremented at reservation time
(pre-call), so the remaining values returned by get_remaining_model_group_usage
already account for the current request. Replaying the in-flight delta on top
double-counted it and understated x-ratelimit-remaining-input/output-tokens by
up to max_tokens on every response. Skip the delta for io-token groups; the
legacy TPM/RPM replay path is unchanged.
* fix(router): clear io-token reservation after reconcile/refund
async_io_token_refund_failure and async_io_token_reconcile_success now clear
the stashed reservation keys from the request metadata once done. Otherwise, on
a model group mixing IO-limited and non-IO deployments, a failed IO call that
retries on a non-IO fallback left the stale sentinel in the shared request
metadata; the fallback's success handler would divert into IO reconciliation
against the already-refunded key, driving the ITPM counter negative and
skipping the non-IO deployment's TPM tracking.
* fix(router): tidy reservation channel lookup and header guard
Consolidate the reservation channel lookup into a single ordered helper shared
by read and clear, so top-level metadata always wins over litellm_params
metadata without the tangled per-iteration fallback.
Also stop gating the router rate-limit header block on the presence of
x-ratelimit-remaining-input/output-tokens. That block only emits those headers
for ITPM/OTPM groups; for a non-IO group backed by a provider that natively
returns input/output token headers, the extra conditions suppressed the
router's own remaining-tokens/requests headers.
* fix(router): strip client-supplied io-token reservation keys
The reservation sentinels (_litellm_itpm_reserved, _litellm_itpm_cache_key,
and the otpm equivalents) are server-only, but metadata is caller-controlled on
proxy requests. An authenticated caller could forge these fields with an
arbitrary cache key so the post-call reconcile/refund path would decrement any
deployment's ITPM/OTPM counter and let it exceed the configured limit. Strip
the reserved keys from the request metadata in set_io_token_rate_limit_request_kwargs,
which runs before the router stashes its own reservation, so only a genuine
server-side reservation is ever read post-call.
* fix(router): track TPM routing load for io-limited deployments
deployment_callback_on_success early-returned for any deployment with itpm/otpm
set, so its total-token usage never landed in the router's TPM routing counter.
TPM-aware routing strategies then saw 0 load for IO deployments and over-routed
to them in mixed model groups. Only skip tracking when neither tpm/rpm nor
itpm/otpm are configured; itpm/otpm enforcement still runs separately in
ModelRateLimitingCheck, so the routing counter and the enforcement counters
stay independent.
* fix(router): expose standard tpm/rpm headers for io-limited groups
get_remaining_model_group_usage returned early for ITPM/OTPM groups, so a group
that also set tpm/rpm never emitted x-ratelimit-remaining-tokens / -requests;
clients and prometheus gauges reading those saw no data. Build both header sets
instead of returning early.
Also simplify the in-flight header replay: only the tpm/rpm counters are
incremented post-response, so the delta now adjusts just those. The itpm/otpm
counters are incremented at reservation time (pre-call), so the input/output
token headers already reflect the request and are left untouched - which
removes the need for the separate io-group special case.
* fix(router): roll back ITPM on any OTPM reservation error; dedup warning per instance
Two follow-ups from review. The pre-call OTPM reservation only rolled back the
ITPM reservation on a RateLimitError, so a transient cache error while reserving
OTPM left the ITPM counter inflated until the TTL expired; catch any exception,
release the ITPM reservation, then re-raise.
Replace the module-level lru_cache warn-once (caching a logging side effect,
which never re-warns in a long-lived process) with an instance-scoped set of
already-warned deployment ids on ModelRateLimitingCheck.
* fix(router): always clear reservation stash on reconcile; don't collapse id-less warning dedup
Clear the reservation in a finally block so a mid-reconciliation cache error
still removes the stash and a duplicate success event can't re-process it.
Dedup the itpm/otpm-vs-tpm/rpm conflict warning per real deployment id; a
deployment with no id no longer collapses every id-less deployment onto the
str(None) key (which would suppress all but the first warning).
* fix(router): skip io reservation when deployment can't be keyed
_get_cache_keys returned a shared 'global_router:None:None:...' key when a
deployment was missing model_info.id or litellm_params.model, so misconfigured
deployments could share one rate-limit bucket. Return None in that case and
skip io reservation for the request.
* fix(router): honor explicit max_tokens=0 in io reservation
_resolve_max_tokens used 'max_tokens or max_completion_tokens', so an explicit
max_tokens=0 fell through to the model default. Only fall back to
max_completion_tokens when max_tokens is absent.
* fix(ci): satisfy lint budget, router coverage, and dashboard schema sync
- Modernize the new itpm/otpm module's type hints to PEP 585 lowercase
generics (Dict/Tuple/List -> dict/tuple/list) to clear the added UP006
violations; ratchet ruff-strict-budget.json's UP006 ceiling down to match.
- Replace three try/except Exception blocks that must stay broad by design
(token_counter and litellm.get_model_info raise untyped exceptions, and an
io-token refund failure must never break the logging pipeline) with
contextlib.suppress(Exception), matching the codebase's existing resolution
for this exact BLE001 pattern.
- Add direct unit tests for get_model_group_io_token_usage (multi-deployment
aggregation and the empty-model-list case) in test_router_helper_utils.py,
satisfying the router function-coverage check.
- Regenerate the dashboard's schema.d.ts so the new itpm/otpm fields on
GenericLiteLLMParams and ModelGroupInfo are reflected in the OpenAPI types.
* fix: enforce io token rate limits consistently
* fix: honor zero max tokens in otpm reservation
* fix(lint): fix UP007 violation and resync ruff-strict-budget.json to base
Convert Union[_Span, Any] to _Span | Any (safe on this repo's Python >=3.10
floor) to clear the new UP007 violation from the TYPE_CHECKING-gated Span
alias.
The previously committed ruff-strict-budget.json ratcheted UP006 down from a
stale base; litellm_internal_staging has since tightened that same ceiling
further on its own. Reset the file to the current base's committed values and
re-ratchet from there so the budget only ever moves down relative to the
actual merge-base, never against a stale snapshot.
* fix(router): attach ITPM/OTPM headers on dict responses and harden reservation
Strip itpm/otpm from provider kwargs, ensure messages are available for ITPM
estimation, honor max_output_tokens on /v1/responses, and propagate rate-limit
headers through /v1/messages dict responses via _hidden_params.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): attach ITPM/OTPM headers to streaming /v1/messages responses
Wrap bare async iterators in HiddenParamsAsyncIteratorWrapper so
set_response_headers can attach rate-limit headers to streaming Anthropic
messages responses that lack a _hidden_params slot.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: ruff format add_retry_fallback_headers.py
Fix CI ruff format check failure on get_hidden_params_dict call site.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(router): extract set_response_headers helpers to fix C901 budget
Move header-attachment logic into add_retry_fallback_headers helpers so
set_response_headers stays under the strict complexity ceiling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: keep IO token reservation when response usage is missing
Missing usage was reconciled as zero and fully refunded the pre-call
reservation, allowing limit bypass on repeated successful calls. Only
adjust counters when usage is resolved from the response or standard
logging fields; otherwise keep the reservation until TTL expires.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: enforce RPM/TPM alongside IO-token limits on mixed deployments
Deployments with both itpm/otpm and tpm/rpm previously returned after the
IO reservation and skipped RPM/TPM checks. Run both paths and refund the
IO reservation only when RPM/TPM rejects after a successful reservation.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: track TPM usage on success for mixed IO+TPM deployments
The early return after IO-token reconciliation in log_success_event and
async_log_success_event skipped the TPM counter increment, so the tpm_key
the pre-call check reads was never written and tpm_limit was never
actually enforced on deployments that also configure itpm/otpm.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: treat total-only usage as unresolved in IO-token reconcile
usage/standard_logging_object entries carrying only total_tokens (no
prompt/completion or input/output breakdown) were treated as resolved
usage, resolving to (0, 0) and refunding the full reservation. Both
_usage_is_present and the standard_logging_object fallback now require an
actual input/output breakdown before reconciling, keeping the reservation
otherwise.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: reserve minimal token when input/output estimation fails
_reservation_value(0, limit) reserved the entire limit whenever token
estimation failed (empty/unsupported input, tokenizer error), letting one
such request claim the whole bucket and 429 every concurrent request to
the deployment until it completed. Reserve 1 token instead so estimation
failures no longer serialize traffic.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: refund IO reservation synchronously before retry deployment pick
On retry, set_io_token_rate_limit_request_kwargs clears reservation
sentinels from the shared kwargs dict before a background failure handler
can refund them, stranding the counter until TTL. Refund and clear any
stale reservation in _update_kwargs_with_deployment before stripping
sentinels for the next attempt.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(io_token_rate_limit_check): use model-specific tokenizer for ITPM estimate; document sync-refund Redis ceiling
Pass the deployment litellm_params.model to token_counter so it uses the
model's native tokenizer instead of the generic fallback, narrowing the
reservation over/under-estimate window between pre-call and post-call
reconcile.
Add a ponytail: comment to refund_stale_reservation_before_retry explaining
the known ceiling: the synchronous DualCache.increment_cache issues a
blocking Redis INCR when a Redis backend is configured. This only fires on
streaming mid-stream retries (non-streaming failures await their failure
handler before the retry picks a new deployment, leaving no sentinels to
refund). Upgrade path: make _update_kwargs_with_deployment async.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
