It's a literal in the OptionalPreCallChecks type union, but
add_optional_pre_call_checks has no handler for it - selecting it from the
Admin UI's new multi-select would save successfully and show as enabled while
the router does nothing with it. Drop it from optional_pre_call_checks'
exposed options until it's actually implemented.
default_litellm_params is merged into every completion call's kwargs, so an
operator can put a shared api_key or an Authorization header under
extra_headers there. Router.get_settings() now returns it (needed so the
Admin UI's Router Settings page can display/edit it), but /get/config/callbacks
forwarded router_settings verbatim regardless of caller role - unlike the
callback and alerting env vars on the same response, which already redact for
non-full-admin callers (e.g. PROXY_ADMIN_VIEW_ONLY). That let a read-only
admin read another admin's upstream provider credentials.
Add the same role gate used for callback/alerting env vars, scoped to
default_litellm_params via the existing SensitiveDataMasker so any
key/secret/token/auth-shaped field is masked for non-full-admin callers while
full admins keep seeing the real value.
Router.update_settings() silently dropped default_litellm_params and
optional_pre_call_checks (not in the allow-list, and optional_pre_call_checks
was never even stored as a readable attribute), so the Admin UI's Router
Settings page could not display or persist either setting - e.g. enabling
cache_control_injection_points or prompt_caching pre-call routing required
editing config.yaml directly.
Router now persists optional_pre_call_checks and returns both fields from
get_settings(). update_settings() merges default_litellm_params instead of
replacing it (a full replace would drop the timeout/max_retries/metadata
defaults Router.__init__ sets), and diffs optional_pre_call_checks against
what's already applied before calling add_optional_pre_call_checks(), since
that method has no dedup guard for prompt_caching/enforce_model_rate_limits
and would otherwise register a duplicate callback on every re-save.
The bridge refresh path decided whether an upstream token-endpoint rejection was invalid_grant by substring-matching the raw response body, so a rejection whose actual error is something else but whose error_description merely contains the string invalid_grant would false-match, map to invalid_grant, and trigger a needless authorization_code re-run
Parse the RFC 6749 section 5.2 error object and compare the error field. A non-JSON body, or an error that is not invalid_grant, now propagates as the upstream error rather than being reinterpreted. The regression test drives an invalid_client rejection whose description contains the string invalid_grant and asserts it is not mapped, mutation-checked against the substring match
The prior fix sent the sealed scope on a refresh, but the re-minted refresh envelope re-seals scope from the upstream response, and RFC 6749 section 5.1 lets an upstream omit scope when it is unchanged. So after one refresh whose response omitted scope, the new envelope sealed scope=None and every subsequent refresh dropped it, letting a stricter upstream narrow the renewed token
When the upstream omits scope on a bridge refresh, seal the scope we requested (which RFC 6749 section 5.1 defines as the granted scope when omitted) into the renewed access and refresh envelopes, so the scope survives the whole refresh chain. The regression test refreshes against an upstream that omits scope, asserts the new refresh envelope still carries it, and refreshes again off that envelope to prove the chain does not lose it, mutation-checked
The refresh envelope seals the upstream scope as the scope to re-request (RefreshCredential), but _prepare_bridge_refresh dropped it, unwrapping only the refresh token, and the exchange added scope to the upstream request only from the client's HTTP form. A DCR/MCP client typically omits scope on refresh, so the sealed scope was never sent and a stricter upstream could narrow or drop the renewed token's scope
Thread the sealed scope through _BridgeRefreshReady.upstream_scope and fall back to it when the client sends none; a client-supplied scope still wins, which RFC 6749 section 6 bounds to the original grant. The regression test drives a refresh where the client omits scope and asserts the upstream POST carries the sealed scope, mutation-checked against both the drop and the fallback
Mutation testing surfaced branches in cost_tracking_settings and common_utils that the suite executed but never asserted on. Pin those behaviors with targeted tests: the returned (model, provider) from _resolve_model_for_cost_lookup for deployments carrying a custom_llm_provider and for deployments missing the litellm_params / model_info keys, plus the exact error-response bodies, the caller-identity lookup arguments, and the member and guard branches in common_utils.
Four fixes to the refresh_token grant for dcr_bridge oauth_delegate, surfaced by an adversarial pass over the exchange path
Route the user-subject re-validation's outage check through the chain-aware classifier, so a transient DB outage (which get_user_object wraps in a bare ValueError) reports as unavailable (a retryable 503) rather than collapsing to no_active_key and an invalid_grant, matching how admission now handles the same wrapper
When the upstream reports its own refresh token as already elapsed (refresh_expires_in non-positive), do not seal it into a full-TTL refresh envelope; return no refresh so the exchange degrades to an access-only response, mirroring how the access grant refuses an already-elapsed access token instead of capping it
When the upstream rejects the sealed refresh token with 400 invalid_grant (revoked or expired at the IdP), return an RFC 6749 invalid_grant response so the OAuth client re-runs authorization_code, rather than surfacing the opaque upstream error it cannot act on
Gate key-subject renewal on the owner's SCIM state, mirroring admission's _reject_if_admitted_owner_scim_deactivated, so an offboarded user cannot keep refreshing a still-active key; the check fails open on a missing owner or a DB blip so a key that outlives its owner record does not get wrongly revoked
Each fix has a mutation-checked regression test
Three review findings on the refresh path, addressed at the root:
_BridgeRefreshReady.upstream_refresh_token was a plain str, the one credential in the envelope/bridge
layer that escaped the SecretStr discipline every other one follows (RefreshCredential.refresh_token,
UpstreamTokenGrant.access_token, EnvelopeKeys.signing_key). A repr or a traceback capturing a local
_BridgeRefreshReady would have logged the raw upstream refresh token. It is now a SecretStr, carried as
the SecretStr open_bridge_refresh_envelope already returns and unwrapped only at the point the exchange
builds the upstream request body.
_prepare_bridge_refresh took a request it never read; on the refresh path identity comes entirely from
the sealed envelope, not the HTTP request, so the parameter was dead and misleadingly implied it read
from the request the way the authorization_code prepare does. Removed, and the caller updated.
_reload_active_user_by_id misclassified a missing user as unresolvable (500). This is the same root
cause as the admission user-reload fix: get_user_object raises a bare Exception for a deleted user
rather than a ProxyException, so its except-Exception arm must fail closed to no_active_key (which the
refresh path maps to invalid_grant) for anything that is not a database-service-unavailable outage,
rather than treating a missing user as an opaque gateway fault. Regression tests cover the missing-user
and DB-outage classifications directly.
The live proof showed a refresh envelope presented at the MCP tool-call edge was rejected, but through
the generic oauth2 arm ("expected a virtual key starting with sk-") rather than the bridge arm, because
the admission routing gate is_bridge_envelope_shaped matched only the access prefix. The rejection was
already fail-closed and never forwarded anything upstream, but the path was imprecise and the unit test
modelled a route the real router did not take.
Match either envelope kind in is_bridge_envelope_shaped so the bridge arm engages for a refresh envelope
too, and have resolve_bridge_envelope return BridgeEnvelopeInvalid for it: a refresh envelope is a valid
gateway credential but only ever presented back to the token endpoint, never usable to authenticate a
tool call. Admission now fails it closed with the bridge arm's own 401 ("Invalid or expired
credential"), live-verified, with the upstream never touched. is_bridge_envelope_shaped has a single
caller (the admission routing gate), so the change is contained.
A dcr_bridge oauth_delegate access envelope is capped at one hour, and until now the mode had no refresh
at all: when the envelope expired the client had to re-run the interactive authorization_code flow. This
adds a second client-held credential, the refresh envelope, so the client renews on a back channel and
only re-authenticates when the refresh envelope expires or the upstream refresh token dies.
The refresh envelope is a distinct llm_refresh_ credential that seals only the upstream refresh token
(never the access token) bound to the same litellm identity and MCP server as the access envelope, under
the same master-key-derived keys, with nothing stored server-side. Both envelopes now carry a signed
kind claim ("access" or "refresh") that open() requires to match, so a refresh envelope can never open as
an access credential even if its wire prefix is swapped (the prefix is not signed; the claim is). A
refresh envelope presented at the MCP tool-call edge is not an access envelope, so admission fails it
closed the same way it already fails any non-access bearer.
At the token endpoint the authorization_code mint now returns a refresh envelope alongside the access
envelope whenever the upstream returned a refresh token, and the refresh_token grant is supported for
bridge servers: the client presents its refresh envelope, the endpoint opens it, re-validates the sealed
litellm key so a revoked key cannot keep refreshing, unwraps the real upstream refresh token, exchanges
it with the upstream IdP, and returns a fresh access envelope. Because the endpoint re-seals a refresh
envelope only when the upstream returns a new refresh token, the design mirrors the upstream's own
rotation policy rather than reinventing it: with a rotating upstream the client rotates and reuse is
detected upstream; with a non-rotating upstream the original refresh envelope stands until its bounded
14-day TTL. Both preconditions and the unwrap run before the exchange, so a rejected refresh never
consumes or rotates an upstream token.
The pure envelope and credential layers stay side-effect free: mint/open share one signing, size, and
kind gate across both envelope kinds, and every failure is a value. Tests cover the refresh round-trip,
the kind-claim and server-id bindings, the revoked-key gate, upstream rotation carried through, the
unwrap sending the real upstream token upstream, and edge rejection of a refresh envelope; the three
security bindings are mutation-checked. Limitation documented in the PR: gateway-enforced refresh
rotation with reuse detection would require server-side state, which this zero-custody mode omits by
design, so the refresh envelope inherits the upstream's rotation posture plus gateway identity binding
and a bounded TTL.
get_user_object catches every DB failure in a broad except and re-raises a bare ValueError (litellm/proxy/auth/auth_checks.py), so a real outage and a missing user look identical and the original error survives only as __context__. The dcr_bridge admission path keyed its 503-vs-401 decision on the exception type, so a transient outage during a user-subject reload surfaced as a 401 rather than a retryable 503, and the regression test injected a raw ConnectionError, a shape get_user_object never produces, so it passed on a fiction
Add PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain, which walks __cause__/__context__ (bounded and cycle-safe) the PEP 3134 way, and route _raise_503_if_db_unavailable through it. Move the user's object_permission resolution inside the single classified try so an outage there is a 503 too, never an opaque 500. Pin get_user_object's wrapping with a contract test that drives the real function, and drive the reload tests with that same faithful shape so a chain-blind regression fails them
_reload_admitted_user returned a bare UserAPIKeyAuth(user_id=...), so the shared
get_allowed_mcp_servers found no key/team/object-permission grants and an interactive SSO client could
admit successfully yet see zero tools on a normal (allow_all_keys=False) server. The key path returns
the full key record whose object permission drives that computation; the user path dropped it.
Resolve the user's own MCP object permission and put it on the returned auth, so the same
get_allowed_mcp_servers the key path uses grants the user their litellm-granted servers and access
groups. This reuses get_object_permission (the id-to-grants resolver keys and teams already use) and
does not duplicate any permission logic; get_user_object does not load object_permission, so it is
resolved from the user's object_permission_id the same way the key and team paths do.
Only the user's own object permission is bound. A UserAPIKeyAuth carries a single team_id while a user
may belong to many teams, so team-inherited MCP grants for a user are a follow-up: they need a
many-teams union get_allowed_mcp_servers does not do off one auth object, and faking one here would be
the kind of half-measure that spawns more bugs. Tests cover the user's object permission riding onto the
admitted auth, and the existing admit/SCIM/missing-user/503 cases still hold.
_reload_admitted_user mirrored only part of _reload_admitted_key's error contract: it caught
ProxyException and HTTPException but had no arm for anything else, so a transient DB outage surfaced as
an opaque 500 instead of the retryable 503 the key path guarantees, and a missing user surfaced as a 500
too. The missing-user case is the subtle one: get_user_object raises a bare Exception for a deleted user
(not a ProxyException like get_key_object does for a missing key), so the ProxyException/HTTPException
clause never caught it and the user_object-is-None branch it was supposed to hit is unreachable on the
production path.
Add the same except-Exception arm the key path uses, with the one deliberate difference the differing
get_user_object contract requires: a database-service-unavailable error still raises the retryable 503,
while a missing user or any other non-outage resolution failure fails closed as a 401 rather than
propagating as a 500. The regression tests now drive the real behavior (get_user_object raising) rather
than a None return that never happens in production, and cover both the 503 outage and the 401
missing-user paths.
Completes the oauth_delegate bridge for real DCR clients (Claude Code, Claude
Desktop), which send no litellm key and cannot use the scripted two-header path.
On the short-circuit bridge arm the gateway now captures the SSO-authenticated
litellm user from the browser session at /authorize and seals it into the OAuth
state; at /callback it seals that user plus the upstream code into a gateway
authorization code the client echoes back; at /token it recovers the user,
exchanges the real upstream code, and mints a user-subject envelope. The user
identity captured in the browser thus rides to the back-channel token call with
nothing stored server-side, and admission opens the envelope under that user. The
scripted key_hash path is unchanged (raw upstream code, key from the request);
without a session the browser is sent through login first.
The scripted two-header client mints under a virtual key it presents at the token
endpoint (key_hash), but the interactive DCR client authenticates via SSO at the
bridged authorize, which yields a user, not a key. Make EnvelopeIdentity a
discriminated subject (subject_type key_hash | user_id) with key_hash_identity /
user_identity constructors, and dispatch admission on it: a key_hash reloads the
key, a user_id reloads the user and admits them as themselves (user-level budget
and SCIM enforced via the same centralized gate; no team bound, since a user
belongs to many teams or none). The interactive producer that mints a user_id
envelope lands in the follow-up commit.
Pass-through endpoints configured with auth=false reach the cost-tracking callback with no key/user/team/end-user, so _should_track_cost_callback returned False and the spend-log write was skipped, leaving the request out of request/usage logs. Track pass-through call types even when unauthenticated so the SpendLog row is still written.
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): rename OldTeams component file to Teams
* fix: show and allow editing team model aliases after team creation
* fix(ui): mark team model_aliases as nullable to match the prisma schema
* fix(guardrails): walk custom_tool_call_output items in _content_utils
* Change _OUTPUT_ITEM_TYPES to Frozenset type
* fix(guardrails): use builtin frozenset generic for _OUTPUT_ITEM_TYPES annotation
Frozenset is not a defined name (typing exports FrozenSet, the builtin is
frozenset), so module import raised NameError and broke every proxy test
suite. The builtin generic is valid on the supported python floor (3.10)
and keeps the UP006 ruff-strict budget at its ceiling, which the typing
alias would exceed
_classify_upstream_lifetime decided "expired" from int(float(expires_in)), which truncates toward
zero, so a positive fractional lifetime in (0, 1) became 0 and was misread as already elapsed. That
rejected the mint with 502 in _finish_bridge_mint after the single-use upstream code had already been
consumed, even though the upstream reported a positive remaining lifetime.
Decide expired on the parsed numeric value rather than its truncated int, so only a genuinely
non-positive value is expired. The envelope works in whole seconds and cannot represent a sub-second
lifetime, so a positive value that truncates to 0 clamps up to the 1s floor instead of being rejected.
Values >= 1 still truncate toward zero so the envelope never claims more life than the upstream stated,
and NaN / Infinity / oversized input still read as unparseable ("unspecified").
Regression covers the classifier (0.5 and 0.001 clamp to 1, 1.9 truncates to 1, -0.5 stays expired) and
the mint (a 0.5s upstream lifetime mints a 200 envelope rather than a 502); reverting to the
truncate-then-check reddens both.
Three findings landed together, all one defect: a resolution step crushed several distinct outcomes
into a single None or a silent default, so the mint's error mapper could not tell them apart and
assigned the wrong status. Identity resolution mapped a database outage to the same None as a missing
credential, which the mint reported as 400 invalid_request, blaming the caller for a gateway outage
while admission statuses the same outage 503/500. Lifetime coercion mapped an explicit non-positive
expires_in to the same None as an absent one, so an upstream token the IdP reports as already dead was
sealed into an hour-long envelope. And the refresh_token grant was run through the upstream exchange
(which can rotate the client's upstream refresh credential) and its result then discarded, even though
a bridge server seals no refresh_token and the client never holds one to present.
Rather than add a mapping branch per finding, the fix changes the return types so a wrong status is not
representable. Each resolution step now returns a precise tagged value instead of None: identity
resolution returns a _ResolvedKey or one of no_active_key / unavailable / unresolvable, classified the
same way admission's _reload_admitted_key classifies the same conditions; upstream-lifetime
classification returns a positive number of seconds, "unspecified" (absent or unparseable, which the
envelope caps), or "expired" (a parseable non-positive value, an already-dead token); and upstream-grant
validation returns a typed grant or one of no_access_token / expired_lifetime. Thin exhaustive mappers
(match plus assert_never) lift each vocabulary into one bridge-mint taxonomy of eight named failures,
and a single _bridge_mint_error_response gives each its truthful RFC 6749 §5.2 status: 400 for the
caller's missing credential or an unsupported grant, 503 for a transient auth-DB outage, 500 for a
gateway that cannot resolve identity or is not configured, and 502 for an upstream response with no
usable token, an already-expired lifetime, or a token too large to seal. Adding a failure mode now
requires a new literal and a match arm the type checker forces, so the class of wrong-status bug cannot
recur silently.
The refresh_token grant is rejected in _prepare_bridge_mint before the exchange with
unsupported_grant_type, so it can never rotate or consume the client's upstream refresh credential;
renewal is re-running authorization_code, as the sealed refresh_token=None already intends. An absent or
unparseable expires_in still mints a capped envelope (the by-design behaviour for an upstream that omits
the field); only an explicitly-dead lifetime is rejected.
Tests cover the resolver's three failure classes (including a real connection-error outage and a missing
prisma_client), the mint statuses for each (503 before the upstream exchange, 500, 502 on an expired
upstream lifetime, and a capped mint on an unknown one), and the refresh-grant rejection before any
exchange. The three findings are mutation-checked: reverting each fix turns its regression test red.
_finish_bridge_mint floored the reported expires_in at 1. Admission expires the
envelope against the JWT's second-truncated exp, so when the mint lands in the same
second that exp falls on (a sub-second upstream lifetime, for instance), the true
remaining life is 0 and reporting 1 tells the client the bearer lives one second past
the point admission already rejects it. Floor at 0 instead so the reported lifetime
never overstates the exp; the value still cannot go negative.
The regression pins the boundary directly: minting at now=100.25 with a 1s upstream
token seals exp=101, and the reported expires_in is max(0, 101 - ceil(100.25)) = 0.
Under the old floor of 1 it reads 1, so the test fails on that mutation.
Also drops the unused mcp_server parameter from _prepare_bridge_mint; identity and
key derivation there never referenced the server.
The dcr_bridge oauth_delegate token mint validated its preconditions in two
places: a pre-exchange guard inside exchange_token_with_server (master_key set,
resolvable litellm identity) and an authoritative re-check inside the post-exchange
_mint_bridge_delegate_token_response. Keeping the two in step by hand is what kept
producing the same class of finding: a precondition guarded on one grant branch but
not the other, master_key checked after the exchange on one path, identity resolved
twice, and each failure raising an ad-hoc HTTPException with its own status and body
shape.
Model the mint as three phases whose failures are values. _prepare_bridge_mint runs
before the exchange, checks every precondition once (master_key, then identity), and
returns either a frozen _BridgeMintReady carrying the resolved key hash and the
master-key-derived envelope keys, or a _BridgeMintError literal. Because every
precondition lives in prepare, and prepare runs before the upstream POST, no failure
can burn the single-use code or rotate a refresh token, for either grant type, by
construction rather than by a guard we have to remember to keep in sync.
_finish_bridge_mint runs after the exchange and has no preconditions left that can
fail; its only failure values are properties of the upstream response itself (no
usable access_token, or a token too large to seal). One mapper,
_bridge_mint_error_response, turns each _BridgeMintError into an RFC 6749 section
5.2-shaped body with a status truthful about where the failure is (400 for the
caller, 500 for gateway config, 502 for the upstream), with an exhaustive match plus
assert_never so a new failure mode cannot be added without a matching status.
Behavior is unchanged for the client. Every failure that previously raised now
returns the same status as an OAuth error body, which is the correct token-endpoint
contract; the three tests that asserted a raised HTTPException now assert the
returned response. _exchange_for_bridge_server additionally asserts the identity
resolver is awaited exactly once for a bridge server and never for a non-bridge one.
Follow-up to the pre-exchange identity gate, which I had only added to the
authorization_code branch and which left the master_key check inside the mint
(after the upstream exchange) - so the very burn-then-fail pattern it was meant to
prevent still applied to refresh_token grants and to a misconfigured gateway.
- Hoist a single pre-exchange gate above the upstream call that covers BOTH grant
types: it fails closed (invalid_request) on an unresolvable litellm identity and
500s on an unset master_key BEFORE the single-use code or refresh token is
exchanged/rotated, so a bad key or a misconfigured gateway never burns the
upstream credential.
- Report expires_in from the envelope JWT's own second-truncated exp (rounding the
elapsed portion up) instead of the raw expires_at - now delta, so the client is
never told the bearer is valid past the ~1s point admission already expires it.
Regression tests assert the upstream exchange is never called on the no-identity
refresh grant and the master_key-unset path, and that the reported expires_in does
not overstate the JWT exp.
Findings from a full adversarial review of the mint path across security,
correctness, error-handling, concurrency, and OAuth-protocol dimensions.
- expires_in coercion is now total: int(float(...)) can raise OverflowError on
Infinity / a giant numeric string, which escaped the ValueError/TypeError catch
and 500'd the token endpoint. Unified to catch OverflowError too.
- Resolve the litellm identity BEFORE exchanging the single-use upstream code, so
a missing or transiently-unresolvable identity fails closed with invalid_request
without burning the code (the mint re-resolves via a cache hit).
- The no-identity failure is now an RFC 6749 5.2-shaped invalid_request
(JSONResponse, top-level error, no-store) instead of a detail-wrapped
HTTPException, matching the BYOK OAuth endpoint.
- EnvelopeTooLarge (upstream token too big to seal) surfaces a 502, not a 500.
- The upstream refresh_token is no longer sealed into the envelope: the edge
never consumes it, so it was dead weight embedding a long-lived upstream
credential in the client bearer and enlarging the envelope; refresh is a
follow-up (a dedicated refresh-envelope).
Security review found no exploitable defect (forgery, cross-server/user replay,
leakage, confused-deputy all closed). Regression tests cover the OverflowError,
the code-not-burned path, the RFC-shaped error, the 502, and the dropped refresh.
Two correctness gaps in the bridge mint. _bridge_grant_from_token_response only
accepted an int expires_in, dropping a float (3600.0) or numeric-string ('3600')
lifetime to None so the envelope fell back to its 1h cap and could outlive a
shorter-lived upstream token; coerce it to a positive int (bool excluded). And
_key_is_active called datetime.fromisoformat on the str|datetime expires outside
the resolver's try, so a malformed stored expiry raised an unhandled 500 instead
of the fail-closed invalid_request; it now fails closed (inactive) on an
unparseable expiry. Regression tests cover int/float/string/bool coercion, the
short-float TTL, and the malformed-expiry fail-closed path.
_resolve_active_litellm_key gated on _active_key_user_id, which returns None both
for blocked/expired keys AND for valid keys with no user_id, so a team-scoped or
service-account key was wrongly rejected with invalid_request at bridge token
exchange. Split the active-state gate (_key_is_active: blocked/expiry only) from
the user_id extraction; the mint seals the key hash, not the user, and admission
already handles a keyless-user key. The per-user token store still gets no user
for such a key, as there is none to key a stored credential by.
The eager access_token = token_response["access_token"] extraction ran before
the dcr_bridge branch, so a missing upstream access_token raised an unhandled
KeyError and _bridge_grant_from_token_response's nil guard (which maps to a clean
502) was dead code. Move the extraction onto the non-bridge result path so the
bridge branch reaches its 502 guard.
The mint bound only user_id/server_id into the envelope, which gave admission
no way to reload the caller's key and enforce its current restrictions. Seal the
hashed authorizing key instead (a one-way digest, not a usable credential), so
admission reloads the live UserAPIKeyAuth by it and the key's team/org/tool
permissions and revocation apply per request.
Extract the token endpoint's key resolution into a shared _resolve_active_litellm_key
so the per-user token store (user_id) and the bridge mint (key hash) derive from one
active-key-gated path, and fail the mint closed with invalid_request when no active
key accompanies the request.
XecGuard's async_logging_hook wrote a bare dict to
standard_logging_object["guardrail_information"] while the typed
contract is Optional[List[StandardLoggingGuardrailInformation]].
Readers that iterated the field walked dict keys, raised on
info.get, or silently dropped the entry from guardrail usage
tracking and spend-log writes
Construct the typed entry and append it to the existing list or
create a new one, matching the shared helper pattern. Record the
configured guardrail name instead of a hardcoded "xecguard" and
pass the GuardrailEventHooks enum for guardrail_mode
* feat(proxy): add expires filter to GET /key/list
Add an opt-in expires query param to GET /key/list so callers can fetch
only expired or only active keys without paginating every page and
filtering client-side. 'expired' matches keys whose expires is in the
past (NULL expires excluded); 'active' matches keys that never expire or
expire in the future. Omitting the param preserves existing behavior for
every caller. An unrecognized value returns HTTP 400 rather than silently
returning all keys.
The filter is pushed to the database via the existing Prisma where
builder so callers avoid pulling the full key table into application
memory.
Resolves LIT-3387
* refactor(proxy): declare VALID_EXPIRES_FILTER_VALUES before its first use
* feat(guardrails): add pre_mcp_call support to Content Filter
* test(guardrails): cover canonical MCP key gate under pre_mcp_call mode
* fix(guardrails): scan MCP arguments per value and gate mixed-mode scans by call type
* fix(guardrails): cap MCP argument scan depth and register the walker with the recursion detector
* test(guardrails): update LIT-4226 UI settings tests for content filter pre_mcp_call support
* fix(guardrails): use builtin generics in MCP scan annotations to satisfy strict-rule budget
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Some tiered Dashscope models price reasoning output above standard output
(output_cost_per_reasoning_token > output_cost_per_token). The reservation charged
all output at the standard rate, so a reasoning-heavy request reserved too little
and concurrent calls could exceed the budget before reconciliation. The reasoning
share is unknown before the request runs, so reserve every output token at the
higher of the two configured rates, for both tiered and flat pricing.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): filter Add-Guardrail mode dropdown per provider
The GET /guardrails/ui/add_guardrail_settings endpoint returned every
GuardrailEventHooks value in one flat supported_modes list, so the Admin
UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving
Content Filter or Tool Permission with pre_mcp_call then failed with a
400 because those guardrails' server-side supported_event_hooks list
excludes it.
Expose each guardrail's supported hooks as a get_supported_event_hooks
classmethod on CustomGuardrail (mirrors the existing get_config_model
pattern) and have the endpoint iterate guardrail_class_registry to build
a supported_modes_by_provider map. The UI Mode dropdown filters by that
map when the selected provider is known and falls back to the global
list otherwise. __init__ now sources its own supported_event_hooks list
from the classmethod so the two sides can't drift.
Also register BedrockGuardrail, ToolPermissionGuardrail, lakera,
lakera_v2, and presidio in guardrail_class_registry so they participate
in the map (they were previously only in guardrail_initializer_registry
and had no class-registry entry).
Behavior change: guardrails that previously had no supported_event_hooks
declared (aim, javelin, azure/text_moderation, cato_networks,
crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx,
prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai,
lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now
validate the configured mode at instantiation. Existing configs where
the mode was silently a no-op will fail at proxy startup with a clear
validation error rather than running as a broken guardrail.
Resolves LIT-4226
* fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form
Address Greptile P1 (startup break) and P2 (edit form UX):
LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported
event_hook, unchanged behavior for the guardrails validated pre-PR).
Setting it to false logs a warning and continues, giving deployments an
opt-out while they fix configs that now surface as errors instead of
silently no-op'ing. Regression test covers both modes.
Edit form now surfaces the currently-saved mode even when it is not in
the filtered per-provider list, so a legacy row (e.g. content_filter
saved with pre_mcp_call before this fix) no longer disappears from the
dropdown; the option renders with a 'not supported by <provider>' note
so the user knows to pick another.
* fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint
Audited every get_supported_event_hooks classmethod against the hooks
each guardrail's own tests exercise and its handler methods. Five were
too narrow and their tests caught it in CI: rubrik gains pre_call,
presidio gains during_call and pre_mcp_call, prompt_security, onyx and
qualifire gain during_call. The remaining classes match either their
original __init__ declarations or their exercised modes exactly.
Cursor review fixes: the Add form now drops selected modes the new
provider does not support when the user switches providers, so a
pre_mcp_call selection cannot ride along into a provider that rejects
it at save; the edit form handles list-shaped stored modes instead of
treating mode as always a string.
Extracted shared toModeArray and getSupportedModesForProvider helpers
into guardrail_info_helpers so both forms use one implementation, typed
the remaining any usages in both forms, removed nested ternaries, and
committed the ratcheted-down eslint metrics and pruned suppressions
The envelope arm bypasses user_api_key_auth, so it never ran
pre_db_read_auth_checks (request-size and body-safety limits, the IP allowlist,
and the general_settings route allowlist) that the normal MCP admission path
runs before any key lookup. A caller blocked by IP or a disallowed proxy route
could be admitted through an envelope where the same principal on the normal
path is rejected. Run those gates before the envelope crypto, mirroring the
pipeline's pre-DB ordering; a blocked IP or route surfaces its own 403.
Alibaba Model Studio (Dashscope) tiered pricing is all-or-nothing: the tier is
selected by a request's total input tokens and every token, input and output, is
billed at that one tier's rate. The reservation path used graduated slicing and,
worse, picked the output tier from the output-token count, so a long-context
request with a large output allowance reserved far less than the provider charges
and could slip past a depleted budget. Select the tier from input tokens and apply
its rates to all input and output tokens.
Reservation also read tiered pricing from only the first deployment in a model
group. A caller could hit an alias whose cheaper deployment was listed first and
exceed the budget once routed to a costlier sibling. Estimate against every
eligible deployment's pricing and reserve the maximum.
Co-authored-by: Cursor <cursoragent@cursor.com>
The envelope arm reloaded the identity and ran _run_centralized_common_checks
but skipped RouteChecks.should_call_route, which the standard pipeline runs
between the builder and common_checks. Because the centralized checks treat MCP
as an inference route and never re-check allowed_routes, a key barred from MCP
routes could mint an envelope at the token endpoint (not itself an MCP route)
and replay it against MCP. Run the route gate before admitting, and clear the
request-scoped budget_reservation, matching the wrapper's sequence; a disallowed
route now surfaces the gate's own 403.
* feat: add silent CLI token refresh for apiKeyHelper support
lite auth print-token prints a valid proxy credential for use as Claude
Code's apiKeyHelper, transparently refreshing it first if the cached JWT
is stale. This unblocks MDM-managed apiKeyHelper deployments (managed
via `lite auth print-token`) that need silent mid-session credential
rotation without restarting the client.
Refresh capability is backed by a virtual key minted with an empty model
list and cli_refresh metadata, kept strictly separate from the actual
(short-lived, real-model-scoped) call credential -- so a leak of the
credential that flows through every LLM request and subprocess env var
can't also self-renew. The refresh flow is single-use: /sso/cli/refresh
mints a fresh JWT + refresh token pair and blocks the presented refresh
token immediately, so a replay can't mint a second pair from it.
Server: /sso/cli/refresh (rotate) and /sso/cli/logout (revoke) endpoints.
lite login now also stores a refresh token; lite logout revokes it
server-side instead of only clearing the local file.
* fix: allow non-admin users to hit CLI refresh routes; resolve apiKeyHelper base_url from token.json
Found via a live end-to-end test against a real proxy + real Claude Code
session: /sso/cli/refresh and /sso/cli/logout were unreachable for any
non-proxy-admin caller, since Depends(user_api_key_auth) pulls in a
route-RBAC gate that 403s any route not on an explicit allowlist. That
made the feature unusable for actual end users, who authenticate as
internal_user. Add both routes to internal_user_routes; the handlers
already do their own fine-grained check (metadata.cli_refresh) same as
/key/block does today.
Also: `lite auth print-token` required an explicit --base-url/
LITELLM_PROXY_URL matching the stored token's origin, defaulting to
localhost:4000 otherwise. But apiKeyHelper is configured bare (no
flags), so this always mismatched a real deployment. Track whether
--base-url was explicitly passed (via click's ParameterSource) and, if
not, resolve the server from token.json directly instead of the CLI
default.
* test: mock refresh-token minting in test_cli_poll_key_tolerates_missing_user_row
Landed on litellm_internal_staging after this branch's refresh-key minting
change; needs the same mock as the other cli_poll_key tests since minting
now runs unconditionally whenever a JWT is generated.
* fix(ci): update test_cli_auth.py for refresh_token contract, regenerate schema.d.ts
_poll_for_authentication now always includes "refresh_token" in its
returned dict, and _handle_team_selection_during_polling returns a dict
instead of a bare JWT string -- test_cli_auth.py predates this branch's
refresh-token work and still asserted the old shapes.
schema.d.ts regenerated via `npm run gen:api` to pick up the new
/sso/cli/refresh and /sso/cli/logout routes (plus unrelated drift from
other PRs merged since it was last generated).
* fix(ci): apply CI's own schema.d.ts diff (enterprise routes I can't generate locally)
Local `npm run gen:api` only sees OSS routes -- this machine's
litellm_enterprise editable install points at a now-deleted temp
directory, so it silently drops enterprise-only routes from the spec.
Applied the exact diff CI's own generation produced instead of
re-running the generator locally.
* fix: close refresh-token race, fail closed on DB down, fix logout base_url
Addresses Greptile review findings on the CLI refresh-token PR:
- cli_refresh_token minted a new JWT + refresh token BEFORE blocking the
presented one. Two concurrent requests bearing the same refresh token
could both pass auth and both mint fresh pairs, yielding four live
credentials from one consumed token. Now the presented token is
consumed atomically first via update_many (only succeeding if it flips
blocked from False/None to True); the loser gets count=0 and is
rejected before anything is minted.
- When prisma_client is None, refresh silently returned a new JWT
without ever being able to mark the presented token consumed, leaving
it valid indefinitely. Now fails closed with a 500 instead.
- `lite logout` sent its revocation POST to ctx.obj["base_url"], which
defaults to localhost:4000 when --base-url isn't passed -- the same
bug print_token had before the base_url_explicit fix, just missed
here. Now resolves the same way: trust the stored token's origin
unless the caller explicitly overrode --base-url.
* fix(ci): satisfy ruff format and narrow token_data type in logout
* fix(security): never trust refresh-token metadata for authorization
Addresses a real privilege-escalation path Veria flagged: cli_refresh_token
read team_id, team_alias, and max_budget straight off the presented
token's own metadata and used them to authorize the new JWT. Since any
authenticated user can self-mint a virtual key with arbitrary metadata
via the ordinary /key/generate endpoint, a self-forged key with
{"cli_refresh": true, "team_id": "<any-team>", "max_budget": 999999999}
would sail through _require_cli_refresh_token's only check
(metadata.cli_refresh == True) and get a JWT scoped to a team the
caller never belonged to, with a budget it never had -- full
cross-team / budget bypass, and a removed team member could keep
refreshing team-scoped sessions indefinitely.
Metadata's team_id is now treated as an untrusted UX hint only: honored
solely if the CALLER (identified by the authenticated key's own
user_id, not client input) is a current member per a fresh
get_user_object lookup. team_alias and max_budget are never read back
from metadata at all -- team_alias comes from a live get_team_object
lookup and max_budget is recomputed with the exact same capping logic
the initial SSO login poll uses. _mint_cli_refresh_token no longer
accepts or stores team_alias/max_budget, only the team_id hint.
Added regression tests proving: a forged/stale team_id is dropped
(falls back to no team, not silently honored), and a forged max_budget
in metadata never reaches the issued JWT.
* fix(ci): catch HTTPException specifically instead of bare Exception (BLE001)
* fix: un-consume refresh token if minting the replacement fails
Greptile flagged a real reliability gap: cli_refresh_token blocks the
presented token atomically, then does several more DB calls before
returning a replacement (user lookup, team lookup, JWT mint, new
refresh-key mint). Since this endpoint exists specifically for fully
unattended apiKeyHelper operation, a single transient failure in that
window (DB hiccup, etc.) permanently stranded the user: their old
token was already dead and no new one was issued, with no recovery
path short of a full interactive browser re-login.
Wrap that window in try/except; on any failure, best-effort revert the
consumed token back to usable (blocked=False) before re-raising, so a
retry can succeed. Standard compensating-action pattern since
generate_key_helper_fn doesn't take an injectable transaction, so
wrapping the whole thing in a real DB transaction isn't practical here.
* fix(security): refresh key had unrestricted model access, not none
Critical bug: _mint_cli_refresh_token used models=[] intending "no LLM
access", but that's backwards in this codebase. Per
_check_model_access_helper: `len(filtered_models) == 0 and len(models)
== 0` -> all_model_access = True. An empty models list on a key with no
team_id means UNRESTRICTED access to every model, not zero access. The
CLI refresh token -- meant to be usable for nothing but silently
exchanging itself for a new JWT -- was actually a fully unrestricted
API key for its entire 90-day lifetime, completely undermining the
whole point of keeping it separate from the short-lived call
credential.
Fixed with two independent layers: allowed_routes hard-restricts the
key to exactly /sso/cli/refresh and /sso/cli/logout (the real enforced
boundary, checked in the shared user_api_key_auth dependency for every
route); models is set to an unmatchable sentinel string as
defense-in-depth in case any code path only consults the models field.
Added an end-to-end regression test that exercises the actual
model-access-control function against a key shaped like the minted
refresh token, rather than only asserting on what arguments were passed
to the key-generation call -- the latter kind of test is exactly what
let the original bug ship, since asserting `models == []` is equally
consistent with "no access" and "unrestricted access" without checking
what the access-control code actually does with that shape.
Also: the compensating-rollback added for reliability un-blocked a
consumed refresh token even when the underlying user no longer exists.
That's a permanent, intentional rejection, not a transient failure --
un-blocking it would let a stale refresh token become valid again for a
different account if the user_id is ever reused/re-registered. Moved
the user-existence check outside the rollback-on-failure block so it
stays permanently blocked.
* refactor: rotate CLI refresh tokens via regenerate_key_fn instead of hand-rolled consume/rollback
The refresh token is already a plain litellm virtual key, so rotation can
delegate to the same atomic DB update /key/regenerate uses instead of a
bespoke update_many + compensating-rollback dance. This makes silent CLI
refresh an Enterprise feature, same as regular key regeneration.
* refactor: replace CLI stateless JWT + refresh-key pair with one self-rotating virtual key
The CLI previously minted two credentials on login: a stateless self-signed
JWT for LLM calls, and a separate DB-backed refresh-only key (scoped away
from ever calling an LLM) just to authorize minting a new JWT. Collapse
this into a single real virtual key, used directly as the LLM bearer token
and re-presented to /sso/cli/refresh to rotate its own secret in place.
This also means the CLI session key now shows up in the Admin UI's Keys
page and can be revoked/regenerated like any other key, rather than being
an invisible, unmanageable stateless token.
* refactor: drop silent CLI refresh, key just expires and requires re-login
/sso/cli/refresh only ever benefited Enterprise deployments (regenerate_key_fn's
gate), while everyone else already fell through to "re-run lite login" on
failure. Cut the endpoint, the rotation logic, and the client-side refresh
path entirely; print-token now just prints the cached key until it hits its
LITELLM_CLI_JWT_EXPIRATION_HOURS duration, then fails fast telling the user
to log in again. Session key itself is unaffected: still a real, revocable
virtual key visible in the Keys UI, `lite logout` still revokes it directly.
* fix(ci): regenerate schema.d.ts after removing /sso/cli/refresh route
* revert: go back to stateless JWT, keep only lite auth print-token
The virtual-key redesign (revocable, Keys-UI-visible credential) wasn't
needed just to support print-token, and cost real server-side surface
(a mint path, a logout-revoke endpoint, migrated tests/docs) for a property
this repo doesn't need yet. Reverting cli_poll_key/_types.py/schema.d.ts
back to the original stateless-JWT design; the only durable addition from
this whole effort is `lite auth print-token` (reads the cached credential,
prints it while fresh, fails with a clear message once it's past
LITELLM_CLI_JWT_EXPIRATION_HOURS) plus the base_url_explicit plumbing it
needs. `lite logout` goes back to clearing the local file only, since a
stateless JWT can't be revoked server-side.
* refactor: move CLI token freshness check to cli_token_utils, drop unnecessary renames
Addresses review: the freshness check is a pure token-shape/timestamp
util, not command logic, so it belongs alongside the other SDK-level
CLI token helpers (load_cli_token, get_litellm_gateway_api_key) rather
than in commands/auth.py. Also reverted a few incidental jwt_token/
session_key variable and string renames that weren't load-bearing.
