* fix(scim): sync team roster and dedup teams for existing-user email upsert
When POST /scim/v2/Users matched an already-existing user by email,
handle_existing_user_by_email raw-wrote the user's teams array but never
touched the team roster, so the user appeared in the group on their profile
yet was absent from the team directly (members_with_roles stayed empty). It
also did not dedup the teams built from repeated SCIM groups.
Route the existing-user team assignment through the same
_handle_team_membership_changes / team_member_add path the PUT update_user
handler uses, so members_with_roles, LiteLLM_TeamMembership, and the user's
teams array stay in sync, and dedup the teams derived from user.groups. The
user_id rewrite to the new userName is preserved and sequenced before the
roster sync so the roster never references a stale primary key.
* fix(scim): surface roster add failures on existing-user email upsert
Route the existing-email upsert's roster sync through patch_team_membership
with a new opt-in raise_on_error flag so a genuine team_member_add failure
propagates instead of being swallowed, and the deduped teams array is only
persisted after the roster sync succeeds. Without this, a failed add left the
endpoint reporting success while user.teams listed a team members_with_roles
never received.
The benign already-a-member case stays a no-op even under the strict path, and
the flag defaults to False so the PUT update_user, PATCH patch_user, and group
callers keep their existing best-effort behavior. SCIM POST is idempotent, so
surfacing the error lets the IdP retry and converge.
* fix(scim): surface roster removal failures symmetrically with adds
Make team_member_delete failures fail loud under the strict roster sync used by
the existing-email upsert, mirroring the add path, so a swallowed removal can no
longer let the user's teams array drop a team the roster still holds. The
idempotent case where the user is already absent from the team stays a no-op,
matching how an add treats the user already being in the team. Best-effort
behavior is preserved for the default raise_on_error=False callers.
* fix(scim): use members_with_roles as the source of truth for group membership
SCIM group provisioning tracked membership inconsistently. Team creation and
the real team endpoints persist membership in members_with_roles (and each
member's user.teams), but the SCIM group PATCH handler and the GET /Groups
listing read the legacy team.members String[] column, which team creation never
populates. Seeding a PATCH result from that empty column made an Okta "add
member" operation recompute the member set from scratch and silently drop
everyone already in the team, so users ended up missing from the groups they
were provisioned into. Reading the same empty column on GET /Groups reported an
empty member list back to the IdP, which drove repeated re-provisioning.
Separately, add_new_member appended the team id to user.teams with an
unconditional array push. Under the concurrent group PATCHes an IdP sends during
a reconcile, each request passed the members_with_roles duplicate check and
pushed, so user.teams accumulated duplicate ids for the same team. A duplicate
also breaks auth logic that keys off the number of teams a user belongs to.
Read current membership from members_with_roles in the SCIM group PATCH seed and
the GET /Groups listing, and make the user.teams append idempotent via a
filtered update that no-ops once the team is present.
Resolves LIT-4283
* fix(scim): address review; atomic user-creation and stop writing legacy members
Keep the concurrent-safe team append but create the user via an atomic upsert
(create-or-update) instead of a check-then-create, so provisioning the same new
user concurrently cannot race into a duplicate-key failure; the team is still
appended idempotently by a filtered update so an existing user gets no duplicate
team id. Stop writing the legacy team.members column in the group PATCH apply so
the only membership record is the source of truth (members_with_roles plus each
member's user.teams), reconciled by team_member_add/team_member_delete.
Tests: existing add_new_member and team-creation mocks updated to the upsert
plus filtered-append shape, and new tests cover atomic creation and that the
PATCH apply does not write the legacy members column.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
get_team_object consults proxy_logging_obj.internal_usage_cache before
user_api_key_cache, but _cache_team_object (the refresh every team
mutation goes through) only wrote user_api_key_cache. With
enable_redis_auth_cache both caches share one Redis, so any request
backfills the internal cache's in-memory tier with the team object and
that copy keeps shadowing the freshly written team until its TTL expires.
The auth builder then wrote the team object it had just read back into
the cache after check 6, clobbering the fresh Redis value with the stale
one, which made the staleness self-sustaining under traffic: keys with
models=["all-team-models"] kept getting 403 team_model_access_denied
for models added via /team/update, and kept access to removed ones.
_cache_team_object now deletes the internal usage cache entry before
writing the refreshed team, and the auth-time write-back is removed so
only authoritative writers (DB reads and team mutations) populate the
team cache, mirroring how key objects already handle this (see
test_auth_does_not_rewrite_cached_key_object_back_into_cache).
The LIT-4000 test pinning the removed write-back is deleted; its
concern (team object cached under the canonical key) is handled by
_cache_team_object inside get_team_object's DB path and pinned by
test_cache_team_object_writes_team_id_and_invalidates_team_alias
Resolves LIT-4391
* fix(proxy): share CLI SSO login sessions across workers without enable_redis_auth_cache
* fix(proxy): make CLI SSO flow state redis-authoritative across workers
The CLI SSO flow is stored in a DualCache whose get_cache is memory-first, so
the worker that served /sso/cli/start keeps serving its stale in-memory flow and
never observes the sso_complete/session_data update another worker writes during
the OAuth callback. Attaching Redis alone is not enough; poll on the original
worker returns pending forever.
Read and write the flow directly through the attached Redis backend when present
so every worker sees the same authoritative state, falling back to the in-memory
DualCache only when no Redis is configured.
* fix(proxy): serialize CLI SSO flow as JSON for the redis round trip
RedisCache stores values via str(value) and parses reads with
json.loads then ast.literal_eval. The completed flow contains a
LitellmUserRoles enum in session_data.user_role, whose repr is not a
parseable literal, so any worker reading the completed flow from redis
raised SyntaxError and returned 400 "CLI login session not found".
Writing the flow as json.dumps makes the round trip lossless (the enum
is a str subclass) and fails loudly at write time if a non-serializable
value is ever added to the flow.
* fix(proxy): point CLI SSO session-not-found hint at configuring Redis
The error message and warning still told users to set enable_redis_auth_cache,
but the CLI SSO session cache now gets Redis unconditionally whenever one is
configured, so that flag no longer affects CLI login
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
The Cache Settings page read only the database row, so a response cache
pointed at Redis purely through REDIS_* env vars showed a blank page
while the cache worked. It also masked credentials on read with a
partial-reveal string and re-persisted whatever the form submitted, so
an admin who edited an unrelated field and pressed Save wrote the mask
string over the real Redis password, breaking auth.
GET /cache/settings now overlays the same REDIS_* kwargs the runtime
resolves from when the stored config leaves a field unset, and redacts
credentials with a fixed marker. POST /cache/settings restores the
stored secret behind any credential echoed back as the marker or omitted,
and drops an env-sourced marker rather than persisting it; the response
no longer echoes plaintext credentials. The connection test resolves a
redacted credential back to the stored value the same way. The dashboard
never prefills a credential and drops the marker from the save payload,
mirroring the Coordination Redis tab.
Resolves LIT-4315
The model_info / get_model_info_with_id endpoint tests drove refactored
endpoints with bare, unspec'd MagicMock routers and models. Because the
mocks were unspec'd, any attribute or method the (refactored) endpoints
newly read auto-materialized a child MagicMock, and whether that child
was reached depended on process-global state (premium_user, and the real
get_available_models_for_user chain reading litellm globals) that sibling
tests in the same xdist worker mutate. When reached, the MagicMock either
unpacked to empty (a, b = mock.method() -> 'not enough values to unpack
(expected 2, got 0)') or leaked into RouterModelInfo(**model_info) and
failed Pydantic str validation. Pass in isolation, fail under xdist.
The original TestModelInfoEndpoint failure (#33807 CI) was the same class
surfaced by merge skew: #33721 added a get_configured_token_limits unpack
to create_model_info_response, and CI's merge commit ran that against the
un-updated bare-mock test before the #33742 band-aid landed.
Fix (test-only, no product change):
- TestModelInfoEndpoint: mock the real seam (get_available_models_for_user),
configure the router methods the endpoint actually calls, return a real
Deployment, and drop the dead proxy_server.get_key_models/get_team_models/
get_complete_model_list patches the refactor had stranded.
- TestGetModelInfoWithIdBlocked: spec the model mock so unset enterprise
columns read as None instead of child MagicMocks.
- test_ProxyConfig_get_model_info_with_id_missing_model_id_raises: pin
premium_user so the asserted AttributeError no longer flips with the
ambient license global.
* feat(spend): track prompt compression saved tokens in daily spend aggregates
Native compression interception now records tokens_before/after/saved into the
request litellm_metadata so savings land in the SpendLog metadata JSON under a
typed compression_savings key. A single normalizer
(extract_compression_saved_tokens) sums that key with Headroom guardrail
tokens_saved; the two writers are disjoint and run at different stages, so
summing never double-counts. The spend-log redactor now preserves purely
numeric compression stats inside guardrail_response so Headroom savings
survive the store_prompts_in_spend_logs=false default. compression_saved_tokens
is threaded through BaseDailySpendTransaction, queue aggregation, the daily
upsert blocks, a new BigInt column on all six daily spend tables, and the
daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups)
* fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader
* feat(spend): aggregate compression and prompt caching dollar savings in daily rollups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): update daily spend aggregation fixtures for savings columns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add Cost Optimization dashboard page
New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows.
Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A `lite login` token 429'd with "Budget has been exceeded! Max budget:
0.25" even when no budget was configured anywhere. cli_poll_key stamped
the minted CLI session token with litellm.max_ui_session_budget ($0.25)
as a fallback whenever the user and team had no budget of their own. That
cap was designed for the Admin UI "Test Key" chat pane; the CLI reused
the same session-token machinery, so it inherited a playground-sized
budget baked into the encrypted token at login (unchangeable without
re-login), which trips fast under real CLI/agent use.
The cap is also redundant: the token already carries user_id and team_id,
so the real user/team budgets are enforced independently at request time.
Pass max_budget=None so the CLI token is governed only by those real
budgets, and drop the now-dead user/team budget lookups. The UI login
token's guard (get_experimental_ui_login_jwt_auth_token) is untouched.
* fix(proxy): tell outdated litellm CLIs to upgrade when CLI SSO login id is legacy sk- format
* fix(cli): surface server error detail when SSO login polling fails and stop on permanent 4xx
* fix(cli): exhaustive, actionable error handling across the CLI SSO login flow
get_group_ids_from_service_principal only read the first page of the
Graph API appRoleAssignedTo response, so tenants with more than 100
groups assigned to the enterprise application silently lost group
memberships during SSO login. Loop over @odata.nextLink with the same
MAX_GRAPH_API_PAGES cap that get_user_groups_from_graph_api already
uses, and warn when the cap is hit.
Ported from #32792 by @saisurya237 so CI can run.
Fixes#32790
Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com>
* fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models'
key_type is not persisted on a key (the proxy maps it to allowed_routes and
drops it), so the keys tables only inspected the models list and rendered
'All Proxy Models' for any key with an empty models array, including SCIM,
Management and Read-only keys that cannot call a single model.
Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a
scope tooltip for those recognized scopes; unrestricted, AI-API and custom
keys keep the existing model-list rendering.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): move key_scope helper to components root
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(keys): persist key_type on virtual keys so the UI reads scope directly
Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy,
and proxy-extras schemas plus an additive migration) and stop dropping the
value in handle_key_type, so management/read_only/llm_api/default keys store
their type alongside the derived allowed_routes. Surface it on the key read
and create response models. The dashboard now prefers the persisted key_type
for the no-inference buckets and keeps the allowed_routes derivation as the
fallback for keys created before the column existed (key_type null), so no
backfill is required.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): add key_type column to LiteLLM_DeletedVerificationToken
The deleted-token archive model inherits key_type from the verification
token, so regenerate/delete flows write key_type into
LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration)
so the archive insert does not fail with FieldNotFoundError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Mutation testing surfaced branches in cost_tracking_settings and common_utils that the suite executed but never asserted on. Pin those behaviors with targeted tests: the returned (model, provider) from _resolve_model_for_cost_lookup for deployments carrying a custom_llm_provider and for deployments missing the litellm_params / model_info keys, plus the exact error-response bodies, the caller-identity lookup arguments, and the member and guard branches in common_utils.
* refactor(ui): rename OldTeams component file to Teams
* fix: show and allow editing team model aliases after team creation
* fix(ui): mark team model_aliases as nullable to match the prisma schema
* feat(proxy): add expires filter to GET /key/list
Add an opt-in expires query param to GET /key/list so callers can fetch
only expired or only active keys without paginating every page and
filtering client-side. 'expired' matches keys whose expires is in the
past (NULL expires excluded); 'active' matches keys that never expire or
expire in the future. Omitting the param preserves existing behavior for
every caller. An unrecognized value returns HTTP 400 rather than silently
returning all keys.
The filter is pushed to the database via the existing Prisma where
builder so callers avoid pulling the full key table into application
memory.
Resolves LIT-3387
* refactor(proxy): declare VALID_EXPIRES_FILTER_VALUES before its first use
- config: KeywordTierRule now strips and drops blank/whitespace keywords (a stray
"" makes _keyword_matches match every prompt, silently forcing that tier for all
traffic); still requires at least one real keyword to remain
- frontend build_complexity_router_config: trim keywords and drop rules left empty so
an unfilled "Add keyword rule" row no longer ships a rule the backend rejects with a
400 in the heuristic (non-semantic) flow, where the client-side semantic guard doesn't run
- proxy clear_cache / delete_model: the auto_router/ prefix also covers quality_router/
and adaptive_router/, so pop the model_name from all four router registries (no-op
where absent) instead of only auto/complexity; otherwise a DB quality_router's stale
entry made reload raise "already exists" and abort, and adaptive left a leak
- frontend ComplexityRouterConfig: only render the Keyword Tier Overrides and Semantic
keyword matching sections when their change handlers are provided, so the edit-auto-
router modal (which omits them) no longer shows interactive-but-dead controls
delete_model popped the auto_routers/complexity_routers registries by the deleted
deployment's model_name without checking it was actually an auto_router/* deployment.
Deleting a regular DB model that merely shares a name with a config-defined router
therefore evicted that router, which add_deployment never restores, leaving it
unroutable until a proxy restart. This is the same cross-tenant DoS clear_cache was
hardened against; mirror its auto_router/ prefix guard here.
Extracts _deployment_name_and_model to read model_name and litellm_params.model from
the deployment (delete_deployment returns the raw model_list dict at runtime despite
its Deployment annotation), and adds a regression test asserting a same-named config
router survives deletion of an unrelated regular model.
Add deterministic keyword-to-tier overrides and optional embedding-based
(semantic) keyword matching to the complexity router, and surface both in the
Add Auto Router UI behind a Router Type selector: "Auto-Router v2 [Recommended]"
(complexity tiers + keyword overrides + semantic matching, the default) and
"Semantic Router [to be deprecated]" (the existing utterance-based router,
unchanged). Keyword-to-tier overrides resolve to the highest tier matched
rather than the first keyword matched, so match order no longer affects the
routing decision.
Backend:
- config: KeywordTierRule model plus keyword_tier_rules, semantic_keyword_matching,
embedding_model, and match_threshold on ComplexityRouterConfig, with a validator
requiring an embedding model and rules when semantic matching is on
- complexity_router: evaluate keyword rules before scoring; lexical matches escalate
to the most-severe matched tier (order-independent), and semantic mode reuses
LiteLLMRouterEncoder + SemanticRouter to match paraphrases by cosine similarity,
falling back to the scorer when nothing matches
- model management: clear complexity_routers on cache reload so config edits take effect
Frontend:
- Add Auto Router tab restores the Router Type radio (Auto-Router v2 recommended
by default, Semantic Router still available) and sends keyword_tier_rules plus
the semantic settings on the recommended path, instead of flattening keywords
into custom_technical_keywords
- client-side guard blocks submit when semantic matching is enabled without an
embedding model or without any keyword tier rules, mirroring the backend validator
- moved the "How Classification Works" explainer below Custom Technical Keywords
and above Keyword Tier Overrides
- remove the Test Connection action from the recommended flow, which can't build a
valid pre-save payload for a router (leaves a TODO for a JSON preview / config
test follow-up)
Tests cover lexical escalation, semantic matching via the real library with injected
embeddings, the semantic config guard, config validation, the reload-clear
regression, and the frontend payload builder
Add a RESTful PATCH /team/{team_id} that partially updates a team using RFC 7386 JSON Merge Patch. team_id comes from the path, and metadata is merged with the team's stored metadata instead of being replaced wholesale the way POST /team/update does: an omitted key is preserved, key: null deletes it, and any other value overwrites, recursing into nested objects. Every other field behaves the same as POST /team/update
The handler delegates to the existing update path, so authorization, budget checks, system-managed-key stripping, metadata encryption, cache refresh, and audit logging are shared rather than reimplemented. POST /team/update is untouched, so the change is purely additive
* fix(proxy): match list/dict guardrail_mode in compliance mode checks
* test(compliance): cover ComplianceChecker guardrail_mode shapes (str/list/dict/None)
* fix(proxy): trust only Mode.default in compliance mode matching (ignore tag overrides)
* fix(proxy): match dict guardrail_mode only when every branch runs in mode (no false-compliant)
* fix(proxy): treat multi-mode guardrail_mode as unresolved (no false-compliant)
The list branch previously counted a guardrail configured with mode:
[pre_call, post_call] under every listed mode. But when the writer cannot
infer the concrete hook that fired (apply_guardrail invocations), the raw
list is logged, and an image-only request that only reaches the post-call
path still records both modes. That let a pre_call compliance check pass on
a request that only ran post_call.
Match the tightened dict semantics: a list now counts for mode only when
every listed mode equals mode. Same trade-off (under-report instead of
false-COMPLIANT). Speculative set support is dropped (spend logs are
JSON-serialized, sets do not cross the wire).
Tests updated to reflect the tightened list semantics, deduplicated (single
TestModeMatching class), and shortened. The invariant is now expressed as
a computed check: True implies every branch runs in the matched mode.
---------
Co-authored-by: Marton Schneider <marton@schneider.co.nl>
* fix(proxy): build redis usage cache from REDIS_* env when cache backend is not Redis
Selecting a semantic (or any non-Redis-KV) response cache left
redis_usage_cache unset, silently downgrading cross-pod rate limits,
parallel-request limits, spend coordination, and the pod lock manager
to per-pod in-memory state. Fall back to a standalone RedisCache built
from REDIS_* environment variables, mirroring the existing
use_redis_transaction_buffer escape hatch, which now shares the same
helper.
Resolves LIT-3861
* feat(proxy): configure the coordination redis independently of the response cache
Adds general_settings.coordination_redis, an explicit block for the Redis
the proxy uses for cross-pod rate limits, parallel-request limits, spend
tracking, the pod lock manager, and shared health checks. Resolution order
is the explicit block, then a plain-Redis response-cache backend, then the
REDIS_* environment. Cluster and sentinel targets are supported, and a
cluster target now builds a RedisClusterCache so cluster-aware consumers
take the cluster path.
Admins can configure it from the Caching page of the dashboard via
/coordination_redis/settings, which reports which source is in effect,
redacts credentials on read, and offers a connection test. Settings saved
there are read back at startup so they take effect on restart.
Also fixes redis client construction so an explicitly configured host
outranks REDIS_URL in the environment. Previously the url branch stripped
the caller's host and port, so an explicit block, or a connection test
typed into the dashboard, silently targeted whatever REDIS_URL named
* fix(ui): move coordination_redis_settings into renamed _components directory
---------
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
LiteLLM_MCPUserCredentials stores BYOK API keys in the same column as per-user
OAuth tokens, so the purge on a mint-relevant config change now deletes only
rows whose payload decodes as an OAuth2 credential, each by its
(user_id, server_id) pair, instead of every row for the server. An api_key
server whose url changes purges nothing. delete_mcp_server now also
invalidates each enumerated user's cached token so a re-created server reusing
the id cannot serve tokens minted for the deleted one, and both cache drops
are best-effort
The snapshot read only feeds the stale-token purge decision; leaving it unguarded meant a failed
read would 500 an edit whose update would have succeeded, and it broke
test_edit_mcp_server_redacts_credentials, whose mocked prisma is not awaitable on the un-patched
get_mcp_server path. A failure now logs and skips the purge, consistent with the purge half already
being best-effort. Adds the first endpoint-level coverage of the edit purge wiring: purge on a
mint-relevant change, no purge when the identity is unchanged, and edit success with purge skipped
when the snapshot read raises
When an IdP denies SSO access it redirects back to /sso/callback with
error and error_description query params and no code param. The callback
previously fell through to the provider token exchange, which failed
with a generic "'code' parameter was not found in callback request"
400 that hides the real denial reason. Raise a 401 that surfaces the
IdP's error and description instead.
Ported from #26640 with conflicts resolved against current staging
key_alias can become the secret name used by external secret manager
integrations (HashiCorp Vault, CyberArk Conjur) when store_virtual_keys is
enabled. Add raise_if_unsafe_secret_name, a shared validation check applied
unconditionally before a secret name reaches either integration or the
/key/generate, /key/update, and /key/regenerate API boundary, independent
of the existing enable_key_alias_format_validation opt-in flag.
Also hardens the Vault URL builder to percent-encode reserved characters
in secret_name (preserving "/" and "@"), and switches the Conjur policy
body to a real YAML serializer instead of raw string interpolation.
* fix(mcp): drop the cached per-user OAuth token when the credential row changes
The v2 authorization_code chain Cached(Refreshing(V2PerUserTokenStore)) caches a positive token
until its expires_at (or 300s without one), and CachedOAuthTokenStore.invalidate had no callers,
so a re-authorization or revocation wrote the DB while egress kept serving the replaced token
from the in-process cache until its TTL. LazyPerUserOAuthTokenStore now exposes invalidate,
MCPServerManager threads it to the write side, and the three credential write sites (the OAuth
callback, the Tools-tab persist endpoint, and the revoke endpoint) drop the cache entry after
the row changes. The v2 refresher's own persist stays untouched; RefreshingTokenStore already
feeds the rotated token back into the cache in the same fetch
* test(mcp): pin cache invalidation on the revoke already-gone branch
Greptile's review flagged that only the happy-path delete asserted the invalidate; a refactor
moving the call inside the try block would silently skip the cache drop when the row was
already deleted by a concurrent request while the cache still held the revoked token. The new
test fails on exactly that mutation
* test(mcp): cover invalidate on the redis-backed lazy store path
Codecov flagged the redis fast path of LazyPerUserOAuthTokenStore.invalidate as unexercised;
the existing invalidate tests only ran the no-redis chain. The new test builds the redis chain
via a fetch and asserts a subsequent invalidate reaches the same store instance without a
rebuild
Add a tag_rpm_limit field to virtual keys so each request tag gets its own independent RPM counter on the v3 rate limiter. A key configured with per-tag limits tracks each tag/group separately, and requests whose tag has no configured limit fall back to the key-level limit. Includes the dashboard UI to manage per-tag limits on key create and edit.
Resolves LIT-3147
* feat(mcp): let users select the entra_obo token_exchange profile in the UI and API
The backend token_exchange arm supports two wire dialects via token_exchange_profile
("rfc8693" default, or "entra_obo" for Microsoft Entra's On-Behalf-Of, the RFC 7523
jwt-bearer grant), but it could only be set through config.yaml. This surfaces it to the
create/update REST API and the dashboard so an admin can create an entra_obo server there,
completing the parity started in the parent PR for the other token-exchange fields.
token_exchange_profile becomes a dedicated column on LiteLLM_MCPServerTable, mirroring the
sibling fields: it is added to the request models, read column-first in
build_mcp_server_from_table with the credentials-blob as a back-compat fallback and a
default of rfc8693, and carried through both runtime-to-table builders so registry
round-trips preserve it. It is a non-secret dialect selector, so it is not scrubbed from
non-admin or virtual-key responses.
In the dashboard a Profile dropdown (RFC 8693 vs Microsoft Entra OBO) is added to the
token-exchange section. Entra OBO carries the target resource in the scope, so selecting it
makes the scope required and hints the api://<app-id>/.default form, while audience and
subject_token_type (which that dialect ignores) are hidden.
* fix(mcp): extend the blob-to-column lift and non-admin scrubbing to token_exchange_profile
token_exchange_profile gets the same storage contract as the other three
token-exchange settings: the column is authoritative, a blob copy is the legacy
shape — lifted into the column on every write and stripped from the stored
blob — and switching auth_type away from token exchange clears it
(_AUTH_FLOW_SCOPED_FIELDS). Both restricted-view sanitizers scrub it for
uniformity, and the edit form's auth-switch payload nulling includes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(mcp): assert every token-exchange setting is configurable via config.yaml
Pins the config surface: token_exchange_endpoint, audience, subject_token_type
and token_exchange_profile load from top-level config keys onto the built
server and through to the resolver spec; omitted keys resolve to their
documented defaults (RFC 8693 subject token type, rfc8693 profile), and
token_exchange servers need no oauth2_flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard
OAuth 2.0 Token Exchange (RFC 8693, a.k.a. OBO) for MCP servers could previously only be
configured through config.yaml; the create/update REST API and the dashboard had no way to
express it. This wires token_exchange_endpoint, audience, and subject_token_type end to end.
These three are persisted as dedicated columns on LiteLLM_MCPServerTable, mirroring how token_url
and oauth2_flow are stored, so the edit form prefills them and they are unaffected by the
credentials-blob clearing on auth_type change. build_mcp_server_from_table reads the columns first
and falls back to the credentials blob so servers persisted before the columns existed still load.
client_id and client_secret continue to ride the existing encrypted credentials path.
On the dashboard, "OAuth Token Exchange (OBO)" is a distinct auth-type option with its own field
section. The McpOAuthMode classifier gains a token_exchange arm keyed off auth_type; the previous
catch-all oauth2 mode was renamed from "obo" to "authorization_code" so the two on-behalf-of
mechanisms are no longer conflated. The token-exchange IdP endpoint and audience are scrubbed from
non-admin and virtual-key responses, matching how token_url is treated.
* fix(ui): gate the MCP list-401 re-auth on authorization_code, not token_exchange
The renamed 401 gate keyed off isTokenExchange, but that condition exists for authorization_code:
when a stored per-user credential is present yet the tools/list 401s (the backend's refresh could
not mint a token), the user must re-authorize via the browser flow. token_exchange has no
gateway-side authorize step, so the Authorize gate never applied to it. The prior isObo flag was
undefined (a compile error) and, per this file's convention and its tests, meant authorization_code;
renaming it to isTokenExchange changed the behavior and broke the mcp_tools auth-gate test for an
authorization_code server whose token expired with no usable refresh. Gate on isAuthorizationCode
instead and drop the now-unused isTokenExchange
* fix(mcp): clear flow-scoped endpoint config when a server's auth_type changes
Switching an existing oauth2 server to oauth2_token_exchange left the old flow's
token_url on the row. The OBO resolver treats token_exchange_endpoint or token_url
as the configured exchange endpoint, so the stale value both suppressed the RFC
9728/8414 discovery this PR adds and sent the exchange grant (client credentials
plus the user's subject token) to the previous flow's token endpoint
update_mcp_server now mirrors its existing stale-credentials rule for the
flow-scoped columns (authorization_url, token_url, registration_url, oauth2_flow,
token_exchange_endpoint, audience, subject_token_type): when auth_type changes,
each one is cleared unless the same request explicitly provides it, so a
deliberate override in the switch request still wins. Updates that keep the
auth_type never touch these columns, which keeps legacy OBO rows that use
token_url as their exchange endpoint working
The edit form sends explicit nulls for the previous flow's fields on an auth type
switch; antd preserves unmounted field values by default, so without this the old
token_url would be re-sent verbatim and read as an explicit override. Transitions
are detected against the persisted auth_type, so saves that keep the auth type
send nothing extra
Reported by Cursor Bugbot on the PR
* fix(mcp): lift legacy blob token-exchange settings into their columns on every write
The three token-exchange settings live in dedicated columns but also exist on
MCPCredentials as the pre-column REST shape. Writes now lift incoming blob
values into the columns (an explicit top-level value wins, including an
explicit null) and strip them from the stored blob; the same-auth credentials
merge migrates legacy rows the same way. The read-time column-or-blob fallback
then only ever serves rows current code has never written, so clearing a column
to re-enable RFC 9728/8414 discovery can no longer be silently undone by a
stale blob copy.
Also asserts the auth-switch clearing fires on the external fields_set path
(PUT /v1/mcp/server).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(mcp): single source for the RFC 8693 default subject_token_type
The default was applied at four egress build sites plus two model defaults,
each with its own copy of the literal. All sites now share
DEFAULT_SUBJECT_TOKEN_TYPE from litellm.types.mcp. A DB-level DEFAULT is
deliberately not used: Prisma writes explicit values on insert, so a column
default would rarely apply, and NULL-means-RFC-default keeps existing rows
correct.
Also documents two review decisions in place: the audience column keeps the
RFC 8693 parameter name (RFC 8707 resource indicators are already a separate
concept named resource in the v2 egress types), and the migration's
out-of-order timestamp is safe under prisma migrate deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: fix import sort order in outbound_credentials/types.py (I001 strict budget)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): purge legacy blob copies when a token-exchange column is written without credentials
The migrate-on-write in the credentials merge lifts blob values into null
columns, which is correct for legacy rows but could repopulate a column an
admin had cleared in an earlier no-credentials update (that path never touched
the blob, so the stale copy survived to be lifted later). An explicit
token-exchange column write (set or clear) now migrates the row even when the
update carries no credentials: untouched null columns are lifted, every blob
copy is stripped, and unrelated blob keys stay as-is. A cleared column can then
never be resurrected, because no write path leaves a blob copy behind.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(mcp): state the blob-to-column lift contract on the legacy credential keys
The three token-exchange keys on MCPCredentials are the pre-column REST shape
(the only REST shape from 2026-05 until this PR). Document on both the blob
type and the request models that the dedicated columns are authoritative and
that writes lift blob values into them and strip the stored copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): scrub subject_token_type in the non-admin and virtual-key sanitizers
The other two token-exchange fields were cleared while subject_token_type was
left visible. It is a public RFC 8693 URN with no disclosure value, but the
sanitizers' rule is that these views receive no token-exchange config at all —
cleared for uniformity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
UI session tokens carry the virtual team_id litellm-dashboard (UI_TEAM_ID),
which is never persisted. The MCP team-permission helpers passed it to
get_team_object anyway, so every dashboard MCP listing raised a 404 per
lookup that was swallowed into per-server 'Failed to get allowed tools for
server' warnings (plus the sibling 'allowed MCP servers for team' and 'MCP
access groups for team' warnings) and wasted DB queries. The 404 also
escaped past the key-level permission handling in
get_allowed_tools_for_server, dropping key tool restrictions for such
sessions.
Short-circuit the virtual team before the DB lookup in the three helpers,
mirroring the existing UI_TEAM_ID handling in agent_permission_handler.
Also reject /team/new with the reserved team_id, since a real row would
bind its budget and permissions to every UI session
Add an opt-in mode so a key that exceeds its own max_budget is throttled to a
globally configured percentage of its TPM/RPM instead of being blocked entirely.
A new litellm_settings global, budget_exceeded_throttle_percentage, sets the
fraction (e.g. 0.1 = 10%). A per-key throttle_on_budget_exceeded flag (stored in
key metadata via the existing management-endpoint metadata routing) opts the key
in. When both are set and the key is over budget, the budget check records the
percentage on a request-scoped budget_throttle_pct instead of raising, and the
rate limiter scales the key's configured TPM/RPM by it. Keys without the flag
keep hard-blocking; team/user/org budgets are unaffected.
The throttle is recomputed from the key's original limits on every request and
the decision is cleared before the auth object is cached, so it never compounds
across requests. Both the budget read-time check and the budget reservation path
honor the opt-in, and both the v3 and legacy rate limiters apply the scaling.
Enabling throttle_on_budget_exceeded is proxy-admin only. It converts an
admin-imposed hard budget block into a soft throttle that keeps spending past
max_budget, so a non-admin must not be able to self-opt-in and bypass their own
spend cap. Both /key/generate and /key/update reject a non-admin setting it to
true (update only gates the transition to enabled, so a non-admin can still edit
other fields and turn the flag off). This matches the feature being wholly
proxy-admin operated: the global percentage is admin-only too.
A key that opts in but has no TPM or RPM limit has nothing to scale, so it stays
hard-blocked rather than serving unlimited requests past its budget (fail-safe).
The global budget_exceeded_throttle_percentage is configurable from the admin UI
(Settings -> General Settings), persisted through litellm_settings so it survives
a restart, not only from config.yaml.
Resolves LIT-3894. Scope for LIT-3893.
