litellm already supports Google, Microsoft and generic OIDC SSO through
fastapi-sso, which has no SAML support; AuthMethod.SAML existed only as an
unused enum value. This adds real SAML 2.0 single sign-on for the admin UI.
A new SAMLAuthHandler validates signed assertions with the OneLogin
python3-saml toolkit and maps them onto a CustomOpenID, then reuses the
shared post-login path every other provider goes through, so provisioning,
role/team mapping and the UI session JWT are unchanged. Both SP-initiated
and IdP-initiated HTTP-POST flows are supported. SP-initiated logins are
bound to the browser that started them via an HttpOnly state cookie plus a
cached AuthnRequest id, and the ACS rejects any response whose InResponseTo
doesn't match; unsolicited (IdP-initiated) responses cannot be browser-bound
so they are rejected unless SAML_ALLOW_UNSOLICITED=true. Replays are rejected
by a consumed-assertion guard whose lifetime tracks each assertion's
NotOnOrAfter, and both the replay guard and the login-state binding go
through the proxy's shared in-memory + Redis cache for multi-instance
deployments. The ACS honors DISABLE_ADMIN_UI and re-applies the
free-SSO-user Enterprise gate after the assertion is validated, so an
unvalidated POST can no longer drive the billable-user count query.
SAML is configurable from the admin UI SSO settings (IdP metadata URL or
inline XML, SP entity ID, and an allow-unsolicited toggle), which persists
the SAML_* environment variables the handler reads, exactly like the Google,
Microsoft and generic OIDC providers.
python3-saml is kept as an optional saml extra; its xmlsec and lxml wheels
bundle the native libraries so no system packages are required, and the
import is guarded so the proxy still starts without the package with the
SAML routes returning a clear 501.
Resolves LIT-4016
Adds GET /v1/tool/spend returning per-tool and daily tool spend with a
deduplicated request total, and a cache leakage breakdown on the Prompt
Caching tab of the Cost Optimization page. Tool-spend rows are validated at
the boundary with pydantic, the endpoint is scoped to proxy admins, date
params are cast to timestamptz for real-Postgres query_raw, and the leakage
math treats litellm-normalized prompt_tokens as cache-inclusive
(uncached = max(0, prompt - cache_read - cache_creation)).
Admits a keyless SSO user (no virtual key) at the aggregate /mcp endpoint from a gateway DCR
session bearer, resolving team/org/SCIM/budget authorization fresh on every call.
- Aggregate DCR front door: stateless /register (sealed llm_dcrc_ client ids), SSO-backed
/authorize + /authorize/complete, and /token minting identity-only session tokens with PKCE,
single-use codes/flows, and rotating refresh tokens.
- Admission: a session-shaped Authorization at the aggregate scope opens via _admit_gateway_session,
reloads the live user, and runs the centralized policy gate; failures return the RFC 9728
invalid_token challenge. Gated on the un-forgeable, server-only mcp_admitted_user_subject marker,
so virtual-key and JWT auth are unchanged.
- Authorization model: an admitted subject is resolved as one plain UserAPIKeyAuth per grant source
(its own grants, plus each team it is a live roster member of), each answered by the SAME resolver
virtual keys use, then unioned. That branch is the FIRST statement of BOTH public resolvers, so no
single-credential prelude runs for it and a fault in a lookup it never uses cannot deny its grants. A source team counts only while it is a live grantor: roster membership, not
blocked, and neither the team nor its owning org over budget (enforced through the SAME
_team_max_budget_check / _organization_max_budget_check owners common_checks uses for keys).
Each team source carries that team's own org, so the existing org
ceiling caps it; for a keyless source the org list only ever intersects (a ceiling must not become
a grant) and an unresolvable ceiling denies rather than silently uncapping, on both the server and
tool axes. _roster_team_object is the single owner of "which teams count": a team whose roster no
longer lists the user neither grants servers nor throttles, in one place.
- Rate limits: the subject is bounded by its user rpm/tpm AND by the per-server mcp_rpm_limit of
the team a call is ATTRIBUTED to — the same single source billing charges, from the same owner. A key charges its one pinned team's bucket; a keyless
subject has no team_id, so admission stamps each granting team's limit map onto the auth
(server-only field, stripped from validated input like the marker) and the limiter emits that
team's mcp_per_team descriptor. Charging every granting team instead would let one cross-team user
drain several teams' SHARED buckets on a single call and block their other members; and a server
the user's OWN grant reaches charges no team bucket at all, because no team provided it. Per-KEY
MCP limits do not apply because there is no key.
- Wrapper channels: the manager-level union treats the admitted subject by the same grant model.
The admin-role short-circuit and the absolute no_mcp_servers early-return are key-credential
rules and never apply to it (a session bearer is a third-party client credential, not the
dashboard, and the subject's opt-out silences only its own source). Operator-open channels
(allow_all_keys, the user's own BYOM submissions) are owned by one operator_open_server_ids
helper that BOTH the server union and the admitted tool resolution consult (suppress-BYOM-when-
explicitly-scoped is a key-credential rule and never applies to the subject, whose user row
carries the DB-default empty mcp_servers), so an open-channel
server is default-open for tools instead of listable but uninvokable.
- Redirect URIs: one owner, validate_redirect_uri_shape, decides redirect-URI hygiene (bad scheme,
fragment, missing host, userinfo, backslash host) and resolves allowlisted native callbacks, shared
by DCR registration and the OAuth endpoints. Registration keeps a deliberately wider trust policy
than validate_trusted_redirect_uri: public dynamic registration accepts any https client, and its
controls are mandatory S256 PKCE plus the consent screen.
- Egress leak-defense: a gateway admission credential (session bearer / bridge envelope) is scrubbed
from EVERY egress header context, anchored to the credential shape, so it can never be forwarded
upstream and replayed.
- Single-use guard: auth-code, refresh and connect-flow claims resolve the proxy's cross-worker redis
cache themselves rather than trusting the cache passed in, and fail CLOSED on a Redis fault instead
of falling back to a per-worker count that a captured id could replay through another worker.
- Sign-in return_to: one shared, never-raising helper persists a safe return_to for every sign-in
branch (SSO/Okta/generic and username/password), and every branch RESUMES through the same
_sso_return_to_redirect the SSO callback uses, so however a deployment signs in the stored value
is honored identically (same-origin path directly; control_plane_url via the one-time login-code
handoff). A stale cookie is ignored rather than failing a completed sign-in.
- Budgets, both halves: ENFORCEMENT (an already over-budget team or its owning org stops being a
grantor, in the source gate) and ACCOUNTING (a team-derived tool call is billed to the granting
team and ITS org, so that budget accumulates and the right organization is charged). A server the
user's own grant reaches bills the user; when several teams grant one server the pick is the
lowest team_id, stable and auditable. Billing rides a COPY, so authorization still sees the full
union, and it is inert when the target server cannot be resolved from the tool name.
Deferred (tracked): client-selected server scoping of the session token (LIT-4680).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(organization): persist cleared fields on /organization/update
Clearing an org field (the Metadata box or a TPM/RPM/max_budget limit) via PATCH /organization/update looked like it saved but reverted on refresh; the partial-update merge could not tell a cleared field from an untouched one and dropped every clear
The endpoint now decides SET vs CLEAR vs UNTOUCHED purely from which keys the raw request body carried, via a pure build_organization_update_plan. Budget nulls flow to update_budget (null clears via exclude_unset), metadata is replace-when-sent (written as {} for the non-nullable Json column), and a budget write on an org with no budget_id creates and links a budget row. This removes the exclude_none dump, both "if v is not None" filters, and the additive _update_dictionary merge
Resolves LIT-3664
* feat(organization): add RESTful PATCH /v2/organization/{organization_id}
Adds a v2 organization-update endpoint with a deterministic partial-update contract, and reverts the v1 /organization/update changes so its public behavior stays untouched
On v2 a field present in the request body is written (null/[]/{} clears, a value sets) and an omitted field is left untouched; presence is read from model_fields_set. Clearing a TPM/RPM/max_budget limit or the metadata now persists instead of being dropped as if it were never sent. Metadata is replace-when-sent and written as {} when cleared, since the org metadata Json column is non-nullable. Budget nulls flow to update_budget, and an org with no budget row gets one created and linked. The endpoint is hidden from the public Swagger docs via include_in_schema=False, and stays typed in the generated dashboard schema
Resolves LIT-3664
* test(organization): cover v2 auth guard, negative budget, and object_permission
Adds v2 endpoint tests that were missing: the real _verify_org_access path rejects a non-admin caller with 403 and writes nothing, a negative max_budget is rejected with 400 before any DB access, and a sent object_permission is passed to the upsert helper with its id linked onto the org write
Refs LIT-3664
* fix(organization): 400 on null-clear of required org fields; drop dead budget upsert
organization_alias and models are non-nullable columns, so a v2 request clearing them with null hit a 500 (NOT NULL violation) and could partially apply the budget half of the request first; the endpoint now returns a 400 with a clear message. Also removes the unreachable "create a budget when the org has none" branch from _apply_organization_budget_updates, since budget_id is a non-nullable FK and every org already has one, so the endpoint no longer needs to link a newly-created budget id
Refs LIT-3664
* fix(organization): let v2 clear object permissions when sent as null
Sending object_permission: null now detaches the org's permission by setting the nullable object_permission_id to null, instead of being a silent no-op, so the endpoint honors its documented "null clears" contract and an admin can actually revoke vector-store/MCP access. Sending a value still merges as before
Refs LIT-3664
* fix(organization): make v2 PATCH atomic, strict, and 422-consistent
Tighten the PATCH /v2/organization/{id} endpoint against standard HTTP
PATCH (RFC 5789 / RFC 7396 JSON Merge Patch) semantics:
- Apply the budget-row and org-row writes in one prisma transaction so a
failure between them can no longer half-apply the patch (RFC 5789 requires
a PATCH to apply atomically). The budget write is inlined as a tx-aware
call mirroring the team-member budget path rather than the standalone
update_budget route handler
- Set extra="forbid" on OrganizationUpdateRequestV2 so an unknown or
misspelled key is a 422 instead of a silently dropped no-op; the contract
is presence-driven, so swallowing unknown keys is unsafe
- Return 422 (not 400) for the hand-rolled field validations (negative
budgets, null-clear of required organization_alias/models, invalid
model_max_budget) so every validation failure matches the 422 that
pydantic already returns for bad values
- Document the per-field clear tokens accurately: null clears budget limits
and metadata, [] clears models, and organization_alias cannot be cleared
Tests cover the single-transaction write path, unknown-field rejection, the
422 status changes, and the budget_reset_at recompute.
* fix(organization): reject empty object_permission on v2 PATCH instead of silently keeping grants
object_permission is a nested merge field on PATCH /v2/organization/{id}: a
sent object merges into the existing permission row (updating one grant list
without touching the others), and null detaches it. An empty {} therefore
merged nothing and left every existing vector-store/MCP grant in place, so an
admin who sent {"object_permission": {}} to strip access silently kept it.
Reject a present-but-empty object_permission with a 422 that points the caller
at null, mirroring how the endpoint already rejects a null clear of the
required organization_alias/models. This keeps merge semantics for non-empty
payloads and does not affect the Admin UI, which only ever sends a fully
populated object or omits the field.
* fix(organization): JSON-serialize model_max_budget on the v2 budget write
model_max_budget is a Json column on the budget table. Route the budget-row
write through jsonify_object so a dict value is serialized the same way
new_budget and the org-row metadata write already do it, keeping every Json
column on this endpoint written consistently.
Raw dicts already round-trip (update_budget writes them unserialized), so this
is not a correctness fix so much as making the one Json column on the budget
path follow the same serialization as the rest of the file. Added a test that
a patched model_max_budget reaches the budget write JSON-serialized.
* refactor(organization): trim v2 docstrings and consolidate planner tests
Trim the verbose docstrings on the v2 endpoint, request model, and the two
pure helpers to the essential contract, and drop a stale line that still
referenced update_budget's exclude_unset (the budget write is inlined now).
Collapse the nine per-case planner tests into one parametrized test asserting
exact budget/org split per body, and fold the two model-validation rejection
cases into one parametrized test. Same 36 test cases run; the planner
assertions get stronger (exact-equality instead of presence/absence) and the
test additions shrink by ~85 lines.
* refactor(organization): inline the v2 update planner into the endpoint
Fold the OrganizationUpdatePlan dataclass and build_organization_update_plan
helper into update_organization_v2. The budget-vs-org split is a few dict
comprehensions built in one shot, so the extra type plus builder was more
ceremony than the job needed. Drops the now-unused dataclass/AbstractSet
imports and the isolated planner unit tests; the split is exercised end-to-end
by the endpoint tests.
* fix(organization): run v2 object permission upsert inside the update transaction
prepare_object_permission_upsert splits the shared helper's read-and-merge
step from its write so the v2 endpoint can upsert the permission row on the
same prisma transaction as the budget and org writes. Previously the upsert
ran before the transaction, so a rolled-back org write left merged grants
live on the permission row the org still pointed at. The upsert record now
pins object_permission_id, since the column's @default(uuid()) would
otherwise mint a fresh-create id different from the one linked on the org.
v1 and the team/key callers of handle_update_object_permission_common keep
their existing behavior
* fix(lint): keep the v2 org PR within the strict-rule budget
The strict gate flagged the PR's new code after the base merge: 11 UP045
Optional fields and a typing.List on OrganizationUpdateRequestV2, Dict
annotations in the new upsert helper and the TypeAdapter, and a B008 from
the v2 endpoint's Depends default. The model and helper now use pipe
unions and builtin generics, and the endpoint takes its auth dependency
via Annotated, which avoids the call-in-default pattern B008 targets
* fix(routes): expose /v2/organization on the backend component allowlist
The component-split coverage test requires every app route on a component;
the new v2 org PATCH belongs with the other management endpoints on the
backend, alongside the existing /v2/key and /v2/team prefixes
* fix(organization): clear budget_reset_at when budget_duration is cleared via v2 PATCH
Resolves LIT-4581
A true_passthrough MCP server created without the at-creation auth step
has no stored client_id, and the tools-page browser flow supplies none,
so GET /v1/mcp/server/oauth/{id}/authorize dead-ended on a 400
missing_client_id. The client-forwarded-token modes forbid the gateway
from persisting an OAuth client, so client acquisition moves into the one
chokepoint every caller crosses: the authorize endpoint.
resolve_ephemeral_dcr_client owns the whole mint policy (mode gate,
authorization-url precondition, required S256 PKCE, redirect trust, then
a TTL-deduped, per-server single-flighted RFC 7591 mint). The minted
client rides the encrypted OAuth state; /callback seals it with the
upstream code and server_id into an llm_ptcode_ gateway code, and
redeem_passthrough_authorization_code recovers it at the token endpoint
(server binding plus required code_verifier) to authenticate the upstream
exchange. Nothing is persisted; every value rides the encrypted blobs, so
it works across replicas.
Client acquisition is one predicate applied across the whole auth-mode
matrix: the gateway mints for a clientless authorize iff true_passthrough
(any dcr_bridge) or oauth_delegate-and-not-dcr_bridge, and the UI
gatewayMintsClientFor mirrors that set exactly so the browser pre-registers
a client through the dcr_bridge front door only for the cells the gateway
does not mint (the interactive oauth_delegate dcr_bridge sign-in and the
legacy oauth2 passthrough). A minted flow runs the bridge short-circuit
arm; the relay front door stays for external clients that present their
own client_id. Both sides are pinned against the same truth table
(test_resolve_ephemeral_dcr_client_mint_set_is_exact and the
gatewayMintsClientFor matrix test) so no mode can silently diverge. The
authorization_code hook and M2M/token-exchange modes are unchanged.
* feat(ui): add react-hook-form + zod form infrastructure
Introduce the shared form layer the dashboard's antd forms will migrate onto,
with no user-visible change yet.
- pin react-hook-form, @hookform/resolvers, and zod (kept on 3.25.76 and
imported via the zod/v4 entrypoint so openai's optional zod ^3 peer still
resolves and npm ci stays clean)
- vendor the base-vega Field family into components/shared/form as forwardRef
components on the repo's cva.config, since base-vega ships no form primitive
and its field source imports class-variance-authority and is React 19 style
- add a FormField bridge that binds a react-hook-form Controller to the Field
layer and wires label, description, and error ids into aria attributes
- add pickDirty, which narrows a submitted body to the top-level keys the user
actually touched so a partial update stops re-sending untouched fields
pickDirty reads dirtiness at the top level because react-hook-form tracks it
per leaf, so an edited array arrives as [true, false] and a cleared list as an
empty array that still carries its default-length dirty markers; the falsy
clear tokens (null, [], {}, 0, false) all survive.
Tests cover the Field primitives, the FormField aria wiring against a live
zod resolver, and pickDirty both as a unit and driven through a real
react-hook-form instance.
* test(ui): lock pickDirty behavior on a pure field-array reorder
react-hook-form compares each array element to its default positionally by
value, so useFieldArray move/swap and a reordered scalar array all mark the
moved indices dirty and pickDirty sends the whole array; a swap of two equal
elements is a value-level no-op and is correctly omitted. Covers the reorder
case a review flagged as untested.
* feat(proxy): publish a typed request body for PATCH /team/{team_id}
The route validated its body into UpdateTeamRequest but read it off the raw
request, so the OpenAPI spec carried no requestBody and the dashboard's
generated client could not type the call at all.
- add PatchTeamRequest, UpdateTeamRequest with an optional team_id, since PATCH
takes the id from the path; a body team_id is still accepted when it matches
- validate the body through PatchTeamRequest before delegating to update_team
- declare the request body on the route and regenerate schema.d.ts
The handler keeps reading the raw body rather than declaring a typed parameter.
FastAPI validates a declared body before the handler runs, which would replace
the 400 for a non-object body with a 422 and move absent-vs-null out of reach of
the RFC 7386 metadata merge; those are pinned by existing tests, so the schema is
declared on the route instead and every error path is unchanged.
Validation is shape-preserving: the body is dumped with exclude_unset so an
omitted field never reaches the write, an explicit null still clears, and a
partial object_permission does not gain sibling sub-keys, which would wipe them
given the column merges rather than replaces.
Tests extend the existing patch harness rather than replacing it.
* refactor(proxy): declare the PATCH /team/{team_id} body as a typed parameter
Replaces the hand-written OpenAPI declaration added earlier in this branch. The
route now takes data: PatchTeamRequest, so FastAPI generates the request body
itself and emits a $ref to the model instead of an inlined copy that would go
stale as fields are added.
The earlier approach was a workaround built on a wrong premise. Declaring the
body does not cost absent-vs-null: model_fields_set preserves it, which is how
POST /team/update already gets its tri-state, and a nested null inside metadata
survives validation untouched, so the RFC 7386 merge is unaffected.
The one real change is the status code for a malformed body. The route answered
400 for a non-object body and 500 for a wrongly typed field, reporting a caller
mistake as a server fault; both are now 422, matching POST /team/update and the
other typed management endpoints. The two tests that pinned the old parse-level
errors are replaced by one that pins the 422 through the ASGI stack, and the
handler drops its manual parsing entirely.
Okta commonly sends SCIM membership removals as a filtered path with no request
body value, e.g. Groups PATCH members[value eq "uid"] and Users PATCH
groups[value eq "tid"]. The patch handlers pulled ids only from op.value, so
these removes were a silent no-op and the member or team was never dropped
Add a linear-time filter parser reused by both the Groups members path and the
Users groups path so the id is taken from the [value eq "..."] filter when
op.value is absent, for add and remove ops. The eq operator is matched
case-insensitively, both quote styles are accepted, and the quoted value is
unescaped. The path-filter fallback only fires when the request body value is
omitted, so an explicit empty value no longer resurrects the filter id, and the
compared value must be quoted per the SCIM filter grammar
_add_team_members_to_team reconciled membership by reading the complete_team_data
snapshot captured at the start of team_member_add, appending in memory, and
writing the whole members_with_roles array back. Two concurrent /team/member_add
calls for the same team read the same snapshot, so the last write wins and one
member is silently lost. This affects every concurrent team member add, including
the SCIM group PATCH op:add path that routes through team_member_add
Reconcile members_with_roles inside a transaction that locks the team row with
SELECT ... FOR UPDATE before re-reading the current membership, so concurrent
writers serialize on the row lock and each appends onto the other's committed
result. The interactive transaction is exposed through a thin PrismaClient.tx()
passthrough and the locked read is encapsulated in
TeamRepository.get_members_with_roles_locked, and the SCIM group PATCH applies
membership as deltas so concurrent adds are not clobbered
SCIM delete_user removed the user from the legacy team.members column and deleted
their team membership rows, but never pruned members_with_roles, which is the
source of truth ScimTransformations reads for GET /Groups/{id}. A deleted user
therefore lingered as a dangling member reference on every team they belonged to
Prune each of the user's teams directly via team_member_delete before deleting
the user row, and only for teams whose members_with_roles actually contain the
user, so a real DB failure surfaces (the endpoint fails loudly and the user is
kept; SCIM DELETE is idempotent, so the IdP retries) while a user who was never
in a team's members_with_roles stays a no-op. patch_team_membership is left
unchanged for its other callers
* fix(scim): sync team roster and dedup teams for existing-user email upsert
When POST /scim/v2/Users matched an already-existing user by email,
handle_existing_user_by_email raw-wrote the user's teams array but never
touched the team roster, so the user appeared in the group on their profile
yet was absent from the team directly (members_with_roles stayed empty). It
also did not dedup the teams built from repeated SCIM groups.
Route the existing-user team assignment through the same
_handle_team_membership_changes / team_member_add path the PUT update_user
handler uses, so members_with_roles, LiteLLM_TeamMembership, and the user's
teams array stay in sync, and dedup the teams derived from user.groups. The
user_id rewrite to the new userName is preserved and sequenced before the
roster sync so the roster never references a stale primary key.
* fix(scim): surface roster add failures on existing-user email upsert
Route the existing-email upsert's roster sync through patch_team_membership
with a new opt-in raise_on_error flag so a genuine team_member_add failure
propagates instead of being swallowed, and the deduped teams array is only
persisted after the roster sync succeeds. Without this, a failed add left the
endpoint reporting success while user.teams listed a team members_with_roles
never received.
The benign already-a-member case stays a no-op even under the strict path, and
the flag defaults to False so the PUT update_user, PATCH patch_user, and group
callers keep their existing best-effort behavior. SCIM POST is idempotent, so
surfacing the error lets the IdP retry and converge.
* fix(scim): surface roster removal failures symmetrically with adds
Make team_member_delete failures fail loud under the strict roster sync used by
the existing-email upsert, mirroring the add path, so a swallowed removal can no
longer let the user's teams array drop a team the roster still holds. The
idempotent case where the user is already absent from the team stays a no-op,
matching how an add treats the user already being in the team. Best-effort
behavior is preserved for the default raise_on_error=False callers.
* fix(scim): use members_with_roles as the source of truth for group membership
SCIM group provisioning tracked membership inconsistently. Team creation and
the real team endpoints persist membership in members_with_roles (and each
member's user.teams), but the SCIM group PATCH handler and the GET /Groups
listing read the legacy team.members String[] column, which team creation never
populates. Seeding a PATCH result from that empty column made an Okta "add
member" operation recompute the member set from scratch and silently drop
everyone already in the team, so users ended up missing from the groups they
were provisioned into. Reading the same empty column on GET /Groups reported an
empty member list back to the IdP, which drove repeated re-provisioning.
Separately, add_new_member appended the team id to user.teams with an
unconditional array push. Under the concurrent group PATCHes an IdP sends during
a reconcile, each request passed the members_with_roles duplicate check and
pushed, so user.teams accumulated duplicate ids for the same team. A duplicate
also breaks auth logic that keys off the number of teams a user belongs to.
Read current membership from members_with_roles in the SCIM group PATCH seed and
the GET /Groups listing, and make the user.teams append idempotent via a
filtered update that no-ops once the team is present.
Resolves LIT-4283
* fix(scim): address review; atomic user-creation and stop writing legacy members
Keep the concurrent-safe team append but create the user via an atomic upsert
(create-or-update) instead of a check-then-create, so provisioning the same new
user concurrently cannot race into a duplicate-key failure; the team is still
appended idempotently by a filtered update so an existing user gets no duplicate
team id. Stop writing the legacy team.members column in the group PATCH apply so
the only membership record is the source of truth (members_with_roles plus each
member's user.teams), reconciled by team_member_add/team_member_delete.
Tests: existing add_new_member and team-creation mocks updated to the upsert
plus filtered-append shape, and new tests cover atomic creation and that the
PATCH apply does not write the legacy members column.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
* fix(proxy): share CLI SSO login sessions across workers without enable_redis_auth_cache
* fix(proxy): make CLI SSO flow state redis-authoritative across workers
The CLI SSO flow is stored in a DualCache whose get_cache is memory-first, so
the worker that served /sso/cli/start keeps serving its stale in-memory flow and
never observes the sso_complete/session_data update another worker writes during
the OAuth callback. Attaching Redis alone is not enough; poll on the original
worker returns pending forever.
Read and write the flow directly through the attached Redis backend when present
so every worker sees the same authoritative state, falling back to the in-memory
DualCache only when no Redis is configured.
* fix(proxy): serialize CLI SSO flow as JSON for the redis round trip
RedisCache stores values via str(value) and parses reads with
json.loads then ast.literal_eval. The completed flow contains a
LitellmUserRoles enum in session_data.user_role, whose repr is not a
parseable literal, so any worker reading the completed flow from redis
raised SyntaxError and returned 400 "CLI login session not found".
Writing the flow as json.dumps makes the round trip lossless (the enum
is a str subclass) and fails loudly at write time if a non-serializable
value is ever added to the flow.
* fix(proxy): point CLI SSO session-not-found hint at configuring Redis
The error message and warning still told users to set enable_redis_auth_cache,
but the CLI SSO session cache now gets Redis unconditionally whenever one is
configured, so that flag no longer affects CLI login
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
The Cache Settings page read only the database row, so a response cache
pointed at Redis purely through REDIS_* env vars showed a blank page
while the cache worked. It also masked credentials on read with a
partial-reveal string and re-persisted whatever the form submitted, so
an admin who edited an unrelated field and pressed Save wrote the mask
string over the real Redis password, breaking auth.
GET /cache/settings now overlays the same REDIS_* kwargs the runtime
resolves from when the stored config leaves a field unset, and redacts
credentials with a fixed marker. POST /cache/settings restores the
stored secret behind any credential echoed back as the marker or omitted,
and drops an env-sourced marker rather than persisting it; the response
no longer echoes plaintext credentials. The connection test resolves a
redacted credential back to the stored value the same way. The dashboard
never prefills a credential and drops the marker from the save payload,
mirroring the Coordination Redis tab.
Resolves LIT-4315
The model_info / get_model_info_with_id endpoint tests drove refactored
endpoints with bare, unspec'd MagicMock routers and models. Because the
mocks were unspec'd, any attribute or method the (refactored) endpoints
newly read auto-materialized a child MagicMock, and whether that child
was reached depended on process-global state (premium_user, and the real
get_available_models_for_user chain reading litellm globals) that sibling
tests in the same xdist worker mutate. When reached, the MagicMock either
unpacked to empty (a, b = mock.method() -> 'not enough values to unpack
(expected 2, got 0)') or leaked into RouterModelInfo(**model_info) and
failed Pydantic str validation. Pass in isolation, fail under xdist.
The original TestModelInfoEndpoint failure (#33807 CI) was the same class
surfaced by merge skew: #33721 added a get_configured_token_limits unpack
to create_model_info_response, and CI's merge commit ran that against the
un-updated bare-mock test before the #33742 band-aid landed.
Fix (test-only, no product change):
- TestModelInfoEndpoint: mock the real seam (get_available_models_for_user),
configure the router methods the endpoint actually calls, return a real
Deployment, and drop the dead proxy_server.get_key_models/get_team_models/
get_complete_model_list patches the refactor had stranded.
- TestGetModelInfoWithIdBlocked: spec the model mock so unset enterprise
columns read as None instead of child MagicMocks.
- test_ProxyConfig_get_model_info_with_id_missing_model_id_raises: pin
premium_user so the asserted AttributeError no longer flips with the
ambient license global.
* feat(spend): track prompt compression saved tokens in daily spend aggregates
Native compression interception now records tokens_before/after/saved into the
request litellm_metadata so savings land in the SpendLog metadata JSON under a
typed compression_savings key. A single normalizer
(extract_compression_saved_tokens) sums that key with Headroom guardrail
tokens_saved; the two writers are disjoint and run at different stages, so
summing never double-counts. The spend-log redactor now preserves purely
numeric compression stats inside guardrail_response so Headroom savings
survive the store_prompts_in_spend_logs=false default. compression_saved_tokens
is threaded through BaseDailySpendTransaction, queue aggregation, the daily
upsert blocks, a new BigInt column on all six daily spend tables, and the
daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups)
* fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader
* feat(spend): aggregate compression and prompt caching dollar savings in daily rollups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): update daily spend aggregation fixtures for savings columns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add Cost Optimization dashboard page
New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows.
Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A `lite login` token 429'd with "Budget has been exceeded! Max budget:
0.25" even when no budget was configured anywhere. cli_poll_key stamped
the minted CLI session token with litellm.max_ui_session_budget ($0.25)
as a fallback whenever the user and team had no budget of their own. That
cap was designed for the Admin UI "Test Key" chat pane; the CLI reused
the same session-token machinery, so it inherited a playground-sized
budget baked into the encrypted token at login (unchangeable without
re-login), which trips fast under real CLI/agent use.
The cap is also redundant: the token already carries user_id and team_id,
so the real user/team budgets are enforced independently at request time.
Pass max_budget=None so the CLI token is governed only by those real
budgets, and drop the now-dead user/team budget lookups. The UI login
token's guard (get_experimental_ui_login_jwt_auth_token) is untouched.
* fix(proxy): tell outdated litellm CLIs to upgrade when CLI SSO login id is legacy sk- format
* fix(cli): surface server error detail when SSO login polling fails and stop on permanent 4xx
* fix(cli): exhaustive, actionable error handling across the CLI SSO login flow
get_group_ids_from_service_principal only read the first page of the
Graph API appRoleAssignedTo response, so tenants with more than 100
groups assigned to the enterprise application silently lost group
memberships during SSO login. Loop over @odata.nextLink with the same
MAX_GRAPH_API_PAGES cap that get_user_groups_from_graph_api already
uses, and warn when the cap is hit.
Ported from #32792 by @saisurya237 so CI can run.
Fixes#32790
Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com>
* fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models'
key_type is not persisted on a key (the proxy maps it to allowed_routes and
drops it), so the keys tables only inspected the models list and rendered
'All Proxy Models' for any key with an empty models array, including SCIM,
Management and Read-only keys that cannot call a single model.
Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a
scope tooltip for those recognized scopes; unrestricted, AI-API and custom
keys keep the existing model-list rendering.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): move key_scope helper to components root
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(keys): persist key_type on virtual keys so the UI reads scope directly
Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy,
and proxy-extras schemas plus an additive migration) and stop dropping the
value in handle_key_type, so management/read_only/llm_api/default keys store
their type alongside the derived allowed_routes. Surface it on the key read
and create response models. The dashboard now prefers the persisted key_type
for the no-inference buckets and keeps the allowed_routes derivation as the
fallback for keys created before the column existed (key_type null), so no
backfill is required.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): add key_type column to LiteLLM_DeletedVerificationToken
The deleted-token archive model inherits key_type from the verification
token, so regenerate/delete flows write key_type into
LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration)
so the archive insert does not fail with FieldNotFoundError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Mutation testing surfaced branches in cost_tracking_settings and common_utils that the suite executed but never asserted on. Pin those behaviors with targeted tests: the returned (model, provider) from _resolve_model_for_cost_lookup for deployments carrying a custom_llm_provider and for deployments missing the litellm_params / model_info keys, plus the exact error-response bodies, the caller-identity lookup arguments, and the member and guard branches in common_utils.
* refactor(ui): rename OldTeams component file to Teams
* fix: show and allow editing team model aliases after team creation
* fix(ui): mark team model_aliases as nullable to match the prisma schema
* feat(proxy): add expires filter to GET /key/list
Add an opt-in expires query param to GET /key/list so callers can fetch
only expired or only active keys without paginating every page and
filtering client-side. 'expired' matches keys whose expires is in the
past (NULL expires excluded); 'active' matches keys that never expire or
expire in the future. Omitting the param preserves existing behavior for
every caller. An unrecognized value returns HTTP 400 rather than silently
returning all keys.
The filter is pushed to the database via the existing Prisma where
builder so callers avoid pulling the full key table into application
memory.
Resolves LIT-3387
* refactor(proxy): declare VALID_EXPIRES_FILTER_VALUES before its first use
- config: KeywordTierRule now strips and drops blank/whitespace keywords (a stray
"" makes _keyword_matches match every prompt, silently forcing that tier for all
traffic); still requires at least one real keyword to remain
- frontend build_complexity_router_config: trim keywords and drop rules left empty so
an unfilled "Add keyword rule" row no longer ships a rule the backend rejects with a
400 in the heuristic (non-semantic) flow, where the client-side semantic guard doesn't run
- proxy clear_cache / delete_model: the auto_router/ prefix also covers quality_router/
and adaptive_router/, so pop the model_name from all four router registries (no-op
where absent) instead of only auto/complexity; otherwise a DB quality_router's stale
entry made reload raise "already exists" and abort, and adaptive left a leak
- frontend ComplexityRouterConfig: only render the Keyword Tier Overrides and Semantic
keyword matching sections when their change handlers are provided, so the edit-auto-
router modal (which omits them) no longer shows interactive-but-dead controls
delete_model popped the auto_routers/complexity_routers registries by the deleted
deployment's model_name without checking it was actually an auto_router/* deployment.
Deleting a regular DB model that merely shares a name with a config-defined router
therefore evicted that router, which add_deployment never restores, leaving it
unroutable until a proxy restart. This is the same cross-tenant DoS clear_cache was
hardened against; mirror its auto_router/ prefix guard here.
Extracts _deployment_name_and_model to read model_name and litellm_params.model from
the deployment (delete_deployment returns the raw model_list dict at runtime despite
its Deployment annotation), and adds a regression test asserting a same-named config
router survives deletion of an unrelated regular model.
Add deterministic keyword-to-tier overrides and optional embedding-based
(semantic) keyword matching to the complexity router, and surface both in the
Add Auto Router UI behind a Router Type selector: "Auto-Router v2 [Recommended]"
(complexity tiers + keyword overrides + semantic matching, the default) and
"Semantic Router [to be deprecated]" (the existing utterance-based router,
unchanged). Keyword-to-tier overrides resolve to the highest tier matched
rather than the first keyword matched, so match order no longer affects the
routing decision.
Backend:
- config: KeywordTierRule model plus keyword_tier_rules, semantic_keyword_matching,
embedding_model, and match_threshold on ComplexityRouterConfig, with a validator
requiring an embedding model and rules when semantic matching is on
- complexity_router: evaluate keyword rules before scoring; lexical matches escalate
to the most-severe matched tier (order-independent), and semantic mode reuses
LiteLLMRouterEncoder + SemanticRouter to match paraphrases by cosine similarity,
falling back to the scorer when nothing matches
- model management: clear complexity_routers on cache reload so config edits take effect
Frontend:
- Add Auto Router tab restores the Router Type radio (Auto-Router v2 recommended
by default, Semantic Router still available) and sends keyword_tier_rules plus
the semantic settings on the recommended path, instead of flattening keywords
into custom_technical_keywords
- client-side guard blocks submit when semantic matching is enabled without an
embedding model or without any keyword tier rules, mirroring the backend validator
- moved the "How Classification Works" explainer below Custom Technical Keywords
and above Keyword Tier Overrides
- remove the Test Connection action from the recommended flow, which can't build a
valid pre-save payload for a router (leaves a TODO for a JSON preview / config
test follow-up)
Tests cover lexical escalation, semantic matching via the real library with injected
embeddings, the semantic config guard, config validation, the reload-clear
regression, and the frontend payload builder
Add a RESTful PATCH /team/{team_id} that partially updates a team using RFC 7386 JSON Merge Patch. team_id comes from the path, and metadata is merged with the team's stored metadata instead of being replaced wholesale the way POST /team/update does: an omitted key is preserved, key: null deletes it, and any other value overwrites, recursing into nested objects. Every other field behaves the same as POST /team/update
The handler delegates to the existing update path, so authorization, budget checks, system-managed-key stripping, metadata encryption, cache refresh, and audit logging are shared rather than reimplemented. POST /team/update is untouched, so the change is purely additive
* fix(proxy): match list/dict guardrail_mode in compliance mode checks
* test(compliance): cover ComplianceChecker guardrail_mode shapes (str/list/dict/None)
* fix(proxy): trust only Mode.default in compliance mode matching (ignore tag overrides)
* fix(proxy): match dict guardrail_mode only when every branch runs in mode (no false-compliant)
* fix(proxy): treat multi-mode guardrail_mode as unresolved (no false-compliant)
The list branch previously counted a guardrail configured with mode:
[pre_call, post_call] under every listed mode. But when the writer cannot
infer the concrete hook that fired (apply_guardrail invocations), the raw
list is logged, and an image-only request that only reaches the post-call
path still records both modes. That let a pre_call compliance check pass on
a request that only ran post_call.
Match the tightened dict semantics: a list now counts for mode only when
every listed mode equals mode. Same trade-off (under-report instead of
false-COMPLIANT). Speculative set support is dropped (spend logs are
JSON-serialized, sets do not cross the wire).
Tests updated to reflect the tightened list semantics, deduplicated (single
TestModeMatching class), and shortened. The invariant is now expressed as
a computed check: True implies every branch runs in the matched mode.
---------
Co-authored-by: Marton Schneider <marton@schneider.co.nl>
* fix(proxy): build redis usage cache from REDIS_* env when cache backend is not Redis
Selecting a semantic (or any non-Redis-KV) response cache left
redis_usage_cache unset, silently downgrading cross-pod rate limits,
parallel-request limits, spend coordination, and the pod lock manager
to per-pod in-memory state. Fall back to a standalone RedisCache built
from REDIS_* environment variables, mirroring the existing
use_redis_transaction_buffer escape hatch, which now shares the same
helper.
Resolves LIT-3861
* feat(proxy): configure the coordination redis independently of the response cache
Adds general_settings.coordination_redis, an explicit block for the Redis
the proxy uses for cross-pod rate limits, parallel-request limits, spend
tracking, the pod lock manager, and shared health checks. Resolution order
is the explicit block, then a plain-Redis response-cache backend, then the
REDIS_* environment. Cluster and sentinel targets are supported, and a
cluster target now builds a RedisClusterCache so cluster-aware consumers
take the cluster path.
Admins can configure it from the Caching page of the dashboard via
/coordination_redis/settings, which reports which source is in effect,
redacts credentials on read, and offers a connection test. Settings saved
there are read back at startup so they take effect on restart.
Also fixes redis client construction so an explicitly configured host
outranks REDIS_URL in the environment. Previously the url branch stripped
the caller's host and port, so an explicit block, or a connection test
typed into the dashboard, silently targeted whatever REDIS_URL named
* fix(ui): move coordination_redis_settings into renamed _components directory
---------
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
LiteLLM_MCPUserCredentials stores BYOK API keys in the same column as per-user
OAuth tokens, so the purge on a mint-relevant config change now deletes only
rows whose payload decodes as an OAuth2 credential, each by its
(user_id, server_id) pair, instead of every row for the server. An api_key
server whose url changes purges nothing. delete_mcp_server now also
invalidates each enumerated user's cached token so a re-created server reusing
the id cannot serve tokens minted for the deleted one, and both cache drops
are best-effort
The snapshot read only feeds the stale-token purge decision; leaving it unguarded meant a failed
read would 500 an edit whose update would have succeeded, and it broke
test_edit_mcp_server_redacts_credentials, whose mocked prisma is not awaitable on the un-patched
get_mcp_server path. A failure now logs and skips the purge, consistent with the purge half already
being best-effort. Adds the first endpoint-level coverage of the edit purge wiring: purge on a
mint-relevant change, no purge when the identity is unchanged, and edit success with purge skipped
when the snapshot read raises
When an IdP denies SSO access it redirects back to /sso/callback with
error and error_description query params and no code param. The callback
previously fell through to the provider token exchange, which failed
with a generic "'code' parameter was not found in callback request"
400 that hides the real denial reason. Raise a 401 that surfaces the
IdP's error and description instead.
Ported from #26640 with conflicts resolved against current staging
key_alias can become the secret name used by external secret manager
integrations (HashiCorp Vault, CyberArk Conjur) when store_virtual_keys is
enabled. Add raise_if_unsafe_secret_name, a shared validation check applied
unconditionally before a secret name reaches either integration or the
/key/generate, /key/update, and /key/regenerate API boundary, independent
of the existing enable_key_alias_format_validation opt-in flag.
Also hardens the Vault URL builder to percent-encode reserved characters
in secret_name (preserving "/" and "@"), and switches the Conjur policy
body to a real YAML serializer instead of raw string interpolation.
