Adds GET /v1/tool/spend returning per-tool and daily tool spend with a
deduplicated request total, and a cache leakage breakdown on the Prompt
Caching tab of the Cost Optimization page. Tool-spend rows are validated at
the boundary with pydantic, the endpoint is scoped to proxy admins, date
params are cast to timestamptz for real-Postgres query_raw, and the leakage
math treats litellm-normalized prompt_tokens as cache-inclusive
(uncached = max(0, prompt - cache_read - cache_creation)).
The v2 credential resolver owns oauth2_token_exchange end to end: any server
with a token-exchange config maps to a non-None TokenExchangeConfig spec, and
that config is in _create_mcp_client's override-exclusion set, so a caller
x-mcp-* override cannot force it back to v1 either. The v1 handler
resolve_mcp_auth reached at spec is None was therefore dead for OBO, including
its warn-then-proceed-unauthenticated fall-through. Delete auth/token_exchange.py
and the exchange branch, dropping the subject_token parameter that only fed it.
Separately, the REST listing and call paths still ran the v1 per-user OAuth
lookup for servers the v2 resolver owns. Unlike the two protocol-path call
sites they gated on auth_type == oauth2 only, with no to_server_spec check, so a
migrated authorization_code server did a DB round-trip whose Authorization
header _resolve_v2_auth then discards. Add the same guard via
_is_v1_resolved_oauth2_server, shared by the per-server lookup and the prefetch
preflight.
Also collapses MCPOAuth2TokenCache.async_get_token's now single-caller
require_client_credentials_flow kwarg and removes the dead
_get_bulk_user_oauth_headers helper (zero callers).
Coverage collection runs against the sharded tests/test_litellm tree, so
the provider tests living only in tests/search_tests were invisible to
codecov (patch coverage reported ~31% despite the suite). Mirror them as
tests/test_litellm/llms/bedrock/search/test_agentcore_search_transformation.py
and add edge-case tests (malformed MCP content blocks, SSE without a JSON
frame, notification-only streams, list request body, error-class mapping).
transformation.py line coverage: 99% (26 tests x2 trees).
Compress the security rationale in the gateway-session admission path of
user_api_key_auth_mcp.py, keeping the load-bearing "why" and dropping the
restatement, and remove a garbled dead comment in get_allowed_tools_for_server
In the tests, hoist the duplicated _team / _admitted_subject fixtures to
module-level factories and parametrize the four fail-closed session-bearer
variants into one case. No behavior change; the 294 tests in the file still pass
Admits a keyless SSO user (no virtual key) at the aggregate /mcp endpoint from a gateway DCR
session bearer, resolving team/org/SCIM/budget authorization fresh on every call.
- Aggregate DCR front door: stateless /register (sealed llm_dcrc_ client ids), SSO-backed
/authorize + /authorize/complete, and /token minting identity-only session tokens with PKCE,
single-use codes/flows, and rotating refresh tokens.
- Admission: a session-shaped Authorization at the aggregate scope opens via _admit_gateway_session,
reloads the live user, and runs the centralized policy gate; failures return the RFC 9728
invalid_token challenge. Gated on the un-forgeable, server-only mcp_admitted_user_subject marker,
so virtual-key and JWT auth are unchanged.
- Authorization model: an admitted subject is resolved as one plain UserAPIKeyAuth per grant source
(its own grants, plus each team it is a live roster member of), each answered by the SAME resolver
virtual keys use, then unioned. That branch is the FIRST statement of BOTH public resolvers, so no
single-credential prelude runs for it and a fault in a lookup it never uses cannot deny its grants. A source team counts only while it is a live grantor: roster membership, not
blocked, and neither the team nor its owning org over budget (enforced through the SAME
_team_max_budget_check / _organization_max_budget_check owners common_checks uses for keys).
Each team source carries that team's own org, so the existing org
ceiling caps it; for a keyless source the org list only ever intersects (a ceiling must not become
a grant) and an unresolvable ceiling denies rather than silently uncapping, on both the server and
tool axes. _roster_team_object is the single owner of "which teams count": a team whose roster no
longer lists the user neither grants servers nor throttles, in one place.
- Rate limits: the subject is bounded by its user rpm/tpm AND by the per-server mcp_rpm_limit of
the team a call is ATTRIBUTED to — the same single source billing charges, from the same owner. A key charges its one pinned team's bucket; a keyless
subject has no team_id, so admission stamps each granting team's limit map onto the auth
(server-only field, stripped from validated input like the marker) and the limiter emits that
team's mcp_per_team descriptor. Charging every granting team instead would let one cross-team user
drain several teams' SHARED buckets on a single call and block their other members; and a server
the user's OWN grant reaches charges no team bucket at all, because no team provided it. Per-KEY
MCP limits do not apply because there is no key.
- Wrapper channels: the manager-level union treats the admitted subject by the same grant model.
The admin-role short-circuit and the absolute no_mcp_servers early-return are key-credential
rules and never apply to it (a session bearer is a third-party client credential, not the
dashboard, and the subject's opt-out silences only its own source). Operator-open channels
(allow_all_keys, the user's own BYOM submissions) are owned by one operator_open_server_ids
helper that BOTH the server union and the admitted tool resolution consult (suppress-BYOM-when-
explicitly-scoped is a key-credential rule and never applies to the subject, whose user row
carries the DB-default empty mcp_servers), so an open-channel
server is default-open for tools instead of listable but uninvokable.
- Redirect URIs: one owner, validate_redirect_uri_shape, decides redirect-URI hygiene (bad scheme,
fragment, missing host, userinfo, backslash host) and resolves allowlisted native callbacks, shared
by DCR registration and the OAuth endpoints. Registration keeps a deliberately wider trust policy
than validate_trusted_redirect_uri: public dynamic registration accepts any https client, and its
controls are mandatory S256 PKCE plus the consent screen.
- Egress leak-defense: a gateway admission credential (session bearer / bridge envelope) is scrubbed
from EVERY egress header context, anchored to the credential shape, so it can never be forwarded
upstream and replayed.
- Single-use guard: auth-code, refresh and connect-flow claims resolve the proxy's cross-worker redis
cache themselves rather than trusting the cache passed in, and fail CLOSED on a Redis fault instead
of falling back to a per-worker count that a captured id could replay through another worker.
- Sign-in return_to: one shared, never-raising helper persists a safe return_to for every sign-in
branch (SSO/Okta/generic and username/password), and every branch RESUMES through the same
_sso_return_to_redirect the SSO callback uses, so however a deployment signs in the stored value
is honored identically (same-origin path directly; control_plane_url via the one-time login-code
handoff). A stale cookie is ignored rather than failing a completed sign-in.
- Budgets, both halves: ENFORCEMENT (an already over-budget team or its owning org stops being a
grantor, in the source gate) and ACCOUNTING (a team-derived tool call is billed to the granting
team and ITS org, so that budget accumulates and the right organization is charged). A server the
user's own grant reaches bills the user; when several teams grant one server the pick is the
lowest team_id, stable and auditable. Billing rides a COPY, so authorization still sees the full
union, and it is inert when the target server cannot be resolved from the tool name.
Deferred (tracked): client-selected server scoping of the session token (LIT-4680).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(organization): persist cleared fields on /organization/update
Clearing an org field (the Metadata box or a TPM/RPM/max_budget limit) via PATCH /organization/update looked like it saved but reverted on refresh; the partial-update merge could not tell a cleared field from an untouched one and dropped every clear
The endpoint now decides SET vs CLEAR vs UNTOUCHED purely from which keys the raw request body carried, via a pure build_organization_update_plan. Budget nulls flow to update_budget (null clears via exclude_unset), metadata is replace-when-sent (written as {} for the non-nullable Json column), and a budget write on an org with no budget_id creates and links a budget row. This removes the exclude_none dump, both "if v is not None" filters, and the additive _update_dictionary merge
Resolves LIT-3664
* feat(organization): add RESTful PATCH /v2/organization/{organization_id}
Adds a v2 organization-update endpoint with a deterministic partial-update contract, and reverts the v1 /organization/update changes so its public behavior stays untouched
On v2 a field present in the request body is written (null/[]/{} clears, a value sets) and an omitted field is left untouched; presence is read from model_fields_set. Clearing a TPM/RPM/max_budget limit or the metadata now persists instead of being dropped as if it were never sent. Metadata is replace-when-sent and written as {} when cleared, since the org metadata Json column is non-nullable. Budget nulls flow to update_budget, and an org with no budget row gets one created and linked. The endpoint is hidden from the public Swagger docs via include_in_schema=False, and stays typed in the generated dashboard schema
Resolves LIT-3664
* test(organization): cover v2 auth guard, negative budget, and object_permission
Adds v2 endpoint tests that were missing: the real _verify_org_access path rejects a non-admin caller with 403 and writes nothing, a negative max_budget is rejected with 400 before any DB access, and a sent object_permission is passed to the upsert helper with its id linked onto the org write
Refs LIT-3664
* fix(organization): 400 on null-clear of required org fields; drop dead budget upsert
organization_alias and models are non-nullable columns, so a v2 request clearing them with null hit a 500 (NOT NULL violation) and could partially apply the budget half of the request first; the endpoint now returns a 400 with a clear message. Also removes the unreachable "create a budget when the org has none" branch from _apply_organization_budget_updates, since budget_id is a non-nullable FK and every org already has one, so the endpoint no longer needs to link a newly-created budget id
Refs LIT-3664
* fix(organization): let v2 clear object permissions when sent as null
Sending object_permission: null now detaches the org's permission by setting the nullable object_permission_id to null, instead of being a silent no-op, so the endpoint honors its documented "null clears" contract and an admin can actually revoke vector-store/MCP access. Sending a value still merges as before
Refs LIT-3664
* fix(organization): make v2 PATCH atomic, strict, and 422-consistent
Tighten the PATCH /v2/organization/{id} endpoint against standard HTTP
PATCH (RFC 5789 / RFC 7396 JSON Merge Patch) semantics:
- Apply the budget-row and org-row writes in one prisma transaction so a
failure between them can no longer half-apply the patch (RFC 5789 requires
a PATCH to apply atomically). The budget write is inlined as a tx-aware
call mirroring the team-member budget path rather than the standalone
update_budget route handler
- Set extra="forbid" on OrganizationUpdateRequestV2 so an unknown or
misspelled key is a 422 instead of a silently dropped no-op; the contract
is presence-driven, so swallowing unknown keys is unsafe
- Return 422 (not 400) for the hand-rolled field validations (negative
budgets, null-clear of required organization_alias/models, invalid
model_max_budget) so every validation failure matches the 422 that
pydantic already returns for bad values
- Document the per-field clear tokens accurately: null clears budget limits
and metadata, [] clears models, and organization_alias cannot be cleared
Tests cover the single-transaction write path, unknown-field rejection, the
422 status changes, and the budget_reset_at recompute.
* fix(organization): reject empty object_permission on v2 PATCH instead of silently keeping grants
object_permission is a nested merge field on PATCH /v2/organization/{id}: a
sent object merges into the existing permission row (updating one grant list
without touching the others), and null detaches it. An empty {} therefore
merged nothing and left every existing vector-store/MCP grant in place, so an
admin who sent {"object_permission": {}} to strip access silently kept it.
Reject a present-but-empty object_permission with a 422 that points the caller
at null, mirroring how the endpoint already rejects a null clear of the
required organization_alias/models. This keeps merge semantics for non-empty
payloads and does not affect the Admin UI, which only ever sends a fully
populated object or omits the field.
* fix(organization): JSON-serialize model_max_budget on the v2 budget write
model_max_budget is a Json column on the budget table. Route the budget-row
write through jsonify_object so a dict value is serialized the same way
new_budget and the org-row metadata write already do it, keeping every Json
column on this endpoint written consistently.
Raw dicts already round-trip (update_budget writes them unserialized), so this
is not a correctness fix so much as making the one Json column on the budget
path follow the same serialization as the rest of the file. Added a test that
a patched model_max_budget reaches the budget write JSON-serialized.
* refactor(organization): trim v2 docstrings and consolidate planner tests
Trim the verbose docstrings on the v2 endpoint, request model, and the two
pure helpers to the essential contract, and drop a stale line that still
referenced update_budget's exclude_unset (the budget write is inlined now).
Collapse the nine per-case planner tests into one parametrized test asserting
exact budget/org split per body, and fold the two model-validation rejection
cases into one parametrized test. Same 36 test cases run; the planner
assertions get stronger (exact-equality instead of presence/absence) and the
test additions shrink by ~85 lines.
* refactor(organization): inline the v2 update planner into the endpoint
Fold the OrganizationUpdatePlan dataclass and build_organization_update_plan
helper into update_organization_v2. The budget-vs-org split is a few dict
comprehensions built in one shot, so the extra type plus builder was more
ceremony than the job needed. Drops the now-unused dataclass/AbstractSet
imports and the isolated planner unit tests; the split is exercised end-to-end
by the endpoint tests.
* fix(organization): run v2 object permission upsert inside the update transaction
prepare_object_permission_upsert splits the shared helper's read-and-merge
step from its write so the v2 endpoint can upsert the permission row on the
same prisma transaction as the budget and org writes. Previously the upsert
ran before the transaction, so a rolled-back org write left merged grants
live on the permission row the org still pointed at. The upsert record now
pins object_permission_id, since the column's @default(uuid()) would
otherwise mint a fresh-create id different from the one linked on the org.
v1 and the team/key callers of handle_update_object_permission_common keep
their existing behavior
* fix(lint): keep the v2 org PR within the strict-rule budget
The strict gate flagged the PR's new code after the base merge: 11 UP045
Optional fields and a typing.List on OrganizationUpdateRequestV2, Dict
annotations in the new upsert helper and the TypeAdapter, and a B008 from
the v2 endpoint's Depends default. The model and helper now use pipe
unions and builtin generics, and the endpoint takes its auth dependency
via Annotated, which avoids the call-in-default pattern B008 targets
* fix(routes): expose /v2/organization on the backend component allowlist
The component-split coverage test requires every app route on a component;
the new v2 org PATCH belongs with the other management endpoints on the
backend, alongside the existing /v2/key and /v2/team prefixes
* fix(organization): clear budget_reset_at when budget_duration is cleared via v2 PATCH
Extract the inline sync streaming post/decode into make_sync_call so it can be
exercised with an injected client, mirroring make_async_call, and add a sync
regression test that each token is forwarded after exactly one pulled frame.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Mirror the sagemaker_chat fix on the native sagemaker/ streaming path: the
sync and async completion handlers read the invocations-response-stream body
with iter_bytes(chunk_size=1024) / aiter_bytes(chunk_size=1024), so httpx
withholds bytes until 1024 accumulate and tokens arrive in gap-then-burst
waves. Drop the fixed chunk size so each decoded event is forwarded as its
bytes arrive.
Also add a boundary-agnostic decoder test proving frames reassemble correctly
regardless of where transport reads split the stream.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): include type in tool_choice disable_parallel_tool_use config for Converse
* fix(bedrock): let parallel_tool_calls-derived disable flag win over raw tool_choice value
Calling an MCP tool on a server the key is not scoped to raised a 403 whose
detail interpolated the caller's allowed List[MCPServer] config objects, so
pydantic's default repr printed authentication_token, client_secret, the AWS
keys, client_private_key, env and static_headers straight back to the caller.
The two sibling denial sites already returned a bare message, so this one was
the lone outlier
MCPServer now renders only server_id, name, transport and auth_type in repr
and str, so a future f-string or log line cannot re-leak a credential field.
Field types and model_dump serialization are unchanged
The huggingface embedding test fixture reloaded
litellm.llms.custom_httpx.http_handler, creating a new HTTPHandler class
object. llm_http_handler keeps the class captured at import time, so any
test running later in the same process that injects a client built from
the reloaded class fails the isinstance check and the mock is silently
discarded, causing a real network call. Under pytest-xdist loadscope this
surfaced as a deterministic failure of
test_accept_header_in_completion_request_jwt whenever an unrelated PR
shifted worker distribution.
Also removes the same reload pattern from the vertex rerank integration
test (both were previously removed in a6df01caec and resurrected by a
merge conflict resolution) and hardens the agentcore victim test by
dropping the bare except that swallowed the real error
Resolves LIT-4581
A true_passthrough MCP server created without the at-creation auth step
has no stored client_id, and the tools-page browser flow supplies none,
so GET /v1/mcp/server/oauth/{id}/authorize dead-ended on a 400
missing_client_id. The client-forwarded-token modes forbid the gateway
from persisting an OAuth client, so client acquisition moves into the one
chokepoint every caller crosses: the authorize endpoint.
resolve_ephemeral_dcr_client owns the whole mint policy (mode gate,
authorization-url precondition, required S256 PKCE, redirect trust, then
a TTL-deduped, per-server single-flighted RFC 7591 mint). The minted
client rides the encrypted OAuth state; /callback seals it with the
upstream code and server_id into an llm_ptcode_ gateway code, and
redeem_passthrough_authorization_code recovers it at the token endpoint
(server binding plus required code_verifier) to authenticate the upstream
exchange. Nothing is persisted; every value rides the encrypted blobs, so
it works across replicas.
Client acquisition is one predicate applied across the whole auth-mode
matrix: the gateway mints for a clientless authorize iff true_passthrough
(any dcr_bridge) or oauth_delegate-and-not-dcr_bridge, and the UI
gatewayMintsClientFor mirrors that set exactly so the browser pre-registers
a client through the dcr_bridge front door only for the cells the gateway
does not mint (the interactive oauth_delegate dcr_bridge sign-in and the
legacy oauth2 passthrough). A minted flow runs the bridge short-circuit
arm; the relay front door stays for external clients that present their
own client_id. Both sides are pinned against the same truth table
(test_resolve_ephemeral_dcr_client_mint_set_is_exact and the
gatewayMintsClientFor matrix test) so no mode can silently diverge. The
authorization_code hook and M2M/token-exchange modes are unchanged.
* feat(ui): add react-hook-form + zod form infrastructure
Introduce the shared form layer the dashboard's antd forms will migrate onto,
with no user-visible change yet.
- pin react-hook-form, @hookform/resolvers, and zod (kept on 3.25.76 and
imported via the zod/v4 entrypoint so openai's optional zod ^3 peer still
resolves and npm ci stays clean)
- vendor the base-vega Field family into components/shared/form as forwardRef
components on the repo's cva.config, since base-vega ships no form primitive
and its field source imports class-variance-authority and is React 19 style
- add a FormField bridge that binds a react-hook-form Controller to the Field
layer and wires label, description, and error ids into aria attributes
- add pickDirty, which narrows a submitted body to the top-level keys the user
actually touched so a partial update stops re-sending untouched fields
pickDirty reads dirtiness at the top level because react-hook-form tracks it
per leaf, so an edited array arrives as [true, false] and a cleared list as an
empty array that still carries its default-length dirty markers; the falsy
clear tokens (null, [], {}, 0, false) all survive.
Tests cover the Field primitives, the FormField aria wiring against a live
zod resolver, and pickDirty both as a unit and driven through a real
react-hook-form instance.
* test(ui): lock pickDirty behavior on a pure field-array reorder
react-hook-form compares each array element to its default positionally by
value, so useFieldArray move/swap and a reordered scalar array all mark the
moved indices dirty and pickDirty sends the whole array; a swap of two equal
elements is a value-level no-op and is correctly omitted. Covers the reorder
case a review flagged as untested.
* feat(proxy): publish a typed request body for PATCH /team/{team_id}
The route validated its body into UpdateTeamRequest but read it off the raw
request, so the OpenAPI spec carried no requestBody and the dashboard's
generated client could not type the call at all.
- add PatchTeamRequest, UpdateTeamRequest with an optional team_id, since PATCH
takes the id from the path; a body team_id is still accepted when it matches
- validate the body through PatchTeamRequest before delegating to update_team
- declare the request body on the route and regenerate schema.d.ts
The handler keeps reading the raw body rather than declaring a typed parameter.
FastAPI validates a declared body before the handler runs, which would replace
the 400 for a non-object body with a 422 and move absent-vs-null out of reach of
the RFC 7386 metadata merge; those are pinned by existing tests, so the schema is
declared on the route instead and every error path is unchanged.
Validation is shape-preserving: the body is dumped with exclude_unset so an
omitted field never reaches the write, an explicit null still clears, and a
partial object_permission does not gain sibling sub-keys, which would wipe them
given the column merges rather than replaces.
Tests extend the existing patch harness rather than replacing it.
* refactor(proxy): declare the PATCH /team/{team_id} body as a typed parameter
Replaces the hand-written OpenAPI declaration added earlier in this branch. The
route now takes data: PatchTeamRequest, so FastAPI generates the request body
itself and emits a $ref to the model instead of an inlined copy that would go
stale as fields are added.
The earlier approach was a workaround built on a wrong premise. Declaring the
body does not cost absent-vs-null: model_fields_set preserves it, which is how
POST /team/update already gets its tri-state, and a nested null inside metadata
survives validation untouched, so the RFC 7386 merge is unaffected.
The one real change is the status code for a malformed body. The route answered
400 for a non-object body and 500 for a wrongly typed field, reporting a caller
mistake as a server fault; both are now 422, matching POST /team/update and the
other typed management endpoints. The two tests that pinned the old parse-level
errors are replaced by one that pins the 422 through the ASGI stack, and the
handler drops its manual parsing entirely.
* fix(anthropic): strip uniqueItems + other unsupported array/object constraints from output_format schema
Anthropic's structured outputs (`output_format`) validate the JSON schema
against a strict subset and reject cross-element / count constraints that a
constrained-decoding grammar cannot enforce, returning a 400
`invalid_request_error`.
`filter_anthropic_output_schema` already stripped the numeric / string /
item-count constraints (minimum, maximum, exclusiveMinimum/Maximum, minLength,
maxLength, minItems, maxItems) but still let these through:
- uniqueItems
- contains / minContains / maxContains
- minProperties / maxProperties
so a request using them fails with e.g. "output_format.schema: For 'array'
type, property 'uniqueItems' is not supported".
This is provider-visible: newer Claude models on the native `output_format`
path (e.g. `azure_ai`) 400, while `vertex_ai` is unaffected because it is
forced onto the permissive tool-use path (#18625 / #19201).
Add the missing keywords to the unsupported-field set and the description map,
and skip the advisory description note for a disabled boolean constraint
(`uniqueItems: false`) so it isn't misdescribed as required.
* fix(anthropic): serialize contains sub-schema in output_format advisory note
Address Greptile review: the `contains` advisory note previously discarded the
sub-schema, so the description only said an item must match "a schema" without
saying which. It now serializes the sub-schema as JSON (e.g. "array must
contain an item matching: {\"type\": \"integer\", \"const\": 1}"), matching the
other stripped constraints which carry their value. Sub-schema (dict/list)
values are json.dumps'd; scalar constraints are unchanged.
* style(anthropic): apply ruff format to output_format filter change
* style(test): ruff format anthropic schema filter tests
* test(anthropic): cover output_format array/object constraint filtering in test_litellm tree
Mirrors the schema-filter tests under tests/test_litellm/ so the coverage
job exercises the new uniqueItems/contains/min-maxProperties handling and the
uniqueItems: false branch.
---------
Co-authored-by: Darien Kindlund <darien@kindlund.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov flagged the ModelResponse-branch lines as uncovered: add async
per-token normalization, plus zero-completion-token fallback tests for
both handlers (the else branch storing plain float seconds).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
log_success_event/async_log_success_event only converted the
end_time - start_time timedelta to float seconds inside the
isinstance(response_obj, ModelResponse) branch, so every embedding /
speech / image response appended a raw timedelta to the latency list
and broke the Redis cache sync with 'Object of type timedelta is not
JSON serializable' (no cross-replica latency sharing for those model
groups + error-log spam). Normalize response_ms to float seconds
up-front in both handlers.
Completes the partial fix from #14040. Fixes#33169
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Okta commonly sends SCIM membership removals as a filtered path with no request
body value, e.g. Groups PATCH members[value eq "uid"] and Users PATCH
groups[value eq "tid"]. The patch handlers pulled ids only from op.value, so
these removes were a silent no-op and the member or team was never dropped
Add a linear-time filter parser reused by both the Groups members path and the
Users groups path so the id is taken from the [value eq "..."] filter when
op.value is absent, for add and remove ops. The eq operator is matched
case-insensitively, both quote styles are accepted, and the quoted value is
unescaped. The path-filter fallback only fires when the request body value is
omitted, so an explicit empty value no longer resurrects the filter id, and the
compared value must be quoted per the SCIM filter grammar
The SSO and Email Server settings pages read only stored config, so a gateway
configured entirely through environment variables rendered every field blank
even though both features were live. Rather than add per-endpoint env fallback,
resolve each setting through one typed config object.
A FieldDescriptor names, for one setting, where it lives in the stored row
(db_key), which process env var carries it (env_var), whether it is a secret,
and its effective default. A pure resolve_fields reconciles a descriptor table
against the stored row and the process environment with a fixed precedence and
reports per-field provenance (db, env, default, or unset). The SSO descriptor
table single-sources the field-to-env mapping that the read and write paths
previously duplicated, so they can no longer drift.
get_sso_settings and the /get/config/callbacks alerting block read through the
resolver instead of their own inline fallbacks. get_sso_settings no longer
decrypts stored values into os.environ; decryption happens once inside the
resolver via the pure helper, so a GET stops mutating the process environment.
The SSO response carries provenance so the UI can distinguish an env-sourced
value from a stored one, and secrets are masked at the endpoint (the resolver
returns them unmasked so the login path could consume them). os.environ remains
the runtime carrier; the SSO login and mail-send paths are unchanged.
The settings pages also submit only fields an admin actually edited, so a
rendered mask or env-sourced value is never written back over a working
secret, and generic_scope is a real SSO form field. Omitting a field from
/update/sso_settings clears it, which provider switching relies on; the deeper
write-path concern that behaviour points at is tracked in LIT-4498.
_add_team_members_to_team reconciled membership by reading the complete_team_data
snapshot captured at the start of team_member_add, appending in memory, and
writing the whole members_with_roles array back. Two concurrent /team/member_add
calls for the same team read the same snapshot, so the last write wins and one
member is silently lost. This affects every concurrent team member add, including
the SCIM group PATCH op:add path that routes through team_member_add
Reconcile members_with_roles inside a transaction that locks the team row with
SELECT ... FOR UPDATE before re-reading the current membership, so concurrent
writers serialize on the row lock and each appends onto the other's committed
result. The interactive transaction is exposed through a thin PrismaClient.tx()
passthrough and the locked read is encapsulated in
TeamRepository.get_members_with_roles_locked, and the SCIM group PATCH applies
membership as deltas so concurrent adds are not clobbered
SCIM delete_user removed the user from the legacy team.members column and deleted
their team membership rows, but never pruned members_with_roles, which is the
source of truth ScimTransformations reads for GET /Groups/{id}. A deleted user
therefore lingered as a dangling member reference on every team they belonged to
Prune each of the user's teams directly via team_member_delete before deleting
the user row, and only for teams whose members_with_roles actually contain the
user, so a real DB failure surfaces (the endpoint fails loudly and the user is
kept; SCIM DELETE is idempotent, so the IdP retries) while a user who was never
in a team's members_with_roles stays a no-op. patch_team_membership is left
unchanged for its other callers
* fix(scim): sync team roster and dedup teams for existing-user email upsert
When POST /scim/v2/Users matched an already-existing user by email,
handle_existing_user_by_email raw-wrote the user's teams array but never
touched the team roster, so the user appeared in the group on their profile
yet was absent from the team directly (members_with_roles stayed empty). It
also did not dedup the teams built from repeated SCIM groups.
Route the existing-user team assignment through the same
_handle_team_membership_changes / team_member_add path the PUT update_user
handler uses, so members_with_roles, LiteLLM_TeamMembership, and the user's
teams array stay in sync, and dedup the teams derived from user.groups. The
user_id rewrite to the new userName is preserved and sequenced before the
roster sync so the roster never references a stale primary key.
* fix(scim): surface roster add failures on existing-user email upsert
Route the existing-email upsert's roster sync through patch_team_membership
with a new opt-in raise_on_error flag so a genuine team_member_add failure
propagates instead of being swallowed, and the deduped teams array is only
persisted after the roster sync succeeds. Without this, a failed add left the
endpoint reporting success while user.teams listed a team members_with_roles
never received.
The benign already-a-member case stays a no-op even under the strict path, and
the flag defaults to False so the PUT update_user, PATCH patch_user, and group
callers keep their existing best-effort behavior. SCIM POST is idempotent, so
surfacing the error lets the IdP retry and converge.
* fix(scim): surface roster removal failures symmetrically with adds
Make team_member_delete failures fail loud under the strict roster sync used by
the existing-email upsert, mirroring the add path, so a swallowed removal can no
longer let the user's teams array drop a team the roster still holds. The
idempotent case where the user is already absent from the team stays a no-op,
matching how an add treats the user already being in the team. Best-effort
behavior is preserved for the default raise_on_error=False callers.
* fix(scim): use members_with_roles as the source of truth for group membership
SCIM group provisioning tracked membership inconsistently. Team creation and
the real team endpoints persist membership in members_with_roles (and each
member's user.teams), but the SCIM group PATCH handler and the GET /Groups
listing read the legacy team.members String[] column, which team creation never
populates. Seeding a PATCH result from that empty column made an Okta "add
member" operation recompute the member set from scratch and silently drop
everyone already in the team, so users ended up missing from the groups they
were provisioned into. Reading the same empty column on GET /Groups reported an
empty member list back to the IdP, which drove repeated re-provisioning.
Separately, add_new_member appended the team id to user.teams with an
unconditional array push. Under the concurrent group PATCHes an IdP sends during
a reconcile, each request passed the members_with_roles duplicate check and
pushed, so user.teams accumulated duplicate ids for the same team. A duplicate
also breaks auth logic that keys off the number of teams a user belongs to.
Read current membership from members_with_roles in the SCIM group PATCH seed and
the GET /Groups listing, and make the user.teams append idempotent via a
filtered update that no-ops once the team is present.
Resolves LIT-4283
* fix(scim): address review; atomic user-creation and stop writing legacy members
Keep the concurrent-safe team append but create the user via an atomic upsert
(create-or-update) instead of a check-then-create, so provisioning the same new
user concurrently cannot race into a duplicate-key failure; the team is still
appended idempotently by a filtered update so an existing user gets no duplicate
team id. Stop writing the legacy team.members column in the group PATCH apply so
the only membership record is the source of truth (members_with_roles plus each
member's user.teams), reconciled by team_member_add/team_member_delete.
Tests: existing add_new_member and team-creation mocks updated to the upsert
plus filtered-append shape, and new tests cover atomic creation and that the
PATCH apply does not write the legacy members column.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
Greptile review: _cache_team_object runs after a successful DB fetch in
get_team_object and after every team mutation's DB write, but
DualCache.async_delete_cache propagates backend errors, so a Redis blip
during invalidation would turn a healthy team lookup into a 404 and a
committed /team/update into a 500. Both the internal usage cache delete
and the alias-key invalidation now log a warning and continue on failure,
matching how DualCache.async_set_cache already swallows write errors.
Worst case on failure is worker-local staleness bounded by the internal
cache's in-memory TTL, the same bound other workers already have
get_team_object consults proxy_logging_obj.internal_usage_cache before
user_api_key_cache, but _cache_team_object (the refresh every team
mutation goes through) only wrote user_api_key_cache. With
enable_redis_auth_cache both caches share one Redis, so any request
backfills the internal cache's in-memory tier with the team object and
that copy keeps shadowing the freshly written team until its TTL expires.
The auth builder then wrote the team object it had just read back into
the cache after check 6, clobbering the fresh Redis value with the stale
one, which made the staleness self-sustaining under traffic: keys with
models=["all-team-models"] kept getting 403 team_model_access_denied
for models added via /team/update, and kept access to removed ones.
_cache_team_object now deletes the internal usage cache entry before
writing the refreshed team, and the auth-time write-back is removed so
only authoritative writers (DB reads and team mutations) populate the
team cache, mirroring how key objects already handle this (see
test_auth_does_not_rewrite_cached_key_object_back_into_cache).
The LIT-4000 test pinning the removed write-back is deleted; its
concern (team object cached under the canonical key) is handled by
_cache_team_object inside get_team_object's DB path and pinned by
test_cache_team_object_writes_team_id_and_invalidates_team_alias
Resolves LIT-4391
* fix(proxy): share CLI SSO login sessions across workers without enable_redis_auth_cache
* fix(proxy): make CLI SSO flow state redis-authoritative across workers
The CLI SSO flow is stored in a DualCache whose get_cache is memory-first, so
the worker that served /sso/cli/start keeps serving its stale in-memory flow and
never observes the sso_complete/session_data update another worker writes during
the OAuth callback. Attaching Redis alone is not enough; poll on the original
worker returns pending forever.
Read and write the flow directly through the attached Redis backend when present
so every worker sees the same authoritative state, falling back to the in-memory
DualCache only when no Redis is configured.
* fix(proxy): serialize CLI SSO flow as JSON for the redis round trip
RedisCache stores values via str(value) and parses reads with
json.loads then ast.literal_eval. The completed flow contains a
LitellmUserRoles enum in session_data.user_role, whose repr is not a
parseable literal, so any worker reading the completed flow from redis
raised SyntaxError and returned 400 "CLI login session not found".
Writing the flow as json.dumps makes the round trip lossless (the enum
is a str subclass) and fails loudly at write time if a non-serializable
value is ever added to the flow.
* fix(proxy): point CLI SSO session-not-found hint at configuring Redis
The error message and warning still told users to set enable_redis_auth_cache,
but the CLI SSO session cache now gets Redis unconditionally whenever one is
configured, so that flag no longer affects CLI login
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(guardrails): add only_scan_new_messages for per-session incremental scanning
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): use fixed TTL constant and revert unrelated test formatting
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): run only_scan_new_messages in the unified apply_guardrail path
The initial wiring lived in BedrockGuardrail.async_pre_call_hook, but the proxy
routes Bedrock through the unified apply_guardrail interface, so the flag had no
effect live. Move incremental selection into apply_guardrail: filter the flat
texts list against per-session scanned hashes, skip the Bedrock call when nothing
is new, and mark hashes only after a successful (non-blocked) scan. Full-context
fallback is preserved when there is no session id, the cache is unavailable, or a
masking guardrail is configured.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover session-id fallbacks and mark_texts_scanned guards
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): fall back to full scan when incremental guardrail masks content, use shared cache
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover generic agent multi-turn incremental scan
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover incremental scan cache resolver fallbacks
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover flag interactions and /v1/messages incremental scan semantics
* feat(guardrails): make GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS env configurable
* test(guardrails): prove skip_system/skip_tool are enforced upstream of incremental scan
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
urlsplit validates the port lazily, so a non-numeric port raised
ValueError out of _redact_mcp_resource_url after the urlsplit try
had already passed; the server loaders now call the helper while
warning about typo'd urls, which would have turned the warning into
a load failure. Resolve hostname and port inside the guard and pin
the malformed-port case in the redaction test
A typo'd MCP server url failed OAuth endpoint discovery silently: every
failure died at debug level, the config loader warned nothing, and the
/authorize 400 blamed "servers with no url" even when a url was set.
_descovery_metadata now records each attempt's outcome and, when a total
failure would leave the server's flow without a needed endpoint, logs one
warning with the trail (urls origin-only, exception text url-stripped).
Both server loaders warn which endpoints stayed unresolved for the
server's flow (client_credentials never needs authorization_url, OBO
needs only token_url) with the remedies; this replaces the DB path's
reason-less warning and closes the config path's no-warning gap. The
authorize/token/register 400 details branch on server shape via one
shared helper and point at the proxy logs. _redact_mcp_resource_url
moves to oauth_utils.py so the manager can import it without a cycle.
Resolves LIT-4658
