Wires the identity-only session token into admission, so a client that completed the gateway
OAuth flow can actually call the aggregate endpoint. The arm sits after the explicit
x-litellm-api-key arm, which keeps winning, and before the generic Authorization arm, which
would otherwise hand the bearer to user_api_key_auth as if it were a litellm key and 401.
The bearer carries no authorization. It reloads the live user record and runs the same
centralized policy gate the key path uses, so deactivating a user, blocking their team, or
exhausting a budget kills outstanding sessions on the next call rather than at expiry. Failures
raise the aggregate challenge, so a spec client re-authorizes instead of giving up; a session
refresh token presented at the tool edge is rejected the same way rather than falling through.
Three ways an admitted subject was quietly weaker than a key, fixed here:
An admin connecting through the front door received every MCP server on the proxy, because
get_allowed_mcp_servers short-circuits to the whole registry for an admin role with no explicit
object permission. That is the dashboard's view-all behavior; a gateway session is excluded from
it, so the connect grid keeps describing what the client can reach.
The org ceiling and the org budget check both return early without an org_id, which the reload
never bound, so neither applied. A ceiling only narrows, so binding it cannot widen access.
Every rate limiter descriptor keys on api_key, team_id or end_user_id, none of which a keyless
subject has, so the limiter enforced nothing at all and logged nothing. The user-level limits
are the only ones this principal can bind, so the reload now carries them.
Extracts _is_aggregate_mcp_scope so the 401 challenge and this arm share one definition of the
aggregate scope instead of two that can drift.
The aggregate front door has to carry three values across a browser round trip: the registered
public client, the authorization request parked while the user signs in and authorizes servers on
the connect grid, and the authorization code the client redeems for a session bearer.
All three are sealed the same way as the identity-only session token: HS256 over strict
`extra="forbid"` claims as the sole type gate, a size cap applied before parsing, a `jti` so a
single-use guard has a stable handle, and a signing key derived from the master key under a domain
label distinct from the session and bridge labels. Sealing the client's `redirect_uris` into the
`client_id` is what makes registration stateless, so it needs no table, no migration, and no
lifecycle surface.
The older `encrypt_value_helper` family that seals the bridge authorization code and OAuth state
was deliberately not reused: it has no TTL, no claims, no kind separation, and no domain-separated
KDF, and its own callers document that values minted before an expiry check was added stay valid
indefinitely.
Openers are total over hostile input and return typed failures, since they run on an
unauthenticated path where an exception is a 500 rather than an OAuth error.
The dashboard `token` cookie is an HS256 JWT signed with the proxy master key, minted from
six places, and none of them stamped an `exp`, so the cookie never expired and a leaked one
stayed valid until the master key rotated.
The same gap broke callers that do the right thing: `_user_id_from_session_cookie` requires
an `exp` claim, so it rejected every genuine login cookie and the MCP SSO interpose was an
unconditional redirect-to-login loop.
Routes all six mints through `encode_ui_session_jwt`, which stamps
`exp = LITELLM_UI_SESSION_DURATION` (default 24h), the same window that already bounds the
virtual key carried inside the payload. The helper lives in its own module because
`login_utils` already imports from `ui_sso`, so a shared helper in either would close an
import cycle for the other.
* test(e2e): cover key max_budget blocks on personal, team, and team-member keys
* refactor(e2e): convert budget enforcement cases to the resources-fixture pattern
The E2ECase class pattern existed only in this file; every other suite uses
plain pytest tests with the resources fixture. Rewrites the nine cases as two
spec classes and removes the now-dead E2ECase protocol and run_case driver
from lifecycle.py
Moves the dashboard's next pin from 16.2.6 to the latest 16.2.x patch and bumps eslint-config-next to match. Regenerating the lock also healed in explicit bundled-dependency records under @tailwindcss/oxide-wasm32-wasi
* test(ui): characterise transform-request panel behaviour before migration
* refactor(ui): migrate transform-request to shadcn
* fix(ui): keep transform-request panels within the fixed-height content fold
* fix(ui): let transform-request flow naturally so the shell scrolls instead of clipping
* test(ui): select the copy button by its accessible name
* test(ui): characterise the old usage page before migrating it
Role- and text-based coverage of the route as it behaves on Tremor, so the
shadcn migration has a regression net it did not get to write. Pins the
DISABLE_EXPENSIVE_DB_QUERIES branch (warning copy, the docs link and its
target, and that every expensive query is skipped), the admin vs non-admin
tab set, the cost cards, and the provider and customer tables
* refactor(ui): migrate old-usage to shadcn
Replaces Tremor with the installed shadcn primitives and the shared recharts
wrappers on the only file the route owns. Tabs, cards, tables, the key select
and the tag multi-select come from src/components/ui; the bar, area and donut
charts come from src/components/shared/charts. Tremor BarList has no shared
equivalent, so Total Spend Per Team is composed from ui/meter, which also means
the per-team totals stay numbers in state instead of pre-formatted strings; a
team total of 1,000 or more used to make the bar widths NaN.
The Database Query Limit Reached warning moves with it: same copy, same docs
link, still short-circuiting every expensive query.
Drops the file's no-restricted-imports suppression and the dead customTooltip,
getTopKeys, DataDict and UserData symbols. The characterisation test from the
previous commit is unchanged and green on both sides
* test(ui): pin prompts panel toolbar and delete behaviour before migration
* refactor(ui): migrate prompts list panel to shadcn
* fix(ui): resolve prompts environment label and hold the delete dialog while deleting
* test(ui): characterise the API reference page before the shadcn migration
Pins the behaviour the migration must preserve: the three SDK tabs and their
accessible names, the default selection, that selecting a tab surfaces that
SDK's snippet wired to the resolved base url, and the title, blurb and docs
link. Written against the current Tremor markup with role and text queries so
it carries over unedited.
* refactor(ui): migrate the API reference page to shadcn
Replaces the Tremor Grid, Text and Tab primitives on the API reference route
with the shadcn Tabs primitive and token utilities, and prunes the file's now
stale no-restricted-imports suppression. Markup only; the characterisation
test added in the previous commit passes unedited.
The wrapper keeps an explicit grid-cols-1 because Tremor's Grid defaults to
numItems=1 and emitted it; the implicit auto column that replaces it sizes to
the widest child and made the code block overflow the viewport.
* test(ui): scope the API reference snippet assertion to the selected tab panel
Asserts against the rendered tabpanel instead of searching every mounted code
block, so the check keeps proving the selected tab drives the snippet even if
the panels are ever kept mounted.
* feat(ui): add react-hook-form + zod form infrastructure
Introduce the shared form layer the dashboard's antd forms will migrate onto,
with no user-visible change yet.
- pin react-hook-form, @hookform/resolvers, and zod (kept on 3.25.76 and
imported via the zod/v4 entrypoint so openai's optional zod ^3 peer still
resolves and npm ci stays clean)
- vendor the base-vega Field family into components/shared/form as forwardRef
components on the repo's cva.config, since base-vega ships no form primitive
and its field source imports class-variance-authority and is React 19 style
- add a FormField bridge that binds a react-hook-form Controller to the Field
layer and wires label, description, and error ids into aria attributes
- add pickDirty, which narrows a submitted body to the top-level keys the user
actually touched so a partial update stops re-sending untouched fields
pickDirty reads dirtiness at the top level because react-hook-form tracks it
per leaf, so an edited array arrives as [true, false] and a cleared list as an
empty array that still carries its default-length dirty markers; the falsy
clear tokens (null, [], {}, 0, false) all survive.
Tests cover the Field primitives, the FormField aria wiring against a live
zod resolver, and pickDirty both as a unit and driven through a real
react-hook-form instance.
* test(ui): lock pickDirty behavior on a pure field-array reorder
react-hook-form compares each array element to its default positionally by
value, so useFieldArray move/swap and a reordered scalar array all mark the
moved indices dirty and pickDirty sends the whole array; a swap of two equal
elements is a value-level no-op and is correctly omitted. Covers the reorder
case a review flagged as untested.
* feat(proxy): publish a typed request body for PATCH /team/{team_id}
The route validated its body into UpdateTeamRequest but read it off the raw
request, so the OpenAPI spec carried no requestBody and the dashboard's
generated client could not type the call at all.
- add PatchTeamRequest, UpdateTeamRequest with an optional team_id, since PATCH
takes the id from the path; a body team_id is still accepted when it matches
- validate the body through PatchTeamRequest before delegating to update_team
- declare the request body on the route and regenerate schema.d.ts
The handler keeps reading the raw body rather than declaring a typed parameter.
FastAPI validates a declared body before the handler runs, which would replace
the 400 for a non-object body with a 422 and move absent-vs-null out of reach of
the RFC 7386 metadata merge; those are pinned by existing tests, so the schema is
declared on the route instead and every error path is unchanged.
Validation is shape-preserving: the body is dumped with exclude_unset so an
omitted field never reaches the write, an explicit null still clears, and a
partial object_permission does not gain sibling sub-keys, which would wipe them
given the column merges rather than replaces.
Tests extend the existing patch harness rather than replacing it.
* refactor(proxy): declare the PATCH /team/{team_id} body as a typed parameter
Replaces the hand-written OpenAPI declaration added earlier in this branch. The
route now takes data: PatchTeamRequest, so FastAPI generates the request body
itself and emits a $ref to the model instead of an inlined copy that would go
stale as fields are added.
The earlier approach was a workaround built on a wrong premise. Declaring the
body does not cost absent-vs-null: model_fields_set preserves it, which is how
POST /team/update already gets its tri-state, and a nested null inside metadata
survives validation untouched, so the RFC 7386 merge is unaffected.
The one real change is the status code for a malformed body. The route answered
400 for a non-object body and 500 for a wrongly typed field, reporting a caller
mistake as a server fault; both are now 422, matching POST /team/update and the
other typed management endpoints. The two tests that pinned the old parse-level
errors are replaced by one that pins the 422 through the ASGI stack, and the
handler drops its manual parsing entirely.
* fix(anthropic): strip uniqueItems + other unsupported array/object constraints from output_format schema
Anthropic's structured outputs (`output_format`) validate the JSON schema
against a strict subset and reject cross-element / count constraints that a
constrained-decoding grammar cannot enforce, returning a 400
`invalid_request_error`.
`filter_anthropic_output_schema` already stripped the numeric / string /
item-count constraints (minimum, maximum, exclusiveMinimum/Maximum, minLength,
maxLength, minItems, maxItems) but still let these through:
- uniqueItems
- contains / minContains / maxContains
- minProperties / maxProperties
so a request using them fails with e.g. "output_format.schema: For 'array'
type, property 'uniqueItems' is not supported".
This is provider-visible: newer Claude models on the native `output_format`
path (e.g. `azure_ai`) 400, while `vertex_ai` is unaffected because it is
forced onto the permissive tool-use path (#18625 / #19201).
Add the missing keywords to the unsupported-field set and the description map,
and skip the advisory description note for a disabled boolean constraint
(`uniqueItems: false`) so it isn't misdescribed as required.
* fix(anthropic): serialize contains sub-schema in output_format advisory note
Address Greptile review: the `contains` advisory note previously discarded the
sub-schema, so the description only said an item must match "a schema" without
saying which. It now serializes the sub-schema as JSON (e.g. "array must
contain an item matching: {\"type\": \"integer\", \"const\": 1}"), matching the
other stripped constraints which carry their value. Sub-schema (dict/list)
values are json.dumps'd; scalar constraints are unchanged.
* style(anthropic): apply ruff format to output_format filter change
* style(test): ruff format anthropic schema filter tests
* test(anthropic): cover output_format array/object constraint filtering in test_litellm tree
Mirrors the schema-filter tests under tests/test_litellm/ so the coverage
job exercises the new uniqueItems/contains/min-maxProperties handling and the
uniqueItems: false branch.
---------
Co-authored-by: Darien Kindlund <darien@kindlund.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Okta commonly sends SCIM membership removals as a filtered path with no request
body value, e.g. Groups PATCH members[value eq "uid"] and Users PATCH
groups[value eq "tid"]. The patch handlers pulled ids only from op.value, so
these removes were a silent no-op and the member or team was never dropped
Add a linear-time filter parser reused by both the Groups members path and the
Users groups path so the id is taken from the [value eq "..."] filter when
op.value is absent, for add and remove ops. The eq operator is matched
case-insensitively, both quote styles are accepted, and the quoted value is
unescaped. The path-filter fallback only fires when the request body value is
omitted, so an explicit empty value no longer resurrects the filter id, and the
compared value must be quoted per the SCIM filter grammar
The SSO and Email Server settings pages read only stored config, so a gateway
configured entirely through environment variables rendered every field blank
even though both features were live. Rather than add per-endpoint env fallback,
resolve each setting through one typed config object.
A FieldDescriptor names, for one setting, where it lives in the stored row
(db_key), which process env var carries it (env_var), whether it is a secret,
and its effective default. A pure resolve_fields reconciles a descriptor table
against the stored row and the process environment with a fixed precedence and
reports per-field provenance (db, env, default, or unset). The SSO descriptor
table single-sources the field-to-env mapping that the read and write paths
previously duplicated, so they can no longer drift.
get_sso_settings and the /get/config/callbacks alerting block read through the
resolver instead of their own inline fallbacks. get_sso_settings no longer
decrypts stored values into os.environ; decryption happens once inside the
resolver via the pure helper, so a GET stops mutating the process environment.
The SSO response carries provenance so the UI can distinguish an env-sourced
value from a stored one, and secrets are masked at the endpoint (the resolver
returns them unmasked so the login path could consume them). os.environ remains
the runtime carrier; the SSO login and mail-send paths are unchanged.
The settings pages also submit only fields an admin actually edited, so a
rendered mask or env-sourced value is never written back over a working
secret, and generic_scope is a real SSO form field. Omitting a field from
/update/sso_settings clears it, which provider switching relies on; the deeper
write-path concern that behaviour points at is tracked in LIT-4498.
_add_team_members_to_team reconciled membership by reading the complete_team_data
snapshot captured at the start of team_member_add, appending in memory, and
writing the whole members_with_roles array back. Two concurrent /team/member_add
calls for the same team read the same snapshot, so the last write wins and one
member is silently lost. This affects every concurrent team member add, including
the SCIM group PATCH op:add path that routes through team_member_add
Reconcile members_with_roles inside a transaction that locks the team row with
SELECT ... FOR UPDATE before re-reading the current membership, so concurrent
writers serialize on the row lock and each appends onto the other's committed
result. The interactive transaction is exposed through a thin PrismaClient.tx()
passthrough and the locked read is encapsulated in
TeamRepository.get_members_with_roles_locked, and the SCIM group PATCH applies
membership as deltas so concurrent adds are not clobbered
A gateway whose native extension is unavailable falls back to the Python
implementation without raising, so it answers /v1/messages normally and the
only difference on the wire is the absent x-litellm-rust header. Nothing in
the suite read that header, so a Rust deployment that had stopped running
Rust produced a fully green e2e run.
Assert the marker on the streamed Messages assertions when E2E_EXPECT_RUST is
set. It stays opt-in because the same suite image also runs against the
standard gateway, which has no Rust path and must keep passing; the two
deployments are already separate Applications, so this is one value on the
Rust instance rather than branching inside the tests.
SCIM delete_user removed the user from the legacy team.members column and deleted
their team membership rows, but never pruned members_with_roles, which is the
source of truth ScimTransformations reads for GET /Groups/{id}. A deleted user
therefore lingered as a dangling member reference on every team they belonged to
Prune each of the user's teams directly via team_member_delete before deleting
the user row, and only for teams whose members_with_roles actually contain the
user, so a real DB failure surfaces (the endpoint fails loudly and the user is
kept; SCIM DELETE is idempotent, so the IdP retries) while a user who was never
in a team's members_with_roles stays a no-op. patch_team_membership is left
unchanged for its other callers
* fix(scim): sync team roster and dedup teams for existing-user email upsert
When POST /scim/v2/Users matched an already-existing user by email,
handle_existing_user_by_email raw-wrote the user's teams array but never
touched the team roster, so the user appeared in the group on their profile
yet was absent from the team directly (members_with_roles stayed empty). It
also did not dedup the teams built from repeated SCIM groups.
Route the existing-user team assignment through the same
_handle_team_membership_changes / team_member_add path the PUT update_user
handler uses, so members_with_roles, LiteLLM_TeamMembership, and the user's
teams array stay in sync, and dedup the teams derived from user.groups. The
user_id rewrite to the new userName is preserved and sequenced before the
roster sync so the roster never references a stale primary key.
* fix(scim): surface roster add failures on existing-user email upsert
Route the existing-email upsert's roster sync through patch_team_membership
with a new opt-in raise_on_error flag so a genuine team_member_add failure
propagates instead of being swallowed, and the deduped teams array is only
persisted after the roster sync succeeds. Without this, a failed add left the
endpoint reporting success while user.teams listed a team members_with_roles
never received.
The benign already-a-member case stays a no-op even under the strict path, and
the flag defaults to False so the PUT update_user, PATCH patch_user, and group
callers keep their existing best-effort behavior. SCIM POST is idempotent, so
surfacing the error lets the IdP retry and converge.
* fix(scim): surface roster removal failures symmetrically with adds
Make team_member_delete failures fail loud under the strict roster sync used by
the existing-email upsert, mirroring the add path, so a swallowed removal can no
longer let the user's teams array drop a team the roster still holds. The
idempotent case where the user is already absent from the team stays a no-op,
matching how an add treats the user already being in the team. Best-effort
behavior is preserved for the default raise_on_error=False callers.
* fix(scim): use members_with_roles as the source of truth for group membership
SCIM group provisioning tracked membership inconsistently. Team creation and
the real team endpoints persist membership in members_with_roles (and each
member's user.teams), but the SCIM group PATCH handler and the GET /Groups
listing read the legacy team.members String[] column, which team creation never
populates. Seeding a PATCH result from that empty column made an Okta "add
member" operation recompute the member set from scratch and silently drop
everyone already in the team, so users ended up missing from the groups they
were provisioned into. Reading the same empty column on GET /Groups reported an
empty member list back to the IdP, which drove repeated re-provisioning.
Separately, add_new_member appended the team id to user.teams with an
unconditional array push. Under the concurrent group PATCHes an IdP sends during
a reconcile, each request passed the members_with_roles duplicate check and
pushed, so user.teams accumulated duplicate ids for the same team. A duplicate
also breaks auth logic that keys off the number of teams a user belongs to.
Read current membership from members_with_roles in the SCIM group PATCH seed and
the GET /Groups listing, and make the user.teams append idempotent via a
filtered update that no-ops once the team is present.
Resolves LIT-4283
* fix(scim): address review; atomic user-creation and stop writing legacy members
Keep the concurrent-safe team append but create the user via an atomic upsert
(create-or-update) instead of a check-then-create, so provisioning the same new
user concurrently cannot race into a duplicate-key failure; the team is still
appended idempotently by a filtered update so an existing user gets no duplicate
team id. Stop writing the legacy team.members column in the group PATCH apply so
the only membership record is the source of truth (members_with_roles plus each
member's user.teams), reconciled by team_member_add/team_member_delete.
Tests: existing add_new_member and team-creation mocks updated to the upsert
plus filtered-append shape, and new tests cover atomic creation and that the
PATCH apply does not write the legacy members column.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
* fix(proxy): share CLI SSO login sessions across workers without enable_redis_auth_cache
* fix(proxy): make CLI SSO flow state redis-authoritative across workers
The CLI SSO flow is stored in a DualCache whose get_cache is memory-first, so
the worker that served /sso/cli/start keeps serving its stale in-memory flow and
never observes the sso_complete/session_data update another worker writes during
the OAuth callback. Attaching Redis alone is not enough; poll on the original
worker returns pending forever.
Read and write the flow directly through the attached Redis backend when present
so every worker sees the same authoritative state, falling back to the in-memory
DualCache only when no Redis is configured.
* fix(proxy): serialize CLI SSO flow as JSON for the redis round trip
RedisCache stores values via str(value) and parses reads with
json.loads then ast.literal_eval. The completed flow contains a
LitellmUserRoles enum in session_data.user_role, whose repr is not a
parseable literal, so any worker reading the completed flow from redis
raised SyntaxError and returned 400 "CLI login session not found".
Writing the flow as json.dumps makes the round trip lossless (the enum
is a str subclass) and fails loudly at write time if a non-serializable
value is ever added to the flow.
* fix(proxy): point CLI SSO session-not-found hint at configuring Redis
The error message and warning still told users to set enable_redis_auth_cache,
but the CLI SSO session cache now gets Redis unconditionally whenever one is
configured, so that flag no longer affects CLI login
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(guardrails): add only_scan_new_messages for per-session incremental scanning
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): use fixed TTL constant and revert unrelated test formatting
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): run only_scan_new_messages in the unified apply_guardrail path
The initial wiring lived in BedrockGuardrail.async_pre_call_hook, but the proxy
routes Bedrock through the unified apply_guardrail interface, so the flag had no
effect live. Move incremental selection into apply_guardrail: filter the flat
texts list against per-session scanned hashes, skip the Bedrock call when nothing
is new, and mark hashes only after a successful (non-blocked) scan. Full-context
fallback is preserved when there is no session id, the cache is unavailable, or a
masking guardrail is configured.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover session-id fallbacks and mark_texts_scanned guards
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): fall back to full scan when incremental guardrail masks content, use shared cache
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover generic agent multi-turn incremental scan
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover incremental scan cache resolver fallbacks
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover flag interactions and /v1/messages incremental scan semantics
* feat(guardrails): make GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS env configurable
* test(guardrails): prove skip_system/skip_tool are enforced upstream of incremental scan
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
The General tab renders generalSettings with TypedDictionary and
prompt-caching rows filtered out, but the Update and Reset handlers
indexed into the unfiltered array, so any row rendered after a
filtered-out entry read another field's value. max_ui_session_budget is
the first General-tab row positioned after the prompt-caching entries,
so its Update sent that row's boolean and failed Dollar validation.
Reset also cleared the local input to null, which reads as unset or
unlimited while the backend had restored the default.
Handlers now resolve the row by field name and drop the index parameter,
and reset displays the row's field_default_value. Component tests drive
the real /config/list ordering through the actual clicks and fail under
either original behavior.
Every dashboard login mints a 24h session key whose max_budget comes from
litellm.max_ui_session_budget, and all dashboard LLM traffic (playground,
auto router per-tier Test Connection probes) spends against and is gated
by that one key. The $0.25 default locked sessions out mid-testing with
"Budget has been exceeded ... Max budget: 0.25" and the setting appeared
in no docs, no UI, and no error text, so it read as a hardcoded cap.
Raise the default to $1. Give the setting an explicit typed arm in the
config loader (float coercion for env-var strings, null disables the
cap). Surface it on the Admin UI General settings tab through the
existing litellm_settings bridge as a new Dollar field type (positive
USD, unbounded above; the existing Float type is validated to (0, 1] for
fractions), with a spec-level default so clearing the field restores $1
instead of silently removing the cap, and enroll it in
LITELLM_SETTINGS_SAFE_DB_OVERRIDES so UI edits propagate to peer workers.
* test(e2e): cover customer chat/messages cost + streaming paths
Fills five uncovered P0 registry cells matching the customer's confirmed stack
(OpenAI SDK, Bedrock, /v1/messages) and their per-request cost dependency:
- /v1/messages logs cost that matches the x-litellm-response-cost header (LIT-4076)
- OpenAI /chat/completions streams real content, and a non-streamed call is costed
- Bedrock Converse /chat/completions returns real content non-streamed and streamed
The streaming checks aggregate delta content and parse every chunk as JSON, so a
clean-but-empty stream or a truncated chunk fails instead of passing on a bare 200.
* test(e2e): add tool-use coverage for openai, bedrock converse, anthropic responses
Function-calling regression guards on the paths the customer's agentic SDK usage
exercises: OpenAI and Bedrock Converse /chat/completions, and Anthropic
/v1/responses. The model is forced to call a weather tool and the test asserts the
returned tool call names the function and carries JSON-parseable arguments with the
expected field, so a dropped tool_call or malformed argument JSON fails instead of
passing on a bare 200. Adds a minimal tool_calls field to the response OutMessage.
* test(e2e): cover bedrock converse responses + thinking
Adds llm.responses.bedrock_converse.basic/tool_use and
llm.chat_completions.bedrock_converse.thinking. The thinking test enables extended
thinking and requires reasoning_content plus a real answer, so a path that drops
the reasoning block fails rather than passing.
* test(e2e): cover bedrock embeddings + openai structured output and reasoning
Bedrock Titan embeddings return a real vector; OpenAI structured output must yield
schema-conforming JSON with the correct extracted values (age==42, not just valid
JSON); an OpenAI reasoning call must report reasoning tokens, so a non-reasoning
fallback fails. Adds response_format to ChatBody and reasoning-token details to Usage.
* test(e2e): cover vision + streaming tool calls on openai and bedrock converse
Vision on both providers must describe the image (not just 200); the streamed
OpenAI tool call is reassembled from its fragments and its argument JSON parsed, so
a stream that never completes the call or splits its JSON fails. Extends ChatMessage
content to a typed text/image union.
* test(e2e): cover openai prompt caching hit on repeated large prefix
A repeated large-prefix prompt must report cached prompt tokens on the second call,
so a cache regression that stops reusing the prefix (and silently re-bills full
input) fails here.
* test(e2e): cover openai audio speech + bedrock rerank and image generation
Marks the OpenAI TTS cell and adds Bedrock Titan rerank (top_n honored, scored) and
Bedrock Titan image generation (returns b64/url), the customer's non-chat AWS
surfaces.
* test(e2e): cover end-user (customer) create persistence
mgmt.end_user.new.happy_path: create an end-user via /customer/new and confirm
/customer/info reports it, the end-user-identity surface the customer relies on for
per-customer controls. Adds customer models + management-client methods.
* test(e2e): enforce key model allow-list on the passthrough route
other.auth.passthrough.model_allowlist_enforced: a key scoped to gemini must be
denied a claude call through the anthropic passthrough route (403), so custom-auth
scoping is not bypassable by going through passthrough instead of /chat/completions.
* test(e2e): address Greptile - assert stream data events, correlate messages spend by key
- streaming: assert len(stream_events) > 1 instead of chunks > 1, since chunks
counts the terminal data: [DONE] marker and would pass a single content event
- messages cost: correlate the spend row by the unique scoped key rather than the
Anthropic response id, which need not equal the proxy spend-log request_id
The UI theme and logging-callback read endpoints reported only stored
config while the features resolve their values from the process
environment, so a gateway configured purely through env vars showed
blank settings pages even though branding rendered and callbacks fired.
/get/ui_theme_settings read only litellm_settings.ui_theme_config;
logo_url and favicon_url now fall back to UI_LOGO_PATH and
LITELLM_FAVICON_URL when the stored config leaves them blank.
process_callback (the logging-callbacks block of /get/config/callbacks)
reported every callback env var as unset unless it lived in the config
environment_variables overlay; it now falls back to os.getenv, matching
the slack block. Secret values stay redacted for non-admins via the
existing callback role gate.
Stored values keep winning over the environment, so the UI-driven flow
is unchanged.
Resolves LIT-4667
The Cache Settings page read only the database row, so a response cache
pointed at Redis purely through REDIS_* env vars showed a blank page
while the cache worked. It also masked credentials on read with a
partial-reveal string and re-persisted whatever the form submitted, so
an admin who edited an unrelated field and pressed Save wrote the mask
string over the real Redis password, breaking auth.
GET /cache/settings now overlays the same REDIS_* kwargs the runtime
resolves from when the stored config leaves a field unset, and redacts
credentials with a fixed marker. POST /cache/settings restores the
stored secret behind any credential echoed back as the marker or omitted,
and drops an env-sourced marker rather than persisting it; the response
no longer echoes plaintext credentials. The connection test resolves a
redacted credential back to the stored value the same way. The dashboard
never prefills a credential and drops the marker from the save payload,
mirroring the Coordination Redis tab.
Resolves LIT-4315
Delete Key moved into the key info page's overflow dropdown (#34116) and the
credentials table's row actions moved into a shared DataTable overflow menu, so
both specs were clicking a button that no longer exists. Point them at the menu
items instead.
Add a CredentialsPanel unit test asserting the update payload drops the masked
api key and keeps the edited api base, so that guard is not held up solely by an
e2e a table migration can silently disarm.
The spend-log metadata schema gained a compression_savings key, so the
gcs pubsub v1 payload now carries it. The golden fixture was never
updated, and the comparator flags any key present in the payload but
absent from the fixture, so test_async_gcs_pub_sub_v1 failed on every
run. Pin the key as null rather than adding it to ignored_keys; the
value is deterministic on this path, so ignoring it would leave the
assertion blind to the field entirely.
* fix(e2e): reference client.proxy in mid-conversation native providers test
EndpointsClient exposes the shared ProxyClient as .proxy and has never had a
.gateway attribute, so these two calls raised AttributeError at runtime and
failed the tests/e2e basedpyright zero-error gate for any PR touching e2e
files. Introduced in 23b5b7d199.
* test(e2e): cover 12 non-core LLM coverage registry cells
Raises Non-Core LLMs registry coverage from 24/50 to 36/50 (overall 51.9%
to 54.8%). Four cells were already asserted by existing tests and only
gain their covers marker (openai embeddings, openai image generation,
openai TTS, cohere rerank); one is dual-marked onto the existing
spend-tracking embeddings test rather than duplicated.
New tests: bedrock and vertex embeddings, streaming TTS (asserts chunked
transfer encoding so a buffered body cannot pass), audio transcriptions
via the realtime suite's wav fixture, moderations flag/pass pair, and
files list/retrieve in the batches suite.
Harness: e2e_http.upload generalized to any form model with a
file_content_type override (batches path unchanged), new stream_binary
primitive + BinaryStream for binary chunked responses, transcribe and
moderations client methods, file retrieve/list client methods.
* fix(e2e): close streamed TTS response on error paths and surface the error body
With stream=True a non-2xx response returned with the body unread, keeping
the socket checked out until garbage collection; the sibling
_streaming_outcome already consumes resp.text on error. The response now
closes on every path and BinaryStream carries a bounded error_body so a
failed stream call is triageable.
* test(e2e): assert streamed TTS response carries no content-length
httpbin.org is an external dependency prone to transient 503s (caused the
stage failure); its echo-body assertion also doesn't exercise a real LLM
provider. Point the custom pass-through endpoint at the real Anthropic
Messages API instead. Anthropic doesn't echo headers back, but it gates
real behavior on two of them, which is enough to prove forwarding: a
static x-api-key configured on the endpoint (never supplied by the caller)
must reach upstream or the call 401s, and an invalid x-pass-anthropic-version
sent by the caller must reach upstream with the prefix stripped, which
Anthropic echoes verbatim in its 400 body. Verified live against a local
proxy and the real Anthropic API: valid version returns a real completion,
invalid version returns the exact marker in the 400 body.
* refactor(ui): migrate Tool Policies table onto the shared DataTable
Splits the old components/ToolPolicies.tsx into a data-owning panel, a thin
DataTable consumer and a getToolPoliciesTableColumns module, all under
components/ToolPolicies/. The hand-rolled tremor table, sort dropdowns and
Prev/Next pager are replaced by the shared DataTable in client mode, so
sorting, pagination and filtering now come from TanStack rather than local
state. Search moves to the toolbar global filter and the four facets (input
policy, output policy, team, key) move into a filter drawer; the facets match
exactly instead of by substring, so filtering on "trusted" no longer also
matches "untrusted"
Inline policy editing is preserved. The two policy columns still render a
PolicySelect directly in the row, with the per-row-per-column saving state and
the in-place row update kept in the panel that owns the data
The 15s live-tail poll is removed in favour of the toolbar refresh action, which
takes the auto-refresh out of the write path of the inline edits. The green
live-tail banner goes with it. The panel now reads through React Query with
window-focus and reconnect refetching disabled, so refresh stays manual; that
also removes the effect that previously needed a set-state-in-effect suppression
The metric cards, the Needs Review banner and the detail swap are unchanged.
Review still scrolls to the row when it is on screen, but no longer jumps
across pages, since the paginated order now lives inside the table
Drops the unused userRole prop threaded from the route through the view into
the table, and prunes the suppressions stranded by the file move
* fix(ui): make Tool Policies inline saves safe against concurrent edits and refresh
Two races in the inline policy editing path, both found by review.
Saving state was a single tool name per column, so starting a second row's save
re-enabled the first row while its PATCH was still in flight, and whichever
save finished first cleared the indicator for whichever row was in the slot.
Track the set of tool names currently saving per column instead, so each cell
disables and re-enables on its own request
A list fetch already in flight when a save landed would resolve afterwards and
overwrite the row with its pre-save snapshot, silently reverting a policy the
user had just changed and the server had already accepted. Cancel in-flight
queries before writing the row, which is the documented React Query ordering
for this; the stale response is then discarded and the refresh can be retried
Tightens the test helpers that hid the second bug: policy values are now
compared exactly rather than with toHaveTextContent, which substring-matches
and so let "untrusted" satisfy an assertion for "trusted"
sharp reaches the dashboard only as an optional dependency of next, which
pins it to ^0.34.5. A caret range on a 0.x version cannot resolve past
0.34.x, and every stable next through 16.2.11 still declares that same
range, so there is no transitive path to the 0.35 line. Add an overrides
entry, matching how the other pinned transitives in this package are
already handled.
The dashboard builds with output: "export" and images.unoptimized, so
sharp is never loaded; this keeps the lockfile current rather than
changing runtime behaviour.
