* fix(scim): use members_with_roles as the source of truth for group membership
SCIM group provisioning tracked membership inconsistently. Team creation and
the real team endpoints persist membership in members_with_roles (and each
member's user.teams), but the SCIM group PATCH handler and the GET /Groups
listing read the legacy team.members String[] column, which team creation never
populates. Seeding a PATCH result from that empty column made an Okta "add
member" operation recompute the member set from scratch and silently drop
everyone already in the team, so users ended up missing from the groups they
were provisioned into. Reading the same empty column on GET /Groups reported an
empty member list back to the IdP, which drove repeated re-provisioning.
Separately, add_new_member appended the team id to user.teams with an
unconditional array push. Under the concurrent group PATCHes an IdP sends during
a reconcile, each request passed the members_with_roles duplicate check and
pushed, so user.teams accumulated duplicate ids for the same team. A duplicate
also breaks auth logic that keys off the number of teams a user belongs to.
Read current membership from members_with_roles in the SCIM group PATCH seed and
the GET /Groups listing, and make the user.teams append idempotent via a
filtered update that no-ops once the team is present.
Resolves LIT-4283
* fix(scim): address review; atomic user-creation and stop writing legacy members
Keep the concurrent-safe team append but create the user via an atomic upsert
(create-or-update) instead of a check-then-create, so provisioning the same new
user concurrently cannot race into a duplicate-key failure; the team is still
appended idempotently by a filtered update so an existing user gets no duplicate
team id. Stop writing the legacy team.members column in the group PATCH apply so
the only membership record is the source of truth (members_with_roles plus each
member's user.teams), reconciled by team_member_add/team_member_delete.
Tests: existing add_new_member and team-creation mocks updated to the upsert
plus filtered-append shape, and new tests cover atomic creation and that the
PATCH apply does not write the legacy members column.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
Greptile review: _cache_team_object runs after a successful DB fetch in
get_team_object and after every team mutation's DB write, but
DualCache.async_delete_cache propagates backend errors, so a Redis blip
during invalidation would turn a healthy team lookup into a 404 and a
committed /team/update into a 500. Both the internal usage cache delete
and the alias-key invalidation now log a warning and continue on failure,
matching how DualCache.async_set_cache already swallows write errors.
Worst case on failure is worker-local staleness bounded by the internal
cache's in-memory TTL, the same bound other workers already have
get_team_object consults proxy_logging_obj.internal_usage_cache before
user_api_key_cache, but _cache_team_object (the refresh every team
mutation goes through) only wrote user_api_key_cache. With
enable_redis_auth_cache both caches share one Redis, so any request
backfills the internal cache's in-memory tier with the team object and
that copy keeps shadowing the freshly written team until its TTL expires.
The auth builder then wrote the team object it had just read back into
the cache after check 6, clobbering the fresh Redis value with the stale
one, which made the staleness self-sustaining under traffic: keys with
models=["all-team-models"] kept getting 403 team_model_access_denied
for models added via /team/update, and kept access to removed ones.
_cache_team_object now deletes the internal usage cache entry before
writing the refreshed team, and the auth-time write-back is removed so
only authoritative writers (DB reads and team mutations) populate the
team cache, mirroring how key objects already handle this (see
test_auth_does_not_rewrite_cached_key_object_back_into_cache).
The LIT-4000 test pinning the removed write-back is deleted; its
concern (team object cached under the canonical key) is handled by
_cache_team_object inside get_team_object's DB path and pinned by
test_cache_team_object_writes_team_id_and_invalidates_team_alias
Resolves LIT-4391
* fix(proxy): share CLI SSO login sessions across workers without enable_redis_auth_cache
* fix(proxy): make CLI SSO flow state redis-authoritative across workers
The CLI SSO flow is stored in a DualCache whose get_cache is memory-first, so
the worker that served /sso/cli/start keeps serving its stale in-memory flow and
never observes the sso_complete/session_data update another worker writes during
the OAuth callback. Attaching Redis alone is not enough; poll on the original
worker returns pending forever.
Read and write the flow directly through the attached Redis backend when present
so every worker sees the same authoritative state, falling back to the in-memory
DualCache only when no Redis is configured.
* fix(proxy): serialize CLI SSO flow as JSON for the redis round trip
RedisCache stores values via str(value) and parses reads with
json.loads then ast.literal_eval. The completed flow contains a
LitellmUserRoles enum in session_data.user_role, whose repr is not a
parseable literal, so any worker reading the completed flow from redis
raised SyntaxError and returned 400 "CLI login session not found".
Writing the flow as json.dumps makes the round trip lossless (the enum
is a str subclass) and fails loudly at write time if a non-serializable
value is ever added to the flow.
* fix(proxy): point CLI SSO session-not-found hint at configuring Redis
The error message and warning still told users to set enable_redis_auth_cache,
but the CLI SSO session cache now gets Redis unconditionally whenever one is
configured, so that flag no longer affects CLI login
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(guardrails): add only_scan_new_messages for per-session incremental scanning
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): use fixed TTL constant and revert unrelated test formatting
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): run only_scan_new_messages in the unified apply_guardrail path
The initial wiring lived in BedrockGuardrail.async_pre_call_hook, but the proxy
routes Bedrock through the unified apply_guardrail interface, so the flag had no
effect live. Move incremental selection into apply_guardrail: filter the flat
texts list against per-session scanned hashes, skip the Bedrock call when nothing
is new, and mark hashes only after a successful (non-blocked) scan. Full-context
fallback is preserved when there is no session id, the cache is unavailable, or a
masking guardrail is configured.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover session-id fallbacks and mark_texts_scanned guards
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): fall back to full scan when incremental guardrail masks content, use shared cache
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover generic agent multi-turn incremental scan
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover incremental scan cache resolver fallbacks
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover flag interactions and /v1/messages incremental scan semantics
* feat(guardrails): make GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS env configurable
* test(guardrails): prove skip_system/skip_tool are enforced upstream of incremental scan
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
The previous check only consulted AWS_REGION* env vars before rejecting
custom hostnames, breaking deployments that configure their region via
the AWS shared config (profile). Resolve through boto3's session (env
vars + shared config) and only error when that chain yields nothing —
never sign with a silently guessed region.
urlsplit validates the port lazily, so a non-numeric port raised
ValueError out of _redact_mcp_resource_url after the urlsplit try
had already passed; the server loaders now call the helper while
warning about typo'd urls, which would have turned the warning into
a load failure. Resolve hostname and port inside the guard and pin
the malformed-port case in the redaction test
A typo'd MCP server url failed OAuth endpoint discovery silently: every
failure died at debug level, the config loader warned nothing, and the
/authorize 400 blamed "servers with no url" even when a url was set.
_descovery_metadata now records each attempt's outcome and, when a total
failure would leave the server's flow without a needed endpoint, logs one
warning with the trail (urls origin-only, exception text url-stripped).
Both server loaders warn which endpoints stayed unresolved for the
server's flow (client_credentials never needs authorization_url, OBO
needs only token_url) with the remedies; this replaces the DB path's
reason-less warning and closes the config path's no-warning gap. The
authorize/token/register 400 details branch on server shape via one
shared helper and point at the proxy logs. _redact_mcp_resource_url
moves to oauth_utils.py so the manager can import it without a cycle.
Resolves LIT-4658
Every dashboard login mints a 24h session key whose max_budget comes from
litellm.max_ui_session_budget, and all dashboard LLM traffic (playground,
auto router per-tier Test Connection probes) spends against and is gated
by that one key. The $0.25 default locked sessions out mid-testing with
"Budget has been exceeded ... Max budget: 0.25" and the setting appeared
in no docs, no UI, and no error text, so it read as a hardcoded cap.
Raise the default to $1. Give the setting an explicit typed arm in the
config loader (float coercion for env-var strings, null disables the
cap). Surface it on the Admin UI General settings tab through the
existing litellm_settings bridge as a new Dollar field type (positive
USD, unbounded above; the existing Float type is validated to (0, 1] for
fractions), with a spec-level default so clearing the field restores $1
instead of silently removing the cap, and enroll it in
LITELLM_SETTINGS_SAFE_DB_OVERRIDES so UI edits propagate to peer workers.
- Refuse to send the server-managed AGENTCORE_GATEWAY_TOKEN to a
caller-supplied api_base (reuses resolve_server_api_key's trusted-host
guard) — closes the token-exfiltration path via
/search_tools/test_connection
- Disable BaseAWSLLM's AWS_BEARER_TOKEN_BEDROCK fallback when signing:
that token is a Bedrock Runtime credential and must not reach an
AgentCore gateway
- Parse SSE responses per spec: join multi-line data fields, iterate
events, and return the JSON-RPC response (result/error) instead of the
first data line — progress notifications no longer shadow the result
- Validate tool_name ends with ___WebSearch so a caller-supplied name
cannot invoke unrelated tools on the same gateway with the proxy's
credentials
- Send the documented maxResults default (10) explicitly instead of
leaving it to the gateway
- Custom gateway hostnames: raise a clear error when no signing region
can be derived and none is configured, instead of signing for a
guessed region
- 7 new unit tests covering each fix (20 total)
* test(e2e): cover customer chat/messages cost + streaming paths
Fills five uncovered P0 registry cells matching the customer's confirmed stack
(OpenAI SDK, Bedrock, /v1/messages) and their per-request cost dependency:
- /v1/messages logs cost that matches the x-litellm-response-cost header (LIT-4076)
- OpenAI /chat/completions streams real content, and a non-streamed call is costed
- Bedrock Converse /chat/completions returns real content non-streamed and streamed
The streaming checks aggregate delta content and parse every chunk as JSON, so a
clean-but-empty stream or a truncated chunk fails instead of passing on a bare 200.
* test(e2e): add tool-use coverage for openai, bedrock converse, anthropic responses
Function-calling regression guards on the paths the customer's agentic SDK usage
exercises: OpenAI and Bedrock Converse /chat/completions, and Anthropic
/v1/responses. The model is forced to call a weather tool and the test asserts the
returned tool call names the function and carries JSON-parseable arguments with the
expected field, so a dropped tool_call or malformed argument JSON fails instead of
passing on a bare 200. Adds a minimal tool_calls field to the response OutMessage.
* test(e2e): cover bedrock converse responses + thinking
Adds llm.responses.bedrock_converse.basic/tool_use and
llm.chat_completions.bedrock_converse.thinking. The thinking test enables extended
thinking and requires reasoning_content plus a real answer, so a path that drops
the reasoning block fails rather than passing.
* test(e2e): cover bedrock embeddings + openai structured output and reasoning
Bedrock Titan embeddings return a real vector; OpenAI structured output must yield
schema-conforming JSON with the correct extracted values (age==42, not just valid
JSON); an OpenAI reasoning call must report reasoning tokens, so a non-reasoning
fallback fails. Adds response_format to ChatBody and reasoning-token details to Usage.
* test(e2e): cover vision + streaming tool calls on openai and bedrock converse
Vision on both providers must describe the image (not just 200); the streamed
OpenAI tool call is reassembled from its fragments and its argument JSON parsed, so
a stream that never completes the call or splits its JSON fails. Extends ChatMessage
content to a typed text/image union.
* test(e2e): cover openai prompt caching hit on repeated large prefix
A repeated large-prefix prompt must report cached prompt tokens on the second call,
so a cache regression that stops reusing the prefix (and silently re-bills full
input) fails here.
* test(e2e): cover openai audio speech + bedrock rerank and image generation
Marks the OpenAI TTS cell and adds Bedrock Titan rerank (top_n honored, scored) and
Bedrock Titan image generation (returns b64/url), the customer's non-chat AWS
surfaces.
* test(e2e): cover end-user (customer) create persistence
mgmt.end_user.new.happy_path: create an end-user via /customer/new and confirm
/customer/info reports it, the end-user-identity surface the customer relies on for
per-customer controls. Adds customer models + management-client methods.
* test(e2e): enforce key model allow-list on the passthrough route
other.auth.passthrough.model_allowlist_enforced: a key scoped to gemini must be
denied a claude call through the anthropic passthrough route (403), so custom-auth
scoping is not bypassable by going through passthrough instead of /chat/completions.
* test(e2e): address Greptile - assert stream data events, correlate messages spend by key
- streaming: assert len(stream_events) > 1 instead of chunks > 1, since chunks
counts the terminal data: [DONE] marker and would pass a single content event
- messages cost: correlate the spend row by the unique scoped key rather than the
Anthropic response id, which need not equal the proxy spend-log request_id
The UI theme and logging-callback read endpoints reported only stored
config while the features resolve their values from the process
environment, so a gateway configured purely through env vars showed
blank settings pages even though branding rendered and callbacks fired.
/get/ui_theme_settings read only litellm_settings.ui_theme_config;
logo_url and favicon_url now fall back to UI_LOGO_PATH and
LITELLM_FAVICON_URL when the stored config leaves them blank.
process_callback (the logging-callbacks block of /get/config/callbacks)
reported every callback env var as unset unless it lived in the config
environment_variables overlay; it now falls back to os.getenv, matching
the slack block. Secret values stay redacted for non-admins via the
existing callback role gate.
Stored values keep winning over the environment, so the UI-driven flow
is unchanged.
Resolves LIT-4667
The Cache Settings page read only the database row, so a response cache
pointed at Redis purely through REDIS_* env vars showed a blank page
while the cache worked. It also masked credentials on read with a
partial-reveal string and re-persisted whatever the form submitted, so
an admin who edited an unrelated field and pressed Save wrote the mask
string over the real Redis password, breaking auth.
GET /cache/settings now overlays the same REDIS_* kwargs the runtime
resolves from when the stored config leaves a field unset, and redacts
credentials with a fixed marker. POST /cache/settings restores the
stored secret behind any credential echoed back as the marker or omitted,
and drops an env-sourced marker rather than persisting it; the response
no longer echoes plaintext credentials. The connection test resolves a
redacted credential back to the stored value the same way. The dashboard
never prefills a credential and drops the marker from the save payload,
mirroring the Coordination Redis tab.
Resolves LIT-4315
The spend-log metadata schema gained a compression_savings key, so the
gcs pubsub v1 payload now carries it. The golden fixture was never
updated, and the comparator flags any key present in the payload but
absent from the fixture, so test_async_gcs_pub_sub_v1 failed on every
run. Pin the key as null rather than adding it to ignored_keys; the
value is deterministic on this path, so ignoring it would leave the
assertion blind to the field entirely.
* fix(e2e): reference client.proxy in mid-conversation native providers test
EndpointsClient exposes the shared ProxyClient as .proxy and has never had a
.gateway attribute, so these two calls raised AttributeError at runtime and
failed the tests/e2e basedpyright zero-error gate for any PR touching e2e
files. Introduced in 23b5b7d199.
* test(e2e): cover 12 non-core LLM coverage registry cells
Raises Non-Core LLMs registry coverage from 24/50 to 36/50 (overall 51.9%
to 54.8%). Four cells were already asserted by existing tests and only
gain their covers marker (openai embeddings, openai image generation,
openai TTS, cohere rerank); one is dual-marked onto the existing
spend-tracking embeddings test rather than duplicated.
New tests: bedrock and vertex embeddings, streaming TTS (asserts chunked
transfer encoding so a buffered body cannot pass), audio transcriptions
via the realtime suite's wav fixture, moderations flag/pass pair, and
files list/retrieve in the batches suite.
Harness: e2e_http.upload generalized to any form model with a
file_content_type override (batches path unchanged), new stream_binary
primitive + BinaryStream for binary chunked responses, transcribe and
moderations client methods, file retrieve/list client methods.
* fix(e2e): close streamed TTS response on error paths and surface the error body
With stream=True a non-2xx response returned with the body unread, keeping
the socket checked out until garbage collection; the sibling
_streaming_outcome already consumes resp.text on error. The response now
closes on every path and BinaryStream carries a bounded error_body so a
failed stream call is triageable.
* test(e2e): assert streamed TTS response carries no content-length
httpbin.org is an external dependency prone to transient 503s (caused the
stage failure); its echo-body assertion also doesn't exercise a real LLM
provider. Point the custom pass-through endpoint at the real Anthropic
Messages API instead. Anthropic doesn't echo headers back, but it gates
real behavior on two of them, which is enough to prove forwarding: a
static x-api-key configured on the endpoint (never supplied by the caller)
must reach upstream or the call 401s, and an invalid x-pass-anthropic-version
sent by the caller must reach upstream with the prefix stripped, which
Anthropic echoes verbatim in its 400 body. Verified live against a local
proxy and the real Anthropic API: valid version returns a real completion,
invalid version returns the exact marker in the 400 body.
Adds get_external to e2e_http.py for absolute third-party GETs (no proxy base url or auth, same Result classification) and rewires fetch_agent_card through it, dropping the urllib.request escape hatch. Creates tests/code_coverage_tests/check_e2e_no_raw_requests.py, the checker tests/e2e/CLAUDE.md already referenced, and wires it into the code-quality workflow so raw HTTP client imports outside the transport fail CI; pre-existing uses (root conftest liveness probe, claude_code version resolver) are grandfathered and exception-type-only imports stay allowed.
GET /batches served from the managed-objects table paged with a
where id > after filter, but the after cursor clients send back is a
batch's unified_object_id (the value returned as .id and last_id), and
id is the table's random-uuid primary key. Comparing the two unrelated
fields, while ordering by created_at desc but filtering with gt, made
pages repeat the same last_id (pagination loops) and silently drop
batches. Switch to Prisma cursor pagination on the unique
unified_object_id column so listing walks every batch exactly once in
reverse-chronological order, matching OpenAI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): add Azure AI Foundry and Anthropic /v1/messages coverage
* test(e2e): add Azure AI Foundry + Anthropic messages coverage for the Rust bridge
* test(e2e): guard against an empty SSE stream in the Azure Foundry tool-use streaming case
JWT auth built UserAPIKeyAuth without user_email even though the resolved
user row and the JWT email claim were both available, so the user_email
label on Prometheus metrics and user_api_key_user_email in
StandardLogging/SpendLogs metadata were always None for JWT traffic.
Plumb user_email through JWTAuthBuilderResult: auth_builder returns the
user row email when set, falling back to the user_email_jwt_field claim
(covers the scope-based proxy-admin path where no user row is loaded).
The JWT branch now stamps it on the proxy-admin return, the standard
valid_token, and the auto-registered virtual key object.
Resolves LIT-4238
The add_deployment and get_credentials background jobs that keep a multi-pod
deployment in sync with config-in-DB objects (models, credentials, guardrails,
general settings, etc.) polled the database on a hardcoded 30s interval, with
no way to trade convergence latency against DB load.
Expose it as the general_setting proxy_config_reload_interval_seconds (env
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS parsed via get_env_int, default 30),
threaded like the existing proxy_batch_polling_interval knob, and surface it on
the admin general-settings page so it is reachable from the dashboard and
persists to the DB for all pods. Non-positive values are rejected at the UI
(gt=0) and fall back to 30s with a warning on the env/config/DB paths.
The async @client wrapper stamped the global litellm.num_retries onto the raised
exception via setattr(e, "num_retries", ...), even on router calls where the
request-level num_retries had already been popped and resolved. async_function_with_retries
then adopted that stamped global value, overwriting the request-level num_retries it had
correctly resolved. So a per-request num_retries (request body or x-litellm-num-retries
header) was silently ignored whenever litellm_settings.num_retries was set.
Only stamp num_retries on the exception when the call itself carried one (an explicit
request value or a deployment's litellm_params.num_retries), never the global fallback.
The router already resolves the global via self.num_retries, so leaving the exception
unset preserves the request-level value and lets the per-deployment path set it when present.
Resolves LIT-4516
