SCIM roster writes were swallowed, so a group or user push returned 200 while the
team roster never received the membership. Surfacing the failure fixes that, but
aborting on the first failed write leaves the rest of the batch unattempted on top
of unrolled-back, which is worse than what it replaces.
Every roster write in a reconciliation is now attempted, and the ones that did not
land are reported together, naming each failed add and remove. Rollback would be the
other option and it is not safe here: the compensating write can fail too, and it can
strip a membership that pre-dated the push. SCIM reconciliation is idempotent, so a
named partial failure is what the IdP's next push needs to close the gap.
The reported status still follows the failures, so a unanimous 404 stays a 404 and
only a batch whose failures disagree falls back to 500.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The read replica never received the operator's DB pool settings, so its
Prisma pool fell back to `num_physical_cpus * 2 + 1` and the configured cap
was not enforced. Both startup paths now pass the same params to the reader:
the CLI, and the componentized entrypoints that go through
`DatabaseURLSettings.apply_to_env`.
Only pool and timeout params are inherited, through a single allowlist both
paths share. Anything that decides which tables a query resolves against
stays on the writer, including entries smuggled in through
`database_extra_connection_params`, so a writer `search_path` cannot repoint
reader queries. Params the operator pinned on the replica URL still win.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: point the live web search, groq and vertex image suites at models that still exist
Three CircleCI jobs on the staging-to-main promotion are red because the models
their live suites call have been retired by the providers, not because anything
in litellm changed.
openai/gpt-4o-search-preview now answers "has been deprecated" (its dated id
gpt-4o-search-preview-2025-03-11 carries deprecation_date 2026-07-23), so the
two web search conformance tests and the web search cost tracking test move to
gpt-5-search-api, the current search model. It keeps mode chat,
supports_web_search and a search_context_cost_per_query map, so the cost
assertion still resolves.
groq/llama-3.1-8b-instant reached its deprecation_date of 2026-08-16 and Groq
answers "does not exist or you do not have access to it". It follows
groq/llama-3.3-70b-versatile to groq/openai/gpt-oss-120b, the same replacement
PR #37422 already picked. The proxy config that job boots routes on a */*
wildcard, so no config change is needed.
vertex_ai/imagen-3.0-fast-generate-001 404s with "was not found or your project
does not have access to it". Google retired the whole Imagen family across
Vertex and the Gemini API, so there is no Imagen id left to point at. The class
is removed rather than repointed: Vertex image generation is already covered
live by TestVertexAIGeminiImageGeneration on vertex_ai/gemini-2.5-flash-image,
and the Imagen request and response transformations keep their offline coverage
in tests/test_litellm/llms/vertex_ai/image_generation/.
Only live call sites move. Remaining references to the old ids sit in offline
cost-map and transformation tests, where the string is a lookup key and no
request leaves the process.
* chore(lint): ratchet the TQ005 ceiling down to the count this branch reached
Removing the retired TestVertexImageGeneration class cleared one TQ005
violation, so the gate demands the limit come down with it.
make lint-budget-update only lowers a limit by the delta a branch cleared, and
this ceiling already sat 2 above the base count, so the tool landed on 2834
while the gate wants the limit at or below the 2832 this branch reached. The
remaining 2 are that stale headroom, which is exactly what the gate is asking
to reclaim.
The edit model is reached through the image generation path with fal's
image_urls param; /v1/images/edits is not wired for fal_ai and errors.
Point supported_endpoints at /v1/images/generations and say so in the
entry notes.
* test(ci): serve /moderations from the canned OpenAI mock
The otel proxy E2E job points its `openai/*` wildcard deployment at the
canned mock, and #37492 made `get_model_list` agree with
`get_available_deployment` on bare model names. /moderations now resolves
`omni-moderation-latest` to that wildcard deployment the way
/chat/completions already did, so the request lands on the mock, which
never implemented the route and answers a bare 404.
Add /moderations and /v1/moderations to the mock, returning an
OpenAI-shaped response with one result per input item.
* style(ci): annotate the new moderations locals as Final
The monitors are separate servers with their own password, so the data node's Entra or
IAM token has no standing there. Dropping the provider only when a Sentinel password was
configured left it in place for unauthenticated monitors, where redis-py sends it as an
AUTH the monitor rejects and async Sentinel discovery fails.
The generic container handler returned response.content for endpoints marked
returns_binary before it ran any status or error check, so a non-2xx answer
from the provider was handed back to the caller as raw bytes. Asking for the
content of a container file that does not exist returned the provider's 404
error body as an opaque payload instead of raising.
Move the check ahead of the binary short-circuit and apply it to every
container file endpoint, falling back to the response text when the error body
is not JSON.
Entering a custom model name on the Add Model form crashed the whole page
to "This page couldn't load" (React error #185, maximum update depth
exceeded), taking the provider credential fields down with it, so the
model could never be created.
ConditionalPublicModelName kept a `tableKey` counter and bumped it from
an effect on every run to force the mappings table to remount. That was
harmless under antd, whose useWatch handed back the stored array. React
Hook Form's useWatch returns a fresh array each render, so the effect's
dependency changed every render, the effect bumped state again, and the
render loop never settled.
The table is driven by its `data` prop, so the remount counter buys
nothing: drop it, key the effects off the selection contents rather than
the array identity, and write model_mappings only when they actually
change. The two `react-hooks/set-state-in-effect` suppressions on this
file, which were recording exactly this bug, go with it.
A JWKS fetch had no retry, so a single connect timeout to the identity provider
failed authentication outright, and once the cached copy expired there was
nothing to fall back on. How that surfaced depended on the outage shape:
httpx.ConnectTimeout was missing from DB_CONNECTION_ERROR_TYPES so it fell
through to the generic auth handler as a 401 with an empty detail, while a read
timeout took the database path and reported a healthy database as unreachable.
Transport failures are now retried three times with a short backoff, and the
last-known-good JWKS stays usable for a bounded window past public_key_ttl.
That window is public_key_stale_ttl, a new config field defaulting to 3600s and
settable to 0 to fail closed. It is checked on every read against the current
setting rather than baked into the cache entry when it is written, so lowering
it binds immediately instead of waiting for entries written under the old value
to age out, which matters because a shared cache survives the restart an
operator performs to make the change take effect. A copy whose write time
cannot be established is not servable. Only httpx.TransportError unlocks the
stale copy, so an identity provider that answers at all, including with a
narrowed key set, revokes on the next refresh. Every stale serve logs the kid
it authenticated, how long ago that copy was refreshed, and how long until it
stops being trusted.
A sustained outage is remembered for 30s per key url, so it costs one fetch per
window instead of three timeouts per request serialised behind the refresh lock.
Non-200 JWKS responses now raise instead of being cached as the key set, which
previously let an error body overwrite the last-known-good copy. An unreachable
identity provider with no cached copy left returns 503 auth_provider_unavailable.
Resolves LIT-5524
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Every pod and uvicorn worker schedules its own CheckBatchCost poller against the
shared managed-object table, so two of them can select the same completed batch in
one polling window and both write an aretrieve_batch spend log for it, counting
that batch's cost twice.
Claim the row with a compare-and-swap on batch_processed, and skip the batch when
another pod already holds it. The claim sits immediately before the spend log is
written rather than before the results fetch, because batch_processed is also what
blocks deletion of the files the fetch reads and what keeps an unbilled row
selectable by later poll cycles, so claiming up front would strand the spend of any
worker that died mid-fetch. A failed spend log write hands the row back.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Some IdPs, ADFS among them, return only `sub` from UserInfo and put the real
identity claims in the ID token or the access token. Those users land in the
Admin UI with no username, email, groups or teams.
Adds an opt-in `GENERIC_INCLUDE_TOKEN_CLAIMS` that merges token claims into the
UserInfo response before the existing `GENERIC_USER_*_ATTRIBUTE` mappings run.
Precedence is UserInfo, then id_token, then access token, and it applies to both
the PKCE and non-PKCE login flows. With the flag unset, behavior is unchanged.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Adds a chat_completions route module to litellm-core, mirroring the messages
route, plus Anthropic Messages and Bedrock Converse provider configs. The
per-model `rust: true` opt-in now covers /chat/completions for both providers.
The core accepts an allowlisted subset (text conversations, non-streaming) and
returns CoreError::Unsupported for anything else, so tool calls, multimodal
content and streaming fall back to the Python path transparently.
Resolves LIT-5698
redis-py awaits a redis_connect_func that is a coroutine function, so
dropping every connect func the async paths cannot convert took away an
auth path that worked.
tiktoken's BPE merge loop is quadratic in the length of a single regex piece, so a long
run of one repeated character turns a multi-MB payload into minutes of CPU. Encoding in
bounded chunks makes that linear, at a drift of at most ~1 token per chunk boundary.
Chunking alone only makes the stall shorter, so the async paths now count in a worker
thread: tiktoken releases the GIL for its Rust encode, so the loop keeps serving other
requests while a count is in flight. The /utils/token_counter endpoint awaits the new
atoken_counter, and the router's async deployment selection counts off-loop and hands
the result to _pre_call_checks instead of making it count inline.
The chunk size knob is bounded to [1, 4096]: a non-positive value used to raise or
silently report zero tokens, and an arbitrarily large one restored the quadratic cost
this exists to remove. Out-of-range and unparseable values warn and fall back to 1024.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Budget reservation tokenized every request twice, once for the max-cost
estimate and once for the input-cost estimate, and again per pricing
candidate. Tokenizing is O(prompt) and ran inline, so admitting one large
request stalled every other request the worker was serving.
Count the input tokens once per request and reuse the counts for both
estimates. Prompts above 30K characters of input text are counted in a
worker thread so the event loop stays free. The size heuristic renders the
body rather than walking its values, so tool-schema property names count
toward the threshold, and it sizes every field the counter tokenizes,
tool_choice included.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The shared logging ThreadPoolExecutor uses an unbounded work queue, so
sync callbacks that fall behind request arrival pin every queued payload
in memory until the task restarts. Cap queued-plus-running work with a
semaphore, shed submissions past the cap, and warn at most once every 30
seconds naming the knob that raises it. No caller of the shared executor
reads the returned future, so shedding is safe.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(helm): compose DATABASE_URL_READ_REPLICA from a reader host secret key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(helm): cover reader host composition and readReplicaUrlKey precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): suppress unused reader host env when readReplicaUrlKey is set
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): emit reader host only when readReplicaUrl composition is active
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Agent registry CRUD (/v1/agents*) sat in agent_routes, which feeds
llm_api_routes, so DISABLE_LLM_API_ENDPOINTS returned "LLM API routes are
disabled for this instance." for every Admin UI Agents tab call. Split the
group the same way MCP is split: agent_inference_routes stays on the data
plane, agent_management_routes joins management_routes, and agent_routes
remains their union for keys configured with allowed_routes=["agent_routes"].
Non-admin callers reached agent CRUD through llm_api_routes before, so the
management paths also join self_managed_routes and the llm_api_routes virtual
key carve-out; the handlers already scope reads by role and 403 non-admin
writes.
Both new groups are tuples, so check_route_access now takes a Sequence and
matches wildcards through a generator instead of materializing an
intermediate list on every call.
A dropped connection anywhere in the budget reset tick used to abort the whole
phase, so every due key, user, team and budget tier stayed unreset until the
next tick ten minutes later. Route the job's DB calls through
call_with_db_reconnect_retry so a transport blip costs one reconnect instead.
Reads replay on any transport error, since re-running a SELECT has nothing to
double-apply. Writes are non-idempotent, a reset assigns spend = 0
unconditionally, so they narrow to DB_RETRY_SAFE_ERROR_TYPES: only a
ConnectError proves the statements never reached the database. A post-send
error like ReadError or ReadTimeout leaves the commit outcome unknown, and
replaying one that already landed would erase whatever was spent since, so
those keep the pre-existing behaviour of failing the tick.
Resolves LIT-5372
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(scim): fail group sync when a member add or user creation fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(scim): apply ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The AUTH exchange it runs is the blocking client API, so on an async
connection send_command and read_response hand back coroutines nobody
awaits and the connect fails outright.
REDIS_URL-based async clients and every async connection pool dropped the
managed-identity credential the caller configured, so they connected
unauthenticated against an auth-enforcing Redis. The conversion from
redis_connect_func to a CredentialProvider now happens once, before any
branch, and covers the url, sentinel, cluster, and pool paths alike.
Also adds credential_provider to the cluster kwargs allowlist, which
silently filtered it out.
Route fal.ai's openai/gpt-image-2 endpoints through a dedicated transformation that maps OpenAI image params (n, size, quality, output_format) into fal's schema, and register the model in the cost map.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): surface the paginated fallback on Cost Optimization
The page streamed its fallback silently: useDailyActivityRange dropped
the hook's progress and cancel fields and CacheLeakageCard only showed
a loading state while empty. Extract the Usage page's fetch banner into
a shared PaginationStatusAlerts component, render it above the tabs,
and note on the cache leakage tables when pages are still arriving.
* fix(ui): gate the cache leakage streaming note on isFetchingMore only
loading also covers a fresh aggregated request over the previous
range's rows, where pagination copy mislabels stale data. Drop the
redundant component comment flagged against the repo comment policy.
