The read replica never received the operator's DB pool settings, so its
Prisma pool fell back to `num_physical_cpus * 2 + 1` and the configured cap
was not enforced. Both startup paths now pass the same params to the reader:
the CLI, and the componentized entrypoints that go through
`DatabaseURLSettings.apply_to_env`.
Only pool and timeout params are inherited, through a single allowlist both
paths share. Anything that decides which tables a query resolves against
stays on the writer, including entries smuggled in through
`database_extra_connection_params`, so a writer `search_path` cannot repoint
reader queries. Params the operator pinned on the replica URL still win.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: point the live web search, groq and vertex image suites at models that still exist
Three CircleCI jobs on the staging-to-main promotion are red because the models
their live suites call have been retired by the providers, not because anything
in litellm changed.
openai/gpt-4o-search-preview now answers "has been deprecated" (its dated id
gpt-4o-search-preview-2025-03-11 carries deprecation_date 2026-07-23), so the
two web search conformance tests and the web search cost tracking test move to
gpt-5-search-api, the current search model. It keeps mode chat,
supports_web_search and a search_context_cost_per_query map, so the cost
assertion still resolves.
groq/llama-3.1-8b-instant reached its deprecation_date of 2026-08-16 and Groq
answers "does not exist or you do not have access to it". It follows
groq/llama-3.3-70b-versatile to groq/openai/gpt-oss-120b, the same replacement
PR #37422 already picked. The proxy config that job boots routes on a */*
wildcard, so no config change is needed.
vertex_ai/imagen-3.0-fast-generate-001 404s with "was not found or your project
does not have access to it". Google retired the whole Imagen family across
Vertex and the Gemini API, so there is no Imagen id left to point at. The class
is removed rather than repointed: Vertex image generation is already covered
live by TestVertexAIGeminiImageGeneration on vertex_ai/gemini-2.5-flash-image,
and the Imagen request and response transformations keep their offline coverage
in tests/test_litellm/llms/vertex_ai/image_generation/.
Only live call sites move. Remaining references to the old ids sit in offline
cost-map and transformation tests, where the string is a lookup key and no
request leaves the process.
* chore(lint): ratchet the TQ005 ceiling down to the count this branch reached
Removing the retired TestVertexImageGeneration class cleared one TQ005
violation, so the gate demands the limit come down with it.
make lint-budget-update only lowers a limit by the delta a branch cleared, and
this ceiling already sat 2 above the base count, so the tool landed on 2834
while the gate wants the limit at or below the 2832 this branch reached. The
remaining 2 are that stale headroom, which is exactly what the gate is asking
to reclaim.
* test(ci): serve /moderations from the canned OpenAI mock
The otel proxy E2E job points its `openai/*` wildcard deployment at the
canned mock, and #37492 made `get_model_list` agree with
`get_available_deployment` on bare model names. /moderations now resolves
`omni-moderation-latest` to that wildcard deployment the way
/chat/completions already did, so the request lands on the mock, which
never implemented the route and answers a bare 404.
Add /moderations and /v1/moderations to the mock, returning an
OpenAI-shaped response with one result per input item.
* style(ci): annotate the new moderations locals as Final
The generic container handler returned response.content for endpoints marked
returns_binary before it ran any status or error check, so a non-2xx answer
from the provider was handed back to the caller as raw bytes. Asking for the
content of a container file that does not exist returned the provider's 404
error body as an opaque payload instead of raising.
Move the check ahead of the binary short-circuit and apply it to every
container file endpoint, falling back to the response text when the error body
is not JSON.
Entering a custom model name on the Add Model form crashed the whole page
to "This page couldn't load" (React error #185, maximum update depth
exceeded), taking the provider credential fields down with it, so the
model could never be created.
ConditionalPublicModelName kept a `tableKey` counter and bumped it from
an effect on every run to force the mappings table to remount. That was
harmless under antd, whose useWatch handed back the stored array. React
Hook Form's useWatch returns a fresh array each render, so the effect's
dependency changed every render, the effect bumped state again, and the
render loop never settled.
The table is driven by its `data` prop, so the remount counter buys
nothing: drop it, key the effects off the selection contents rather than
the array identity, and write model_mappings only when they actually
change. The two `react-hooks/set-state-in-effect` suppressions on this
file, which were recording exactly this bug, go with it.
A JWKS fetch had no retry, so a single connect timeout to the identity provider
failed authentication outright, and once the cached copy expired there was
nothing to fall back on. How that surfaced depended on the outage shape:
httpx.ConnectTimeout was missing from DB_CONNECTION_ERROR_TYPES so it fell
through to the generic auth handler as a 401 with an empty detail, while a read
timeout took the database path and reported a healthy database as unreachable.
Transport failures are now retried three times with a short backoff, and the
last-known-good JWKS stays usable for a bounded window past public_key_ttl.
That window is public_key_stale_ttl, a new config field defaulting to 3600s and
settable to 0 to fail closed. It is checked on every read against the current
setting rather than baked into the cache entry when it is written, so lowering
it binds immediately instead of waiting for entries written under the old value
to age out, which matters because a shared cache survives the restart an
operator performs to make the change take effect. A copy whose write time
cannot be established is not servable. Only httpx.TransportError unlocks the
stale copy, so an identity provider that answers at all, including with a
narrowed key set, revokes on the next refresh. Every stale serve logs the kid
it authenticated, how long ago that copy was refreshed, and how long until it
stops being trusted.
A sustained outage is remembered for 30s per key url, so it costs one fetch per
window instead of three timeouts per request serialised behind the refresh lock.
Non-200 JWKS responses now raise instead of being cached as the key set, which
previously let an error body overwrite the last-known-good copy. An unreachable
identity provider with no cached copy left returns 503 auth_provider_unavailable.
Resolves LIT-5524
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Every pod and uvicorn worker schedules its own CheckBatchCost poller against the
shared managed-object table, so two of them can select the same completed batch in
one polling window and both write an aretrieve_batch spend log for it, counting
that batch's cost twice.
Claim the row with a compare-and-swap on batch_processed, and skip the batch when
another pod already holds it. The claim sits immediately before the spend log is
written rather than before the results fetch, because batch_processed is also what
blocks deletion of the files the fetch reads and what keeps an unbilled row
selectable by later poll cycles, so claiming up front would strand the spend of any
worker that died mid-fetch. A failed spend log write hands the row back.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Some IdPs, ADFS among them, return only `sub` from UserInfo and put the real
identity claims in the ID token or the access token. Those users land in the
Admin UI with no username, email, groups or teams.
Adds an opt-in `GENERIC_INCLUDE_TOKEN_CLAIMS` that merges token claims into the
UserInfo response before the existing `GENERIC_USER_*_ATTRIBUTE` mappings run.
Precedence is UserInfo, then id_token, then access token, and it applies to both
the PKCE and non-PKCE login flows. With the flag unset, behavior is unchanged.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Adds a chat_completions route module to litellm-core, mirroring the messages
route, plus Anthropic Messages and Bedrock Converse provider configs. The
per-model `rust: true` opt-in now covers /chat/completions for both providers.
The core accepts an allowlisted subset (text conversations, non-streaming) and
returns CoreError::Unsupported for anything else, so tool calls, multimodal
content and streaming fall back to the Python path transparently.
Resolves LIT-5698
tiktoken's BPE merge loop is quadratic in the length of a single regex piece, so a long
run of one repeated character turns a multi-MB payload into minutes of CPU. Encoding in
bounded chunks makes that linear, at a drift of at most ~1 token per chunk boundary.
Chunking alone only makes the stall shorter, so the async paths now count in a worker
thread: tiktoken releases the GIL for its Rust encode, so the loop keeps serving other
requests while a count is in flight. The /utils/token_counter endpoint awaits the new
atoken_counter, and the router's async deployment selection counts off-loop and hands
the result to _pre_call_checks instead of making it count inline.
The chunk size knob is bounded to [1, 4096]: a non-positive value used to raise or
silently report zero tokens, and an arbitrarily large one restored the quadratic cost
this exists to remove. Out-of-range and unparseable values warn and fall back to 1024.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Budget reservation tokenized every request twice, once for the max-cost
estimate and once for the input-cost estimate, and again per pricing
candidate. Tokenizing is O(prompt) and ran inline, so admitting one large
request stalled every other request the worker was serving.
Count the input tokens once per request and reuse the counts for both
estimates. Prompts above 30K characters of input text are counted in a
worker thread so the event loop stays free. The size heuristic renders the
body rather than walking its values, so tool-schema property names count
toward the threshold, and it sizes every field the counter tokenizes,
tool_choice included.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The shared logging ThreadPoolExecutor uses an unbounded work queue, so
sync callbacks that fall behind request arrival pin every queued payload
in memory until the task restarts. Cap queued-plus-running work with a
semaphore, shed submissions past the cap, and warn at most once every 30
seconds naming the knob that raises it. No caller of the shared executor
reads the returned future, so shedding is safe.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(helm): compose DATABASE_URL_READ_REPLICA from a reader host secret key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(helm): cover reader host composition and readReplicaUrlKey precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): suppress unused reader host env when readReplicaUrlKey is set
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): emit reader host only when readReplicaUrl composition is active
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Agent registry CRUD (/v1/agents*) sat in agent_routes, which feeds
llm_api_routes, so DISABLE_LLM_API_ENDPOINTS returned "LLM API routes are
disabled for this instance." for every Admin UI Agents tab call. Split the
group the same way MCP is split: agent_inference_routes stays on the data
plane, agent_management_routes joins management_routes, and agent_routes
remains their union for keys configured with allowed_routes=["agent_routes"].
Non-admin callers reached agent CRUD through llm_api_routes before, so the
management paths also join self_managed_routes and the llm_api_routes virtual
key carve-out; the handlers already scope reads by role and 403 non-admin
writes.
Both new groups are tuples, so check_route_access now takes a Sequence and
matches wildcards through a generator instead of materializing an
intermediate list on every call.
A dropped connection anywhere in the budget reset tick used to abort the whole
phase, so every due key, user, team and budget tier stayed unreset until the
next tick ten minutes later. Route the job's DB calls through
call_with_db_reconnect_retry so a transport blip costs one reconnect instead.
Reads replay on any transport error, since re-running a SELECT has nothing to
double-apply. Writes are non-idempotent, a reset assigns spend = 0
unconditionally, so they narrow to DB_RETRY_SAFE_ERROR_TYPES: only a
ConnectError proves the statements never reached the database. A post-send
error like ReadError or ReadTimeout leaves the commit outcome unknown, and
replaying one that already landed would erase whatever was spent since, so
those keep the pre-existing behaviour of failing the tick.
Resolves LIT-5372
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(scim): fail group sync when a member add or user creation fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(scim): apply ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): surface the paginated fallback on Cost Optimization
The page streamed its fallback silently: useDailyActivityRange dropped
the hook's progress and cancel fields and CacheLeakageCard only showed
a loading state while empty. Extract the Usage page's fetch banner into
a shared PaginationStatusAlerts component, render it above the tabs,
and note on the cache leakage tables when pages are still arriving.
* fix(ui): gate the cache leakage streaming note on isFetchingMore only
loading also covers a fresh aggregated request over the previous
range's rows, where pagination copy mislabels stale data. Drop the
redundant component comment flagged against the repo comment policy.
* fix(ptu): refuse an incomplete config.yaml reservation the way the endpoints do
POST /model/new answers 400 when PTU fields are set without a team_id, a
ptu_effective_from, or the count and rate together. config.yaml ran none of
those checks, so the same deployment loaded and served, billing per token
while accruing no flat cost, with nothing logged.
The rule moves into litellm_core_utils.ptu_pricing so both paths state it
once. Registration refuses such a deployment and names it, and the proxy's
ignore_invalid_deployments keeps that to the one entry. Only enforced while
PTU cost attribution is enabled, so a proxy that never opted in is unchanged.
* refactor(ptu): build the refusal message in the module that owns the rule
router.py raised a message it composed itself, which put proxy-facing
wording on the shared SDK surface. ptu_config_error now takes the
deployment name and returns the whole sentence; the endpoints still ask
without a name and their 400 bodies are unchanged.
JSON-RPC 2.0 types `id` as string, integer or null, but
LiteLLMSendMessageResponse annotated it as a bare required `str`. Pydantic v2
dropped v1's int-to-str coercion, so an upstream agent echoing an integer id was
rejected outright, and a null id, which section 5 requires for an error that
cannot be correlated to a request, was rejected too. Both surfaced as -32603 with
a pydantic ValidationError in the message: five distinct 500s on
/a2a/{agent_id}, across message/send and tasks/get.
Everything around the model already handled the full union: the endpoint reads
the id off the body as Any, its helpers are typed `str | int | None`, the error
builder takes `object`, and the streaming path passes the id through untouched.
The response model was the only narrowing left.
Backfilling an id the agent omitted keeps the caller's type too, since JSON-RPC
requires the response id to equal the request id and a caller that sent 7 cannot
correlate a response carrying "7".
`bool` is excluded from the integer half even though it subclasses `int`, so a
boolean id is stringified rather than relayed as 1 or 0, where it would collide
with a real integer id another in-flight request may be using.
* feat(ci): freeze the conftest save/restore inventory so it can only shrink
* fix(ci): resolve the named constant a conftest save loop iterates
* fix(ci): match the snapshot shape instead of a list of blessed dict names
* feat(ci): fail a branch that clears TQ violations without lowering the ceiling
A limit that only ever falls is not the same as one that falls when it can.
Clearing violations and leaving the ceiling above the new count let the same
violations return later under a limit nobody moved, so the gate now fails on
that and names `make lint-budget-update` as the fix. It needs both head below
base and head below limit, so headroom already in the base is never blamed on
the branch that happens to run next.
Drops the seeded-rule exemption from the ratchet along with it. Its stated
reason was that the base tree predates a rule introduced on this branch, but
base counts are measured with the current checker, so such a rule is counted at
the base too and its grandfathered total was never at risk of reading as fixed.
Removing the exemption is what lets a newly seeded rule ratchet like the six
that came before it.
The base scan is skipped when the branch touches neither the test tree nor the
checker, since neither count can have moved.
* feat(ci): assert .github/workflows holds only workflows, correctly named
* style(tests): annotate the hygiene test module's names with Final
* fix(ci): report a .yaml workflow as a naming finding, not a stray
GitHub reads .yml and .yaml alike, so WF001 telling you to move a valid
.yaml workflow to .github/scripts/ was wrong advice. WF001 now covers only
files that are not workflows at all, and the .yml spelling this directory
keeps moves to WF004, which says to rename rather than relocate.
WF001 also never looked into subdirectories, since GitHub does not read
them either; the message now says so. The directory is injected rather
than read off a module constant, so the cases are testable without
monkeypatching.
* test: merge three stranded twins into the files that shadow them
The second mirror's last four files each share a filename with a live test, so
the previous commit could not move them. Three of the four turn out to be plain
additions: their classes collide with nothing in the live file, so the tests are
extra coverage that has sat unrun rather than a competing version of anything.
Appending them takes the three files from 156 collected tests to 196, and all
196 pass. The 40 recovered are 13 OCI cases covering key normalization,
credential validation, complete-URL building and image-url transformation, 15
management-endpoint cases covering empty-value handling and the premium check,
and 12 DeepSeek thinking-parameter cases.
One assertion had to change. test_map_reasoning_effort_none_does_not_enable_thinking
asserted that reasoning_effort='none' leaves no thinking key, while the handler
maps it to {'type': 'disabled'} on purpose, documented in map_openai_params as
the OpenAI-style way to ask for thinking off. The test's stated intent holds,
since disabled does not enable anything, so it now asserts the disabled mapping
instead of the key's absence. Two imports moved to module scope for the
appended code, and no live test was touched.
test_discoverable_endpoints.py is the one left. Its twin grew from 1268 lines
to 9434, 25 of its assertions fail against today's code, and only 5 of its 19
tests have no counterpart, so deciding what survives that rewrite is a
judgement about the endpoints rather than a merge. The allowlist now holds
exactly that file and that reasoning.
* test(oci): stop the OCI suite reading credentials from the environment
validate_environment falls back to os.environ for every OCI credential and only
defaults the region when OCI_REGION is unset, so on a machine with OCI
configured the missing-credential test finds credentials it never passed and the
default-region test builds a URL for the ambient region. The suite then passes
or fails depending on who runs it.
A fixture drops the seven OCI variables for the four classes this branch added
and for TestOCIChatConfig, which had the same dependency before any of this and
fails the same way: with OCI_USER and friends exported, two of its cases fail on
origin/litellm_internal_staging today.
clean env: 83 passed
ambient OCI env: 83 passed
Same numbers either way, where the pre-existing file gave 68 passed / 2 failed
under the second.
Coverage is the single biggest time lever on the unit shards: the legacy
no-coverage workflow ran the same directory in about 5 minutes against 11 to 13
with coverage on. coverage.py's sys.monitoring backend (PEP 669) is the cheapest
core it ships, and it is not in use here today.
It has to be asked for explicitly. coverage 7.14 only defaults to sysmon from
Python 3.14 (`SYSMON_DEFAULT = CPYTHON and PYVERSION >= (3, 14)`) and these
shards pin 3.12, so without `COVERAGE_CORE` they get the slow tracer.
The audit left open whether sysmon survives turning on branch coverage. It does
not, at this Python. coverage gates branch measurement under sysmon on
`branch_right_left`, which needs newer than 3.14.0a5; on 3.12 it refuses and
falls back to the default core with a `no-sysmon` warning. Verified directly
against Python 3.12.13 with coverage 7.14.0:
$ COVERAGE_CORE=sysmon python -m coverage run --branch --source=. run.py
CoverageWarning: Can't use core=sysmon: sys.monitoring can't measure
branches in this version, using default core (no-sysmon)
So this speedup and `branch = true` are mutually exclusive until the runners
move to 3.14. Nothing here turns branch coverage on, so the two never collide
in this change, but whoever does turn it on is choosing to give this back.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
MCP egress prefixed the configured scheme unconditionally, but callers legitimately supply
both a bare token (from a stored credential) and an already-schemed value (passed through
from the caller's x-mcp-auth or Authorization header). The second shape produced
Authorization: Bearer Bearer <jwt>, which upstream servers reject as a malformed token. It
presented intermittently because a resolved stored credential arrives via extra_headers and
overwrites the doubled header, so only users without one always failed.
strip_auth_scheme drops one leading scheme before the header is rebuilt. It matches the
scheme case-insensitively per RFC 7235 and requires a credential behind it, so both a token
that merely begins with the scheme text and a scheme with nothing behind it are left intact.
MCPAuth.authorization stays verbatim because that auth type means the caller owns the whole
header value.
For MCPAuth.basic the normalization has to happen in update_auth_value rather than at
header-build time: to_basic_auth has already encoded the whole "Basic <credentials>" string
by then, so no prefix is left to find. A schemed value whose remainder decodes is already
encoded and is reused; one that does not decode is the bare pair with the scheme written in
front of it, and is encoded rather than forwarded as an invalid header.
The same doubling reached OpenAPI-backed servers through _format_byok_openapi_auth_header. A
non-BYOK server short-circuits _resolve_byok_mcp_auth_header, so that formatter also receives
the deprecated global x-mcp-auth, which is already a complete header value.
store_user_oauth_credential refused to overwrite any existing row that did not
decode as an OAuth2 payload, which conflated two states: a live BYOK secret that
reads back as plaintext, and ciphertext written under a LITELLM_SALT_KEY the proxy
no longer holds. The second is unrecoverable by any caller, so refusing preserved
nothing and instead wedged the user out of the OAuth flow permanently, since
re-authorizing is their only recovery.
The guard now raises only when the existing value is genuinely readable. An
undecryptable row is logged and replaced by the newly authorized token.
Both read paths were equally silent: get_user_oauth_credential and
list_user_oauth_credentials (which backs the bulk prefetch) each dropped an
undecryptable row indistinguishably from "user never authorized", so an operator
saw an upstream 401 and no hint that a credential had failed to decrypt. Both now
warn with the user and server ids, never the stored value.
AKS workload identity injects AZURE_CLIENT_ID, AZURE_TENANT_ID, and
AZURE_FEDERATED_TOKEN_FILE into the pod, and never a client secret.
Reading that bare client id as a managed identity sent the pod to IMDS,
which has no identity attached to it, so the token request failed and the
federated token was never exchanged.
AZURE_FEDERATED_TOKEN_FILE now wins over the bare client id and infers
DefaultAzureCredential, whose chain reaches WorkloadIdentityCredential
before ManagedIdentityCredential. DefaultAzureCredential passes
AZURE_CLIENT_ID to both legs, so a plain user-assigned managed identity
still reaches the same identity it does today.
This is the credential path Azure recommends for passwordless Postgres on
AKS, and it also fixes the Azure OpenAI token provider, which infers its
credential the same way.
