* fix(ollama_chat): give each streamed tool call its own index
Ollama streams parallel tool calls one per chunk with no top-level index, and the delta
numbers the calls of each chunk from zero, so every call left the proxy as index 0 and
OpenAI-style clients appended the arguments of each later call onto the first one. Number
the calls with a counter that lives for the whole stream instead.
Fixes#33678
* fix(ollama_chat): answer 400 when replayed tool call arguments are not JSON
A chat history that already holds a merged tool call from an older stream
used to fail with a 500 APIConnectionError. It now answers 400 naming the
tool call. The streaming tests use Ollama's real tool call shape and drop
a case the merge base already passed.
* fix(ollama_chat): parse replayed tool call arguments with the shared repair parser
Empty, null, and truncated arguments in a replayed assistant tool call now go through parse_tool_call_arguments, so they are forwarded as an object instead of answering 500. Arguments that still are not a JSON object answer 400 naming the call id.
* fix(ollama_chat): send {} for null replayed arguments and keep argument text out of the repair warning
Replayed tool call arguments of the JSON string "null" used to reach Ollama and return 200. The shared parser turns them into None, so they now go out as {} like empty or missing arguments instead of answering 400.
The parser's repair warning logged up to 200 characters of the arguments. It now logs only the tool name, context, and length.
* fix(ollama): keep replayed tool arguments out of a rejected replay's error chain
* fix(ollama): keep the repair warning's argument preview unless message logging is off
The shared repair warning lost its 200-character argument preview for every provider. It now logs the same preview as before by default, and logs redacted-by-litellm instead when turn_off_message_logging is on.
* test(ollama): cover short and boundary-length repair warning previews
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(minimax): forward reasoning_effort instead of dropping it
* fix(minimax): forward reasoning_effort instead of dropping it
* fix(minimax): gate reasoning_effort on the cost map and map MiniMax-M3.1-Flash
Forward reasoning_effort for every MiniMax model unless its cost map entry says it
does not reason, so a MiniMax model newer than the map keeps the caller's effort
instead of losing it.
Add a minimax/MiniMax-M3.1-Flash entry with MiniMax's published pay-as-you-go
pricing and its accepted effort levels (low, medium, high, xhigh, max), and a
minimax-reasoning-baseline fallback rule that marks unmapped minimax/MiniMax-M*
ids as reasoning-capable, so thinking is kept for them too.
* refactor(minimax): gate reasoning_effort on supports_reasoning like thinking
Forward reasoning_effort when the cost map, or the MiniMax M-series fallback rule, says the model supports reasoning, the same check thinking already uses. An unlisted MiniMax model outside the M series now drops it, with the existing drop warning.
* refactor(minimax): build supported params in one expression and type the new tests
Build the MiniMax supported params list without mutating it, give the new test helpers precise types, and assert invariants (both cost map files agree, prices are positive and ordered, every declared effort level is forwarded) instead of pinning MiniMax's prices and effort levels as literals.
* fix(minimax): treat every unlisted minimax/ model as reasoning-capable
Widen the MiniMax fallback rule from M-series ids to the whole minimax/ namespace, matching the W&B rule, so an unlisted MiniMax model keeps reasoning_effort and thinking. Exact cost map entries still win, so mapped speech, video and image models are unaffected.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(auth): grant access groups through principal attribute rules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): satisfy cache tracing and recursion CI checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): close access group rule authorization bypasses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): reject self-editable identity rule attributes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Centralize JWT model authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Cover centralized key checks and name denied JWT models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Test centralized authorization for custom auth and JWT mappings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve model normalization before budget validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use a controlled clock for WebSocket session deadline tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Unify JWT credential admission and simplify team selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Simplify request-time access group authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use real requests in custom auth budget fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When the proxy parks the success log of a non-streaming response and the
router then discards that response (a safeguard refusal it falls back
from), the parked log was sent in place of the serving attempt's, so the
spend log and cost header reported only the refused attempt and the
serving attempt was never billed.
Carry the parked response as a billed attempt owed by whichever attempt
ends the request, the same way a superseded stream is already billed, so
one spend row and the cost header cover every billed attempt.
Fixes#49103
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(cost-map): add Claude Fable 5.2 on the Anthropic API, Vertex AI and Claude Platform on AWS
Adds claude-fable-5-2, vertex_ai/claude-fable-5-2 (plus @default) and
bedrock/claude_platform/claude-fable-5-2 at $8/$40 per MTok, $10 and $16
cache writes, $0.20 cache reads, 50% batch discount and a 2x fast mode
multiplier, and points claude-fable-latest and vertex_ai/claude-fable-latest
at the new rows. Bedrock converse rows ship separately in #50707.
* fix(cost-map): drop fast mode from the Vertex Fable 5.2 rows, the Vertex cost path does not bill it
* Merge main, drop the Anthropic API and Claude Platform rows that #50715 already landed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prompt_templates): keep moved private helpers importable from common_utils
The split moved 83 private helpers out of common_utils, so code importing them by
their old path got ImportError. A module __getattr__ now serves each one from its
new module. Also renames the new test files to the directory's naming convention
* fix(prompt_templates): hide the old-path getattr from type checkers and serve only released names
Type checkers stop seeing the module __getattr__, so a typo'd import from common_utils
fails lint again. The old path now serves only the 69 moved helpers that common_utils
had in a tagged release, since the other 14 were never importable from it
* fix(prompt_templates): let patches through the old common_utils path reach moved helpers and warn on old-path use
* fix(prompt_templates): sync old-path patches of re-exported helpers and restore moved helpers when warnings are errors
* fix(prompt_templates): reset a moved helper on old-path delete instead of removing it, and warn on old-path writes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL
* fix(prometheus): carry live spend for the user, org and customer budget gauges too
The remaining budget gauges for users, orgs and customers read the same cached rows as the key and
team gauges, so they read high for about a minute after a charge. Auth now carries the live counter
for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded.
The x-litellm-key-spend header and logging spend fields keep their current values.
* fix(prometheus): show committed spend in the remaining budget gauges
Reads twin counters that take only charged cost, so a worst-case budget hold
never shows as spent and a request that finishes last cannot set a gauge back.
The twins exist only while the Prometheus logger is registered, and every path
that resets a spend counter drops them. Failed requests now set the org gauge
from the carried org row and committed spend, not the cached org row.
* fix(prometheus): count a charge the cached customer row already holds once
* fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets
Auth now reads committed spend through the admission MGET and names the counters a charge
may move, so the master key and unbudgeted entities get no twin. Organization spend edits
and the global spend reset drop their twins, and a long request no longer reads as charged.
* fix(prometheus): read the committed org spend on failed requests without the auth object
* test(prometheus): cover a failed request with no carried org
* fix(prometheus): move the committed counters for charges with no auth on the row
A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them.
* fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes
Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a
charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed
counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys
and teams in bounded pages.
* fix(prometheus): count a charged failure's own cost in the org gauge
A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request
* fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog
* fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete
Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins
* fix(prometheus): keep committed spend values out of logger dumps of pass-through rows
CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values
* fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster
The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files
* fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(bedrock): add Claude Fable 5.2 and Mythos 5.2 offer index prices
* fix(cost-map): absorb verified prices from #50559
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): remove unsupported Gemini cache-write rates from #50504
Co-authored-by: Chester <42577861+superdiaodiao@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): remove retired Friendli models from #50692
Co-authored-by: Lee-Si-Yoon <76679207+Lee-Si-Yoon@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): give the eu and us-gov Mythos 5.2 profiles the base model's native structured output flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): give the eu and us-gov Mythos 5.2 profiles the base model's native structured output flag
* feat(anthropic): add claude-fable-5-2
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): add tool and caching flags to claude-fable-5-2
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): carry supports_mid_conversation_system on claude-fable-5-2
* fix(anthropic): declare Fable 5.2 mid-conversation system support
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): deduplicate concurrently added Fable capability
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): resolve merge conflict with main, keep main's claude-fable-5-2 row
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chester <42577861+superdiaodiao@users.noreply.github.com>
Co-authored-by: Lee-Si-Yoon <76679207+Lee-Si-Yoon@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(proxy): build plugin session claims from FIPS provider services in fips_mode
With general_settings.fips_mode on, the per-plugin key comes from the
salt key KBKDF and the Fernet token is built from ssl.RAND_bytes,
AES-128-CBC and hmac.digest. Standard mode is unchanged.
* test(proxy): freeze the clock in the fips_mode plugin claim test
* feat(proxy): add OpenSearch index pass-through with per-key read/write grants
Adds /opensearch/<index>/<api> for LiteLLM vector store indexes on OpenSearch
stores. A key or team needs read or write on the index in
allowed_vector_store_indexes, and the proxy signs each call with the vector
store's AWS identity, so callers never hold AWS credentials
The route maps the LiteLLM index name to the store's provider index, refuses
any path, query parameter, or body field that could reach another index, and
sends the exact signed bytes upstream with only the signed headers
* fix(proxy): let OpenSearch _bulk and _msearch NDJSON bodies through auth
opensearch-py sends _bulk and _msearch NDJSON with content-type application/json, and the
auth-time body read parsed it as one JSON document and answered 400 before the route ran.
Auth now skips parsing OpenSearch NDJSON bodies the same way it skips Azure Speech audio.
* feat(proxy): accept a _bulk _index naming the path index and allow named search pipelines
A _bulk action line may name the index in the URL path, and LiteLLM rewrites it to the
real index, which is what opensearch-py's helpers.bulk sends. The search_pipeline query
param now goes through so hybrid search works, while an inline pipeline stays refused.
Also types the type-checking Router stub with _router_init's parameters again, which
make lint's tests/e2e check needs since the Rust core change.
* fix(opensearch): answer 401 when the AWS credential source cannot read its token file
* refactor(proxy): check OpenSearch request bodies without recursive helpers
* fix(opensearch): answer 401 when AWS credentials fail to refresh at signing time
* fix(opensearch): keep auth off OpenSearch bodies and widen the read list to index checks
Auth no longer parses any OpenSearch request body, so a document field named model, api_base or base_url is written
instead of answering 401 or 403. GET and HEAD on the index and HEAD on _doc and _source are reads, each operation
checks the exact grant the list names, an index whose store name two stores share answers 400, and an _msearch
header that names the path index is sent to the mapped index
* fix(opensearch): need read for writes that return documents and keep document fields out of LiteLLM params
* fix(opensearch): keep documents out of LiteLLM hooks, keep main's credential statuses, make _msearch admin only, and bill only searches
* fix(proxy): bill only an exact /opensearch/<index>/_search path
A document read such as /opensearch/docs/_doc/_search ended in /_search and was counted as a billable search. The check now matches the index segment and _search exactly
* fix(proxy): only proxy admins set vector store index grants, and bill no OpenSearch document call
An index grant now opens document writes through /opensearch, so key generate, update, bulk update,
regenerate, team new and update, /user/new, and bulk user creation answer 403 when a non-admin
changes allowed_vector_store_indexes, top-level or under metadata. Resending the stored grants is not a change
An /opensearch path is matched before the LLM route suffixes, so a document id such as search or
embeddings never makes a document call count as a billable request
* fix(proxy): treat an empty index grant list from a non-admin as a change
An explicit empty top-level allowed_vector_store_indexes passed the admin-only check as unchanged and then
replaced the stored grants. It now counts as a change, so a non-admin gets 403. The new grant tests also
carry parameter types
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): derive the web identity session policy per AWS caller
Each AWS caller (Bedrock embeddings, batches, files, SageMaker, Polly, S3 Vectors, and the S3 and SQS loggers) now sends its own STS session policy on AssumeRoleWithWebIdentity, so a role that allows those actions is no longer capped by the Bedrock-only policy. The default Bedrock policy is unchanged, and the credential cache key includes the policy so callers never share a narrower session.
* fix(bedrock): let async invoke write its S3 output under the web identity session policy
Bedrock writes an async-invoke job's output with the caller's session, so the embedding session policy now allows s3:PutObject only when Bedrock makes the call (aws:CalledViaLast bedrock.amazonaws.com).
* fix(opensearch): give OpenSearch vector store search its own web identity session policy
* fix(bedrock): give every remaining AWS caller its own web identity session policy
Secrets Manager, knowledge base ingestion, model listing, the S3 config file reader, the SageMaker chat handler, and the Comprehend Medical and Transcribe pass-throughs each send a session policy with only the actions they call
Bedrock embeddings now use the Bedrock policy, so chat and embeddings share one STS session. That policy adds async-invoke status, S3 reads and writes only through Bedrock, KMS only through S3, and Mantle token counting
bedrock:InvokeModel gets its own statement without aws:SecureTransport, since Bedrock checks a Nova Sonic realtime stream as InvokeModel with that key false
KMS grants are limited by kms:ViaService, file downloads get a read-only S3 policy, and batches and speech config export skip underscore attributes
* test(batches): expect config export to skip every underscore attribute
The session policy class attribute is private, so batches config export no longer returns single-underscore names
* fix(bedrock): scope knowledge base ingestion role grants to the roles it creates
Knowledge base ingestion only creates and passes litellm-bedrock-kb-* roles, so its session policy no longer lets the session create, edit, or pass any other role, the proxy's own included
* fix(bedrock): give embeddings and the agent-runtime pass-through their own web identity session policies
* fix(bedrock): grant enforced guardrails and the remaining agent-runtime operations over web identity
Every web identity ceiling that grants bedrock:InvokeModel now also grants bedrock:ApplyGuardrail, so an
account or organization enforced guardrail no longer denies embeddings, the agent-runtime pass-through,
or realtime
The agent-runtime ceiling grants every POST operation the pass-through forwards, adding flow executions,
document content, and ingested document ACLs, and a test derives that list from botocore
Only the realtime caller gets bedrock:InvokeModel without the aws:SecureTransport condition, since chat and
embeddings never open a bidirectional stream
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): price expanded wildcard /model/info rows as the model their calls route to
* fix(proxy): read wildcard routing through the router registry entry points
* fix(proxy): price a wildcard row only when the router sends its name there
Use the router's own route candidates for the listed name, so an alias, a routing group, a more specific pattern, or a team's own deployment keeps a wildcard row unpriced
* fix(proxy): show no price on a wildcard row whose calls go to another deployment
* fix(proxy): keep the listed price on catch-all wildcard rows and match the edit path to the read path
* fix(proxy): price a wildcard row aliased to a sibling name as the alias target
* fix(proxy): price wildcard rows through the caller's alias renames and show nothing for rows another deployment takes
* fix(proxy): keep same-model wildcard row prices, skip deleted team aliases, use the caller's team
* fix(proxy): keep live team aliases when skipping deleted team alias targets
* fix(proxy): show no wildcard row price when the same-model deployment serving it sets custom rates
* fix(model_info): recognize a replayed wildcard row on a worker that has not loaded the wildcard
* test(proxy): feed wildcard discovery real provider data in model info tests
* fix(proxy): price a wildcard row another deployment serves as the model its calls bill
A row whose calls reach another deployment now shows that deployment's model
and limits, and goes blank only when those deployments bill more than one model
or declare their own rate. A name no route serves keeps its own price, and a
replayed row stores none of the metadata it showed.
Drop the router's per-call scan of every deployment for a team key, since the
team and name indexes already cover those names.
* fix(proxy): price a wildcard row aliased to an aliased group as the group its calls reach
An alias target that is itself aliased onward listed the onward group's deployments too, so the row showed no price while calls billed the first target's model. The billed model now comes only from the route candidates
* fix(proxy): price a wildcard row billed by another deployment without the wildcard's base_model
A row aliased to another deployment priced as that deployment's model, but the lookup still swapped in the source wildcard's base_model, so the row showed a rate its calls never bill
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cli): end the local login when logout cannot check the OS keychain
With no usable keyring, lite logout kept a secret-free token.json as a note
that the keychain went unchecked. Every reader took that note for a live
keychain-backed login, so lite whoami, lite auth print-token, and the bare
lite shell all reported a login the user had just ended, and a keychain that
answered again could pair the note with an earlier login's entry and sign
the machine back in.
Logout now removes token.json whatever the keychain answers. When the
keychain could not be checked and there was a login to end, it records that
in a separate owner-only keychain_unchecked file beside it, written after
the removal so a full disk still has room for it. A repeat logout still
warns while that file is there, and a logout that confirms the keychain is
clean removes it.
* fix(cli): keep warning after a logout the keychain refused to clear
A keychain that will not give up its entry gets the same note as one that
could not be checked, so a later logout made while the keychain is out of
reach does not report a clean logout while the credential is still there.
* fix(cli): revoke the login a logout left in an unreachable keychain once it answers
A logout made while the keychain was locked ended the local login and dropped the metadata that pairs with the refresh token still in the keychain, so the next logout had nothing to revoke. The not-cleared note now keeps that record without its secret, and a later logout pairs it with the keychain entry and revokes it before erasing.
* fix(cli): revoke a locked-keychain login on the next login, and only once after a refused erase
A login made after a logout that could not reach the keychain now revokes the
login it replaces, read from the keychain_not_cleared note, as the merge base
did through the kept token.json. A keychain that refused the erase had already
handed that logout the refresh token, so its note no longer pairs with the entry
and the next logout does not revoke the same token again.
* fix(cli): keep the first login revocable after a second locked-keychain logout
A logout whose login kept its secret in token.json never reached the
keychain, so it now leaves the note an earlier logout wrote in place, or
writes an empty one, instead of overwriting it. The first login's refresh
token stays revocable once the keychain answers, and a keychain-less
machine no longer keeps user and team ids in the note.
Also documents the limits that remain in the CLI README and drops the
docstrings from the tests this PR added.
* fix(cli): revoke the keychain's login after a login made while the keychain was locked
* fix(cli): stop lite from reading the stored login before login and logout
The cli group resolved the stored API key before every command, and that read moved a
locked-keychain login's secret over the keychain entry the logout note names, so logout and
login never saw the first login to revoke it.
* fix(cli): keep the login a locked logout left in the keychain revocable
While the keychain still holds the login an earlier logout could not
end, reading the stored login now serves the secret token.json kept and
leaves that keychain entry alone, whatever the login times say. A clock
that stepped back no longer serves the ended login, and commands other
than logout and login no longer move the newer secret over it. A silent
renewal that writes over the entry revokes that login on the proxy.
* fix(cli): keep a login revocable when its note cannot be written, and never block logout on an earlier login's proxy
A locked logout whose note cannot take the login's record now puts token.json back, so the next logout that reaches the keychain still revokes that login, as the base did. A logout that ends both a current login and an earlier one left in the keychain warns when the earlier login's proxy cannot record the revocation, as lite login does, instead of refusing to log out of the current proxy.
* fix(cli): move a locked keychain login's token file into its note instead of putting it back
* fix(cli): never tell a logout to delete the record of a keychain login it could not move aside
A read-only ~/.litellm made the locked-keychain logout say to delete token.json, which would leave the keychain's login unrevoked. It now says the logout did not happen and asks to make the directory writable. The README also says an older lite cannot finish a logout this release started.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): total_timeout bounds a whole non-streaming request
A `total_timeout` body field or `x-litellm-total-timeout` header (the header wins) gives a non-streaming
/v1/chat/completions, /v1/messages or /v1/responses request one budget for preparation, retries,
moderation and post-call processing. Past it the proxy answers 408 and cancels the upstream call, and
a retry whose backoff cannot fit ends early with the error it would have retried. A body value on a
streaming or background request is refused with a 400, a header value leaves it unbounded. A timed out
request is billed like a client disconnect.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): ignore any total_timeout header on requests it cannot bound, and never interrupt pre-call admission
A globally configured x-litellm-total-timeout that is malformed no longer fails the streaming and
background requests it is ignored on. The deadline holds while pre-call admission runs, since admission
reserves shared quota before it records it, and an expired deadline then ends the request before
dispatch, where the failure path hands the reservation back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): a total_timeout cuts slow pre-call hooks short, holding only the rate limit reservation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(key/info): return 400 instead of 500 when no key is provided
When GET /key/info is called without a ?key= query parameter and the
authenticated principal has no virtual key (e.g. an admin-UI session
token whose UserAPIKeyAuth.api_key is None), info_key_fn passed
hashed_key=None to Prisma find_unique(where={"token": None}), which
raised MissingRequiredValueError and surfaced as an unhandled 500.
Add an explicit None/empty guard that raises a clean 400 ProxyException
before the DB lookup, and add a regression test asserting that prisma
find_unique is never called with token=None.
Fixes#43571
Signed-off-by: apex-mochen <2756823972@qq.com>
* test: remove redundant assertion comment
* fix(key/info): remove redundant None check to satisfy basedpyright delta gate
After if not key: raise, key is narrowed to str. The subsequent
if key is not None: guard is always true and triggers basedpyright
reportUnnecessaryComparison, which fails the delta gate.
Collapse to a single hash_token_if_needed(token=key) call.
Signed-off-by: apex-mochen <2756823972@qq.com>
---------
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(accounting): stop delivery metrics from scanning retained receipts on every flush
The backlog query now reads only pending and parked deliveries, so its cost follows the backlog instead
of retained receipt history. Batch count and table size are measured once per check interval per pod,
and the storage and delayed-delivery alerts repeat at most once per interval instead of every flush
Fixes#50475
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): prove the storage query reruns only when its interval is due
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): read the check interval on each flush so YAML environment_variables apply
The writer is built at proxy import, before the config's environment_variables load, so reading
the interval there ignored values set in config.yaml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(router): wait for router accounting instead of sleeping in user-config limit tests
Each request ended with a 0.3s sleep and hoped the router's usage writes had landed, so under CI load a
concurrent request could read counters mid-update. The tests now flush the logging worker and wait for
every task the requests scheduled, and a retired router's counter flush lands before the next router reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(completion): accept a tool_choice that forces an OpenAI hosted tool
litellm.completion rejected tool_choice={"type": "web_search"} before any request, while /v1/responses took the same choice. The chat validator now forwards a hosted tool type unchanged and a provider gate turns it into a 400 off openai, azure and litellm_proxy, where the choice would otherwise be dropped or mangled.
Fixes#50566
* fix(utils): keep a non-string tool_choice type a 400, not a TypeError
A list or dict in tool_choice["type"] is unhashable, so the hosted tool lookup needs the string check first. The bridge test that used file_search as an untranslatable choice now uses a type that really is unknown.
* refactor(completion): forward any Responses API tool_choice type instead of keeping a hosted tool allowlist
Chat completions now accepts any tool_choice dict with a string type it has no shape for and sends it as is on openai, azure and litellm_proxy, where OpenAI validates the type itself (a typo is still a 400, from OpenAI). The provider gate keeps the 400 everywhere else. The HOSTED_TOOL_CHOICE_TYPES list is gone, so a new OpenAI hosted tool needs no change here.
* Revert "refactor(completion): forward any Responses API tool_choice type instead of keeping a hosted tool allowlist"
This reverts commit 5a6d5a8e08.
* fix(utils): add tool_search to the hosted tool_choice types
OpenAI lists it among the supported tool_choice.type values (live error on 2026-11-08), alongside the ones already in the set.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): let one streaming request opt into usage.cost with x-litellm-include-cost-in-streaming-usage
The global include_cost_in_streaming_usage flag moved every stream off the
fast path. A request can now send x-litellm-include-cost-in-streaming-usage: true
to get usage.cost on its usage-carrying stream event (/v1/messages and the
Anthropic and OpenAI pass-throughs) while requests without the header keep the
fast path. The global flag keeps working unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: type the new stream cost header tests
* fix(types): keep reportUnknownArgumentType at its merge-base count
* fix(proxy): the stream cost header only accepts true
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): a cancelled or retried attempt releases its io-token reservation and least-busy count
A routed attempt cancelled mid-call (client disconnect with cancel_on_disconnect) kept its ITPM/OTPM
reservation and its least-busy in-flight count, and retries sharing one logging object leaked the
count of every attempt after the first. Each async attempt now runs under its own attempt id, so the
attempt and its own callbacks release what it holds exactly once, in tasks a cancellation can't stop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): settle an io-token reservation once even through a logged copy of its metadata
/v1/messages and /v1/responses stash the reservation in litellm_metadata, which the logging object
copies. A cancelled attempt refunded it, then the proxy's failure logging refunded the copy again.
Each stash now carries an id, and a reservation already settled under that id is not settled again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): keep spent tokens counted, refund a previous attempt's stash, and fit every call in flight
A request cancelled after its provider answered now keeps the tokens it spent counted instead of
refunding them. A retry refunds the reservation its previous attempt left behind even though it runs
under a new attempt. The least-busy reservation store holds twice the keys, since each call in flight
now keeps one more.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): never evict a settled reservation id before its counters' window has passed
Settled ids lived in a capacity-bounded cache, so past 10,000 settlements in two minutes a late copy
of an evicted reservation was refunded a second time. They now leave only when their window expires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): start releases eagerly and keep attempt settlement out of router_utils
A release that never waits (an in-memory counter) now finishes before its caller goes on, as it did
when the caller awaited it inline, so a websocket turn's usage is counted by the time its logging is
flushed. Each reservation carries its own id, and the attempt settlement module moves to
litellm_core_utils, since litellm/utils.py may not import from router_utils.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): count only finished answers as spent, and declare the least-busy redis family
A stream's wrapper returning is not an answer: a stream cancelled before its first chunk now hands
back its whole reservation. Only a non-streaming result marks its attempt as answered. The module
level least-busy shared decrement carries its redis family, as every direct redis caller must.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
57 of the 62 tests in this file are hard-skipped because they need a live
Prisma DB, so they never run. This removes the 20 whose behavior a running
integration or e2e test already asserts, and the 8 that assert nothing
meaningful or can never pass. The imports and the one helper only they used
go with them. The other 29 skipped tests stay as a list of behavior still to
port into tests/integration.
A Google metadata-server identity token carries no jti, so Anthropic lets it be exchanged more than once. The shared
on-disk token store still answered a mandatory refresh with the stored token minted from that same assertion, which
was the almost expired token the refresh existed to replace. Google references now skip the store, so every refresh
is a real exchange and a Kubernetes token file keeps today's cross-worker reuse.
ANTHROPIC_IDENTITY_TOKEN was always wrapped as oidc/env/ANTHROPIC_IDENTITY_TOKEN, so a value that was itself an
oidc/google/ reference was posted to Anthropic as the literal assertion text. A value starting with oidc/ is now the
reference, with oidc/env_path/ still rejected, and a raw token in the variable resolves exactly as before.
Fixes#50553
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Refresh the WIF bearer token on the connection error retry
* Only refresh the bearer when the retry goes to the host the token was minted for
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hold MODEL_RECONCILE_LOCK around credential create, update and delete
The config reload snapshots the credential table, awaits config rows, then prunes and
upserts litellm.credential_list from that snapshot. A credential write landing inside
that window was undone in memory until the next reload. The three endpoints now take
the same lock the reload holds.
Fixes#50273
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): take MODEL_RECONCILE_LOCK before the credential update reads its row
Two PATCHes queued behind a reload read the same old row and the second
overwrote the first. The relay now keeps enough bytes to spot a split
after marker, and the new unit tests are fully typed.
* test(credentials): annotate the reconcile lock test locals with Final
* refactor(proxy): rename MODEL_RECONCILE_LOCK to DB_RECONCILE_LOCK
The lock gates every db-to-memory reconcile and now the credential writes too, so
the model-only name undersold what it protects.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
On an auto-router request the LLM span reported the name the caller asked
for (the router or an alias of it) as gen_ai.request.model (OTel v2) and
gen_ai.response.model (v1 and v2). Both now name the model group the router
picked, read from the request's routing_decision. In v2 the span name and the
vendor keys built from the request model (Langfuse, OpenInference, Langtrace,
Weave) follow. Requests that were not auto-routed are unchanged.
* fix(responses): surface mid-stream upstream errors that arrive without an event type
Bedrock's native Responses route can end a stream with a bare {"error": {...}}
chunk. It parsed as a typeless GenericEvent and crashed the iterator with
AttributeError, so clients saw that instead of the upstream error. Such chunks
are now read as error events, which raise the mapped upstream error and reach
the client as response.failed.
* refactor(responses): inline the untyped error chunk check to keep LIT001 at the base count
* fix(mcp): derive MCP bridge and session keys with the FIPS provider's KBKDF
With OpenSSL enforcing FIPS, hashlib.scrypt raises, so every MCP session
token and bridge envelope request failed with a 500. Under FIPS the keys
now come from LITELLM_SALT_KEY through SP 800-108r1 KBKDF run by the
provider's EVP_KDF. Without FIPS the scrypt keys are unchanged
* fix(mcp): keep master key rotation invalidating MCP tokens under FIPS
Pass the master key as the KBKDF context, so rotating it changes the
three FIPS-mode keys as it changes the scrypt keys
* test(mcp): annotate new FIPS key test locals with Final
* fix(mcp): derive the FIPS MCP keys only when general_settings.fips_mode is on
On a FIPS OpenSSL 1.1.1 host (RHEL 8) scrypt still works, so keying the switch on
openssl_enforces_fips() sent those hosts to the OpenSSL 3 only KBKDF and broke MCP
sessions and bridge envelopes that worked on main. fips_mode_active() is the same
reader B1 adds, set by enforce_fips_mode at startup.
* refactor(fips): move the salt key KBKDF into encrypt_decrypt_utils
The KBKDF helper derives keys from LITELLM_SALT_KEY, so it now sits next to the other salt key helpers, and fips.py goes back to main's version.
* Pin the MCP key KBKDF fetch to the FIPS provider
Fetch KBKDF with the property query fips=yes, so a change to the process's default
properties after boot can't move the derivation to a non-FIPS provider. The shared
fips_openssl test fixture drops the pin, since the test OpenSSL has no FIPS provider.
* feat(guardrails): add prebuilt credential patterns for JWT, and OpenAI, Anthropic, Gemini API keys
* fix(guardrails): prevent quadratic backtracking in jwt_token pattern
* test(guardrails): move credential pattern tests into test_patterns.py
* feat(guardrails): remove jwt_token to keep credential patterns strictly bounded
* fix(guardrails): mask whole provider keys and keep openai_api_key to OpenAI's key shapes
End each pattern so a key ending in "-" is masked in full, require OpenAI's
T3BlbkFJ marker on bare sk- keys so other vendors' sk- keys and LiteLLM
virtual keys are left alone, add sk-None- user keys, and replace the
registration-only tests with whole-match, length-boundary, and negative cases
* fix(guardrails): require the T3BlbkFJ marker on every OpenAI key form and match keys after escaped separators
openai_api_key now needs the marker on the sk-proj-, sk-svcacct-, sk-admin- and sk-None- forms too, as gitleaks and TruffleHog do. anthropic_api_key and gemini_api_key also start a match right after a literal \n or \t or a percent-encoded byte, so keys in pasted JSON strings and URL-encoded logs are masked.
* fix(guardrails): bound the OpenAI key regex, match Gemini prefixes case-sensitively, and hold back long keys in streamed replies
* fix(guardrails): size the streaming holdback per pattern, stop OpenAI key tails before the next key, and allow non-ASCII text after a key
* fix(guardrails): mask content filter pattern matches found in the original text so overlapping patterns cannot split a key
* fix(guardrails): let a block pattern see the text after earlier masking patterns, as before
Masking patterns still match the original text and merge overlapping matches into one tag, but a BLOCK pattern checks the text with the masking patterns listed before it applied, so a request main masked and passed is not blocked now. Detections name only the patterns whose tag was written.
* fix(guardrails): keep content filter masks in order and hold back the longest OpenAI key
Masking patterns run in configured order on the masked text as on main, so a contextual
pattern still matches next to an earlier tag, and each also matches the original text so an
earlier tag cannot split a key. Detections follow the configured pattern, not its name, and
the OpenAI streaming holdback covers the pattern's 419-character longest match.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(cost-map): move -latest model aliases into the cost map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: load the finalized local cost map in the moonshot alias test and assert alias pricing by behavior
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: compare alias cost map rows to their targets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): give each -latest alias its own alias_of row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): keep alias_of rows out of alias lists and add them to the registry schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 91e6cf2c27)
* fix(cost-map): let alias lists win over alias_of rows in both loaders and never serve a malformed pointer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit a54c42218a)
* feat(cost-map): add -latest alias rows for the tier 1 and 2 hosting providers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-map): derive hosted alias targets from the cost map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): skip alias rows when checking cache-write rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chatgpt): round-trip encrypted reasoning through /v1/messages
The chat bridge stored encrypted reasoning on reasoning_items, the Anthropic
adapter never read that field, stream chunks that only carried thinking
blocks were dropped, and the messages adapter stripped the signed blocks
before a chatgpt request could turn them back into reasoning items.
Co-authored-by: DDT <1786035110@qq.com>
* fix(anthropic): keep Responses reasoning on tool turns and retry once when a backend cannot decrypt it
* fix(anthropic): stop logging rejected encrypted reasoning
Quote the retry helper return type so importing litellm no longer evaluates a string alias with |. The warning keeps provider, model, status, and error code, and leaves the ciphertext out.
* fix(anthropic): keep encrypted reasoning on OpenAI thinking and xAI web search turns, and stream signatures only to Messages clients
The keep check now knows when the handler moves an OpenAI thinking turn to Responses, and when web_search_options sends an xAI turn there.
The Responses-to-chat stream iterator adds the signed reasoning chunk only when the caller is /v1/messages, so /v1/chat/completions streams stay unchanged.
The reasoning retry fires only when the translated request still carries encrypted reasoning.
* fix(anthropic): keep each streamed reasoning summary with its own item, and read the Messages marker before the bridge call
* fix(anthropic): send openai/ Messages turns again once without encrypted reasoning a backend rejects
The openai/ Responses path on /v1/messages had no retry, so a chatgpt/ block left in history answered 400 on every later openai/ turn. That path now sends the turn once more without encrypted reasoning, the same way the chat bridge does. The retry matches only invalid_encrypted_content or a "could not be verified" message naming the content, so the exception mapper's own hint no longer triggers it. The xAI web search keep path is backed out, since xAI sends no encrypted reasoning over the bridge.
* fix(anthropic): keep every non-streamed reasoning item for Messages clients and read a deployment's reasoning_summary in the keep check
* fix(anthropic): log only the outcome of a turn sent again without rejected reasoning
A /v1/messages turn whose replayed reasoning the backend rejects is sent once more without it. The rejected first attempt ran the failure callbacks on the shared logging object, which logged a failure for a turn that then succeeded, kept the retry's own failure from being logged, and freed the parallel-request slot while the retry was still in flight.
The first attempt now runs inside resent_on, and the client wrapper skips failure logging and the budget unbind for a failure that matches it. The retry runs outside that context, so its own outcome logs as usual.
* test(anthropic): type the parameters of the failure-recording test loggers
* fix(anthropic): skip the SDK num_retries retry for an attempt that is sent again
* fix(anthropic): stop SDK retries from resending a rejected block or clearing the global num_retries
A transient 429 or 500 on a /v1/messages turn that carries a replayed encrypted
block was retried by an SDK call's num_retries loop, which sent the block again
until every retry was rejected. On async calls the loop then raised the first
transient error, so the turn failed instead of being sent again without the
block. The retry loop now stops at that rejection and passes it on, so the turn
is sent again once without encrypted blocks.
A rejected first try on a sync call also cleared the process-wide
litellm.num_retries, so later calls in the process were no longer retried. That
setting now stays as configured.
* fix(anthropic): stop a cancelled SDK retry from starting another call
The resent-failure retry predicate returned true for BaseException
subclasses such as asyncio.CancelledError, so a cancelled
acompletion_with_retries or aresponses_with_retries call could start
another provider call. Retry only Exception subclasses again, as
tenacity's default did, while still skipping a resent failure.
* test(anthropic): type the retry helper parameter in the cancellation test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(cost-map): read alias_of pointer rows in the catalog and the loader
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): keep alias_of rows out of alias lists and add them to the registry schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): let alias lists win over alias_of rows in both loaders and never serve a malformed pointer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): preserve streamed tool argument fragments verbatim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): send the schema-less stream test the null properties form providers accept
* fix(anthropic): unwrap values when a schema-less stream sends its whole tool input in one piece
* fix(anthropic): unwrap only a one-piece values input and never emit Infinity in schema-less streams
* fix(anthropic): keep JSON mode answers verbatim when they cannot be written back as strict JSON
A tool input holding a number past a float or an integer past Python's digit limit now passes through as the model wrote it on the stream and non-stream paths, instead of coming back with Infinity or raising. One-piece values rows for false, null, a plain object and too-deep nesting pin the remaining unwrap branches.
* fix(anthropic): reject non-finite floats through pydantic instead of a raising parse hook
* fix(anthropic): keep split JSON mode answers whole when a field cannot be written back as strict JSON
* fix(anthropic): keep out-of-range numbers as the model wrote them in JSON mode answers
Decode tool input with parse hooks that keep a number past a float or an integer past the digit limit as its source text, and write it back unchanged, so held, wrapped and name and value split answers stay whole instead of falling back to raw blocks.
* fix(anthropic): keep numbers that round to zero as the model wrote them in JSON mode answers
* refactor(anthropic): encode JSON mode answers without a recursive helper
* fix(anthropic): decode deeply nested JSON mode answers without slow recursive validation
* fix(anthropic): type the decoded JSON mode answer as object and test non-object answers
json.loads output now goes through the existing TypeAdapter(object), which returns it unchanged, so no Any reaches the object guard. New stream and non-stream cases check that a list, string or number answer comes back as written.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(redis): share one native circuit breaker between RedisCache and the Rust stores
RedisCache and the native Redis client each kept their own breaker, so once
the cache learned the coordination Redis was wedged, a spend-counter write
through the native store still paid a full socket timeout. Every Redis target
now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected
into both the RedisCache guard and the native client. The Python FSM is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(rust): cargo fmt the shared breaker
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(redis): close the breaker when the recovery probe gets a Redis reply error
A command error such as WRONGTYPE means Redis answered, but the probe was left
unsettled and its drop reopened the breaker for another recovery timeout. Both
the RedisCache guard and the native client now settle such a probe as a success.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test
The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw.
* fix(redis): key the shared breaker by server and close the three audit gaps
The breaker registry now keys on the Redis server (host and port, unix socket,
cluster node set or sentinel service) instead of the full connection target, so
caches that differ only in db, credentials or url spelling share one breaker.
A READONLY reply from a replica mid-failover no longer counts as Redis answering,
so it can't reset a run of failures. A new RedisCache refreshes the breaker state
gauge, which the deleted Python breaker did at construction. A shared native client
is only reused by a store that passes the same breaker.
* refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations
lib.rs now only declares modules and re-exports, with resolution, target, tls,
auth and endpoint each in their own file. Derived FromPyObject already reads a
struct field as the attribute of the same name, so #[pyo3(attribute)] on the
breaker and store settings did nothing.
* refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot
Derived FromPyObject already reads each field as the attribute of the same name.
The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in
the same order.
* refactor(redis): return a HashMap from the three memory-tier snapshots
The snapshots read cache-memory's HashMap, so there is no order to keep.
PyO3 builds the same dict from a std HashMap, which replaces the hand-written
PyDict loops in the spend counter and router usage stores and the IndexMap in
the cooldown store. The rate-limit slot registry keeps IndexMap because it
hands back a Python dict in the order Python wrote it.
* refactor(redis): share Redis breakers and clients through one RedisHosts per host
redis-connect's RedisHosts now owns the per-server breaker and the shared clients
that python-bridge kept in two process-wide statics. A client takes breaker
settings and admits through its server's breaker, so no caller can pair a client
with a breaker of its own, and clients are keyed by Resolution alone. This drops
RedisClient::guarded_by and Breaker::same.
python-bridge keeps one RedisHosts per process and calls it with the GIL released.
The gateway builds one per connect, so its coordination and response cache on the
same server now share a breaker. RedisStoreSettings.breaker is back to breaker
settings, as on main.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(vertex_ai): run auth once per Vertex pass-through request
Drive /vertex_ai, /vertex-ai and /vertex_ai/discovery requests through the real
proxy app with a counting custom_auth, and assert each request authenticates
once and is served as the caller that auth returned. The generate-content
routes fail today because the route authenticates a second time.
* fix(vertex_ai): authenticate Vertex pass-through requests once
vertex_proxy_route already resolves the caller through Depends(user_api_key_auth),
but _base_vertex_proxy_route ignored it and authenticated the request again. A
custom_auth function ran twice per request, and a key with max_budget reserved
its estimated cost twice, so the unreleased first reservation exhausted the
budget after one or two calls.
The base route now uses the caller it is given and authenticates once itself
only when none is passed, which is the discovery route's case.
* fix(vertex): read the caller's key only from x-litellm-api-key or Authorization on Vertex routes
The single auth check on /vertex_ai and /vertex-ai now reads the same two headers main's effective check did, so a key sent only in another key header gets 401 again instead of authenticating.
* fix(vertex): authenticate the discovery route through the same Vertex key check
Both Vertex pass-through routes now resolve the caller once through
_authenticate_vertex_caller and hand it to _base_vertex_proxy_route, which
no longer re-runs auth. x-litellm-api-key accepts the same scheme prefixes
on /vertex_ai/discovery as on /vertex_ai and every other proxy route.
* fix(proxy): size a Gemini request's budget hold from its prompt and maxOutputTokens
A Gemini generateContent body's budget hold now counts its contents and
systemInstruction and reads generationConfig.maxOutputTokens. Before, it held
the model's whole context window, about $0.85 on gemini-3.8-flash, so a key
with a small budget got 422s on back-to-back calls it could afford
Also removes get_litellm_virtual_key, which nothing calls since the Vertex
routes stopped using it
* fix(proxy): count every Gemini system part and candidateCount in the budget hold
A Gemini generateContent body with several systemInstruction text parts lost
every part after the first when LiteLLM translated it to a chat request. That
dropped the later system text for non-Gemini models behind
/v1beta/models/<model>:generateContent, and made the budget hold count only the
first part. The adapter now keeps every text part, as a plain string when there
is one part and as text blocks when there are several.
The output hold now also multiplies by generationConfig.candidateCount, the
same way chat bodies use n, so a request asking for several candidates holds
budget for all of them.
* fix(proxy): keep the context-window hold for Gemini bodies the prompt count cannot see
A Gemini generateContent body with media, cachedContent, other roles, or
non-function tools keeps the full context-window hold, since the chat
translation drops or undercounts those parts. The hold also reads the
snake_case generation_config output fields, and several systemInstruction
text parts reach non-Gemini models joined into one system string
* fix(proxy): read only camelCase Gemini output fields in the budget hold
/v1beta forwards only generationConfig.maxOutputTokens to non-Gemini models, so a snake_case cap the model never receives must not shrink the hold. Snake_case bodies now get the default output hold.
* fix(proxy): count snake_case candidate_count in the Gemini budget hold
* fix(vertex_ai): skip malformed system parts and read Gemini output fields only on Gemini bodies
A systemInstruction part whose text is not a string is skipped, as main did, instead of failing the request. The budget hold reads generationConfig output and candidate fields, and counts contents, only when the body has no messages, prompt, input, query or documents field. An output cap of 0 or below counts as no cap. Test recorders keep tuples instead of mutating lists.
* fix(proxy): take the budget hold's request shape from the route
On generateContent routes the hold counts only contents and reads its output
cap and multiplier from generationConfig, so stray fields of other APIs can no
longer hide the prompt. A max_tokens of 0 holds no output again, while negative
values and a Gemini maxOutputTokens of 0 or below hold the default output
* fix(proxy): hold one candidate for a count too large to price and read system parts like content parts
* fix(proxy): keep the spend already counted when a budget hold is far over the budget
* fix(proxy): count candidateCount in the budget hold only for models that generate the candidates
A non-Gemini model reached through /v1beta generateContent never receives candidateCount, so a huge count held the whole budget for the length of the call while the model produced one answer
* fix(proxy): hold a candidate count only when every deployment generates it, and drop n when every deployment drops it
* fix(proxy): count candidateCount when any deployment generates it and always count n
The stricter rules from the previous commit let a key spend past max_budget through a Gemini name no deployment serves, an allowed_openai_params n, and a router fallback. Restore the earlier rules and add regression tests for those three shapes.
* fix(proxy): cap the candidateCount hold at main's default output hold
A candidateCount like 1e300 sent to a mixed Gemini and Claude group, or through a key alias from a Gemini name to Claude, held a whole key while Claude served the call and answered a strict key 422. On generateContent routes the counted candidates now raise the output hold at most to the default output hold main used for the model, so no count makes a call hold more than on main. Chat n is unchanged.
* fix(proxy): hold every Gemini candidate up to the merge base's whole hold
The previous cap stopped the candidateCount output hold at the default
16384 tokens. Because the hold now counts the Gemini prompt instead of the
whole input window, a strict key admitted a call like 4 candidates of 8000
tokens and then spent past max_budget, where the merge base refused it.
Every candidate is now held at its output cap, like chat n, and the whole
hold is clamped at what the merge base held for the same call (the model's
input window plus the default output), never below one candidate's hold.
A huge count still holds no more than the merge base did. Chat n is
unchanged
* fix(proxy): let a Gemini output cap that /v1beta drops only raise the hold
The Vertex and Gemini pass-throughs forward every spelling of the output
cap, but /v1beta passes only generationConfig.maxOutputTokens to other
models. The hold read only that spelling, so a snake_case cap above the
default output held less than the pass-through could spend. A cap in any
other spelling now raises the hold to it and never lowers it
* fix(proxy): hold Gemini candidates of a model with no context window up to the default output
A Gemini model the cost map has no context window for, priced in its deployment, held one
candidate for any candidateCount, so parallel calls could pass their budget holds. Its
candidates are held again up to its counted prompt plus the default output hold
* fix(proxy): keep main's Gemini hold unless the request caps its output
An uncapped generateContent call, or one capped only by a snake_case
spelling, now holds what main holds (the context window plus the default
output). A call with generationConfig.maxOutputTokens holds its counted
prompt plus cap times candidateCount, never clamped to the deployment's
max_output_tokens and never more than main's hold. A model with no context
window keeps main's no-hold, and candidateCount always counts on
generateContent routes. Also keeps get_litellm_virtual_key in
pass_through_endpoints/common_utils.py, so an out-of-tree importer
does not break
* fix(proxy): keep main's budget hold on generateContent routes
The PR now leaves the generateContent budget estimate as main has it.
A live risk check showed the counted Gemini hold under-reserves (the
prompt counter undercounts Gemini bodies, tiered and image output rates
are not read, and request fields the handler merges later are not
counted), so that estimate moves to a follow-up. The single auth pass
on the Vertex pass-through and the systemInstruction adapter fix stay
* fix(vertex_ai): keep main's Gemini adapter and pass a missing Authorization as no key
The Gemini adapter change for later systemInstruction parts goes back to
main, since it is not the double-auth fix this PR is for. The Vertex auth
dependency now hands a missing Authorization header to the auth check as
no key, like main's first auth pass, so custom auth and the 401 text see
what they see on main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(caching): don't cache an Anthropic Messages stream whose content blocks are all empty
A non-Anthropic model routed through the Messages adapter turns an empty reply into a single empty text block. The no-output check counted any content_block_start as output, so the async stream cache stored that empty reply and replayed it on every repeat. Judge each content_block_start and content_block_delta by its payload instead, like chat streams judge a choice: a block carries output when a field other than type holds something besides "", [] or null. Empty replies are no longer stored and stored ones become misses. A bare {} is not treated as empty, since the only block fields that hold one (tool_use input) come with an id and a name
Fixes#49931
* fix(caching): treat thought-signature-only and non-streamed empty replies as no output
* fix(caching): treat whitespace-only replies as no output
* fix(caching): keep whitespace replies and text completion fills cached
* refactor(caching): drop the unused streamed-choice branch of the output check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): preserve Responses SSE event framing
* style(proxy): format Responses stream terminator condition
* fix(proxy): validate Responses guardrail event names
* fix(proxy): keep Responses first-event errors and keepalive failures in Responses framing
* test(proxy): use controlled clocks for Responses keepalive failures
* fix(proxy): end late Responses failures with an error event the SDK raises on
Errors that arrive after keepalive pings, and guardrail error strings yielded
mid-stream, now end /v1/responses with a named `error` event carrying the
same error object chat sends, so the OpenAI SDK still raises APIError with a
clean message and provider_specific_fields. Background polling passes the
same framing and records an `error` event as a failed response. The first
data line of a frame is found by splitting on newlines only, so a U+2028 in
an error message no longer hides a first-event error.
* fix(proxy): match OpenAI's Responses error event and stream late passthrough blocks as block responses
* fix(proxy): keep the reason when a background Responses request fails
A background poll of a request a post-call guardrail blocked saved a
null code and message, because the guardrail's error event nests them.
A request whose first chunk was an error polled as completed with no
error. Both now poll failed with the same code and message a late
failure gets.
* test(proxy): type the first-chunk poll test parameters
* fix(proxy): keep first-chunk error detection to lone error frames and handle string poll errors
* fix(proxy): keep the HTTP status on late Responses errors and end Responses streams on error events
A late error body without a code now takes the response's status, an upstream error event counts as the stream's terminal event, and a Responses stream that fails before its first chunk ends with an error event instead of data: [DONE].
* test(proxy): type the first-chunk failure test's parameters
* fix(proxy): keep cursor chat framing and JSON-encode Responses hook errors
* fix(proxy): encode non-dict Mapping Responses chunks as JSON objects
* fix(proxy): encode mappings and models nested in Responses stream chunks as JSON objects
* test(proxy): freeze the hook-output fixture
* fix(proxy): keep masked mappings and odd keys safe in Responses stream frames
Only read-only dict views and pydantic models nested in a hook chunk or error event expand into JSON objects. Other mappings, such as httpx.Headers, keep their own string form so masked values stay masked. Keys JSON cannot take become strings, and a chunk JSON still cannot encode goes out as the plain data frame it was before.
* fix(proxy): never fail a Responses stream frame on a value JSON cannot encode
Only dicts and read-only dict views expand into JSON, at the top level too, so a header mapping yielded as a whole chunk keeps its masked string form. Any failure while encoding a chunk falls back to the plain data frame the base sent, and an error event that cannot encode still goes out with its code and message, so a late block never drops the connection.
* fix(proxy): keep what a nested value's repr hides out of Responses stream frames
* fix(proxy): print values in Responses stream frames the way each base path did
* fix(proxy): send string and number subclasses whose repr shows their value as that value in Responses stream frames
* fix(proxy): print container subclasses and SDK models in Responses stream frames the way base did
* fix(proxy): read hook container storage the way repr does and send error-event models as str()
* fix(proxy): frame streamed Responses reads and send unreadable hook values as base did
Track pre-framed and raw SSE frames so a late failure never follows a terminal event, accept CRLF and CR line breaks in a first named error, frame GET /v1/responses/{id}?stream=true as Responses events, print a whole hook proxy chunk once and fall back to its text, ignore a non-integer status_code in a hook error frame, read a late-block dict subclass through items() when that view differs from its storage, and keep the order and inline form base's repr printed for nested mappings
* fix(proxy): match base on Responses stream edge cases
Read a decimal-string or integral-float status_code as its integer again, so
"429" and 429.0 still map to rate_limit_exceeded. Print each hook value's repr
once and reuse it, so a repr that changes between calls cannot reach the
client. Track only pre-framed sequence numbers from 0 to 2**63-1, so a huge
one cannot break the failure event. Build nested JSON with one frame per level
again, which restores the nesting depth sent as JSON. Count a pre-framed frame
as terminal from its data type, and a named error frame only when its data
carries an error, the way the OpenAI SDKs read them.
* fix(proxy): print hook proxies once and keep base's chunk tracking
Read-only mappings in hook chunks now emit the text from their first print, hook
chunks are read without their error field, only a non-empty error object makes a
framed error event terminal, and framed sequence numbers stay below 2**63-1 while
hook chunks keep base's unbounded tracking
* fix(proxy): print each read-only mapping item once and poll framed error events as failed
A MappingProxyType in a hook chunk or error detail now expands from the
mapping it wraps, so each item is printed once, the way base printed it.
A value whose repr changes between calls stays masked, a value whose repr
works once is kept, and deep nesting keeps one stack frame per level
The background reader takes each frame's type from the same terminal rule
the stream uses, so an event: error frame with an error object polls
failed even when its data names another type or none
* fix(proxy): print self-holding hook values once and poll how the stream ended
A container that holds itself now goes out as the marker its repr prints, like {...}, [...] or mappingproxy({...}), and the dict behind a read-only mapping joins the cycle check, so no item is printed a second time. A dict whose item repr adds a key keeps going after the items already shown, the way dict's repr does, instead of printing the whole dict again
A whole read-only mapping chunk that cannot go out as JSON is printed once by the formatter, so a print that fails fails the stream as it did on the base
The background poll reads the event type from the block it parses, as on the base, and keeps the first error frame the stream treats as terminal anywhere in a chunk. That frame fails the poll only when no response.completed, response.failed or response.incomplete follows it
* fix(proxy): name hook chunk events from the sent JSON and keep polls on frames they cannot read
A mapping chunk now takes its event: line from the JSON it sends, so a masked type value never reaches that line, and a chunk the event model cannot read goes out as a plain data frame instead of failing the stream
A dict whose item repr deletes or swaps keys keeps printing every key not yet shown, and an error code past the integer digit limit reads as no status
The background poll skips a frame whose shape it cannot read, as base skipped frames it could not parse, stops reading a chunk at [DONE], and lets only the last terminal frame in a chunk decide its status
A Responses stream's first chunk answers as a JSON error only for codes 400 and up
* fix(proxy): read every frame of a polled chunk and name model chunks from the sent JSON
A background poll parsed only the first data frame of each chunk, so a terminal event behind an error or delta frame in the same chunk was lost, and the poll stored a status its own stream never ended with. It now reads every frame up to [DONE]
A terminal event whose response is a string or null kept no status and polled completed. Its status now comes from the event type, and final output items that are not objects are skipped, so the rest of the final output is still stored
Model chunks now take their event line from the JSON they send, as mapping chunks already do, so a serializer that drops type no longer sends event: response.completed over {}
* fix(proxy): read polled chunks as SSE events and keep draining past a mid-chunk [DONE]
A background poll split a chunk into data lines, so one event with two data lines parsed as two payloads, and a frame with an integer past the digit limit raised a ValueError the parse guard did not catch, which failed the poll and dropped its spend row. The poll now joins each event's data lines the way an SSE reader does and skips any frame it cannot parse
A [DONE] behind another frame in one chunk ended the poll and left the stream unbilled. Only a chunk that opens with [DONE] ends the poll now, as on main, and the poll stops reading a chunk at its first terminal event, so two terminal events in one chunk poll the first one as main did
* fix(proxy): JSON-encode only plain hook chunks on Responses streams
A hook chunk whose whole tree is builtin JSON types streams as its named
event. Any other chunk keeps the base data frame, so nothing a custom
repr hides reaches the wire. Error event details encode the way the chat
error frame does.
* fix(proxy): skip containers already visited when checking hook chunks for JSON
* fix(responses): keep streaming past an event with no type
The Responses stream iterator read event.type directly when it noted a
yielded event, so an event whose JSON had no type, like a custom hook's
dict that the proxy now sends as JSON, raised AttributeError and ended
the stream. Read it with getattr, as the iterator's other checks do.
* fix(proxy): keep hook dicts with no type on the base Responses frame
A hook dict with no type is not a Responses event, so it gets no event
name either way. Sending it as JSON only made litellm SDKs that read
type on every event stop the stream. Keep such dicts on the frame the
base sends, so every SDK version reads them as before.
* fix(responses): keep a background poll's first terminal status and take a first-chunk error code from its HTTP status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* Add per-key and team prompt storage restrictions for spend logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Apply spend-log prompt restrictions to passthrough requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Apply team prompt storage restrictions to queued requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend-logs): honor tenant prompt storage in the collector and on "false" strings
The spend-event collector decided whether to ship request and response
bodies from the global setting alone, so an opted-out key or team still
sent prompts through the collector transport. It now uses the same
key/team check as the spend log writer.
A key or team setting of the string "false" was ignored and prompts kept
being stored. It now counts as false, the same way the global setting
parses strings.
Splits the 42-case tenant storage test into one test for the policy and
one for the pass-through logging contexts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(spend-logs): carry key/team prompt storage into failure, batch and side-call rows
Failure rows, executed batch rows, semantic index embeddings, websearch
interception calls and errors rejected before request setup built their
logging metadata without the key and team metadata, so a restricted key
or team still had prompts stored on those rows. They now stamp it through
one helper, LiteLLMProxyRequestSetup.get_tenant_logging_metadata.
The dashboard toggle now shows a saved string "false" as off, matching
the server, and its label and help text come from the translation file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
