* fix(anthropic): time-limit /v1/messages pump waits on a stalled client
The upstream pump waited on client_detached with no timeout, both when the relay queue was full and when waiting for the relay to consume a queued upstream error. client_detached is only set by the relay generator's finally block, so a client that resets its connection while stalled could leave the pump parked forever, holding its request, logging object and collected chunks.
Both waits now give up after ANTHROPIC_MESSAGES_STREAM_CLIENT_STALL_TIMEOUT_SECONDS (default 120), treat the client as gone, and fall through to the existing detached billing path.
* fix(anthropic): bound the detached /v1/messages drain and end a relay whose pump gave up
A provider that stalled after the client left kept the detached pump, the open provider stream and every chunk it
had buffered for billing alive until the provider read timed out, if it ever did. Pumps that never read another
chunk after the disconnect are not counted against the detached-drain cap, so nothing limited them
A detached drain still reading ANTHROPIC_MESSAGES_DETACHED_STREAM_DRAIN_TIMEOUT_SECONDS (default 600) after the
client left now bills what it read and closes the provider stream. The deadline starts when the relay closes or
when the client stall timeout gives up on the client, so it also covers a read that was already stalled
Once the stall timeout gives up, the pump stops feeding the relay. A client that resumed reading then waited on an
empty queue forever. The relay now ends with the incomplete-stream error once the pump is gone. The stall timeout
is read with get_env_float so fractional values parse
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): end a stream cleanly when the pump gives up on its last put
When the upstream had already finished and only the end-of-stream sentinel was left, a client that resumed after
the stall timeout got every chunk through message_stop and then an extra incomplete-stream error event. The relay
now only adds that error when the client has not already received a message_stop or error event
The new test pins the case where the stall timeout fires on the final sentinel put, so the pump exits with the
request billed and a resumed client still sees a clean, complete stream
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): give a stalled /v1/messages client 600 s before the pump gives up on it
The 120 s default could cut off a client that was only slow. 600 s matches the usual request_timeout, so a stalled
client and a stalled provider get the same budget. SSE keepalive pings do not change this, because they block on the
same stalled write. A stuck pump now lives at most about 20 minutes, which keeps only a handful alive at once
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic_messages): keep queued chunks when dropping an unconsumed upstream error
_exception_left_unconsumed emptied the whole relay queue to check whether the
provider error was still in it, so a client that resumed after the error wait
expired lost every chunk buffered ahead of the error. Re-queue everything but
the error, add a regression test for the resumed client, and type the new test
helpers.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add the CollieAi guardrail integration
* refactor(guardrails): tighten the CollieAi wrapper and its tests
The loader takes its importer as a parameter, so a test covers the fallback for a package that is too old, and the install hint names the Python the package needs
The config model is frozen, the Garden preset's phases and default_on are pinned by a test, and comments the code already says are gone
* fix(guardrails): use the generated Collieai provider key in the dashboard
The dashboard builds provider keys from the proxy's provider list, so collieai becomes Collieai. The static CollieAi key came first in the provider map, which left saved CollieAi guardrails without their display name and logo. The map, the Garden preset and the card now use Collieai, and a test covers the lookup
The preset expectation is a named value now, which keeps the dashboard under its inline object lint budget
* refactor(guardrails): drop the CollieAi wrapper's function docstrings
AGENTS.md keeps comments to complex logic and tool input, and these only restated what each function returns
* fix(guardrails): declare CollieAi's phases, read COLLIEAI_API_KEY and plate its logo
The dashboard offered every mode for CollieAi, and a guardrail saved with a mode the package refuses was reported as created but never loaded. The wrapper now declares pre_call and post_call, so the form offers only those
api_key falls back to COLLIEAI_API_KEY as its description says, and the navy logo gets a plate in dark mode like DeepKeep's
The tests now also cover an entry's own name, phase and default_on, the key fallback, and the modes the dashboard offers
* fix(guardrails): reject CollieAi guardrails the package refuses and stop env-keyed rebuilds
A Garden create whose key or settings the package refuses now answers 400 and
saves nothing, instead of saving a guardrail that checks nothing. Every setting
on the entry now reaches the package, like the class-path route. DB guardrails
keyed with an os.environ/ reference are no longer rebuilt on every DB poll.
The Garden card says which calls CollieAi checks.
* fix(guardrails): load CollieAi stream_scope and import errors as setup errors, default the Garden card off
Pass stream_scope into the package constructor inside the setup-error wrap so a refused scope answers 400 on create and stops a config-file proxy at startup. Drop settings cleared to null so the package default applies. Say an installed package failed to load, with the error, instead of asking to install it, and keep any import-time exception from breaking the guardrail routes. Make api_key optional in the form since COLLIEAI_API_KEY is the fallback, and open the Garden card with Always On off like the other partner presets.
* fix(guardrails): leave CollieAi form fields empty and share one guardrail setup error
The Garden form no longer pre-fills api_base, deadline_ms or forward_conversation_id, so a
create saves only what the admin typed and the package keeps its own defaults. Setup failures
raise GuardrailSetupError from custom_guardrail, the missing-package hint covers the Docker
image, and the card copy no longer repeats the package's eligibility rules.
* test(guardrails): cover env-reference api_base in the DB poll comparison
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* perf(router): list a group's deployments once per registry change on the acompletion path
* perf(router): drop the new docstrings
* perf(router): keep the counter cache's mutable-ok on its annotation
* perf(router): define the counter cache's alias in the registry
* feat(azure_ai): support gpt-image edits on Azure AI Foundry
Foundry hosts gpt-image deployments on the same /openai/deployments/{deployment}/images/edits route as Azure OpenAI, but the azure_ai image-edit factory only knew MAI and FLUX, so gpt-image-2 and the gpt-image-2.5 models raised a ValueError before any request was sent
Reuse AzureImageEditConfig for the request, URL, and multipart handling, and resolve the api_base and credentials with the azure_ai helpers
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(azure_ai): default gpt-image edits to an edit-capable api-version
Foundry gpt-image edits now fall back to 2025-04-01-preview when no api_version is set, since the shared Azure default 2025-02-01-preview answers 404 on /images/edits. The config returns merged headers instead of updating the caller's dict, and the tests check the request each gpt-image model sends
* fix(azure_ai): ignore the Azure OpenAI api-version env for Foundry gpt-image edits
The proxy CLI always sets AZURE_API_VERSION to 2025-02-01-preview, which
Foundry answers with 404 on image edits. gpt-image edits now read only
litellm_params, AZURE_AI_API_VERSION, and litellm.api_version before the
2025-04-01-preview default, matching the other azure_ai edit configs
* fix(azure_ai): route custom Foundry deployments, require a Foundry api_base, and honor caller auth for gpt-image edits
A deployment name the edit factory does not recognize now uses the gpt-image route instead of raising, since Foundry deployment names are free-form. A gpt-image edit with no Foundry api_base raises like the MAI and FLUX configs instead of borrowing AZURE_API_BASE. The shared edit handler passes the call's extra_headers to validate_environment, so a caller's api-key or Authorization header is sent alone
* fix(azure_ai): keep the shared edit handler unchanged and send caller auth headers like azure_ai chat
---------
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(databricks): price claude-fable-5-2 from the proprietary pricing page
* test(databricks): list claude-fable-5-2 among the rows with published cache DBU rates
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): wait for child readiness before supervisor cancellation
* test(proxy): sync supervisor cancellation test on a socketpair handshake
Replace the readiness marker poll and timed sleeps with one-byte socketpair
messages. The child reports its SIGTERM handler is installed, reports the
stop, and holds its exit until the parent releases it, so the assertion only
passes when the cancelled supervisor waits for the child. A pytest-timeout
marker bounds a deadlock.
* test(proxy): drive supervisor cancellation cleanup with a controlled child
Replace the real child and socket handshake with a scripted process returned
by asyncio.create_subprocess_exec and a stop observed through os.killpg. The
child only exits after the stop and a checkpoint, so returncode is 0 only when
the cancelled supervisor stops the child and waits for it without being
cancelled.
* test(proxy): start the supervisor cancellation only once the reporter waits
The scripted child signals when a wait begins, and the test cancels only after
the exit reporter is blocked on the stop, so cancellation reaches it without
relying on task start order.
* test(proxy): await background failure spend counters before asserting
Failure spend counters finish in a background task when the counter store is
slower than the wait budget, so the tests read stale values on a slow runner.
Hold the counter store on a release event with a zero wait budget, capture the
background task, and await it before asserting the counters.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(fips): hash MCP advisory-lock keys with SHA-256 in fips_mode and compute the OCI body digest as a security hash
* test(fips): move the lock-key tests into test_mcp_env_vars.py and type the OCI digest test
* feat(bedrock): search knowledge bases by image and return image hits
vector_stores.search and asearch take a query_image (base64 jpeg, png, gif or webp), and query is optional when it is sent. Bedrock sends it as an IMAGE retrievalQuery, and a caller's own retrievalQuery in extra_body still wins
Bedrock IMAGE hits now come back as image content items instead of being dropped. The media type comes from the source file mime type metadata, then the data URI label, then application/octet-stream
A request with both query and query_image, with neither, or with a malformed image gets a 400. So does query_image on a provider without image search, unless drop_params is on. The proxy accepts query_image in place of query on /v1/vector_stores/{id}/search
Spend logs redact image data when prompts are not stored. Emulated file_search skips image-only hits, RAG rerank maps its indices to the text documents it ranked, and the dashboard tester renders image hits
* fix(vector-stores): keep image hits to image searches and reject a caller retrievalQuery
Text searches on a multimodal Bedrock knowledge base return the same text hits as before, so chat, RAG and logging payloads stay unchanged. Image hits come back only when the caller sent query_image
A caller retrievalQuery now answers 400 pointing at query_image, or is dropped under drop_params, instead of replacing the built query unchecked
A file_search chat tool that carries query_image or rewrite_query searches again instead of failing, and a non-string query next to an image answers 400 instead of 500. query_image is keyword-only on search and asearch
* fix(vector-stores): redact query images and keep store search params out of image checks
* fix(vector-stores): redact images nested in extra_body and keep a store's extra_body retrievalQuery out of searches
* fix(vector_stores): keep a caller's image query through store and router params
A deployment's retrievalQuery no longer reaches a routed search, a caller's query fields survive a store's own extra_body, and an extra_body image stops before the provider. file_search forwards rewrite_query, both retrieval query spellings are redacted, and the cost error stops echoing the response when message logging is off
* fix(cost): leave image search responses out of the cost error text
* fix(vector_stores): accept line-wrapped base64 in query_image data
Bedrock takes inlineContent wrapped with \n or \r\n line breaks, as plain base64 writes it, so the check ignores only those before decoding and the data goes to Bedrock as sent
* fix(vector_stores): drop rewrite_query false from the chat file_search tool
False asks for the default search every store already runs, and MongoDB
rejects the field outright, so forwarding it made a MongoDB chat lose the
store's context for a no-op flag. True is still forwarded.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(health): price connection tests using their resolved mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): avoid deployment pricing on alternate probes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): preserve pricing for equivalent model names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): avoid authentication when comparing health check models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover health probe pricing without a deployment model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): pin the clock in the rpm rate limit test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(messages): apply modify_params rewrites on rust
* feat(messages): mint anthropic workload identity tokens on rust
* feat(messages): apply prompt caching injection on rust
* feat(messages): serve the easy python-only detectors on rust
* feat(messages): retry rejected thinking signatures on rust
* feat(messages): serve response cache reads on rust
* feat(messages): retry rejected strict tools on rust
* refactor(messages): delete the python path for anthropic messages
Provider anthropic on /v1/messages is now RUST_REQUIRED, so it runs on Rust
without LITELLM_RUST or litellm.rust(True) and never falls back to Python.
A call the rules would send to Python (a MESSAGES_PYTHON_FEATURES feature or a
detector that raises) gets UnsupportedParamsError naming the feature, raised
from one place in litellm/messages/dispatch.py. Other providers keep the
Python fallback. A call whose arguments the handler signature rejects still
gets Python's TypeError.
Deleted Python code that only served anthropic Messages:
- workload identity in AnthropicMessagesConfig.validate/avalidate (subclasses
were never eligible; the Rust host mints the token itself)
- the anthropic Foundry beta-row and affinity-base branch in
async_anthropic_messages_handler, and the "forward" mode of the beta filter
that only that branch used
The proxy MCP tools detector now uses the light tool precheck so a function
tool never imports the gateway handler, since a raising detector now refuses.
* feat(messages): forward message threads on rust
The Rust path for anthropic /v1/messages dropped the thread body field, so
a create or continue turn reached Anthropic without its thread or the
message-threads-2026-08-12 beta. It now matches the Python handler from
#50976:
- the host forwards thread with the other body params
- a non-null thread adds the message-threads beta
- a well-formed continue turn keeps advisor blocks in history and derives
no compaction, per-turn-control or tool-change beta from its messages
- the recorded httpx_response carries the request that was sent, for JSON
and streamed calls, so the proxy's thread ownership hook resends the
create turn's betas on a continue as it does for Python
Upstream error.details already reached the exception message unchanged, so
thread_not_found relays and keeps the deployment out of cooldown on both
paths. Parity cases cover create, continue, streaming, sync, retries,
thread_not_found and the ownership hook round trip.
* feat(messages): serve mock responses on rust
* feat(messages): send caller clients and unsupported tls through python on rust
* feat(messages): run pre-request hooks on rust
* feat(messages): run agentic loop hooks on rust
* feat(messages): run the safeguards classifier on rust
* feat(messages): run registered interceptors on rust
* fix(messages): match python stream cache gate, logging and interceptor order on rust
Store a streamed Messages response in the cache only when Python would: a
message_stop event line, no provider error frame and at least one frame with
output. Leave dict results untouched by mark_rust_response so hidden params
never reach result dicts, logging objects or hook digests. Ask interceptors
whether they handle the call after the cache lookup, not at decode time, so a
cache hit never calls can_handle. Emit the track_llm_api_timing service event
for the provider exchange. Bill and log a stream superseded by a router
fallback as the failure of its error frame, with the partial usage warning.
Hand litellm_logging_obj to deployment pre-call hooks as @client does, so the
autorouter baseline cache sees retried requests.
* fix(messages): forward python-only json values to anthropic as python does
Accept any max_tokens, sampling scalar and messages collection that
Python main accepts, and carry values serde_json cannot hold (ints beyond
64 bits, NaN and infinities, strings with lone surrogates) as validated
raw JSON leaves written back verbatim on the wire, so the upstream body
matches Python's json.dumps
* fix(messages): match python's body reads, error bodies, encoded heads and tls warnings on rust
* fix(messages): log python's plain provider header names on rust streams
* fix(messages): match python's request params, logged params and validation on rust
Read a non-bool stream by Python truthiness and send it as given, log the
caller's stream flag as Python's handler does, raise the raw Python errors
main raises for malformed messages, tools, model_info and hook-set
litellm_params or providers, read the tool custom-field drop from
drops_unsupported_params, log the caller's nested fields that
additional_drop_params strips and the Python values of raw JSON leaves,
keep a missing stop_reason absent, and map provider-config rejections
(missing max_tokens, unknown reasoning effort) as Python's HTTP handler
maps them, with Python's text, effort order, model and provider
* fix(messages): match python's tls curve names, redirects, auth token, header encoding and timeouts on rust
* fix(messages): match python on hooked streams, empty cache writes and mcp cache hits
* fix(messages): join already buffered stream reads and trim the env api key as python does on rust
* fix(messages): deliver stream bytes read before a cut, keep wide reply numbers and follow python's redirect limit on rust
A stream cut by a transport error now hands the caller the bytes already
buffered before raising, as aiohttp does for a caller that reads them before
the event loop sees the cut, so the frames and their usage reach the caller and
the failure log. Provider replies keep integers wider than 64 bits exactly and
read numbers that overflow a float as infinity, through the response, the
response cache and the logged result. The provider client follows up to 20
redirects as httpx does, and running out raises Python's text. A
provider_specific_header value the ascii codec cannot encode fails before the
send like a caller extra header. The deployment success hook gets the stream as
AnthropicMessagesStreamingResponse, as Python's handler hands it.
* feat(messages): serve proxy mcp tools on rust
* test(messages): cover hook, agentic and interceptor edge cases in the parity suite
* fix(messages): keep provider raw-json marked strings, drop non-string tool_result texts and key clamped streams as python does on rust
* fix(messages): match python's api key encoding, malformed api_base and string provider header errors on rust
A header value Python's client cannot encode as ASCII now fails before the send
whichever header carries it, so a caller api_key with a non-ASCII character
raises Python's InternalServerError instead of a raw UnicodeEncodeError after
the reply. A caller api_base without an http(s) scheme, or with whitespace,
control or non-ASCII characters, is sent through Python's client at the URL
Python builds, so its relative-URL and invalid-URL errors match, and a base
urlsplit cannot split raises Python's bare ValueError before the send. A
provider_specific_header of the wrong shape is scoped through Python's
ProviderSpecificHeaderUtils first, so a plain string raises Python's
AttributeError.
* fix(messages): keep integer temperature and top_p digits a float cannot hold and raise python's typeerror for a string max_tokens while capping the thinking budget on rust
temperature and top_p now deserialize through recognized::exact_float, which keeps an integer
where float(n) != n (beyond 2**53) as an unrecognized value with its exact digits, so it is sent
upstream and logged as the integer python forwards, and comparisons still go through scalar.rs
cap_thinking_budget_to_max_tokens compares max_tokens <= 1024 first, so a max_tokens python
cannot order against an int (a string, list or dict) raises TypeError "'<=' not supported between
instances of 'str' and 'int'" in both the reasoning effort translation and the adaptive effort
reshape. rust now returns ErrorDetail::UnorderableTypes there, the bridge raises it as a python
TypeError marked as a late rejection, and map_failure hands it to the caller unmapped
* test(messages): format the non-string tool_result parity helper
* fix(messages): flag and log the call before python's pre-send raises on rust
The bridge now hands settings() and map_failure() whether the call is async.
A provider_specific_header of the wrong shape sets the aanthropic_messages
flag before re-raising, as Python's handler has already flagged the call. A
base urlsplit cannot split also logs the resolved model, provider and optional
params first, as Python raises it only after its logger is updated. The
TypeError raised while capping the thinking budget for a string max_tokens
sets the flag too
* feat(messages): serve the web search short-circuit override on rust through the intercept host call
* fix(messages): give the web search entry kwargs its mutable-ok reason
* test(messages): drop the body hook gap the rust short-circuit closes
* fix(messages): name the partially unknown handler types on the short-circuit host call
* feat(guardrails): write guardrail changes to the audit log
Creating, updating, patching or deleting a guardrail, and registering, approving or rejecting a team guardrail submission, now each write one row to LiteLLM_AuditLog with the guardrail before and after the change. Credentials the guardrails table encrypts at rest are redacted from those rows, and the routes take the litellm-changed-by header like key, team and model routes do. The Admin UI audit page gets a Guardrails table filter.
Resolves LIT-10639
* fix(guardrails): audit an allowlisted guardrail snapshot and drop litellm-changed-by
Guardrail audit rows now keep the guardrail id, name, info, status, team and the provider, mode and default_on settings, and name (never show) any other provider setting an update changed. Provider settings can hold credentials under any name, for example an apikey header, so no other value is written. The changer always comes from the caller's API key, so the litellm-changed-by header is gone. PUT audits the saved guardrail instead of the copy the live sync adds headers to.
Resolves LIT-10639
* fix(guardrails): never let the audit snapshot fail a saved guardrail change
The audit snapshot now reads guardrail settings stored as a JSON string and records empty settings when it cannot read them, so a saved approve or reject can no longer turn into a 500. Only the provider, mode and default_on settings are ever copied, so this cannot leak a credential. The create entry also records the active status the new guardrail is saved with.
Resolves LIT-10639
* test(guardrails): run the guardrail audit tests through the real registry and in-memory handler
The audit tests no longer replace GUARDRAIL_REGISTRY or the in-memory handler, and no test edits its inputs. Only the Prisma guardrails table is faked, returning exactly what the registry writes, so the PUT test now checks the real live sync adding x-api-key to the saved guardrail's headers.
Resolves LIT-10639
The test spawned a real Python child that installs a SIGTERM handler, slept
0.3s, then cancelled the supervisor and expected the child to exit 0. CI runs
with --cov, and pytest-cov starts coverage in every spawned Python process,
so the child is often still starting when SIGTERM lands and dies with -15.
open_children now takes the process spawner the same way it takes the
platform, and only needs the platform to signal and stop children. The test
uses a fake child whose wait() blocks on an event and a fake platform that
records stop_group, so it checks the shielded stop-then-wait with no real
process and no sleep. It also checks the child is still being waited on,
because anyio's wait_all_tasks_blocked on asyncio can return one loop tick
before a task group finishes.
* fix(fallback): serialize fallback writes and wait for replicas on read-back
POST and DELETE /fallback now read, edit and write router_settings in one
writer transaction under the same per-param advisory lock the config section
writer takes, so concurrent writes on any replica no longer drop each other's
rules. The cache is invalidated and the router updated only after commit.
The Terraform provider now polls the read after a litellm_fallback create or
update until the proxy serves the written list, within new create/update
timeouts, instead of clearing the ID on a stale 404 or saving a stale list.
Refresh 404 still removes the resource, and error responses are not retried.
Fixes#47888
* fix(fallback): bound provider requests by the apply context and inject the fallback repository
The provider's fallback write and read-back requests now carry the apply
context through a context-aware MakeRequest variant, so a cancelled apply
sends nothing and the create/update timeout also ends a hung request.
The POST and DELETE /fallback routes take a RouterFallbackRepositoryDep that
returns a RouterFallbackStore. Router settings parsing moved into
RouterFallbackRepository, which edits the rules through the locked
ConfigRepository operation.
Unit tests now check the outcome of the race and rollback through a fake
lock instead of pinning the lock SQL. They also cover a racing config
section write, a 404 delete inside the lock, route resolution of the
repository and the no-database path.
* refactor(fallback): keep atomic edits in the config repository
* fix(terraform): release hung test handlers and audit MakeRequestWithContext calls
The hung-request cancellation test now releases its handler through a test
owned channel before closing the server, since the server does not notice
the client's cancel while the POST body is unread.
The endpoint audit now reads the method and path of MakeRequestWithContext
calls from their shifted argument positions, and skips only MakeRequest's own
forwarding call inside the helper files, so POST /fallback counts as covered
again and dynamic paths elsewhere still fail closed.
* test(fallback): keep stored-rule regressions with their helpers
* fix(fallback): read saved rules without Terraform changes
* fix(fallback): preserve valid rules when stored entries are malformed
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(bedrock): forward the compact-2026-09-04 `compaction` param on Invoke /v1/messages
The compact-2026-09-04 beta is a top-level `compaction` request parameter
({"type": "summarize"}), not a context_management edit. The Bedrock Invoke
allowlist is built from BedrockInvokeAnthropicMessagesRequest, which had no
`compaction` field, so the field was dropped and the request reached Bedrock
as an ordinary turn. The bedrock beta map also dropped compact-2026-09-04, so
adopting requests lost the beta.
- add `compaction` to BedrockInvokeAnthropicMessagesRequest
- add compact-2026-09-04 to the bedrock beta map (Invoke only, not Converse)
- add the beta automatically when a `compaction` param is present
Ref: https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-compaction.html
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(bedrock): send `compaction` only with its beta, and add the beta for an adopted block
Follow-ups to forwarding the compact-2026-09-04 `compaction` parameter on
Bedrock Invoke:
- `compaction` stays in the body only when it is an object and
compact-2026-09-04 survives the route's beta filter. bedrock_mantle has no
entry for that beta, so a mantle request keeps the earlier strip instead of
reaching Bedrock with the field and no beta (400 "Extra inputs are not
permitted"). A null or non-object value is dropped the same way.
- A request that adopts the signed compaction block now gets the beta even
without a client anthropic-beta header, as the Anthropic route does
(carries_signed_compaction_block).
- The compaction tests pin the bundled beta config (local_beta_headers_config)
so they do not read the remote copy an earlier test left cached.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(bedrock): type the compaction helper and field as read-only views
The type discipline gate rejects a bare dict or set annotation and a TypedDict field without ReadOnly; the helper only reads the request and grows the caller's beta set in place.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep the compaction helper within the basedpyright ceiling
Read the request's messages through a typed adapter and delete the dropped compaction key, so the new call adds no unknown member or argument diagnostics.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(bedrock): drop comments that restate the compaction checks and type the new test params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Homen Shum <317650355+homenshum-cheiron@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The secret manager lanes ran the proxy against a Vault dev container and a
Conjur container that we start ourselves, so nothing in them needed a real
third-party service. They have not run a single test since late October,
because the co-located proxy cannot boot in the lane pod, and they were
soft_fail outside the promotion gate, so they only added noise.
The AWS test in the folder was skipped on every lane, since no aws lane
ever existed.
The four coverage registry rows go with the tests, because the registry is
the e2e denominator and this behavior belongs in the integration tier if it
gets covered again. The Vault and CyberArk clients keep their unit tests.
The API Reference page was already behind a deprecation banner. This deletes
the /ui/api-reference route and its components, the Developer Tools sidebar
entry and page description, the legacy ?page=api_ref redirects, and the i18n
strings only that page used.
Old ?page=api_ref and ?page=api-reference bookmarks now fall through to the
dashboard like any other unknown page id, since there is no replacement page.
Tests that listed the page are updated: the sidebar and legacy-route vitest
files, the uiHref examples, the e2e sidebar fixtures and Internal Viewer nav
check, and the UI path detection unit test.
* fix(proxy): name the deployment in stream error alerts on every provider and route
The llm_exceptions alert for a cut stream on Anthropic chat, /v1/completions,
native /v1/responses and native /v1/messages had no API Base, model_group or
deployment line. The shared exception mapper now attaches its debug info at its
single exit, mid-stream fallback errors inherit it from the error they wrap,
and the native relays add it to the same error object they re-raise.
* fix(proxy): keep the prompt out of attached alert debug info and name the deployment on more stream exits
* fix(proxy): name the deployment on stream errors that arrive already mapped
exception_type returned errors that were already litellm types before it attached the debug info, so a finish_reason=abort chunk or the max streaming duration timeout still sent an alert with no deployment. It now builds the info first and attaches the prompt-free copy on that early return too, and native Responses streams attach it on their generic error exit
Also adds tests for the dropped `Messages: None` line and the mapping-crash exit
* fix(proxy): read the router's deployment from litellm_metadata on Messages and Responses errors
/v1/messages and /v1/responses keep the router's model_group, deployment and api_base in litellm_metadata, so their errors before the stream named only the model in alerts. The debug info now reads litellm_metadata first, falls back to metadata, and takes API Base from it when the call passes none
get_first_chars_messages returns an empty string again for input that is not a mapping, as it did before this branch
* fix(proxy): merge both metadata buckets when error alerts name the deployment
A pre_call guardrail adds litellm_metadata to a chat request while the router keeps model_group and deployment in metadata, and generic routes such as moderations keep the key alias and team in metadata and the deployment in litellm_metadata. Reading only one bucket dropped lines the merge base showed. The debug info now merges both, and litellm_metadata wins when both carry the same key, so a caller's own metadata cannot replace the router's values
* fix(proxy): keep a reused error's alert details on the request that raised it
An exception instance raised for more than one request, such as an SDK mock_response exception or a handler that raises one shared error, kept the first request's Key Name, model_group and deployment in every later alert
The attached details now record the call id that set them. A later call with a different id replaces them, and a layer that re-maps the error without the request keeps them
* fix(proxy): drop alert details from an error object raised by more than one request
One exception object raised by two concurrent requests (a shared mock_response error, or a
handler that raises one error object for every request) let a request's alert show the other
request's Key Name and deployment, since the later attach replaced the earlier one. An error
object tagged by one call id and attached again under another now drops its details for good,
so no alert names the wrong request. One request re-mapping its own error, with or without a
call id, keeps the lines it already has
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(rate-limit): name the limit that rejected a request in a rate_limited_by header on the 429
* test(rate-limit): expect rate_limited_by in the exact-headers assertion
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer x-litellm-call-id on every token count response
The token count log line names a litellm_call_id, but count responses
never carried it, so a client that sent no id of its own could not find
its count's log line. Every count route (Anthropic Messages, Responses
input_tokens, Gemini countTokens, Bedrock count-tokens, and
/utils/token_counter) now answers the id on success and on errors, and
a crash answers the JSON 500 the Anthropic and Gemini routes already used.
* fix(proxy): keep the gateway's crash answers on count routes and stamp Responses opt-in bodies
A crash on /utils/token_counter or /v1/responses/input_tokens goes back to the
gateway's generic 500, or 503 when the database is down, now with the
x-litellm-call-id header. Responses count rejections carry error.litellm_call_id
when include_call_id_in_error_body is on
* test(proxy): count through count_request_tokens in the token counter loop tests
The token_counter route now also takes the response it sets x-litellm-call-id on, so these tests call the shared count function they exercise
* fix(proxy): keep a count crash's own exception and log pre-count Responses failures
The call id now reaches the crash handler through request.state instead of a wrapper exception, so logs and Sentry see the error the count raised. A Responses count that fails before counting writes its count line for any error, so its header names a line
* fix(proxy): keep the token counter's injected args out of its API docs
Tests the opt-in error body id on an Anthropic count the proxy rejects before counting
* fix(proxy): keep a count route's own HTTP error when adding the call id
The wrapper raised a new HTTPException with the same status, detail and headers, so a subclass lost its type and error tracking saw a new exception. It now adds the header to the original and raises it again
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): reset a deleted customer's admission spend counter so a re-created one is not refused
/customer/delete only dropped the committed twin, so spend:end_user:<id> kept the old spend and a
re-created customer with the same id stayed over budget. Zero the counter and its DB floor marker on
every worker after the delete commits, the same way a key spend reset does.
* fix(proxy): drop a deleted customer's spend counter before zeroing it
A Redis that rejects writes but still serves reads, such as one at its memory limit, kept the old
counter when the zeroing SET failed, so the re-created customer stayed refused. Delete the counter
first, which Redis still allows in that state, then write zero and the floor.
* test(proxy): type the customer delete spend counter test parameters
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* refactor(typing): clear 340 basedpyright Any errors (wave 1)
* refactor(typing): clear 497 basedpyright Any errors (wave 2)
* refactor(typing): clear 74 basedpyright Any errors (wave 3)
* refactor(typing): clear 143 basedpyright Any errors (wave 4)
* refactor(typing): clear 106 basedpyright Any errors (wave 5)
* fix(typing): keep router fence visibility and py310-safe Self import
Bind the Protocol-narrowed routers in pass_through_endpoints and rag
ingestion to router-named locals so router_surface_check still sees the
aadapter_completion and aocr uses, and import Self from typing_extensions
in prisma_client for check_py310_typing_imports.
* perf(core_helpers): fast-path dicts and lists in the checked JSON helpers
as_str_mapping now tries the concrete dict check before the Mapping ABC
check, and iter_items returns lists before the runtime-checkable Protocol
check. Both return the same object as before, and roughly halve the
overhead CodSpeed flagged on the A2A text-extraction benchmarks.
* refactor(realtime): keep the base TypeError for a non-mapping translation expires_after
* fix(proxy): restore the require_attributes import the main merge dropped
* test: cover the retyped realtime, vector store, agent, stream, video and provider response paths
* fix(realtime): name the public request types in non-mapping TypeErrors
keyword_arguments wraps expires_after and transcription_session so a non-mapping value raises the same TypeError text as before the typing pass. Tests narrow Greptile's pegasus case with isinstance and freeze its model.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(otel): let a team or key OTel v2 destination add its own request metadata keys
A team or key callback can now set baggage_metadata_keys in its callback_vars,
a comma-separated list of requester_metadata.<key> paths. The tenant fan-out
adds them as litellm.metadata.<key> to that destination's spans only, on top of
the global LITELLM_OTEL_BAGGAGE_METADATA_KEYS; the operator's exporter and other
destinations never see them. Values are bound when each span ends, an attribute
the span already carries wins, and names of proxy identity fields are rejected
when saved, along with lists over 20 keys.
Resolves LIT-9545
* fix(otel): refuse request headers in a destination's baggage_metadata_keys
The proxy copies the request headers into requester_metadata.headers with
credential headers masked, but a header a gateway in front of it added
(an IAP or OIDC assertion, an internal token) stays as sent. A team or key
admin listing requester_metadata.headers.<name> could export it to an
endpoint the team controls. Entries under requester_metadata.headers are
now rejected when saved, so exporting a header stays an operator decision
in LITELLM_OTEL_BAGGAGE_METADATA_KEYS. Also drops an unused import.
* fix(otel): hold destination metadata to the configured span limits
A destination's extra litellm.metadata.* attributes were added to a span
view built from a plain mapping, so they skipped the limits the SDK
applies in set_attribute: a 1 MiB caller value survived an operator's
OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT. Values are now cut to the length
limit, and keys past the attribute-count limit are dropped and counted
instead of evicting an attribute the span already carries.
The per-destination views are built in one Final tuple ahead of the
forwarding loop, since a Final cannot be assigned inside the loop.
* fix(otel): apply destination metadata after the destination view, within the provider's limits
The span limits were checked against the span as it ended, before the
destination view strips prompt content and backend-only attributes, so a
model-call span full of llm.input_messages.* could drop the team's keys on
a no_content destination that had room. The values are now bound when the
span ends and added as the last step before export, counted against what
that destination actually sends, so their dropped count also survives the
view. The limits come from the tracer provider the fan-out is attached to,
read from the live span at start, instead of a separate env-built SpanLimits.
* refactor(otel): consolidate span limits, destination validators and metadata key parsing
Reads a span's limits in one guarded place that the emitter and the fan-out
share, replaces the two parallel lists of per-destination callback-var
validators with one mapping, splits and strips a baggage_metadata_keys list
once, drops a filter the post-view merge already enforces, and folds the
repeated fan-out test setup into one helper. No behavior change.
* ci: retrigger checks stranded by the GitHub Actions outage
* fix(proxy): stream the guardrail block response when the block lands after an SSE keepalive ping
* docs(proxy): keep the keepalive docstring's reasons together
* fix(proxy): stream the /v1/messages block after a keepalive ping and always end a late block stream
A passthrough guardrail block on a streamed /v1/messages request that landed
after the first keepalive ping still ended as an api_error event. The route now
hands its block stream to the keepalive relay, as chat and text completions do.
If a route's block stream itself fails after a ping, the relay sends the usual
late failure frame instead of raising, so the stream always ends.
The passthrough tests now drive the real routes with a real guardrail.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(completion): answer 400 for a chat message without a role
A message with no role was filled with role assistant and sent upstream, so the caller's text reached the
model as its own earlier reply and the request returned 200. Now a missing, null, or empty role answers 400
with OpenAI's wording, naming the message index. A role-less turn that carries tool_calls is still sent as
assistant, since only an assistant writes tool calls.
* fix(completion): treat a role-less message as the caller's error
The 400 now raises MalformedRequestFieldError with OpenAI's param and code and the provider name, so it no longer counts toward the deployment's failure rate, cools it down, or fires an llm_exceptions alert
* fix(completion): keep the assistant fill for a role-less function_call turn
* refactor(completion): type the role-less call turn check and the request kwargs
Read role, tool_calls and function_call off the cleaned message copy, and validate completion's kwargs into a typed mapping before naming the provider on the 400
* fix(completion): keep the alert and failure-count exclusions to the role error
Raise MissingMessageRoleError, a MalformedRequestFieldError subclass, so a malformed tools 400 alerts and counts as on main, and fill assistant whenever a role-less turn has a tool_calls or function_call key
* fix(completion): keep assistant-only role-less turns and skip SDK and PagerDuty alerts
A role-less turn carrying any assistant-only key, null values included, is still
filled as assistant, so a dumped assistant message or a reasoning turn keeps working.
The Router's alerting_config alert and the PagerDuty failure count skip the role error,
like the proxy alert already does.
* fix(utils): drop the string code from the missing-role 400 so the Errors tab counts it as 400
The spend log stored code missing_required_parameter as the error code, which
the Usage page Errors tab cannot parse as a status, so every role-less request
landed under no status. The malformed tools 400 leaves code out the same way.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The team-model and team-key transactions in /team/delete used Prisma's 2s
default max_wait, so a busy writer pool failed the delete with P2028 while
every plain query in the same request waited up to the 60s pool timeout. Both
now get the 30s team-delete budget for max_wait, and the key transaction gets
it for timeout too.
* test(integration): run baseline accounting store tests in a private schema
The accounting workers share their database with two gateway proxies, and every
spend-log tick on those proxies projects any dirty baseline comparison it finds.
When one of them published the test's scope between its last append and its own
project() call, the test saw "unchanged" instead of "published".
The module fixture now copies the twelve tables the store touches into a fresh
schema and points DATABASE_URL at it for the module, so the gateway proxies,
which only read public, can no longer publish these rows.
* test(integration): wait for H9's own prometheus increments before it returns
H9 stopped as soon as both workers' sample files already counted `other`, so
its own chat's increment could still be in flight and land inside the next
cell's window, which then saw two requests instead of one.
H9 now takes a before scrape, counts the chats it sends, and finishes with the
same _overflowed wait every other TestCapped cell uses.
* feat(gemini): file-backed batch jobs on /v1/batches with output download
Adds GeminiBatchesConfig (create + retrieve) behind custom_llm_provider=gemini,
registers it in ProviderConfigManager, and implements file content retrieval on
GoogleAIStudioFilesHandler so GET /v1/files/{output_file_id}/content returns the
batch output. Gemini runs the OpenAI-shaped JSONL the proxy uploads today unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): type the file content request params so no pyright rule grows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(gemini): price completed batch output rows so spend lands on the key
Gemini writes the usage of an OpenAI-shaped output row as camelCase
(promptTokens, completionTokens, totalTokens) with thinking tokens only in
totalTokens. Map it onto Usage with the thinking tokens folded into
completion_tokens, so the shared batch cost path bills them at the gemini
_batches rates.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): parse nanosecond file timestamps
Gemini stamps files with nine fractional digits, which strptime's %f
rejects, so every retrieve of a batch output file logged a ValueError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): parse file timestamps on Python 3.10 too
fromisoformat on 3.10 takes neither a Z suffix nor more than six fractional digits.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): price native batch output rows, stream output downloads, reject unsupported batch metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): detect native batch rows by key, anchor the streaming file download handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(httpx): make anchor_handler_to public so the file content stream can use it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): supervise the opt-in Rust forwarding server
With LITELLM_RUST on, `litellm` and `python -m gateway.launch` start a
supervisor that runs the Rust gateway on the public port and Python on a
private loopback listener. Python still serves every route and sees each
request as if the client had connected to it directly.
Rust sends the client's TCP peer in `x-litellm-rust-peer`. The supervised
server turns server-level proxy headers off, and a middleware restores
the peer and then applies the operator's own trust, so client IPs,
trusted proxies and every IP-based setting behave the same with Rust on
or off.
The supervisor is transparent to Python. Signals pass through, Python's
exit status is the result, and both children shut down when the
supervisor's end of their stdin closes, so nothing outlives it. Rust
starts only once Python listens. When forwarding cannot start (no
gateway binary, TLS, a Unix socket, Hypercorn or Granian), the entry
point warns and serves from Python alone.
Every forwarding decision lives in litellm/proxy/rust_forward, paired
with the gateway-forward crate. OS differences sit behind a Platform
protocol, with Windows supported for local CLI runs. AGENTS.md files in
the touched folders state the boundary. The `--rust_gateway` flag is
gone in favor of the environment variable.
* refactor(proxy): switch Rust forwarding on with LITELLM_RUST_FIRST
Forwarding gets its own switch, independent of LITELLM_RUST, which keeps
selecting only the Rust paths inside the SDK. Neither turns the other on.
---------
Co-authored-by: narmstrong-berri <narmstrong@berri.ai>
Extract serving_port() and configure_database() as they were inline, turn the
LITELLM_RUST parser into a reusable RustSwitch type, and reformat
tests/unit/proxy/test_proxy_cli.py to the 120 column limit.
run_gunicorn_server takes private_listener, an inherited socket or an assigned
loopback port. With one, gunicorn binds it instead of host:port and trusts no
peer's forwarding headers, since every peer on it is the local process in front.
Both are gunicorn's own settings, bind and forwarded_allow_ips, so the uvicorn
worker is unchanged. The reload options take factory. No caller passes either
yet, so every entry point serves as before.
* feat(proxy): count a rate-limit fallback in x-litellm-attempted-fallbacks and name it in x-litellm-fallback-reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count every fallback candidate a local rate limit rejected in x-litellm-attempted-fallbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count fallback attempts without a mutable list and keep self.data on the served request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tls): exclude RSA key exchange from outbound cipher list in FIPS mode
* fix(tls): apply the FIPS cipher filter to SSL contexts clients already hold
enforce_fips_mode runs after load_config, so guardrail and logger clients
built during config load held an SSL context without the RSA key exchange
exclusion. enforce_fips_mode now updates every cached context in place, and
the SDK no longer imports the proxy's FIPS module.
* fix(tls): refuse to start in FIPS mode when SSL_SECURITY_LEVEL has only RSA key exchange
Excluding RSA key exchange from an SSL_SECURITY_LEVEL that lists only RSA
key-exchange suites leaves no TLS 1.2 suite, so set_ciphers raised on every
client build. enforce_fips_mode now checks the configured list first and
refuses to start with a FipsModeError that names the setting. The cipher
list test no longer assumes the host has RSA key-exchange suites.
* fix(tls): offer only TLS 1.3 instead of refusing to start when no TLS 1.2 suite without RSA key exchange is left
* chore(tls): warn when fips_mode leaves a cipher list with only TLS 1.3
* style(tls): split the TLS 1.3 fallback warning to fit the line limit
* fix(proxy): exclude RSA key exchange before config-load secret reads in FIPS mode
get_config reads secrets (AWS, GCP, HashiCorp, CyberArk secret managers) over the shared
outbound SSL context before enforce_fips_mode runs, so with general_settings.fips_mode true
those first TLS handshakes still offered RSA key exchange. Turn the exclusion on in get_config
from the raw general_settings, right before the secret manager starts, reusing the existing
fips_mode_setting (now public) and exclude_rsa_key_exchange. enforce_fips_mode still owns the
refusal and the final state; fips_mode given as an os.environ/ reference is still handled there.
* test(proxy): skip the config-load RSA key exchange test on an OpenSSL without RSA suites
On an OpenSSL whose default cipher list offers no RSA key exchange suite, fips and standard mode read secrets the same way, so the test has nothing to tell apart and skips, as the http_handler fallback test already does.
* fix(tls): exclude TLS 1.2 ChaCha20 suites from the outbound cipher list in FIPS mode
ChaCha20-Poly1305 is not a FIPS approved algorithm. FIPS providers already refuse it, so on the Chainguard
image this changes nothing, but fips_mode now drops it from TLS 1.2 explicitly next to RSA key exchange.
Python's set_ciphers cannot touch TLS 1.3 suites, which still come from OpenSSL's config.
exclude_rsa_key_exchange is renamed to exclude_rsa_key_exchange_and_chacha20 to match.
* fix(proxy): add tzdata to the proxy extra so a legacy TZ doesn't stop startup
* style(proxy): drop the inline comment on the tzdata requirement
* test(proxy): check tzdata is installed with the proxy extra on every platform
* fix(proxy): set the first spend report run in UTC so a scheduler zone that differs from the system clock can't skip it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): track self-heal for test_with_id_security_off_reads_and_cancels_add_no_rows_and_no_replayed_turns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): preserve response turn rows when id security is disabled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
