* perf(ci): cache the Rust build the unit shards compile from scratch
Every unit shard installs the workspace, and the root package builds through
maturin, so each of the eleven jobs compiles litellm-rust/crates/python-bridge
in release mode before a single test runs. That step measured 2m40s a shard on
2026-08-21, which is more wall clock than the entire unit tier spends running
tests, and none of it was cached: the uv cache covers wheels it downloads, not
wheels it builds, and a path dependency whose source moves every commit can
never hit that cache anyway.
A composite action now exports CARGO_TARGET_DIR to a fixed workspace path and
caches it alongside the Cargo registry, keyed on Cargo.lock. Cargo rebuilds only
what changed, so a warm job pays for the bridge crate rather than its whole
dependency graph. Measured locally, that is 34s cold against 8s warm, including
after a Python-only or Rust-only edit.
The absolute path matters: uv builds the wheel from its own working directory,
so a relative target directory lands the artifacts where nothing can find them
again.
* perf(ci): cache the Rust build in the other four workflows that sync the workspace
code-quality, mcp, documentation and the schema.d.ts check each install the
workspace and so each compile the bridge from scratch, measured at 138s, 177s,
163s and 154s on 2026-08-21. The lint job pays the same and is left to #37783,
which already owns that file's setup section.
* perf(ci): cache the Rust build in the lint job too
* fix(ci): cache cargo's own target directory instead of redirecting it
uv builds the wheel in place, so cargo already writes to litellm-rust/target,
which test-rust.yml has cached all along. Redirecting CARGO_TARGET_DIR bought
nothing and cost a GITHUB_ENV write that zizmor rejects as a code-execution
path.
* chore(ci): raise the job backstop for the added setup step
The cargo cache is a fifth bounded setup step, so the base's setup ceiling goes
30m to 35m and every job budget follows: 55 to 60, and proxy-server's 95 to 100.
check_workflow_startup_safety enforces exactly this sum, and failed on the first
push without it.
* perf(ci): cache the Rust build in the four remaining workflows that sync
Six workflows were wired; ten install the workspace. The four left out still
compile the pyo3 bridge from scratch.
test-terraform-provider.yml is the one that matters per PR: its endpoint-drift
job triggers on any change under litellm/proxy/**. The other three are a
scheduled load check, a manual mutation run, and the staging-push counts
publisher, whose gate syncs the project inside scripts/type_check_gate.py
rather than in a workflow step, so nothing in the file names the build it pays
for.
completion() aliased the caller's header mapping instead of copying it,
then merged the provider-scoped headers into that same object. The router
shares one header dict across fallback attempts, so the credential written
on an Anthropic attempt was still present when a later Bedrock or Vertex
attempt read the dict, defeating the provider scoping.
Copy the mapping before merging so each attempt sees only its own headers.
ARCHITECTURE.md promised the clean end_turn for every intercepted request.
A request that streams all the way through and one on /v1/responses both
still hand back the internal tool call, so say that plainly instead. Also
note that where the refused call was the only block left, the turn can come
back with no text in it.
On AgenticLoopSafetyError, note that the chat completions loop still raises
a plain ValueError from its own copy of the rails, so nobody writes an
except for this type expecting it to cover that surface too.
get_llm_provider ran in the messages handler and again inside completion,
so a provider/vendor/model id lost its vendor segment and reached upstream
bare. Pass the caller's unresolved model down the bridge instead, and move
the responses marker into the canonical provider/responses/model slot.
Reporting stays provider-local on both bridges: message_start names the id
the provider itself knows, through a shared local_model_name helper.
Fixes#37716
Adversarial review of the new multipart keying turned up collisions where
two different provider requests computed the same replay key, which is the
dangerous failure for a replay harness: the second request silently gets the
first one's response instead of missing loudly.
- a part counts as an upload when it has a filename or declares its own
content type, and the declared content type joins the identity, so two
uploads of the same bytes under the same field no longer collapse
- the uploaded parts contribute a JSON list of [field, filename, type]
triples instead of a "field:filename" string, so a separator inside a
filename can no longer impersonate a field boundary
- repeated field names get a "name[n]" suffix with a literal "[" doubled
first, so a repeated field and a literally indexed one stay distinct
- a field value that is not UTF-8 is stored as a base64 sha256 digest;
base64 rather than hex because the canonicalizer rewrites 64-character
hex runs to <sha256> and folded every binary value onto one key
- a field whose name reads as a credential is stored as <secret>. This
stays key-preserving because the key is recomputed from the stored
request rather than saved beside it, so the live request carrying the
real value still matches its redacted fixture
- the uploaded byte length leaves the key. The canonicalizer absorbs
timestamp and id drift inside a file, and that drift moves the count,
so keeping it there made re-records miss
Also stops a lookalike parameter such as "xboundary=" from being read as
the multipart boundary, and gives the OpenAI batch backend model a single
constant instead of three copies of the literal.
BUNDLE_FORMAT_VERSION goes to 3 because all of this moves recorded keys.
A bundle recorded under the old rules now fails naming both versions
instead of missing on every call.
The chunk loop read `limit + 1` rows at a time, so a small limit whose
matches sit far behind the newest rows advanced a couple of rows per
query. A `purpose` that matches only the last of 10000 owned rows at
`limit=1` cost 5001 sequential find_many calls for one HTTP request,
which any authenticated caller could ask for on purpose.
Once a scan has to continue past its first chunk, widen the chunk to
FILE_LIST_CONTINUATION_CHUNK_SIZE. That same case now costs 21 queries.
The first chunk keeps its `limit + 1` size, so a page the newest rows
already fill still costs exactly one query and reads nothing extra.
Rows whose blob will not parse drop out of a page the way a filter does,
so they get the bound too, not just the purpose filter.
The floor only changes how many round trips a page costs, never what it
returns: chunk boundaries do not affect a keyset scan, so the page is
still `matches[:page_size]`, `has_more` is still `len(matches) >
page_size`, and empty data still implies `has_more` false.
When the bounded loop cap or the repeated tool-call fingerprint guard refused a
rerun, the raise escaped the parent agentic frame and the client got the raw model
turn back: HTTP 200 carrying an unresolved tool_use block for the internal
litellm_web_search tool and stop_reason "tool_use". The client never declared that
tool, so it had no way to answer it and the conversation could not continue
The safety check now raises AgenticLoopSafetyError, a ValueError subclass, and
_call_agentic_completion_hooks catches it and returns a finalized response: the
blocks belonging to the refused tool calls are dropped, and stop_reason is closed
out to end_turn when nothing the client declared is still waiting. Refused blocks
are matched by the ids and names of the tool calls the rail refused rather than by
hardcoding the web search tool name
Only the non-streaming anthropic messages path ends the turn this way. A streaming
caller has already sent the original message by the time the hooks run, so a
finalized turn would arrive as a second message rather than replace the first, and
the responses surface carries a pydantic model this finalizer does not rewrite.
Both keep raising, exactly as they did before
Also adds max_agentic_loops to websearch_interception_params so the ceiling can be
set once for the whole feature. A per deployment litellm_params.max_agentic_loops
still wins over it, and the field stays on the proxy's untrusted root list so a
client cannot raise its own ceiling
The managed file listing cut the page to `limit` first and applied the
purpose filter in Python afterwards, so a page whose rows all failed the
filter came back as `data: []` with `has_more: true`. openai-python stops
paging the moment `data` is empty, so `files.list(purpose="batch", limit=1)`
returned nothing at all instead of every batch file.
Read successive keyset chunks until the page holds `limit + 1` matches or
the caller's rows run out, then return at most `limit` of them. `data` is
now non-empty whenever matching files remain, its last id is always a
usable cursor, and `has_more: false` only ever means the caller has seen
everything. Rows whose stored blob will not parse drop out in the same
loop, so they cannot empty a page either.
That also makes the `next_cursor_id` escape hatch on `build_list_page`
dead, so it goes back to what it was for the batch and vector-store
listings that share it.
Also move `validate_file_list_limit` up into the list_files route, so the
target_model_names and provider branches reject an out-of-range limit the
same way the managed file store already did.
Chat completions, embeddings, the non-streaming /v1/messages tests, and the
OpenAI batch deployment now register through the provider edge, so
E2E_FIXTURE_MODE=record captures their provider calls and replay serves them
back offline. None of them was wired before, so record was a silent no-op over
these suites and replay quietly went live instead of using the bundle
Multipart uploads now key on their parsed parts: every ordinary form field,
plus the field name, filename, content digest, and length of each file part.
The boundary is envelope rather than content, so it stays out of the digest
instead of changing the key on every run. A body that does not parse as its
declared envelope still has the boundary normalized away before hashing, so
the fallback is at least stable, and it records a name that says why
Binary uploads hash byte for byte. Canonicalizing them first meant decoding
with errors="replace", which collapsed every invalid byte to one U+FFFD and
gave two different PDFs of the same length the same key
Bundles stay out of the repo: they hold verbatim provider response bodies and
expire seven days after recording. Publishing them for CI is LIT-5748, and
streaming fidelity is LIT-5742
* fix(pricing): add undated azure aliases for gpt-audio-mini and gpt-realtime-mini
Azure deployments are commonly created against the undated model name,
and the cost-tracking docs say to set base_model to azure/<model> — but
only the dated -2025-10-06 entries existed for these two models (the
openai provider has undated aliases for both). base_model:
azure/gpt-audio-mini therefore resolved to nothing and, depending on the
fallback path, text tokens billed at $0 while audio tokens billed fine.
Mirror the -2025-10-06 entries as undated aliases, exactly like the
undated openai entries mirror their newest dated variant.
Fixes#33170
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(pricing): assert undated azure audio aliases exactly mirror their dated entries
Review follow-up: COST_FIELDS missed realtime-specific cost keys
(cache_creation_input_audio_token_cost, cache_read_input_token_cost,
input_cost_per_image). Full-entry equality catches drift on every field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pricing): mirror updated mode=realtime on the undated gpt-realtime-mini alias
Upstream changed the dated entry's mode from chat to realtime after this
branch was cut; the undated alias must stay a byte-for-byte mirror.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pricing): mirror the new deprecation_date onto the undated gpt-audio-mini alias
* test(pricing): use shared local_model_cost_map fixture so get_model_info's lru_cache never crosses maps
---------
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(anthropic_messages): gate sampling params on /v1/messages like /chat/completions
/v1/messages forwarded temperature/top_p/top_k raw to models that removed
sampling params (supports_sampling_params: false — Claude 4.7+/Fable 5),
producing provider 400s that router fallbacks mask as silent model
downgrades. The chat path already gates these via
AnthropicModelInfo._apply_sampling_param; reuse it in
get_requested_anthropic_messages_optional_param so both endpoints agree:
drop under drop_params, else raise the clean client-side 400.
Fixes#35053
* test(anthropic): drive new sampling-param tests off the kwarg, not the global
The five tests added here set `litellm.drop_params = True` under a manual
try/finally. That trips TQ005 (module-global mutation, 10 new violations
over the ceiling) and it leaks process-wide if the finally is ever
skipped, which is what the save/restore conftest exists to paper over.
`get_requested_anthropic_messages_optional_param` already takes
`drop_params` as a kwarg, and that is the path /v1/messages actually
uses, so pass it directly. `monkeypatch.setattr` pins the global to
False so each test proves the per-request flag alone is sufficient and
cannot pass on a leaked global.
Verified: TQ gate clean, all 10 tests pass, and the 3 that assert the
new gating still fail with the fix in utils.py reverted.
---------
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
The unscoped GET /v1/files limit check accepted 0, which OpenAI's minimum
of 1 does not allow, and the route's except block rebuilt every error with
getattr(e, "status_code", 500). ProxyException has no status_code, so the
400 it raises went out as a 500 and the OpenAI SDK retried it three times.
Errors now go through handle_exception_on_proxy, the helper the sibling
batches route already uses, and the unknown-cursor error is a ProxyException
so it carries type invalid_request_error and param after instead of the
literal "None". The cursor still 400s whether the file belongs to someone
else or does not exist at all
* fix(mcp): resolve admin OAuth sessions to the same server set the connect page shows
* fix(mcp): bind admitted admin rows through the entitlement ceiling, not the credential scope clause
add_provider_specific_headers_to_request tagged the client's Authorization header
with the same provider list as anthropic-beta and anthropic-version, so an
sk-ant-oat subscription token was sent to AWS Bedrock and Google Vertex AI as
well. On Bedrock it replaced the SigV4 signature, or the deployment's own API
key, and AWS answered 403 "Invalid API Key format". On Vertex it went out as a
second Authorization header next to the Google one and Google answered 401
ACCESS_TOKEN_TYPE_UNSUPPORTED.
The credential and those API headers need different scopes, so a request can now
carry more than one ProviderSpecificHeader entry. The API headers keep the
provider list they already had and the credential gets its own entry scoped to
anthropic alone. get_provider_specific_headers takes either a single entry or a
sequence and merges only the entries whose provider list matches, so callers that
pass one entry keep working unchanged.
Bedrock SigV4 signing and the deliberate extra_headers Authorization pass-through
in _sign_request are left alone.
The new test set LITELLM_LOCAL_MODEL_COST_MAP and reassigned litellm.model_cost
by hand, leaking both into every test that ran after it and skipping the
get_model_info cache clear. The conftest fixture already does this properly and
restores the original map on the way out.
A page whose rows are all dropped by the purpose filter, or by a row
that does not parse, used to come back with an empty data list, has_more
true and last_id null, so the caller had no cursor to advance with and
stopped one page short of files it owns. last_id now falls back to the
last row the page read.
Also drops the OpenAIFilesPurpose import that the widened purpose
annotation left unused.
The docstring claimed azure pricing mirrors the openai family, which stopped
being true when gpt-5.6-sol took its promotional cut and azure did not. Azure
publishes no sol rate of its own today, so the entries stay where they are.
OpenAI's model page for gpt-5.6 serves the GPT-5.6 Sol page and states
that the gpt-5.6 alias routes requests to GPT-5.6 Sol, so the alias bills
at Sol's rates. The registry entry was left on the pre-cut rates while
gpt-5.6-sol took the cut, overbilling gpt-5.6 callers by 25 percent on
input and 50 percent on output.
All 23 cost fields on gpt-5.6 now match gpt-5.6-sol, and a regression
test pins the two entries together so they cannot drift again.
The owner-scoped listing read every row the caller owns in one query, so an
admin key that owns every file on the proxy pulled the whole table into one
response. Page it with a keyset cursor on unified_file_id instead, and accept
limit and after on GET /v1/files so a client can walk the pages. limit follows
what OpenAI documents for that route: 1 to 10000, default 10000.
An after cursor is resolved inside the caller's own scope, so an id they do not
own gets a 400 rather than a page, and has_more now reflects whether another
row exists instead of always being false.
Refs #37714
The websocket routes under /openai_passthrough and /openai had no e2e
coverage, so nothing catches the regression from issue #36088, where both
prefixes carried HTTP routes only and refused every upgrade with a 403
before a socket ever existed.
Two tests cover it. The realtime one opens /openai_passthrough/v1/realtime
and asserts OpenAI's own session.created frame comes back, which proves the
route is registered and relayed upstream. The responses one asserts
/openai/v1/responses accepts the upgrade, since a responses.connect socket
waits for the client to speak first and has no opening frame to check.
A refused upgrade is an HTTP response rather than a close frame, so both
assert on the handshake. ws_base_url moves into e2e_config now that a
second suite needs it
The three lite-image keys landed on the deploy branch separately while this
branch was open, so merging left every key defined twice in both price maps.
The merge is clean as text and the file still parses, but JSON keeps the last
occurrence of a repeated key, so the first copy's supported_endpoints,
supported_modalities and supports_system_messages were being dropped without
any error.
Each key is now one entry, placed next to its gemini-3.1-flash-image sibling
rather than at the end of the file.
supports_reasoning goes to false on all three, matching every other Gemini
image model. Leaving it off is not neutral: _supports_factory falls through to
the vertex_ai provider config, which answers true, and reasoning_effort then
gets forwarded to an image endpoint that rejects it. That was fixed for the
rest of the family in 75dd70a678 and these entries had drifted back.
Also fills in what the entries were missing against Google's published
pricing: the Vertex implicit cache read rate, batch rates on the Vertex
routes, and the pdf/video input flags.
The two overlapping test files are folded into one, and the price map suite
grows a duplicate-key guard so the next clean-but-lossy merge fails loudly.
* fix(logging): preserve uvicorn color_message args during secret redaction
SecretRedactionFilter clears record.args after substituting record.msg, but
uvicorn's colorized formatter re-renders the separate color_message extra
field against record.args at emit time. With args cleared, uvicorn prints
the raw "%s://%s:%d" template instead of the actual startup URL whenever
output goes to a TTY (colors on).
* fix(logging): narrow color_message fallback to TypeError
Bare except-Exception-pass on the color_message substitution pushed the
BLE001 and S110 strict-rule budgets over their ceiling. Narrow to the one
exception the %-format can actually raise and give it a real fallback
instead of silently swallowing it.
* refactor(logging): move color_message substitution into a helper
The two record.color_message stores put LIT011 over its ceiling. Building the
value in a pure helper leaves one store, marked rebind-ok since scrubbing a
record in place is the logging.Filter contract.
Also pins the ordering that makes the substitution safe: it has to run before
args are cleared, which puts it before the extra-field loop that redacts the
result, so a secret arriving through record.args is still scrubbed out of
color_message.
* fix(proxy): group Codex turns under one session id
Codex puts its conversation uuid in an unprefixed `session-id` header
(`session_id` on builds before the codex-api split), so
`get_chain_id_from_headers` never matched it: the `x-<vendor>-session-id`
regex requires an `x-` prefix. Codex also sends no request metadata the
Anthropic `metadata.user_id` path could parse and no traceparent, so every
turn fell through to a freshly generated per-call trace id and landed as its
own row in the logs.
Read the unprefixed `session-id` / `thread-id` (and the older `session_id` /
`conversation_id`) names, gated on the Codex user agent. Those names are
generic enough that an unrelated client could send one meaning something
else, and colliding values across callers would merge their traces, so the
bare-header path stays Codex-only.
* fix(proxy): match every first-party Codex originator
`is_codex_user_agent` tested `startswith("codex_")`, but the Codex TUI sends
`codex-tui` with a hyphen, and often bare with no version at all. Real values
seen in the wild are `codex-tui` and
`codex-tui/0.149.0 (Mac OS 26.5.1; arm64) ghostty/1.3.1 (codex-tui; 0.149.0)`.
codex-rs's own `is_first_party_originator` lists `codex-tui`, `codex_cli_rs`,
`codex_vscode` and a `Codex ` prefix, which agree only on the `codex` stem.
Match that stem plus a separator so no spelling is missed and an unrelated
`codexfoo` client still is. This also repairs the pre-existing gap where
`should_auto_drop_params_for_agentic_cli` (called on the request path at
litellm_pre_call_utils.py:2049) never fired for the Codex TUI.
* refactor(proxy): take headers as a read-only Mapping in the Codex session lookup
The auto-router savings figure was computed only inside the spend-update
writer, downstream of where logging callbacks consume the standard logging
payload, so Datadog-style callbacks never received it. Compute it once in
the payload builder, stamp it as a top-level payload field beside
cost_breakdown, thread it into the spend log metadata, and have both
spend-writer call sites read the recorded value with recomputation as the
fallback for rows written before the field shipped. Internal sub-calls
(classifier, shadow eval) are never stamped, and a caller-forged metadata
value is discarded by the unconditional overwrite.
Resolves LIT-5973
* fix(mcp): strip root_path before matching the per-server MCP route spelling
The 401 challenge for a gateway-managed oauth2 MCP server advertises the
protected-resource metadata URL in the spelling the client connected on, so a
strict RFC 9728 section 3 client lands on a document whose `resource` equals the
URL it actually called. That spelling test compared `_original_path` against the
root-relative `/{server}/mcp` shape, but `_original_path` and `scope["path"]`
are raw request-line paths that still carry the deployment's `root_path`
On a SERVER_ROOT_PATH deployment the prefix therefore made the legacy test fail
and every request fell through to the standard `/mcp/{server}` branch. A client
connecting on `/litellm/github/mcp` was pointed at the standard-pattern
document, which serves `resource = {base}/litellm/mcp/github`; that is not the
URL the client called, so a strict client aborts discovery before the MCP
request fires
Route the path through `get_route_relative_request_path` first, which removes
`root_path` on a segment boundary the same way
`litellm.proxy.auth.auth_utils.get_request_route` already does for the rest of
the MCP auth path, so `/litellmfoo` is not truncated under `root_path=/litellm`
* fix(mcp): make the gateway-managed 401 challenge root-path aware
The gateway-managed authorization_code challenge in process_mcp_request
built its AS-metadata URL from two root-path-unaware pieces:
- it matched the caller's spelling against `scope["_original_path"]`, a
raw request-line path that still carries the deployment prefix, so on a
SERVER_ROOT_PATH deployment the `/mcp/{server}` branch never matched and
every request fell through to the legacy one-segment form
- it hardcoded `/.well-known/oauth-authorization-server` without the
root-path segment the discovery route decorators bake in, so the URL
404'd under a sub-path deployment regardless of which branch was taken
Route the spelling match through get_route_relative_request_path and the
well-known root through well_known_root_suffix, the same two helpers the
discovery route registrations derive their paths from, so the advertised
URL cannot drift from the route that serves it.
Root-mounted deployments are unaffected: both helpers are no-ops when
SERVER_ROOT_PATH is unset.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
A ConnectionError/TimeoutError on one node of the async Redis Cluster client
made redis-py tear down every node's connections and force every other
concurrent caller through the shared reinit lock, turning one client-side
timeout under event-loop saturation into a proxy-wide latency spike while
Redis itself stayed healthy. Confirmed live against a local 3-master
cluster: pausing one node made 100% of concurrent commands to the other
two, untouched nodes stall for the full pause duration; after this change,
zero.
LiteLLMAsyncRedisCluster overrides only the ConnectionError/TimeoutError
branch of _execute_command to reset the one node that failed, mirroring
what a plain non-cluster Redis client already does when a pooled
connection errors. Every other branch (MOVED, ASK, CLUSTERDOWN,
slot-not-covered) is unchanged, since those already carry real evidence
the topology changed.
