get_llm_provider ran in the messages handler and again inside completion,
so a provider/vendor/model id lost its vendor segment and reached upstream
bare. Pass the caller's unresolved model down the bridge instead, and move
the responses marker into the canonical provider/responses/model slot.
Reporting stays provider-local on both bridges: message_start names the id
the provider itself knows, through a shared local_model_name helper.
Fixes#37716
Adversarial review of the new multipart keying turned up collisions where
two different provider requests computed the same replay key, which is the
dangerous failure for a replay harness: the second request silently gets the
first one's response instead of missing loudly.
- a part counts as an upload when it has a filename or declares its own
content type, and the declared content type joins the identity, so two
uploads of the same bytes under the same field no longer collapse
- the uploaded parts contribute a JSON list of [field, filename, type]
triples instead of a "field:filename" string, so a separator inside a
filename can no longer impersonate a field boundary
- repeated field names get a "name[n]" suffix with a literal "[" doubled
first, so a repeated field and a literally indexed one stay distinct
- a field value that is not UTF-8 is stored as a base64 sha256 digest;
base64 rather than hex because the canonicalizer rewrites 64-character
hex runs to <sha256> and folded every binary value onto one key
- a field whose name reads as a credential is stored as <secret>. This
stays key-preserving because the key is recomputed from the stored
request rather than saved beside it, so the live request carrying the
real value still matches its redacted fixture
- the uploaded byte length leaves the key. The canonicalizer absorbs
timestamp and id drift inside a file, and that drift moves the count,
so keeping it there made re-records miss
Also stops a lookalike parameter such as "xboundary=" from being read as
the multipart boundary, and gives the OpenAI batch backend model a single
constant instead of three copies of the literal.
BUNDLE_FORMAT_VERSION goes to 3 because all of this moves recorded keys.
A bundle recorded under the old rules now fails naming both versions
instead of missing on every call.
The chunk loop read `limit + 1` rows at a time, so a small limit whose
matches sit far behind the newest rows advanced a couple of rows per
query. A `purpose` that matches only the last of 10000 owned rows at
`limit=1` cost 5001 sequential find_many calls for one HTTP request,
which any authenticated caller could ask for on purpose.
Once a scan has to continue past its first chunk, widen the chunk to
FILE_LIST_CONTINUATION_CHUNK_SIZE. That same case now costs 21 queries.
The first chunk keeps its `limit + 1` size, so a page the newest rows
already fill still costs exactly one query and reads nothing extra.
Rows whose blob will not parse drop out of a page the way a filter does,
so they get the bound too, not just the purpose filter.
The floor only changes how many round trips a page costs, never what it
returns: chunk boundaries do not affect a keyset scan, so the page is
still `matches[:page_size]`, `has_more` is still `len(matches) >
page_size`, and empty data still implies `has_more` false.
When the bounded loop cap or the repeated tool-call fingerprint guard refused a
rerun, the raise escaped the parent agentic frame and the client got the raw model
turn back: HTTP 200 carrying an unresolved tool_use block for the internal
litellm_web_search tool and stop_reason "tool_use". The client never declared that
tool, so it had no way to answer it and the conversation could not continue
The safety check now raises AgenticLoopSafetyError, a ValueError subclass, and
_call_agentic_completion_hooks catches it and returns a finalized response: the
blocks belonging to the refused tool calls are dropped, and stop_reason is closed
out to end_turn when nothing the client declared is still waiting. Refused blocks
are matched by the ids and names of the tool calls the rail refused rather than by
hardcoding the web search tool name
Only the non-streaming anthropic messages path ends the turn this way. A streaming
caller has already sent the original message by the time the hooks run, so a
finalized turn would arrive as a second message rather than replace the first, and
the responses surface carries a pydantic model this finalizer does not rewrite.
Both keep raising, exactly as they did before
Also adds max_agentic_loops to websearch_interception_params so the ceiling can be
set once for the whole feature. A per deployment litellm_params.max_agentic_loops
still wins over it, and the field stays on the proxy's untrusted root list so a
client cannot raise its own ceiling
A parenthesised query term joined to a top-level VALUES list sat behind
strip_parens, so an insert reading `VALUES (1) UNION ALL (SELECT ...)` copied a
whole table past the gate. A VALUES list now bounds an insert only while no set
operation sits beside it at that same level.
PL/pgSQL also parks dynamic SQL in a variable through a query's INTO and through
the bare `=` it takes as the assignment operator, and assigned_names read
neither, so a rewrite handed to a later EXECUTE went unseen. A bare `=` counts
only where the words ahead of it make it an assignment rather than a test.
The managed file listing cut the page to `limit` first and applied the
purpose filter in Python afterwards, so a page whose rows all failed the
filter came back as `data: []` with `has_more: true`. openai-python stops
paging the moment `data` is empty, so `files.list(purpose="batch", limit=1)`
returned nothing at all instead of every batch file.
Read successive keyset chunks until the page holds `limit + 1` matches or
the caller's rows run out, then return at most `limit` of them. `data` is
now non-empty whenever matching files remain, its last id is always a
usable cursor, and `has_more: false` only ever means the caller has seen
everything. Rows whose stored blob will not parse drop out in the same
loop, so they cannot empty a page either.
That also makes the `next_cursor_id` escape hatch on `build_list_page`
dead, so it goes back to what it was for the batch and vector-store
listings that share it.
Also move `validate_file_list_limit` up into the list_files route, so the
target_model_names and provider branches reject an out-of-range limit the
same way the managed file store already did.
Postgres takes the row source parenthesised, so `INSERT INTO "t" ("a") (SELECT
...)` copies a whole table at boot. Reading only the unparenthesised text let it
through: 777eb8af10 caught it, then e7dea842c3 traded it away to stop a VALUES
list joined to a query by a set operation from bounding nothing.
Read the top level first so set operations still count, then fall back to the
whole statement when no top-level VALUES bounds the insert. `TABLE t` is a row
source as much as a `SELECT` is, and it was passing too
Chat completions, embeddings, the non-streaming /v1/messages tests, and the
OpenAI batch deployment now register through the provider edge, so
E2E_FIXTURE_MODE=record captures their provider calls and replay serves them
back offline. None of them was wired before, so record was a silent no-op over
these suites and replay quietly went live instead of using the bundle
Multipart uploads now key on their parsed parts: every ordinary form field,
plus the field name, filename, content digest, and length of each file part.
The boundary is envelope rather than content, so it stays out of the digest
instead of changing the key on every run. A body that does not parse as its
declared envelope still has the boundary normalized away before hashing, so
the fallback is at least stable, and it records a name that says why
Binary uploads hash byte for byte. Canonicalizing them first meant decoding
with errors="replace", which collapsed every invalid byte to one U+FFFD and
gave two different PDFs of the same length the same key
Bundles stay out of the repo: they hold verbatim provider response bodies and
expire seven days after recording. Publishing them for CI is LIT-5748, and
streaming fidelity is LIT-5742
Four more ruff rules for code the test suite runs but never checks. F601 is the
one that paid: the duplicate key it flagged in a get_form_data fixture was the
mock reproducing the production bug fixed in the previous commit.
B025 removed two unreachable handlers, one of them a pytest.skip shadowed by an
earlier `pass`, so an upstream Vertex flake reported green having asserted
nothing. F632 turned an `is ""` identity check, which passes only on CPython
interning, into the `== ""` it meant. B023 fixed three closures over loop
variables, all latent today but one iteration-order change away from checking the
last case N times.
get_form_data collapsed the FormData multidict with dict(form) before the loop
that rebuilds `foo[]` arrays ever ran, so a request sending
timestamp_granularities[]=word and timestamp_granularities[]=segment reached the
provider as ["segment"] with the first value silently dropped. Read the multidict
with multi_items() instead.
The test could not catch it because its mock was a plain dict carrying the same
key twice, which Python collapses exactly the way the bug did. Every request.form
mock that fed get_form_data now returns real FormData.
* fix(pricing): add undated azure aliases for gpt-audio-mini and gpt-realtime-mini
Azure deployments are commonly created against the undated model name,
and the cost-tracking docs say to set base_model to azure/<model> — but
only the dated -2025-10-06 entries existed for these two models (the
openai provider has undated aliases for both). base_model:
azure/gpt-audio-mini therefore resolved to nothing and, depending on the
fallback path, text tokens billed at $0 while audio tokens billed fine.
Mirror the -2025-10-06 entries as undated aliases, exactly like the
undated openai entries mirror their newest dated variant.
Fixes#33170
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(pricing): assert undated azure audio aliases exactly mirror their dated entries
Review follow-up: COST_FIELDS missed realtime-specific cost keys
(cache_creation_input_audio_token_cost, cache_read_input_token_cost,
input_cost_per_image). Full-entry equality catches drift on every field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pricing): mirror updated mode=realtime on the undated gpt-realtime-mini alias
Upstream changed the dated entry's mode from chat to realtime after this
branch was cut; the undated alias must stay a byte-for-byte mirror.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pricing): mirror the new deprecation_date onto the undated gpt-audio-mini alias
* test(pricing): use shared local_model_cost_map fixture so get_model_info's lru_cache never crosses maps
---------
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A dollar-quoted payload is read as its own region rather than as a handed-off
string, so the marker belongs on the rewrite inside it. Pin that placement, and
pin that a marker on a DO block header never covers the block's body.
* fix(anthropic_messages): gate sampling params on /v1/messages like /chat/completions
/v1/messages forwarded temperature/top_p/top_k raw to models that removed
sampling params (supports_sampling_params: false — Claude 4.7+/Fable 5),
producing provider 400s that router fallbacks mask as silent model
downgrades. The chat path already gates these via
AnthropicModelInfo._apply_sampling_param; reuse it in
get_requested_anthropic_messages_optional_param so both endpoints agree:
drop under drop_params, else raise the clean client-side 400.
Fixes#35053
* test(anthropic): drive new sampling-param tests off the kwarg, not the global
The five tests added here set `litellm.drop_params = True` under a manual
try/finally. That trips TQ005 (module-global mutation, 10 new violations
over the ceiling) and it leaks process-wide if the finally is ever
skipped, which is what the save/restore conftest exists to paper over.
`get_requested_anthropic_messages_optional_param` already takes
`drop_params` as a kwarg, and that is the path /v1/messages actually
uses, so pass it directly. `monkeypatch.setattr` pins the global to
False so each test proves the per-request flag alone is sufficient and
cannot pass on a leaked global.
Verified: TQ gate clean, all 10 tests pass, and the 3 that assert the
new gating still fail with the fix in utils.py reverted.
---------
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
The unscoped GET /v1/files limit check accepted 0, which OpenAI's minimum
of 1 does not allow, and the route's except block rebuilt every error with
getattr(e, "status_code", 500). ProxyException has no status_code, so the
400 it raises went out as a 500 and the OpenAI SDK retried it three times.
Errors now go through handle_exception_on_proxy, the helper the sibling
batches route already uses, and the unknown-cursor error is a ProxyException
so it carries type invalid_request_error and param after instead of the
literal "None". The cursor still 400s whether the file belongs to someone
else or does not exist at all
EXPLAIN ANALYZE runs the statement it wraps rather than only planning it, but
ANALYZE sits in the keyword set, so it stood in for the keyword underneath and a
rewrite left under one reached boot unflagged.
* fix(mcp): resolve admin OAuth sessions to the same server set the connect page shows
* fix(mcp): bind admitted admin rows through the entitlement ceiling, not the credential scope clause
DO takes its body as a string literal, and dollar quoting is a convenience
rather than a requirement. A migration spelling the body in single quotes
got its rewrite through untouched, since nothing was reading that literal
as SQL. It is ordinary syntax rather than an attempt to hide anything, so
the miss was reachable by accident.
The module docstring now also records where concatenated dynamic SQL stops
being readable, which is a keyword split across fragments that do not hold
it. Every fragment is scanned, so the shapes people actually write are all
still caught.
add_provider_specific_headers_to_request tagged the client's Authorization header
with the same provider list as anthropic-beta and anthropic-version, so an
sk-ant-oat subscription token was sent to AWS Bedrock and Google Vertex AI as
well. On Bedrock it replaced the SigV4 signature, or the deployment's own API
key, and AWS answered 403 "Invalid API Key format". On Vertex it went out as a
second Authorization header next to the Google one and Google answered 401
ACCESS_TOKEN_TYPE_UNSUPPORTED.
The credential and those API headers need different scopes, so a request can now
carry more than one ProviderSpecificHeader entry. The API headers keep the
provider list they already had and the credential gets its own entry scoped to
anthropic alone. get_provider_specific_headers takes either a single entry or a
sequence and merges only the entries whose provider list matches, so callers that
pass one entry keep working unchanged.
Bedrock SigV4 signing and the deliberate extra_headers Authorization pass-through
in _sign_request are left alone.
The new test set LITELLM_LOCAL_MODEL_COST_MAP and reassigned litellm.model_cost
by hand, leaking both into every test that ran after it and skipping the
get_model_info cache clear. The conftest fixture already does this properly and
restores the original map on the way out.
A marker on an EXECUTE now covers the SQL that EXECUTE runs, so it goes
where the migration reads rather than inside the string. A literal whose
first line sat below its EXECUTE was missing the marker entirely, and the
documented placement failed CI.
A literal assigned with := counts as SQL only when an EXECUTE in the same
body runs that variable by name. An error message naming a DELETE the
application handles is text, and the only way to silence it before was a
marker claiming a bounded data migration that was not there at all.
A page whose rows are all dropped by the purpose filter, or by a row
that does not parse, used to come back with an empty data list, has_more
true and last_id null, so the caller had no cursor to advance with and
stopped one page short of files it owns. last_id now falls back to the
last row the page read.
Also drops the OpenAIFilesPurpose import that the widened purpose
annotation left unused.
The docstring claimed azure pricing mirrors the openai family, which stopped
being true when gpt-5.6-sol took its promotional cut and azure did not. Azure
publishes no sol rate of its own today, so the entries stay where they are.
OpenAI's model page for gpt-5.6 serves the GPT-5.6 Sol page and states
that the gpt-5.6 alias routes requests to GPT-5.6 Sol, so the alias bills
at Sol's rates. The registry entry was left on the pre-cut rates while
gpt-5.6-sol took the cut, overbilling gpt-5.6 callers by 25 percent on
input and 50 percent on output.
All 23 cost fields on gpt-5.6 now match gpt-5.6-sol, and a regression
test pins the two entries together so they cannot drift again.
The owner-scoped listing read every row the caller owns in one query, so an
admin key that owns every file on the proxy pulled the whole table into one
response. Page it with a keyset cursor on unified_file_id instead, and accept
limit and after on GET /v1/files so a client can walk the pages. limit follows
what OpenAI documents for that route: 1 to 10000, default 10000.
An after cursor is resolved inside the caller's own scope, so an id they do not
own gets a 400 rather than a page, and has_more now reflects whether another
row exists instead of always being false.
Refs #37714
The websocket routes under /openai_passthrough and /openai had no e2e
coverage, so nothing catches the regression from issue #36088, where both
prefixes carried HTTP routes only and refused every upgrade with a 403
before a socket ever existed.
Two tests cover it. The realtime one opens /openai_passthrough/v1/realtime
and asserts OpenAI's own session.created frame comes back, which proves the
route is registered and relayed upstream. The responses one asserts
/openai/v1/responses accepts the upgrade, since a responses.connect socket
waits for the client to speak first and has no opening frame to check.
A refused upgrade is an HTTP response rather than a close frame, so both
assert on the handshake. ws_base_url moves into e2e_config now that a
second suite needs it
The three lite-image keys landed on the deploy branch separately while this
branch was open, so merging left every key defined twice in both price maps.
The merge is clean as text and the file still parses, but JSON keeps the last
occurrence of a repeated key, so the first copy's supported_endpoints,
supported_modalities and supports_system_messages were being dropped without
any error.
Each key is now one entry, placed next to its gemini-3.1-flash-image sibling
rather than at the end of the file.
supports_reasoning goes to false on all three, matching every other Gemini
image model. Leaving it off is not neutral: _supports_factory falls through to
the vertex_ai provider config, which answers true, and reasoning_effort then
gets forwarded to an image endpoint that rejects it. That was fixed for the
rest of the family in 75dd70a678 and these entries had drifted back.
Also fills in what the entries were missing against Google's published
pricing: the Vertex implicit cache read rate, batch rates on the Vertex
routes, and the pdf/video input flags.
The two overlapping test files are folded into one, and the price map suite
grows a duplicate-key guard so the next clean-but-lossy merge fails loudly.
Routing rerank through get_request_headers also picked up its
AWS_BEARER_TOKEN_BEDROCK branch. Bedrock API keys are only valid for
Bedrock and Bedrock Runtime actions, not for Agents for Amazon Bedrock
Runtime ones, and rerank is served by bedrock-agent-runtime, so AWS
rejects a bearer-signed rerank call. Opt the rerank handler out of the
bearer path so it keeps signing with SigV4.
scan() recursed into a dollar-quoted body with the sliced text but kept absolute
offsets, so line_of counted newlines in the slice against a position past its end.
Any DO $$ block below the first line reported a wrong line, which also misaligned
the -- data-migration-ok: markers: an unrelated marker earlier in the file could
exempt a rewrite inside a block, and a marker sitting right above one failed to.
Line numbers now always count against the whole migration text.
EXECUTE was treated as harmless while its quoted SQL was masked, so a rewrite
handed over as a string walked through the gate. The literal an EXECUTE runs is
now scanned like a dollar-quoted body.
INSERT was classified by searching the whole statement for SELECT, so a bounded
INSERT ... VALUES holding a scalar subquery, or led by a helper CTE, was flagged
as INSERT ... SELECT. A top-level VALUES now bounds the insert, and a VALUES
buried in a subquery still does not.
* fix(logging): preserve uvicorn color_message args during secret redaction
SecretRedactionFilter clears record.args after substituting record.msg, but
uvicorn's colorized formatter re-renders the separate color_message extra
field against record.args at emit time. With args cleared, uvicorn prints
the raw "%s://%s:%d" template instead of the actual startup URL whenever
output goes to a TTY (colors on).
* fix(logging): narrow color_message fallback to TypeError
Bare except-Exception-pass on the color_message substitution pushed the
BLE001 and S110 strict-rule budgets over their ceiling. Narrow to the one
exception the %-format can actually raise and give it a real fallback
instead of silently swallowing it.
* refactor(logging): move color_message substitution into a helper
The two record.color_message stores put LIT011 over its ceiling. Building the
value in a pure helper leaves one store, marked rebind-ok since scrubbing a
record in place is the logging.Filter contract.
Also pins the ordering that makes the substitution safe: it has to run before
args are cleared, which puts it before the extra-field loop that redacts the
result, so a secret arriving through record.args is still scrubbed out of
color_message.
