On mapped pass-through routes, of which /vertex_ai is one,
user_api_key_auth accepts the caller key from a header literally named
litellm_user_api_key and applies it last, so it overrides every other source.
The credential-less filter neither dropped it nor resolved the caller key from
it, so a virtual key there reached Google past a real x-goog-api-key, and a
bring-your-own Authorization could be stripped when auth actually came from that
header. Drop it by name and resolve it at highest precedence.
The resolver placed both operator-configured key headers at the top of its
precedence, but user_api_key_auth only overrides with litellm_key_header_name;
a pass_through_endpoints litellm_user_api_key is checked last. So a request that
authenticated via Authorization while also sending a pass-through header could
have the wrong value chosen, leaving the authenticated Authorization key
forwarded. Order the resolver exactly like get_api_key: override first, built-in
headers next, pass-through header last.
user_api_key_auth also accepts the caller key from a pass_through_endpoints
entry's headers.litellm_user_api_key, not just litellm_key_header_name. Drop
every operator-configured caller-key header by name and treat them as
top-precedence caller-key sources, so a virtual key sent through one is never
forwarded to Google.
The LIT-4761 streaming-classification tests passed only the bring-your-own
Google OAuth token in Authorization and mocked get_litellm_virtual_key, a shape
that cannot authenticate in production. The credential-less filter now resolves
the caller key by auth precedence, so a lone Authorization value reads as the
key and is stripped. Send the virtual key in x-litellm-api-key, matching a real
request, so Authorization is preserved and the classification assertions run.
The filter's own Bearer-only stripping missed the other schemes
user_api_key_auth accepts, so a virtual key echoed as `Authorization: Basic
<key>` alongside a higher-precedence auth header did not match the caller key
and was forwarded to Google. Reuse the auth module's _get_bearer_token so the
comparison strips exactly what authentication does (Bearer / bearer / Basic /
AWS4-HMAC-SHA256), falling back to the raw value for a bare token.
The credential-less filter derived the caller key only from x-litellm-api-key,
Authorization, and the custom header, but the route authenticates through
Depends(user_api_key_auth), which also accepts the key from x-goog-api-key. A
virtual key sent only in x-goog-api-key therefore authenticated yet was kept as
a preserved upstream header and forwarded to Google. Resolve the caller key by
the same precedence get_api_key uses and value-strip exactly that, so a key in
x-goog-api-key is stripped while a real Google key alongside a higher-precedence
virtual key is preserved.
The hand-rolled drop set missed Ocp-Apim-Subscription-Key, so a caller
Azure APIM secret in that header was forwarded to Google on the
credential-less branch. Derive the name-drop set from the canonical
SpecialHeaders.litellm_credential_header_names(), minus Authorization and
x-goog-api-key which double as real Google credentials and are value-stripped
instead. New credential headers added there are now dropped automatically.
The router hop _ageneric_api_call_with_fallbacks canonicalises the passthrough
call type onto litellm_metadata, and the cost callback reads spend attribution
from that bucket while only backfilling user_api_key* keys from metadata. The
helper was building on metadata, so agent_id and user_api_end_user_max_budget
were silently dropped before the callback ever saw them. Build and pass the
attribution under litellm_metadata so every field survives.
user_api_key_auth also authenticates a caller from the operator-configured
general_settings.litellm_key_header_name, reading that header straight off
the request, so a virtual key sent there survived the credential-less Vertex
forwarding filter and reached Google alongside a real bring-your-own
credential. Value-strip every header whose value matches the caller's key
from any accepted source, including that custom header.
On the credential-less Vertex passthrough branch, drop every header that
can only carry LiteLLM caller auth (x-litellm-api-key, api-key, x-api-key)
by name, since Google never consumes them, and strip the virtual key by
value from Authorization / x-goog-api-key, which may instead hold a genuine
bring-your-own Google credential. This closes the residual leak where a
distinct caller secret in api-key or x-api-key still reached upstream.
Adds a regression asserting the value-based strip also drops the caller's
virtual key when it is duplicated into the api-key and x-api-key headers,
while a genuine bring-your-own Google credential still forwards.
The credential-less Vertex passthrough dropped the caller's LiteLLM
virtual key only from Authorization by exact match. A caller who sent
the same key in x-goog-api-key (which doubles as a real Google
credential) had it accepted as a credential and forwarded upstream.
Drop the virtual key by value across every forwarded header, normalizing
any Bearer prefix, so no header name carries it to Google.
When no Vertex credential is configured (no default_vertex_config, no matching
use_in_pass_through deployment, no vector-store credential), the Vertex passthrough
took the bring-your-own-credentials branch and forwarded the entire incoming header
set upstream to Google. That set included whichever header carried the caller's
LiteLLM virtual key: x-litellm-api-key, or Authorization when get_litellm_virtual_key
read the key from there. The proxy's own secret was sent to a third-party provider.
The credential-less branch now drops x-litellm-api-key and the Authorization value
that equals the virtual key, keeping a genuine bring-your-own Google credential
(an OAuth token in Authorization, or x-goog-api-key) so real BYO passthrough still
works. When neither survives, the request fails with a clean 401 telling the operator
no credential is configured, instead of forwarding the virtual key.
Regression coverage in the mapped test path asserts the 401-and-never-forwarded
behavior for both leak vectors and that a real Google credential still passes through
with the virtual key stripped.
The /vllm and /azure router-model passthrough branches called
llm_router.allm_passthrough_route directly with no request metadata,
so the cost callback saw no user_api_key and no
user_api_key_budget_reservation. Spend for a budgeted virtual key hit
neither the key's spend nor the spend logs, and the reservation minted
at auth into the shared Redis counter was never released, drifting the
counter up until the key falsely tripped BudgetExceededError.
Thread the authenticated key's attribution metadata into both calls via
the same builder add_litellm_data_to_request uses, so the cost callback
attributes spend and reconciles the reservation. Regression tests cover
both branches.
POST /v1/videos without an input_reference file now goes out as
multipart/form-data the way the OpenAI SDK always sends it, instead of a
JSON body that OpenAI-compatible backends (SGLang Diffusion, vLLM-Omni)
reject; gemini, vertex, and runwayml keep their JSON bodies
/v1/images/edits on the openai/azure/openai-compatible path now forwards
unknown provider params (e.g. seed) and honors extra_body, matching
/v1/images/generations, and aimage_edit forwards
extra_headers/extra_query/extra_body instead of dropping them
Generic pass-through no longer downgrades a file-less multipart form to
application/x-www-form-urlencoded
* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
The field is declared optional on OpenAIFileObject and its own docstring says it
is absent on every upload guardrails did not touch, but the /v1/files routes have
no response_model, so FastAPI falls through to jsonable_encoder with exclude_none
off and serialises the unset default as an explicit null. Every create and
retrieve response on a proxy with no guardrails configured at all picked up a
litellm_batch_guardrail: null it never had before, and so did every row of a file
list, since those rows are the same object.
A wrap serializer drops the key only when nothing set it, so the populated report
still reaches the wire intact, including a record whose guardrail is null. The
managed-files list route spreads a stored file_object blob rather than the model,
so rows persisted before this lands keep their null until it is dropped there too.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
get_form_data collapsed the FormData multidict with dict(form) before the loop
that rebuilds `foo[]` arrays ever ran, so a request sending
timestamp_granularities[]=word and timestamp_granularities[]=segment reached the
provider as ["segment"] with the first value silently dropped. Read the multidict
with multi_items() instead.
The test could not catch it because its mock was a plain dict carrying the same
key twice, which Python collapses exactly the way the bug did. Every request.form
mock that fed get_form_data now returns real FormData.
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
Per-model budgets were three separate things pretending to be one. The
enforcement check, the post-call increment and the info endpoints each derived
their own cache key, so a budget could refuse traffic at 429 while /key/info
reported zero usage, and a Bedrock model id never matched a budget keyed on the
bare family name. /user/new echoed a model_max_budget back and stored an empty
dict, and nothing enforced a user-scoped per-model budget at all.
One owner now builds the counter key from the configured budget model, and
enforcement, the increment and the info endpoints all read it. Bedrock ids
resolve through the model-cost map. Auth carries the user's budget onto the
token on every branch that reaches the spend hook, including JWT and
auto-registration. Native passthrough attaches the three budget metadata keys
its StandardLoggingUserAPIKeyMetadata does not carry, so /anthropic/... and
/bedrock/... traffic is counted and capped like /v1/chat/completions.
The dashboard gains the per-model budget editor it never had, on the key create,
key edit and internal-user edit forms. It is read-only without an enterprise
license, matching the write gate the proxy already enforces, and an untouched
budget is left out of an update so an unrelated edit cannot trip that gate.
The editor hydrates from either BudgetConfig spelling, since model_max_budget is
a plain dict that the proxy stores exactly as the client sent it, and it carries
through the fields it does not model. Without both, editing one model would drop
another model row entirely and silently discard its tpm_limit and rpm_limit.
/user/info refreshes its local copy of the user field by field after a save, so
model_max_budget joins that list. Left out, a saved cap read back as the old one
when the form was reopened, and clearing the row to recover would then wipe the
value that had actually persisted.
A zero-dollar cap is the strictest limit expressible, not the absence of one,
so it is enforced rather than skipped on falsiness, spend exactly at the cap is
refused the way every sibling budget check already refuses it, and a counter
that was never written reads as zero spend rather than as unknown. The usage
endpoints read every counter in one batched lookup, so a large model_max_budget
cannot fan out into one concurrent cache call per configured model.
Every auth path honours the same zero-cost skip flag, so none of them can refuse
a free request that another serves. The custom-auth helper gains the flag it
never had, which also changes its pre-existing key and end-user checks.
The compaction summary gate checks the user scope alongside the key and end-user
ones. This file propagates all three budgets into the summary subrequest, so
enforcing only two let compaction increment a counter it could not be refused by.
Custom auth attaches the user's budget to the token unconditionally, since the
post-call spend hook reads it there: gating the attach on the same condition as
enforcement left the counter uncharged whenever the request was not itself
enforceable. An entry that will not validate is skipped rather than raised on,
so one malformed scope cannot abort every other scope's increment or turn a
config typo into a 500.
The edit forms re-seed the budget editor when a different key or user is loaded.
Its rows are seeded once and cannot re-read their own value prop, so without
this a save wrote the previously loaded record's budgets onto the current one.
Only the built-in provider pass-through routes carry the budget metadata.
get_model_from_request deliberately resolves no model for a user-defined
pass-through, since its body is forwarded verbatim and names an upstream model,
so attaching there would charge a counter nothing on that route can refuse.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* test(lint): ban blind pytest.raises(Exception) with ruff B017
A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError
a refactor introduces satisfies it exactly as well as the rejection the test was
written for, so the crash reads as a pass and the test never goes red.
All 111 existing sites are narrowed here. A runtime probe recorded the concrete
exception each one actually catches, and each site now names that type. Where
the code under test genuinely raises a bare Exception, the site pins a stable
slice of the message with match= instead.
Two sites tell on themselves. The shared responses-API cancel test raises
"custom_llm_provider is required but passed as None" rather than talking to a
provider at all, because cancel_responses takes a provider, not a model. And
test_bedrock_guardrails_with_streaming was the only test in its file still
passing without AWS credentials, because the NoCredentialsError boto3 raised
long before the guardrail ran satisfied the blind raises.
* fix(test): widen the openai batch-dispatch assertion to OpenAIError
The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one
the SDK raises OpenAIError while building the client, long before any 404, so CI
went red. OpenAIError covers both and still rejects a TypeError from a refactor.
- accept the Live SDK's models/<id> and LiteLLM's vertex_ai/<id> when rewriting the setup model
- keep a dict service account intact instead of stringifying it
- treat same-target deployments holding different credentials as ambiguous
- guard both websocket states before every close so a second close cannot raise
- build the sendable close codes from the public CloseCode enum
The /vertex_ai/live WebSocket passthrough only ever looked at
default_vertex_config and the DEFAULT_VERTEXAI_* env vars, so a proxy whose
Vertex credentials live in the DB as a model entry with use_in_pass_through
had nothing to authenticate with. The upgrade still succeeded and the socket
then closed with a bare 1000 on the first client frame, which gave the client
no way to tell a misconfiguration from a normal end of session.
Credentials now also resolve from the router deployments flagged
use_in_pass_through, preferring the one matching the requested model, and a
failure to mint an access token closes 1011 with a reason naming both ways to
configure it. Upstream closes other than a plain 1000 are relayed to the client
with their code and reason, so Google's own errors reach the caller. The setup
frame's model is rewritten to the full projects/.../publishers/google/models
resource path, which is what Vertex expects and what lets a bare model id or a
gateway alias work over this route.
* feat(bedrock): add a config toggle to disable agent-runtime pass-through
The /bedrock pass-through dispatches agents, knowledge bases, flows, rerank,
retrieveAndGenerate, generateQuery and optimize-prompt to bedrock-agent-runtime,
so an operator who only wants to expose model invoke and converse has no way to
narrow that surface
Adds general_settings.disable_bedrock_agent_runtime_passthrough. When set, those
routes are rejected with a 403 before credentials are fetched or the request is
signed. Plain bedrock-runtime model pass-through is unaffected, and the setting
defaults to off, so existing deployments behave exactly as before
The branch is inverted to an early return for the non-agent-runtime case so the
toggle can reject outright instead of falling through to model extraction, which
would surface a confusing 400 about an unparseable model
* style(bedrock): drop redundant docstrings from the agent-runtime toggle
A model with a long time-to-first-token leaves the proxy's response completely
idle, so any hop with an idle read timeout (AWS ALB and nginx both default to
60s) drops a connection that is perfectly healthy and would have delivered its
tokens shortly after.
The keepalive engines LiteLLM already ships wrap the response object, so they
fill a gap once the upstream has answered and then gone quiet. They cannot fill
the gap before it answers at all, and that is where the whole wait is spent:
measured against api.openai.com/v1/chat/completions with gpt-5.6 at
reasoning_effort high, the response headers and the first body byte both arrive
at 37.90s. Nothing has entered the ASGI response phase by then.
The upstream call is now raced against the keepalive interval, and when it
loses, the SSE response is opened immediately and ": ping" comments, which every
conformant SSE client ignores, fill the wire until the real response is ready to
be replayed onto it. One seam per funnel: base_process_llm_request covers every
native route, create_pass_through_route covers every passthrough route.
Committing the status line that early is the cost. A failure discovered after
the first ping reaches the client as an SSE error frame under a 200 rather than
as an HTTP error status, and LiteLLM's own x-litellm-* response headers are not
yet known. keepalive_ping_has_fired already documents the same trade-off for the
existing engines. Both are why this stays off until an operator sets
litellm_settings.sse_keepalive_ping_interval_seconds.
Separately, the passthrough relay reached neither engine even for mid-stream
gaps, which is the shape of #32491 and #24929, so the relayed bytes get the same
treatment, gated on the upstream declaring text/event-stream and only emitted
between complete frames so a binary transport (AWS event streams on /bedrock)
and a stall halfway through a frame are both left alone.
Fixes#34819
Config-driven pass_through_endpoints pointed at a comprehendmedical.*.amazonaws.com
target were being claimed by the Comprehend Medical logging handler through the
hostname arm, which overrode their operator-set cost_per_request and relabeled
their spend rows. Only the built-in /comprehendmedical routes tag the provider,
so match on that alone.
Also mirror /comprehendmedical into the helm ingress and terraform gateway
prefix lists that hand-copy gateway/routes/allowlist.py
- decode upstream first frame as utf-8 instead of ascii
- reject model-restricted keys at connect to match HTTP model enforcement
- log the actual request path for /openai_passthrough traffic
