* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
The field is declared optional on OpenAIFileObject and its own docstring says it
is absent on every upload guardrails did not touch, but the /v1/files routes have
no response_model, so FastAPI falls through to jsonable_encoder with exclude_none
off and serialises the unset default as an explicit null. Every create and
retrieve response on a proxy with no guardrails configured at all picked up a
litellm_batch_guardrail: null it never had before, and so did every row of a file
list, since those rows are the same object.
A wrap serializer drops the key only when nothing set it, so the populated report
still reaches the wire intact, including a record whose guardrail is null. The
managed-files list route spreads a stored file_object blob rather than the model,
so rows persisted before this lands keep their null until it is dropped there too.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
Per-model budgets were three separate things pretending to be one. The
enforcement check, the post-call increment and the info endpoints each derived
their own cache key, so a budget could refuse traffic at 429 while /key/info
reported zero usage, and a Bedrock model id never matched a budget keyed on the
bare family name. /user/new echoed a model_max_budget back and stored an empty
dict, and nothing enforced a user-scoped per-model budget at all.
One owner now builds the counter key from the configured budget model, and
enforcement, the increment and the info endpoints all read it. Bedrock ids
resolve through the model-cost map. Auth carries the user's budget onto the
token on every branch that reaches the spend hook, including JWT and
auto-registration. Native passthrough attaches the three budget metadata keys
its StandardLoggingUserAPIKeyMetadata does not carry, so /anthropic/... and
/bedrock/... traffic is counted and capped like /v1/chat/completions.
The dashboard gains the per-model budget editor it never had, on the key create,
key edit and internal-user edit forms. It is read-only without an enterprise
license, matching the write gate the proxy already enforces, and an untouched
budget is left out of an update so an unrelated edit cannot trip that gate.
The editor hydrates from either BudgetConfig spelling, since model_max_budget is
a plain dict that the proxy stores exactly as the client sent it, and it carries
through the fields it does not model. Without both, editing one model would drop
another model row entirely and silently discard its tpm_limit and rpm_limit.
/user/info refreshes its local copy of the user field by field after a save, so
model_max_budget joins that list. Left out, a saved cap read back as the old one
when the form was reopened, and clearing the row to recover would then wipe the
value that had actually persisted.
A zero-dollar cap is the strictest limit expressible, not the absence of one,
so it is enforced rather than skipped on falsiness, spend exactly at the cap is
refused the way every sibling budget check already refuses it, and a counter
that was never written reads as zero spend rather than as unknown. The usage
endpoints read every counter in one batched lookup, so a large model_max_budget
cannot fan out into one concurrent cache call per configured model.
Every auth path honours the same zero-cost skip flag, so none of them can refuse
a free request that another serves. The custom-auth helper gains the flag it
never had, which also changes its pre-existing key and end-user checks.
The compaction summary gate checks the user scope alongside the key and end-user
ones. This file propagates all three budgets into the summary subrequest, so
enforcing only two let compaction increment a counter it could not be refused by.
Custom auth attaches the user's budget to the token unconditionally, since the
post-call spend hook reads it there: gating the attach on the same condition as
enforcement left the counter uncharged whenever the request was not itself
enforceable. An entry that will not validate is skipped rather than raised on,
so one malformed scope cannot abort every other scope's increment or turn a
config typo into a 500.
The edit forms re-seed the budget editor when a different key or user is loaded.
Its rows are seeded once and cannot re-read their own value prop, so without
this a save wrote the previously loaded record's budgets onto the current one.
Only the built-in provider pass-through routes carry the budget metadata.
get_model_from_request deliberately resolves no model for a user-defined
pass-through, since its body is forwarded verbatim and names an upstream model,
so attaching there would charge a counter nothing on that route can refuse.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* test(lint): ban blind pytest.raises(Exception) with ruff B017
A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError
a refactor introduces satisfies it exactly as well as the rejection the test was
written for, so the crash reads as a pass and the test never goes red.
All 111 existing sites are narrowed here. A runtime probe recorded the concrete
exception each one actually catches, and each site now names that type. Where
the code under test genuinely raises a bare Exception, the site pins a stable
slice of the message with match= instead.
Two sites tell on themselves. The shared responses-API cancel test raises
"custom_llm_provider is required but passed as None" rather than talking to a
provider at all, because cancel_responses takes a provider, not a model. And
test_bedrock_guardrails_with_streaming was the only test in its file still
passing without AWS credentials, because the NoCredentialsError boto3 raised
long before the guardrail ran satisfied the blind raises.
* fix(test): widen the openai batch-dispatch assertion to OpenAIError
The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one
the SDK raises OpenAIError while building the client, long before any 404, so CI
went red. OpenAIError covers both and still rejects a TypeError from a refactor.
- accept the Live SDK's models/<id> and LiteLLM's vertex_ai/<id> when rewriting the setup model
- keep a dict service account intact instead of stringifying it
- treat same-target deployments holding different credentials as ambiguous
- guard both websocket states before every close so a second close cannot raise
- build the sendable close codes from the public CloseCode enum
The /vertex_ai/live WebSocket passthrough only ever looked at
default_vertex_config and the DEFAULT_VERTEXAI_* env vars, so a proxy whose
Vertex credentials live in the DB as a model entry with use_in_pass_through
had nothing to authenticate with. The upgrade still succeeded and the socket
then closed with a bare 1000 on the first client frame, which gave the client
no way to tell a misconfiguration from a normal end of session.
Credentials now also resolve from the router deployments flagged
use_in_pass_through, preferring the one matching the requested model, and a
failure to mint an access token closes 1011 with a reason naming both ways to
configure it. Upstream closes other than a plain 1000 are relayed to the client
with their code and reason, so Google's own errors reach the caller. The setup
frame's model is rewritten to the full projects/.../publishers/google/models
resource path, which is what Vertex expects and what lets a bare model id or a
gateway alias work over this route.
* feat(bedrock): add a config toggle to disable agent-runtime pass-through
The /bedrock pass-through dispatches agents, knowledge bases, flows, rerank,
retrieveAndGenerate, generateQuery and optimize-prompt to bedrock-agent-runtime,
so an operator who only wants to expose model invoke and converse has no way to
narrow that surface
Adds general_settings.disable_bedrock_agent_runtime_passthrough. When set, those
routes are rejected with a 403 before credentials are fetched or the request is
signed. Plain bedrock-runtime model pass-through is unaffected, and the setting
defaults to off, so existing deployments behave exactly as before
The branch is inverted to an early return for the non-agent-runtime case so the
toggle can reject outright instead of falling through to model extraction, which
would surface a confusing 400 about an unparseable model
* style(bedrock): drop redundant docstrings from the agent-runtime toggle
A model with a long time-to-first-token leaves the proxy's response completely
idle, so any hop with an idle read timeout (AWS ALB and nginx both default to
60s) drops a connection that is perfectly healthy and would have delivered its
tokens shortly after.
The keepalive engines LiteLLM already ships wrap the response object, so they
fill a gap once the upstream has answered and then gone quiet. They cannot fill
the gap before it answers at all, and that is where the whole wait is spent:
measured against api.openai.com/v1/chat/completions with gpt-5.6 at
reasoning_effort high, the response headers and the first body byte both arrive
at 37.90s. Nothing has entered the ASGI response phase by then.
The upstream call is now raced against the keepalive interval, and when it
loses, the SSE response is opened immediately and ": ping" comments, which every
conformant SSE client ignores, fill the wire until the real response is ready to
be replayed onto it. One seam per funnel: base_process_llm_request covers every
native route, create_pass_through_route covers every passthrough route.
Committing the status line that early is the cost. A failure discovered after
the first ping reaches the client as an SSE error frame under a 200 rather than
as an HTTP error status, and LiteLLM's own x-litellm-* response headers are not
yet known. keepalive_ping_has_fired already documents the same trade-off for the
existing engines. Both are why this stays off until an operator sets
litellm_settings.sse_keepalive_ping_interval_seconds.
Separately, the passthrough relay reached neither engine even for mid-stream
gaps, which is the shape of #32491 and #24929, so the relayed bytes get the same
treatment, gated on the upstream declaring text/event-stream and only emitted
between complete frames so a binary transport (AWS event streams on /bedrock)
and a stall halfway through a frame are both left alone.
Fixes#34819
Config-driven pass_through_endpoints pointed at a comprehendmedical.*.amazonaws.com
target were being claimed by the Comprehend Medical logging handler through the
hostname arm, which overrode their operator-set cost_per_request and relabeled
their spend rows. Only the built-in /comprehendmedical routes tag the provider,
so match on that alone.
Also mirror /comprehendmedical into the helm ingress and terraform gateway
prefix lists that hand-copy gateway/routes/allowlist.py
- decode upstream first frame as utf-8 instead of ascii
- reject model-restricted keys at connect to match HTTP model enforcement
- log the actual request path for /openai_passthrough traffic
* fix(proxy): track spend for OpenAI passthrough /v1/embeddings
OpenAI passthrough embeddings returned 200 but wrote no spend because the
route was unsupported and Cohere's /v1/embed prefix stole the match.
* fix(proxy): clear embeddings lint and Greptile comment nits
Inline embeddings cost tracking to avoid new LIT001/002 hits, trim
redundant doc comments, and cover the Cohere /v1/embeddings collision.
* fix(proxy): drop unreachable embeddings TypeError guard
convert_to_model_response_object with response_type=embedding already
returns EmbeddingResponse; the isinstance check was dead patch coverage.
POST /azure_ai/indexes carries no index name, so
get_azure_ai_search_index_from_endpoint returns None,
is_vector_store_index never matches any segment, and the request falls
through to the generic Azure passthrough on the proxy's own
AZURE_API_BASE and AZURE_API_KEY without ever reaching
is_allowed_to_call_vector_store_endpoint. A non-admin could therefore
create a Search index whenever AZURE_API_BASE points at the Search
service.
The earlier lifecycle commit made this look covered. Its test asserts
that POST /indexes?api-version=... is refused with "Only proxy admins can
create", but it calls the permission gate directly, and that gate is
exactly what the route skips for a path with no index name, so the guard
was verified in isolation while the route stayed open.
Gate the service-level create on the route itself, before the segment
loop, with assert_proxy_admin_for_vector_store_index_management. Scope it
to POST on a path whose last segment is indexes, mirroring the
endswith("/indexes") branch the lifecycle helper already uses, so the
managed-index paths and ordinary Azure OpenAI passthrough traffic are
untouched.
Add route-level tests: a non-admin is refused with the admin-only message
and never reaches the passthrough handler, an admin still creates, and the
new predicate is parametrized over the service-level, per-index, and
non-Search paths.
The Azure passthrough scanned every URL segment for one matching a registered
index, authorized against that, then forwarded the original path. A caller with
a grant on a managed index named e.g. "index" or "docs" could send
POST /azure_ai/indexes/{victim}/docs/index: the scan matched the trailing
segment and authorized on the caller's own index while Azure applied the batch
write to {victim} on the same Search service, enabling cross-index document
uploads or deletions.
Resolve the index positionally from the /indexes/{name} segment and require
that exact name to be the one authorized and credentialed, so the authorized
index and the physical target can never diverge. Add a pure helper plus
regression tests covering positional extraction and the route-level cross-index
attack.
PostgreSQL rejects NUL in jsonb with 22P05, and the tags go into the managed
object's CREATE payload, so one poisoned tag aborts the whole row insert rather
than just that column. With no LiteLLM_ManagedObjectTable row, CheckBatchCost
never discovers the batch, so a batch that really ran and billed at the provider
produces no spend at all. The create-time write is fire and forget, so nothing
retries it.
This regressed in #36468, which started passing request_tags and
persist_attribution from the Anthropic passthrough; before that no
caller-supplied string reached the column.
Sanitize in the shared helper that builds the value, matching how
spend_tracking_utils already handles LiteLLM_SpendLogs.request_tags. Both the
Anthropic and the Vertex passthrough build tags through that one helper, so this
covers both. Rename it to _sanitized_str_tuple since it no longer merely
coerces.
A successful pass-through request left its pre-call budget reservation in
the shared Redis spend counter. `_init_kwargs_for_pass_through_endpoint`
built the request metadata from the sanitized key fields only, so
`_PROXY_track_cost_callback` resolved `budget_reservation = None` and
`increment_spend_counters` added the actual cost on top of a reservation
nobody released. The counter drifted above real spend on every request
until the key falsely tripped BudgetExceededError, while the Postgres
spend stayed far below the limit. The failure path was unaffected because
it releases `user_api_key_dict.budget_reservation` directly.
The reservation is now set alongside the other internal keys, after the
client-supplied metadata merge, so a request body cannot forge one that
names arbitrary counter keys.
A stream can terminate with a response.failed event that still reports
consumed tokens; those were rebuilt as None and logged at zero spend.
Parse response.failed alongside completed and incomplete, matching the
buffered path, which prices any terminal response that reports usage.
The Anthropic batch create never persisted the creating key's hashed token or its
request tags on the managed object, so when CheckBatchCost billed the batch hours
later there was nothing to attribute it to. Key spend, key budgets and tag spend
never moved for batch usage.
Persist both from the create, the way the Vertex passthrough already does, and
register the batch only from the collection route. An id-scoped route cannot
rebuild the unified object id, because it embeds the model and the model comes
from the create's request body, so it could only claim a row it did not create or
fail the model_object_id unique constraint.
The shared metadata helpers, the route predicate and the registration-result
logging now live in batch_attribution instead of being copied per provider. The
Anthropic write previously logged success unconditionally, before the
fire-and-forget task had run.
Resolves LIT-5288
Streams that terminate with response.incomplete (e.g. max_output_tokens
reached) carry real usage in the terminal event but were rebuilt as None
and logged at zero spend, letting callers bypass budget enforcement.
Parse response.incomplete alongside response.completed when
reconstructing the streamed response.
Addresses review feedback: the ResponseCompletedEvent SSE parsing now lives
in OpenAIResponsesAPIConfig next to the other Responses stream event handling,
and the proxy logging handler calls it. Adds coverage for streams that end
without a response.completed event.
A poll of a Vertex passthrough batch wrote nothing to the managed-object row,
so status and file_object stayed frozen at the create-time snapshot and
GET /v1/batches served a stale status and an empty output file id for the life
of the batch. Only the create may claim a batch, but every observation of one
may refresh its state.
store_unified_object_id takes create_if_missing, which the poll clears: it
refreshes status and file_object through update_many, and leaves a row that is
absent absent rather than creating one owned by the observer, since created_by
and team_id are written by whoever reaches the create branch. The update payload
is now shared with the upsert so it cannot drift into writing api_key,
request_tags, created_by or team_id.
The passthrough identity re-assertion that was previously part of this PR ships
separately in #36121, so this PR keeps only the batch attribution work.
The creating key owns user_api_key_alias only when it actually has one. Guarding
the overwrite on the presence of a key rather than on a resolved alias nulled the
field out for every key generated without key_alias, and for any key rotated or
deleted before its batch finished, losing the creating user's alias that the spend
row previously carried. The guard now matches the team-alias line below it.
The native files and batches routes declare /{provider}/v1/... and their routers are mounted before the passthrough router, so /openai_passthrough/v1/files and /openai_passthrough/v1/batches matched them with provider="openai_passthrough" and 500'd on the LlmProviders lookup instead of reaching openai_proxy_route.
Move the dedicated /openai_passthrough prefix onto its own router mounted ahead of the batches and files routers. /openai/... and every other provider prefix keep their current behavior.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
