The two vision tests pointed at a Wikipedia-hosted cat photo, so every run
depended on upload.wikimedia.org staying up and unthrottled. It throttled,
and the 429 surfaced as a bedrock APIConnectionError, which reads as a
gateway failure rather than what it was.
The image is now a fixture in the repo, passed as a data URL. That also puts
the two providers on the same bytes: litellm downloads the image itself for
bedrock, while openai is handed the link and fetches it from its own servers,
so the hosted URL quietly meant the two tests were not testing the same thing.
The image was generated for this repo rather than borrowed, so nothing here
carries a third-party license. Also drops a stale comment about openai prompt
caching that sat above the vision helper; no caching test uses it.
The /v1/messages validator checked a tool_use block's name and id but not its
input, so a block whose location came back empty or wrong still passed, while
the chat side rejected the same damage. That gap predates this branch; it is
worth closing here because the point of the change is that every parallel call
is checked rather than counted.
AnthropicContentBlock now declares input as a typed field. It already survived
on extra="allow", but reaching it from a test needs a real field to keep the
e2e basedpyright gate at zero. Serialization is unchanged: bodies are dumped
with exclude_none, so a block without an input still replays exactly as before.
The together backend is picked as the cheapest chat row that supports both
tools and reasoning, which currently resolves to together_ai/openai/gpt-oss-120b.
That row is marked supports_parallel_function_calling, so one weather prompt
can legitimately come back as several get_weather calls. Both tool tests
asserted exactly one call, so a parallel answer failed them even though the
gateway handled it correctly.
They now check every returned call instead of counting them: each one has to
be a get_weather naming Paris, with an id a tool result can answer. Dropping,
misnaming, or mangling a call is still red; only the count is the model's
business. The round trips answer every call rather than just the first, which
is also what the Anthropic Messages spec asks for.
The shared SelectContent wrapper defaulted alignItemWithTrigger to true,
which puts Base UI's positioner into item-aligned mode and places the
popup so the active item sits on top of the trigger. In that mode the
side and sideOffset the wrapper passes two lines above are ignored, and
the popup reports data-side="none".
The overlap only becomes visible once the items are tall enough to
matter, which is why the autorouter Template picker shows it clearly:
its options are three-line cards, so the popup covers both the select
box and its own label.
No call site in the dashboard asked for item-aligned mode. 21 of them
across 15 files already passed alignItemWithTrigger={false} by hand to
undo the default, and the remaining 127 inherited the bug. Flipping the
default makes side and sideOffset live, so collision handling works and
a select with no room below now flips above the trigger rather than
covering it. The 21 hand-written opt-outs are deleted as redundant.
`test_gemini_chat_returns_content_and_logs_cost` asks gemini-2.5-flash to
"reply with the single word pong" under `max_tokens=32`, and has been seen
returning no content at all:
completion_tokens=29, reasoning_tokens=29, content=None
gemini-2.5-flash defaults to dynamic thinking, and `max_tokens` maps to
`maxOutputTokens`, which on the 2.5 family counts thinking tokens as well as
visible output. So the model is free to spend the entire budget on thoughts and
emit nothing, which is exactly what the usage above shows.
Raising the limit alone does not fix this. Dynamic thinking on 2.5 Flash is
documented up to 24576 tokens, so no budget small enough to be reasonable for a
one-word smoke test is safe. The fix is to take thinking out of the picture:
`reasoning_effort="none"` maps to `thinkingConfig.thinkingBudget=0` for the 2.5
family, so the whole limit is available to visible output. Verified against this
checkout:
get_optional_params(model="gemini-2.5-flash", custom_llm_provider="gemini",
max_tokens=32)
-> {'max_output_tokens': 32} # no thinkingConfig at all
get_optional_params(model="gemini-2.5-flash", custom_llm_provider="gemini",
max_tokens=64, reasoning_effort="none")
-> {'max_output_tokens': 64,
'thinkingConfig': {'thinkingBudget': 0, 'includeThoughts': False}}
This mirrors what the OpenAI tool tests in this same file already do with
gpt-5.6 for the same failure mode. `max_tokens` goes to 64 for headroom; with
thinking disabled that is ample for a one-word answer.
Neither `covers` claim changes: the call still exercises the gemini chat
translation path and still produces a costed SpendLogs row.
`_cacheable_system_block` embedded the per-run marker in all 300 paragraphs, so
the block's token count moved with the marker's own tokenization. Measured over
40 random markers the size ranged 3611-5408 tokens (median 4509): 15% of runs
landed under the 4096-token minimum cacheable prefix of Haiku 4.5, despite the
docstring claiming the prompt was comfortably above it.
When the system block is under the minimum, no cache entry is written at the
system breakpoint. The entry at the second breakpoint still gets written,
because system + first user turn clears the minimum -- which is why the failures
report a large cache_creation with cache_read stuck at 0
(`cache_creation_input_tokens=5610 cache_read_input_tokens=0`, and 5610 is the
whole prefix, not the user turn's share). `_prime_prompt_cache` rotates the user
turn on every attempt, so that second entry never prefix-matches the next
attempt either. Every attempt re-creates the full prefix, cache_read never rises
above 0, and the loop burns its 60s deadline:
prompt cache never became readable in full within 60.0s
That is the single most frequent flake in the e2e suite, 9 of 38 runs, and it
hits all three provider classes identically because they share this helper.
Move the marker out of the repeated paragraph so it appears once, and size the
block at 1500 paragraphs. The prefix is now 8056-8060 tokens across markers --
spread 4 tokens instead of 1797, and 1.97x the minimum in the worst case. The
same marker-per-repetition pattern in `_first_turn_user_text` is fixed the same
way. Both copies of the helpers stay byte-identical.
The test claiming mgmt.key.generate.happy_path signed in and then only read
/key/list, so nothing proved the session key an admin's sign-in mints is
actually accepted on /key/generate. It now does what an admin filling in
Create New Key does: POST /key/generate under the session key, read the new
key back from /key/info, see it in the dashboard's own /key/list, and drive
real traffic through it to confirm its model scope is enforced.
Adds ManagementClient.generate_key with the same caller_key seam update_key
and key_list already use, so the suite can call the route as the master key
or as a virtual key. Also wraps the over-long models import.
Refusing the dashboard session key on /key/generate turns only this test red;
the master-key generate, the key edit, and regenerate stay green.
Google withdrew gemini-live-2.5-flash-preview-native-audio-09-2025 from the
Vertex Live API. Every session dies at setup:
received 1007 (invalid frame payload data)
gemini-live-2.5-flash-preview-native-audio-09-2025 is not supported in the live api.
The client sees session.created (the proxy synthesizes it on connect) and then
nothing, so both vertex_ai realtime tests time out waiting for session.updated.
Confirmed by probing the Vertex Live endpoint directly with the e2e stack's own
credentials:
gemini-live-2.5-flash-preview-native-audio-09-2025 -> 1007, not supported
gemini-live-2.5-flash-native-audio -> setupComplete
so this swaps to the non-preview sibling, which is the same native-audio class
and is what the cost map already carries for vertex_ai.
Not a litellm regression. The suspicion fell on #38395 because it removed the
native-audio speechConfig strip, but the setup payload this suite sends is
byte-identical either side of that change: the strip only fires when a client
sends a voice, and the e2e SessionConfig has no voice field. Google's rejection
names the model, not a field.
The gemini (Google AI Studio) provider keeps the -09-2025 id, which still works
there; only the Vertex endpoint dropped it.
The two `surface: ui` cells in the coverage registry, mgmt.key.generate.happy_path
and mgmt.key.update.happy_path, had no covering test. The existing key tests all
call /key/generate and /key/update with the master key, which is not how the
dashboard reaches those routes: an admin signs in, the proxy mints a UI session
key scoped to the litellm-dashboard team, and every subsequent create or edit is
written under that session key.
TestDashboardKeyRoutes covers that path. The first test signs in through
/v2/login, decodes the master-key-signed session JWT the way the dashboard does,
and asserts the minted key carries the admin role and the dashboard team, then
that it can actually read the key inventory the Virtual Keys page renders. The
second edits a key under that session key and asserts both halves of the
contract: /key/info reports the new models and limits with the alias untouched,
and the gateway flips enforcement to match.
ManagementClient grows dashboard_login plus caller-aware key_list and update_key,
so a test can say who is driving a management route instead of always implying
the master key. update_key returns its Result rather than raising, which lets a
caller poll a route that is only transiently refusing; a freshly minted session
key is briefly unauthorized while the auth cache picks up its user row.
litellm-e2e-ui 68 failed the test this PR was meant to stabilise: "fallback
never took effect", streak 4 of a required 5, 60s timeout. Requiring a
consecutive streak of 200s after the fallback is set was wrong. It asserts that
the fallback path succeeds five times running, which is a reliability claim the
test never intended to make, and the path is inherently retry-ish because the
broken primary is attempted first on every call. One intermittent non-200
resets the streak, so a mostly-working fallback never converges.
The two directions are not symmetric:
before the write proving NO replica serves it -> needs every replica
after the write proving the fallback serves it -> one success is the claim
So the control keeps a multi-sample window and the success assertion goes back
to polling for a first sighting, on the wider 60s budget rather than the
original 30s that expired on litellm-e2e-ui 63.
Also drops the two local rebinds Greptile flagged against the repo's
no-reassignment convention: the streak counter is gone with the helper it lived
in, and the cache-round loop is now a lazy generator consumed by next().
The case was skipped because /budget/update 500d on any model_max_budget.
#38430 fixes that by serializing the update payload before the write, so
the case now passes against a proxy carrying that change and there is
nothing left for the skip to hide.
Merge this after #38430; on staging alone the case still fails with the
same 500 it was skipped for.
Two e2e tests fail on timing rather than on litellm behaviour. Measured over the
last ~35 litellm-e2e / litellm-e2e-ui runs:
routerSettings.spec.ts:254 9/35 runs (7 flaky-on-retry, 2 hard failures)
test_cost_headers_e2e.py 1/29 runs it appeared in
Router fallback control
-----------------------
The e2e stack runs replicaCount 2 with proxy_config_reload_interval_seconds 7,
and every request is routed independently, so an observation of the new config
only proves the replica that served it reloaded. patchRouterSettings returns as
soon as /config/update returns, and clearBrokenFallback never waits at all, so a
retry's one-shot control assertion could be answered by a sibling replica still
holding the previous attempt's fallback. That is exactly the observed pair of
errors: "fallback never took effect" on the first attempt and "broken primary
unexpectedly succeeded on its own" on the retry.
Both assertions now poll for a consecutive streak spanning more than one reload
cycle, mirroring the PROPAGATION_TIMEOUT / settle_propagation doctrine the Python
suite already applies in e2e_config.py.
Cost-header cache read
----------------------
The prime and measure calls fired back to back with no gap, and each retry threw
away the prefix it had just paid to prime in favour of a fresh one. OpenAI
publishes a primed prefix asynchronously and routes cache lookups by
prompt_cache_key, so the test was rerolling the least likely path to a hit.
Each round now pins a prompt_cache_key and re-reads the same primed prefix up to
CACHE_REREADS times before rotating, so a fresh prefix is spent only after the
primed one has genuinely failed to become readable.
No production code changes; prompt_cache_key is added to the e2e ChatBody model,
which serializes exclude_none and so is inert for every other caller.
The endpoint answers a probe that exceeds HEALTH_CHECK_TIMEOUT_SECONDS with
HTTP 200 and an in-body "Timeout exceeded", which the harness's status-code
rerun policy cannot see. The suite's parallel Bedrock load can push a Mantle
probe past that cap transiently, so only that exact error is retried, three
bounded attempts with visible prints; any other error verdict still fails
immediately.
Adds live e2e coverage for the Bedrock combinations behind recent customer
incidents: llm_provider-* response-header forwarding on /chat/completions
(nonstream and stream), regional us.anthropic.* inference-profile ids over
the invoke route, and the Admin UI Test Connection probe for a
responses-mode Bedrock Mantle deployment. Registers the matching cells in
the coverage registry and publishes the provider x feature matrix table in
its README.
Adds a scheduled GitHub Actions lane on top of the merged record/replay
transport. A Saturday cron records the `replayable` e2e tests against the
real providers and publishes the fixture bundle as a private
`e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull
that artifact by its pinned digest, verify the checksum before extracting,
and replay it with provider credentials set to bogus values, so a run that
ever reached a real provider fails instead of passing.
An egress sentinel pins the provider hostnames to a local sink for the whole
replay job and counts every connection that reaches them; the job asserts
that count is zero, so hermeticity is proven by measurement. A red Saturday
publishes no bundle, so the next weekday finds nothing fresh and fails loudly
rather than replaying a week-old recording, and the transport's seven-day
freshness gate hard-fails any bundle that has drifted too far. The lane also
runs on demand from the Actions tab with a record/replay `mode` input.
Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic
test now counts to twenty so its recorded response banks several content
deltas, matching the assertion that the stream arrives incrementally.
A downstream disconnect mid-relay was recording the chunk whose write never
landed, so replay would hand back a byte the record run never delivered. Append
each chunk after its yield returns, and label the truncation from the generator
close, so the recording holds exactly what the proxy received.
The record/replay harness stored a streamed provider response as one
buffered body, so a replayed stream arrived coalesced and the
/v1/messages streaming test could not be edge-wired. Keep each SSE
transfer chunk in the bundle in the order the provider sent it (a new
streamed response shape at BUNDLE_FORMAT_VERSION 4) so replay reproduces
the provider's split points, the recorded usage chunk keeps its
position, and a mid-stream upstream error replays as the same
mid-stream error rather than a clean body.
Resolves LIT-5742
* test(e2e): send no-cache on every cacheable request body, opt in only where a hit is the assertion
The e2e proxy runs with the response cache on, so any test that re-sends an
identical chat, messages, responses, completions, embeddings or rerank body
reads back a redis copy of an earlier call instead of reaching the provider.
Five tests in the last week failed that way. Default cache: {"no-cache": true}
on those request models and pass cache=None only in the two tests whose
assertion is the cache hit itself.
* test(e2e): give image edits and OCR a 180s client timeout
Both routes wait on providers that can legitimately take longer than the
60s transport-wide request timeout (gpt-image edits, Azure Document
Intelligence), and a client-side read timeout there fails a green request.
post/upload now accept a per-call timeout like get already does; only those
two call sites use it.
* test(e2e): rerun once on network errors and upstream 5xx only
Assertion failures still fail on the first attempt; only an outcome whose
error string carries the e2e_http network kind or a 5xx status gets one
more try. Test Engine records every attempt, so the flake rate stays
visible while a single provider blip no longer reds the rc run.
* test(e2e): let the reseed burst survive one upstream failure and print why
The burst is the precondition, not the property: one 5xx among six
concurrent calls still leaves five workers racing the cold counter, which
is what the reseed assertion measures. Two or more failures still abort,
and the failing bodies are now in the message instead of only the status
codes.
* test(e2e): keep polling Jaeger through a transient query failure
poll_traces_for_call already waits up to POLL_TIMEOUT for spans to land,
but a single refused connection to the query API failed the test on the
spot. Jaeger restarted twice during today's gate runs (19:05 and 19:41
UTC, each under a minute) and took ten and three otel tests with it while
the same tests passed on the rc build minutes later. A network failure
now counts as not-yet inside the same deadline; if Jaeger is still
unreachable when the deadline passes the test fails with that error, and
any non-network failure still fails immediately.
The mid-conversation system tests prime the prompt cache by re-sending an
identical /v1/messages body until its usage shows the full prefix read back
three times in a row. The e2e stack runs with the litellm response cache on,
so every resend after the first is served from redis with the first call's
usage and the streak can never form; the three unflagged-model tests have
failed on every litellm-e2e build since the consecutive-read check landed.
Send cache: {"no-cache": true} on RichMessagesRequest, as test_cache_control
already does, so each resend reaches the provider.
The two fallback tests sent the same "say hi" / max_tokens=16 body to the
gpt-5.5 fallback, so one empty (finish_reason=length) completion served the
second test from the response cache and failed both. Give each test a unique
prompt and leave gpt-5.5 enough tokens to emit text.
Adversarial review of the new multipart keying turned up collisions where
two different provider requests computed the same replay key, which is the
dangerous failure for a replay harness: the second request silently gets the
first one's response instead of missing loudly.
- a part counts as an upload when it has a filename or declares its own
content type, and the declared content type joins the identity, so two
uploads of the same bytes under the same field no longer collapse
- the uploaded parts contribute a JSON list of [field, filename, type]
triples instead of a "field:filename" string, so a separator inside a
filename can no longer impersonate a field boundary
- repeated field names get a "name[n]" suffix with a literal "[" doubled
first, so a repeated field and a literally indexed one stay distinct
- a field value that is not UTF-8 is stored as a base64 sha256 digest;
base64 rather than hex because the canonicalizer rewrites 64-character
hex runs to <sha256> and folded every binary value onto one key
- a field whose name reads as a credential is stored as <secret>. This
stays key-preserving because the key is recomputed from the stored
request rather than saved beside it, so the live request carrying the
real value still matches its redacted fixture
- the uploaded byte length leaves the key. The canonicalizer absorbs
timestamp and id drift inside a file, and that drift moves the count,
so keeping it there made re-records miss
Also stops a lookalike parameter such as "xboundary=" from being read as
the multipart boundary, and gives the OpenAI batch backend model a single
constant instead of three copies of the literal.
BUNDLE_FORMAT_VERSION goes to 3 because all of this moves recorded keys.
A bundle recorded under the old rules now fails naming both versions
instead of missing on every call.
Chat completions, embeddings, the non-streaming /v1/messages tests, and the
OpenAI batch deployment now register through the provider edge, so
E2E_FIXTURE_MODE=record captures their provider calls and replay serves them
back offline. None of them was wired before, so record was a silent no-op over
these suites and replay quietly went live instead of using the bundle
Multipart uploads now key on their parsed parts: every ordinary form field,
plus the field name, filename, content digest, and length of each file part.
The boundary is envelope rather than content, so it stays out of the digest
instead of changing the key on every run. A body that does not parse as its
declared envelope still has the boundary normalized away before hashing, so
the fallback is at least stable, and it records a name that says why
Binary uploads hash byte for byte. Canonicalizing them first meant decoding
with errors="replace", which collapsed every invalid byte to one U+FFFD and
gave two different PDFs of the same length the same key
Bundles stay out of the repo: they hold verbatim provider response bodies and
expire seven days after recording. Publishing them for CI is LIT-5748, and
streaming fidelity is LIT-5742
The websocket routes under /openai_passthrough and /openai had no e2e
coverage, so nothing catches the regression from issue #36088, where both
prefixes carried HTTP routes only and refused every upgrade with a 403
before a socket ever existed.
Two tests cover it. The realtime one opens /openai_passthrough/v1/realtime
and asserts OpenAI's own session.created frame comes back, which proves the
route is registered and relayed upstream. The responses one asserts
/openai/v1/responses accepts the upgrade, since a responses.connect socket
waits for the client to speak first and has no opening frame to check.
A refused upgrade is an HTTP response rather than a close frame, so both
assert on the handshake. ws_base_url moves into e2e_config now that a
second suite needs it
The passthrough tests and their coverage registry rows pointed at the internal
ticket id, which does not resolve for anyone following a link from
status.litellm.ai. Each test docstring and registry rationale now names the
GitHub issue it pins: #36086 for the two prefix routing cases, #36087 for the
file list cursors, #36523 for streamed Responses cost, and #36646 for
embeddings spend.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
The canonical request already folds params and form into the digest, but
nothing asserted it, so dropping either from canonicalize() left all 92
fixture tests green. Two GETs differing only in query string, or two
uploads differing only in a form field, would share a replay pool and
FIFO-pop each other's recorded response.
Resolves LIT-5890
The two tests that assert on reasoning cost read reasoning_tokens off the
response and required it to be nonzero, without ever asking the model to
reason. Both now send reasoning_effort, so the assertion rests on a
parameter the test sets rather than on the model's default behavior.
The cache-breakdown test sends it on its prime call too: OpenAI's prefix
cache keys on the reasoning setting as well as the tokens, so priming at
a different effort never produces a read.
