litellm-e2e-ui 68 failed the test this PR was meant to stabilise: "fallback
never took effect", streak 4 of a required 5, 60s timeout. Requiring a
consecutive streak of 200s after the fallback is set was wrong. It asserts that
the fallback path succeeds five times running, which is a reliability claim the
test never intended to make, and the path is inherently retry-ish because the
broken primary is attempted first on every call. One intermittent non-200
resets the streak, so a mostly-working fallback never converges.
The two directions are not symmetric:
before the write proving NO replica serves it -> needs every replica
after the write proving the fallback serves it -> one success is the claim
So the control keeps a multi-sample window and the success assertion goes back
to polling for a first sighting, on the wider 60s budget rather than the
original 30s that expired on litellm-e2e-ui 63.
Also drops the two local rebinds Greptile flagged against the repo's
no-reassignment convention: the streak counter is gone with the helper it lived
in, and the cache-round loop is now a lazy generator consumed by next().
Two e2e tests fail on timing rather than on litellm behaviour. Measured over the
last ~35 litellm-e2e / litellm-e2e-ui runs:
routerSettings.spec.ts:254 9/35 runs (7 flaky-on-retry, 2 hard failures)
test_cost_headers_e2e.py 1/29 runs it appeared in
Router fallback control
-----------------------
The e2e stack runs replicaCount 2 with proxy_config_reload_interval_seconds 7,
and every request is routed independently, so an observation of the new config
only proves the replica that served it reloaded. patchRouterSettings returns as
soon as /config/update returns, and clearBrokenFallback never waits at all, so a
retry's one-shot control assertion could be answered by a sibling replica still
holding the previous attempt's fallback. That is exactly the observed pair of
errors: "fallback never took effect" on the first attempt and "broken primary
unexpectedly succeeded on its own" on the retry.
Both assertions now poll for a consecutive streak spanning more than one reload
cycle, mirroring the PROPAGATION_TIMEOUT / settle_propagation doctrine the Python
suite already applies in e2e_config.py.
Cost-header cache read
----------------------
The prime and measure calls fired back to back with no gap, and each retry threw
away the prefix it had just paid to prime in favour of a fresh one. OpenAI
publishes a primed prefix asynchronously and routes cache lookups by
prompt_cache_key, so the test was rerolling the least likely path to a hit.
Each round now pins a prompt_cache_key and re-reads the same primed prefix up to
CACHE_REREADS times before rotating, so a fresh prefix is spent only after the
primed one has genuinely failed to become readable.
No production code changes; prompt_cache_key is added to the e2e ChatBody model,
which serializes exclude_none and so is inert for every other caller.
Adds a scheduled GitHub Actions lane on top of the merged record/replay
transport. A Saturday cron records the `replayable` e2e tests against the
real providers and publishes the fixture bundle as a private
`e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull
that artifact by its pinned digest, verify the checksum before extracting,
and replay it with provider credentials set to bogus values, so a run that
ever reached a real provider fails instead of passing.
An egress sentinel pins the provider hostnames to a local sink for the whole
replay job and counts every connection that reaches them; the job asserts
that count is zero, so hermeticity is proven by measurement. A red Saturday
publishes no bundle, so the next weekday finds nothing fresh and fails loudly
rather than replaying a week-old recording, and the transport's seven-day
freshness gate hard-fails any bundle that has drifted too far. The lane also
runs on demand from the Actions tab with a record/replay `mode` input.
Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic
test now counts to twenty so its recorded response banks several content
deltas, matching the assertion that the stream arrives incrementally.
* test(e2e): send no-cache on every cacheable request body, opt in only where a hit is the assertion
The e2e proxy runs with the response cache on, so any test that re-sends an
identical chat, messages, responses, completions, embeddings or rerank body
reads back a redis copy of an earlier call instead of reaching the provider.
Five tests in the last week failed that way. Default cache: {"no-cache": true}
on those request models and pass cache=None only in the two tests whose
assertion is the cache hit itself.
* test(e2e): give image edits and OCR a 180s client timeout
Both routes wait on providers that can legitimately take longer than the
60s transport-wide request timeout (gpt-image edits, Azure Document
Intelligence), and a client-side read timeout there fails a green request.
post/upload now accept a per-call timeout like get already does; only those
two call sites use it.
* test(e2e): rerun once on network errors and upstream 5xx only
Assertion failures still fail on the first attempt; only an outcome whose
error string carries the e2e_http network kind or a 5xx status gets one
more try. Test Engine records every attempt, so the flake rate stays
visible while a single provider blip no longer reds the rc run.
* test(e2e): let the reseed burst survive one upstream failure and print why
The burst is the precondition, not the property: one 5xx among six
concurrent calls still leaves five workers racing the cold counter, which
is what the reseed assertion measures. Two or more failures still abort,
and the failing bodies are now in the message instead of only the status
codes.
* test(e2e): keep polling Jaeger through a transient query failure
poll_traces_for_call already waits up to POLL_TIMEOUT for spans to land,
but a single refused connection to the query API failed the test on the
spot. Jaeger restarted twice during today's gate runs (19:05 and 19:41
UTC, each under a minute) and took ten and three otel tests with it while
the same tests passed on the rc build minutes later. A network failure
now counts as not-yet inside the same deadline; if Jaeger is still
unreachable when the deadline passes the test fails with that error, and
any non-network failure still fails immediately.
The two tests that assert on reasoning cost read reasoning_tokens off the
response and required it to be nonzero, without ever asking the model to
reason. Both now send reasoning_effort, so the assertion rests on a
parameter the test sets rather than on the model's default behavior.
The cache-breakdown test sends it on its prime call too: OpenAI's prefix
cache keys on the reasoning setting as well as the tokens, so priming at
a different effort never produces a read.
The final streaming usage frame only carries usage.cost when the proxy runs
with litellm_settings.include_cost_in_streaming_usage: true, and that flag is
readable only off the module-level litellm setting. There is no header, key,
or management route that turns it on per request, so a test cannot ask the
shared e2e proxy for it, and the proxy's config does not live in this repo.
The registry row stays as an uncovered gap with the reason recorded, rather
than being deleted, so the behavior is still on the list of things we want
covered once the gateway config is reachable.
The StreamOptions model, ChatBody.stream_options, Usage.cost, and
AnthropicMessagesResponse.id existed only for that test, so they go with it.
Seven live e2e tests covering cost-tracking regressions that currently ship
unnoticed: cache-write tokens billed at the cache-creation rate (#34046),
per-component cost_breakdown on the spend row (#31686), cache reads billed at
the cache-read discount on streamed calls (#34812), cache tokens surviving the
anthropic-messages to Responses bridge (#34957), priority-tier rates applied to
input, output and reasoning (#35923, #35925), the per-component response cost
headers summing to the total (#36965), and cost injected into the final usage
frame of an /openai passthrough stream (#36503).
Every test registers its own deployment with a distinct custom rate per
component, so a component billed at the wrong rate cannot pass. The shared
helpers in cost_rows.py encode the one thing the two surfaces disagree on: the
spend row's input_cost is gross of cache while the response's cost-input header
is net of it.
Replaces the test-side fixture transport with an in-process provider-edge
HTTP server the proxy's deployments point their api_base at. Record forwards
provider calls verbatim and writes them to the bundle; replay answers them
from the bundle with zero provider calls while key auth, routing, cost
calculation, and spend-log writes still execute against the live proxy and
database. Drift comes back as HTTP 599 naming the computed and closest
recorded keys. Request headers are never stored and responses are kept
byte-identical between modes from the proxy's side of the socket.
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
This reverts commit dcb4e5033c.
The suites landed without the proof-of-fix and QA runbook the PR body
itself flagged as outstanding, so the coverage they claim is unverified
against a live proxy
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
* fix(e2e): address greptile and veria review on vendor strategy suite
Store search tool keys as os.environ refs and resolve them in SearchAPIRouter.
Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells
* fix(e2e): drop search_api_router os.environ expansion from vendor suite
Keep the PR test-only. Search tools register without an api_key so the
proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a.
* test(e2e): drop search e2e suite from vendor strategy PR
Remove the /v1/search coverage file and its registry rows so this PR
no longer carries search endpoint testing.
The cache-hit and paid rows for the two driver calls flush from different
pods on independent update_spend timers, so waiting only for the cache-hit
row can return a half-arrived result set where the paid-row assertion then
fails on an empty list. Requiring both row kinds in the poll predicate lets
the existing deadline absorb the slower flush without weakening any assertion
* fix(e2e): reference client.proxy in mid-conversation native providers test
EndpointsClient exposes the shared ProxyClient as .proxy and has never had a
.gateway attribute, so these two calls raised AttributeError at runtime and
failed the tests/e2e basedpyright zero-error gate for any PR touching e2e
files. Introduced in 23b5b7d199.
* test(e2e): cover 12 non-core LLM coverage registry cells
Raises Non-Core LLMs registry coverage from 24/50 to 36/50 (overall 51.9%
to 54.8%). Four cells were already asserted by existing tests and only
gain their covers marker (openai embeddings, openai image generation,
openai TTS, cohere rerank); one is dual-marked onto the existing
spend-tracking embeddings test rather than duplicated.
New tests: bedrock and vertex embeddings, streaming TTS (asserts chunked
transfer encoding so a buffered body cannot pass), audio transcriptions
via the realtime suite's wav fixture, moderations flag/pass pair, and
files list/retrieve in the batches suite.
Harness: e2e_http.upload generalized to any form model with a
file_content_type override (batches path unchanged), new stream_binary
primitive + BinaryStream for binary chunked responses, transcribe and
moderations client methods, file retrieve/list client methods.
* fix(e2e): close streamed TTS response on error paths and surface the error body
With stream=True a non-2xx response returned with the body unread, keeping
the socket checked out until garbage collection; the sibling
_streaming_outcome already consumes resp.text on error. The response now
closes on every path and BinaryStream carries a bounded error_body so a
failed stream call is triageable.
* test(e2e): assert streamed TTS response carries no content-length
tests/e2e/conftest.py's pytest_sessionfinish truncated LiteLLM_SpendLogs
against whatever DATABASE_URL resolved to, gated only by "an e2e test body
ran". Pointed at a shared or staging DB, a routine local run wiped real
spend data. It also reached the truncate helper through a sys.path.insert
into quota_management/spend_tracking/spend_e2e_client.py, a cross-suite
import-by-path hack it then unwound in a finally.
The cleanup now routes through a new run_spend_log_cleanup in a top-level
tests/e2e/e2e_db.py, which fires the destructive truncate only when the
operator set E2E_RESET_SPEND_LOGS=1 and an e2e test actually ran. Any other
value (unset, 0, true, empty) leaves the DB untouched, so presence of the
variable alone or a test run alone never arms the truncate. The decision
plus the injectable truncate callable live in that pure helper, and conftest
is a thin adapter that supplies os.environ.get(...), the session stash, and
reset_spend_logs.
reset_spend_logs itself moved from spend_e2e_client.py into e2e_db.py
(implementation unchanged), sitting next to e2e_config and lifecycle so both
conftest and any suite import it by name; the sys.path munging is gone.
Nothing else imported reset_spend_logs, so spend_e2e_client.py drops the
definition, its __all__ entry, and the now-unused os import.
Add a live spend-tracking e2e that drives a streaming anthropic-format
/v1/messages request through litellm's anthropic-messages -> OpenAI Responses
adapter and asserts the consumed stream writes exactly one SpendLogs row with
nonzero cost and token counts, attributed to the calling key under
custom_llm_provider openai and the /v1/messages call_type.
The deployment is a Responses-only OpenAI model (gpt-5.3-codex), so a served,
costed row proves the Responses path was taken; the chat-completions bridge
would have failed at OpenAI on an endpoint the model does not expose. Adds a
streaming /v1/messages method to the shared Gateway and the suite client, the
model to the inline compose config and driver-model registration, a coverage
registry row (quota_management.spend_tracking.messages_bridge.logs_cost), and
the matching variant vocab entry. The _summarize spend-row detail also gains
call_type and custom_llm_provider so a failed assertion prints the fields it
asserts on.
Resolves LIT-4546
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
The e2e docs claimed `e2e`-marked tests skip when no proxy answers the
liveness probe, but the harness has always hard-failed: conftest.py's
pytest_runtest_setup calls pytest.fail, its module docstring states
"hard failures only ... never skip", and logging/conftest.py forbids
skipping outright. Align the docs to the code so the single most
important contract reads the same everywhere; a dead proxy turns a run
red instead of being silently skipped and mistaken for a pass. The
per-suite conftest docstrings that described the shared hook as a
"proxy liveness skip" are corrected to "liveness gate" for the same
reason.
Also scope the no-unit-tests hard rule to what it means: never
substitute a unit test for e2e feature coverage, while explicitly
allowing tests that cover the harness itself (e.g.
coverage_registry/test_collector.py), which carry no e2e marker and
run whether or not a proxy is up.
No product code and no harness logic changed.
Resolves LIT-4554
