Both anchoring tests read the trigger's box before the click and the popup's box the
instant it turns visible. Base UI places the popup asynchronously and opening it can
shift the trigger, so both boxes could be sampled before the layout settled. The run
on 1eedaa3a43 missed by 4.2px (expected >= 446.015, got 441.799) on a tree with no UI
changes at all, having passed on 21092d633b, which differs only in a deleted python
test and a budget json.
Each assertion now re-reads both boxes under expect.poll. The conditions themselves
are unchanged: the popup must sit at or below the trigger's bottom edge in the first
test and must not overlap it in the second. Polling cannot mask a genuinely misplaced
popup, since one that never lands correctly still fails when the poll times out.
(cherry picked from commit d8679508d4)
litellm-e2e-ui 68 failed the test this PR was meant to stabilise: "fallback
never took effect", streak 4 of a required 5, 60s timeout. Requiring a
consecutive streak of 200s after the fallback is set was wrong. It asserts that
the fallback path succeeds five times running, which is a reliability claim the
test never intended to make, and the path is inherently retry-ish because the
broken primary is attempted first on every call. One intermittent non-200
resets the streak, so a mostly-working fallback never converges.
The two directions are not symmetric:
before the write proving NO replica serves it -> needs every replica
after the write proving the fallback serves it -> one success is the claim
So the control keeps a multi-sample window and the success assertion goes back
to polling for a first sighting, on the wider 60s budget rather than the
original 30s that expired on litellm-e2e-ui 63.
Also drops the two local rebinds Greptile flagged against the repo's
no-reassignment convention: the streak counter is gone with the helper it lived
in, and the cache-round loop is now a lazy generator consumed by next().
(cherry picked from commit b95801172c)
Two e2e tests fail on timing rather than on litellm behaviour. Measured over the
last ~35 litellm-e2e / litellm-e2e-ui runs:
routerSettings.spec.ts:254 9/35 runs (7 flaky-on-retry, 2 hard failures)
test_cost_headers_e2e.py 1/29 runs it appeared in
Router fallback control
-----------------------
The e2e stack runs replicaCount 2 with proxy_config_reload_interval_seconds 7,
and every request is routed independently, so an observation of the new config
only proves the replica that served it reloaded. patchRouterSettings returns as
soon as /config/update returns, and clearBrokenFallback never waits at all, so a
retry's one-shot control assertion could be answered by a sibling replica still
holding the previous attempt's fallback. That is exactly the observed pair of
errors: "fallback never took effect" on the first attempt and "broken primary
unexpectedly succeeded on its own" on the retry.
Both assertions now poll for a consecutive streak spanning more than one reload
cycle, mirroring the PROPAGATION_TIMEOUT / settle_propagation doctrine the Python
suite already applies in e2e_config.py.
Cost-header cache read
----------------------
The prime and measure calls fired back to back with no gap, and each retry threw
away the prefix it had just paid to prime in favour of a fresh one. OpenAI
publishes a primed prefix asynchronously and routes cache lookups by
prompt_cache_key, so the test was rerolling the least likely path to a hit.
Each round now pins a prompt_cache_key and re-reads the same primed prefix up to
CACHE_REREADS times before rotating, so a fresh prefix is spent only after the
primed one has genuinely failed to become readable.
No production code changes; prompt_cache_key is added to the e2e ChatBody model,
which serializes exclude_none and so is inert for every other caller.
(cherry picked from commit 84dfc18f6b)
The two vision tests pointed at a Wikipedia-hosted cat photo, so every run
depended on upload.wikimedia.org staying up and unthrottled. It throttled,
and the 429 surfaced as a bedrock APIConnectionError, which reads as a
gateway failure rather than what it was.
The image is now a fixture in the repo, passed as a data URL. That also puts
the two providers on the same bytes: litellm downloads the image itself for
bedrock, while openai is handed the link and fetches it from its own servers,
so the hosted URL quietly meant the two tests were not testing the same thing.
The image was generated for this repo rather than borrowed, so nothing here
carries a third-party license. Also drops a stale comment about openai prompt
caching that sat above the vision helper; no caching test uses it.
(cherry picked from commit ff418ffb9c)
Google withdrew gemini-live-2.5-flash-preview-native-audio-09-2025 from the
Vertex Live API. Every session dies at setup:
received 1007 (invalid frame payload data)
gemini-live-2.5-flash-preview-native-audio-09-2025 is not supported in the live api.
The client sees session.created (the proxy synthesizes it on connect) and then
nothing, so both vertex_ai realtime tests time out waiting for session.updated.
Confirmed by probing the Vertex Live endpoint directly with the e2e stack's own
credentials:
gemini-live-2.5-flash-preview-native-audio-09-2025 -> 1007, not supported
gemini-live-2.5-flash-native-audio -> setupComplete
so this swaps to the non-preview sibling, which is the same native-audio class
and is what the cost map already carries for vertex_ai.
Not a litellm regression. The suspicion fell on #38395 because it removed the
native-audio speechConfig strip, but the setup payload this suite sends is
byte-identical either side of that change: the strip only fires when a client
sends a voice, and the e2e SessionConfig has no voice field. Google's rejection
names the model, not a field.
The gemini (Google AI Studio) provider keeps the -09-2025 id, which still works
there; only the Vertex endpoint dropped it.
(cherry picked from commit a215ecaf3d)
The shared SelectContent wrapper defaulted alignItemWithTrigger to true,
which puts Base UI's positioner into item-aligned mode and places the
popup so the active item sits on top of the trigger. In that mode the
side and sideOffset the wrapper passes two lines above are ignored, and
the popup reports data-side="none".
The overlap only becomes visible once the items are tall enough to
matter, which is why the autorouter Template picker shows it clearly:
its options are three-line cards, so the popup covers both the select
box and its own label.
No call site in the dashboard asked for item-aligned mode. 21 of them
across 15 files already passed alignItemWithTrigger={false} by hand to
undo the default, and the remaining 127 inherited the bug. Flipping the
default makes side and sideOffset live, so collision handling works and
a select with no room below now flips above the trigger rather than
covering it. The 21 hand-written opt-outs are deleted as redundant.
(cherry picked from commit 71449b9c55)
* test(e2e): send no-cache on every cacheable request body, opt in only where a hit is the assertion
The e2e proxy runs with the response cache on, so any test that re-sends an
identical chat, messages, responses, completions, embeddings or rerank body
reads back a redis copy of an earlier call instead of reaching the provider.
Five tests in the last week failed that way. Default cache: {"no-cache": true}
on those request models and pass cache=None only in the two tests whose
assertion is the cache hit itself.
* test(e2e): give image edits and OCR a 180s client timeout
Both routes wait on providers that can legitimately take longer than the
60s transport-wide request timeout (gpt-image edits, Azure Document
Intelligence), and a client-side read timeout there fails a green request.
post/upload now accept a per-call timeout like get already does; only those
two call sites use it.
* test(e2e): rerun once on network errors and upstream 5xx only
Assertion failures still fail on the first attempt; only an outcome whose
error string carries the e2e_http network kind or a 5xx status gets one
more try. Test Engine records every attempt, so the flake rate stays
visible while a single provider blip no longer reds the rc run.
* test(e2e): let the reseed burst survive one upstream failure and print why
The burst is the precondition, not the property: one 5xx among six
concurrent calls still leaves five workers racing the cold counter, which
is what the reseed assertion measures. Two or more failures still abort,
and the failing bodies are now in the message instead of only the status
codes.
* test(e2e): keep polling Jaeger through a transient query failure
poll_traces_for_call already waits up to POLL_TIMEOUT for spans to land,
but a single refused connection to the query API failed the test on the
spot. Jaeger restarted twice during today's gate runs (19:05 and 19:41
UTC, each under a minute) and took ten and three otel tests with it while
the same tests passed on the rc build minutes later. A network failure
now counts as not-yet inside the same deadline; if Jaeger is still
unreachable when the deadline passes the test fails with that error, and
any non-network failure still fails immediately.
The mid-conversation system tests prime the prompt cache by re-sending an
identical /v1/messages body until its usage shows the full prefix read back
three times in a row. The e2e stack runs with the litellm response cache on,
so every resend after the first is served from redis with the first call's
usage and the streak can never form; the three unflagged-model tests have
failed on every litellm-e2e build since the consecutive-read check landed.
Send cache: {"no-cache": true} on RichMessagesRequest, as test_cache_control
already does, so each resend reaches the provider.
The two fallback tests sent the same "say hi" / max_tokens=16 body to the
gpt-5.5 fallback, so one empty (finish_reason=length) completion served the
second test from the response cache and failed both. Give each test a unique
prompt and leave gpt-5.5 enough tokens to emit text.
Adversarial review of the new multipart keying turned up collisions where
two different provider requests computed the same replay key, which is the
dangerous failure for a replay harness: the second request silently gets the
first one's response instead of missing loudly.
- a part counts as an upload when it has a filename or declares its own
content type, and the declared content type joins the identity, so two
uploads of the same bytes under the same field no longer collapse
- the uploaded parts contribute a JSON list of [field, filename, type]
triples instead of a "field:filename" string, so a separator inside a
filename can no longer impersonate a field boundary
- repeated field names get a "name[n]" suffix with a literal "[" doubled
first, so a repeated field and a literally indexed one stay distinct
- a field value that is not UTF-8 is stored as a base64 sha256 digest;
base64 rather than hex because the canonicalizer rewrites 64-character
hex runs to <sha256> and folded every binary value onto one key
- a field whose name reads as a credential is stored as <secret>. This
stays key-preserving because the key is recomputed from the stored
request rather than saved beside it, so the live request carrying the
real value still matches its redacted fixture
- the uploaded byte length leaves the key. The canonicalizer absorbs
timestamp and id drift inside a file, and that drift moves the count,
so keeping it there made re-records miss
Also stops a lookalike parameter such as "xboundary=" from being read as
the multipart boundary, and gives the OpenAI batch backend model a single
constant instead of three copies of the literal.
BUNDLE_FORMAT_VERSION goes to 3 because all of this moves recorded keys.
A bundle recorded under the old rules now fails naming both versions
instead of missing on every call.
Chat completions, embeddings, the non-streaming /v1/messages tests, and the
OpenAI batch deployment now register through the provider edge, so
E2E_FIXTURE_MODE=record captures their provider calls and replay serves them
back offline. None of them was wired before, so record was a silent no-op over
these suites and replay quietly went live instead of using the bundle
Multipart uploads now key on their parsed parts: every ordinary form field,
plus the field name, filename, content digest, and length of each file part.
The boundary is envelope rather than content, so it stays out of the digest
instead of changing the key on every run. A body that does not parse as its
declared envelope still has the boundary normalized away before hashing, so
the fallback is at least stable, and it records a name that says why
Binary uploads hash byte for byte. Canonicalizing them first meant decoding
with errors="replace", which collapsed every invalid byte to one U+FFFD and
gave two different PDFs of the same length the same key
Bundles stay out of the repo: they hold verbatim provider response bodies and
expire seven days after recording. Publishing them for CI is LIT-5748, and
streaming fidelity is LIT-5742
The websocket routes under /openai_passthrough and /openai had no e2e
coverage, so nothing catches the regression from issue #36088, where both
prefixes carried HTTP routes only and refused every upgrade with a 403
before a socket ever existed.
Two tests cover it. The realtime one opens /openai_passthrough/v1/realtime
and asserts OpenAI's own session.created frame comes back, which proves the
route is registered and relayed upstream. The responses one asserts
/openai/v1/responses accepts the upgrade, since a responses.connect socket
waits for the client to speak first and has no opening frame to check.
A refused upgrade is an HTTP response rather than a close frame, so both
assert on the handshake. ws_base_url moves into e2e_config now that a
second suite needs it
The passthrough tests and their coverage registry rows pointed at the internal
ticket id, which does not resolve for anyone following a link from
status.litellm.ai. Each test docstring and registry rationale now names the
GitHub issue it pins: #36086 for the two prefix routing cases, #36087 for the
file list cursors, #36523 for streamed Responses cost, and #36646 for
embeddings spend.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
The canonical request already folds params and form into the digest, but
nothing asserted it, so dropping either from canonicalize() left all 92
fixture tests green. Two GETs differing only in query string, or two
uploads differing only in a form field, would share a replay pool and
FIFO-pop each other's recorded response.
Resolves LIT-5890
The two tests that assert on reasoning cost read reasoning_tokens off the
response and required it to be nonzero, without ever asking the model to
reason. Both now send reasoning_effort, so the assertion rests on a
parameter the test sets rather than on the model's default behavior.
The cache-breakdown test sends it on its prime call too: OpenAI's prefix
cache keys on the reasoning setting as well as the tokens, so priming at
a different effort never produces a read.
The final streaming usage frame only carries usage.cost when the proxy runs
with litellm_settings.include_cost_in_streaming_usage: true, and that flag is
readable only off the module-level litellm setting. There is no header, key,
or management route that turns it on per request, so a test cannot ask the
shared e2e proxy for it, and the proxy's config does not live in this repo.
The registry row stays as an uncovered gap with the reason recorded, rather
than being deleted, so the behavior is still on the list of things we want
covered once the gateway config is reachable.
The StreamOptions model, ChatBody.stream_options, Usage.cost, and
AnthropicMessagesResponse.id existed only for that test, so they go with it.
Five e2e tests over routes a customer drives through the gateway, each one
pinning a fix that currently has no live coverage.
The dedicated /openai_passthrough prefix used to be swallowed by the
provider-scoped /{provider}/v1/files and /{provider}/v1/batches routes, which
bound "openai_passthrough" as a provider name and failed inside the gateway
before ever reaching OpenAI. Two tests now upload a file and list batches
through that prefix and assert OpenAI's own objects come back.
Streamed /openai_passthrough/v1/responses and /openai_passthrough/v1/embeddings
are relayed to OpenAI but still have to be costed, since the customer budgets
against this traffic. Both used to land a row the gateway could not use: the
streamed responses call logged a zero-cost row under a random id, and
embeddings wrote no row at all. Each test now reconciles the logged spend and
token counts against the response the caller was actually served.
GET /v1/files narrowed its data to the caller's own rows but left first_id and
last_id addressing the shared provider account's page, handing any caller raw
provider file ids belonging to other tenants. The new test asserts both cursors
address rows in the page the caller can see.
ResourceManager.defer now accepts any callable rather than one returning None,
so a delete that answers with a response model can be deferred as-is.
On models without supports_mid_conversation_system, a system entry between
an assistant tool_use turn and the user tool_result turn became a user turn
in that position and the provider rejected the request ("tool_use ids were
found without tool_result blocks immediately after"). That run of entries
now goes right after the tool_result turn, where consecutive user turns
merge upstream. The converted turn also carries only role and content, as
the hoist did, so an entry with extra keys no longer 400s with "Extra
inputs are not permitted".
The e2e cache priming re-sends the identical first turn until its own cache
entry reads back before the reminder turn goes out, since Vertex can take a
few seconds to serve a freshly written entry.
/audio/transcriptions answers a model-less request with one of two 400s
depending on whether any wildcard deployment is registered at the time, and
every suite shares one proxy, so run order decided which message came back.
The assertion pinned only the no-wildcard wording, so it went red whenever
the model-access-group suite had registered its wildcards first. It now
accepts either message and still holds the error to naming the model
Verified against a live proxy in both states: with a wildcard registered
(the message CI was seeing) and with none (the message the assertion
expected), the suite passes 3/3 either way
Seven live e2e tests covering cost-tracking regressions that currently ship
unnoticed: cache-write tokens billed at the cache-creation rate (#34046),
per-component cost_breakdown on the spend row (#31686), cache reads billed at
the cache-read discount on streamed calls (#34812), cache tokens surviving the
anthropic-messages to Responses bridge (#34957), priority-tier rates applied to
input, output and reasoning (#35923, #35925), the per-component response cost
headers summing to the total (#36965), and cost injected into the final usage
frame of an /openai passthrough stream (#36503).
Every test registers its own deployment with a distinct custom rate per
component, so a component billed at the wrong rate cannot pass. The shared
helpers in cost_rows.py encode the one thing the two surfaces disagree on: the
spend row's input_cost is gross of cache while the response's cost-input header
is net of it.
Adds e2e coverage for batches terminal state and cost write-back, failure
paths, per-backend file content downloads, and two-gateway routing (LIT-5730).
Replaces the test-side fixture transport with an in-process provider-edge
HTTP server the proxy's deployments point their api_base at. Record forwards
provider calls verbatim and writes them to the bundle; replay answers them
from the bundle with zero provider calls while key auth, routing, cost
calculation, and spend-log writes still execute against the live proxy and
database. Drift comes back as HTTP 599 naming the computed and closest
recorded keys. Request headers are never stored and responses are kept
byte-identical between modes from the proxy's side of the socket.
* fix(auth): resolve bare model names against wildcard deployments in model access groups
* test(e2e): cover model access group permission checks on keys and teams
Greptile flagged that lazy per-slug pool initialization could double-build
under concurrent replay calls, splitting consumption across a discarded
pool. Pools are now built once at ReplaySource construction and per-key
consumption is a single atomic deque pop, with a barrier-synchronized
regression test that fails 10/10 under the lazy-init mutant
Replay previously matched interactions by transport verb and path in
recorded order, so a request whose body drifted from the recording
silently replayed the stale response, and reordering two independent
calls broke replay even though both were recorded. Match keys are now
canonical: fixture_canonical.py strips volatile headers and credential
fields, replaces unique markers, generated ids, uuids, and timestamps
with fixed placeholders, sorts object keys, and hashes what remains, so
a key is stable across runs and machines while any real content drift
is a hard ReplayMiss naming the computed key, the closest recorded key
with its file, and a content diff, with no fallthrough to a live call.
Matching is order-independent across distinct keys and FIFO within one
key. Recording now also redacts credential body and form fields (not
just auth headers) so provider keys never land in bundles.
Resolves LIT-5741
The strict-denial pin only asserted a 401, so any unrelated 401 (a bad key, a deleted key) would have kept it green while tag routing silently broke. The harness now keeps the 401 response body, the way it already does for 429s, and the pin asserts the tag-routing denial message.
* test(ui): repoint the e2e locators at the post-antd form controls
Nine Playwright specs went red after the tremor and antd removals, none of
them because the product broke. The specs selected on markup those libraries
owned: tremor's TextInput stamped data-testid="base-input", the antd toast
facade rendered .ant-notification, and the team member modal's email field was
an .ant-select. Removing the libraries deleted those hooks silently.
Repoint each onto a user-facing locator that survives the next migration:
getByLabel for the key name, the MCP tool argument and the two cache pricing
fields, getByRole("combobox") for the team member email search, and the toast
container for the add-model success message.
The pricing fields needed a source change to be reachable at all. antd's
Form.Item used to assign the field name as the input id and tie the label to
it; the react-hook-form rewrite renders FieldLabel with no htmlFor and lets
FormField generate an opaque control id, so both cache inputs lost their
accessible name and could only be told apart by placeholder, which they share.
Pin the id back to the field name and point the label at it.
* fix(ui): let FormField own the pricing field label instead of hand-rendering one
The previous commit gave the cache cost inputs an accessible name by rendering
a FieldLabel with htmlFor next to the FormField. FieldLabel forwards Label's
props, and Label only accepts children, so next build failed type checking.
FormField already renders a label wired to the control id it generates, and
FormField.test.tsx covers that association, so passing label through is both
type-safe and less markup. The read-only branch keeps its plain FieldLabel,
which has no control to point at.
Key and team router_settings set enable_tag_filtering on the request kwargs,
and get_deployments_for_tag already treats that as authoritative, but
_select_pre_routing_strategy only consulted the router-wide flag, so tagged
auto-router markers still captured untagged requests from keys that enabled
filtering. The e2e auto-router module now enables tag filtering through
key-level router_settings instead of flipping /config/update module-wide,
which was denying concurrently running tagged requests from other suites on
the shared per-build CI proxy.
