The two tests that assert on reasoning cost read reasoning_tokens off the
response and required it to be nonzero, without ever asking the model to
reason. Both now send reasoning_effort, so the assertion rests on a
parameter the test sets rather than on the model's default behavior.
The cache-breakdown test sends it on its prime call too: OpenAI's prefix
cache keys on the reasoning setting as well as the tokens, so priming at
a different effort never produces a read.
The final streaming usage frame only carries usage.cost when the proxy runs
with litellm_settings.include_cost_in_streaming_usage: true, and that flag is
readable only off the module-level litellm setting. There is no header, key,
or management route that turns it on per request, so a test cannot ask the
shared e2e proxy for it, and the proxy's config does not live in this repo.
The registry row stays as an uncovered gap with the reason recorded, rather
than being deleted, so the behavior is still on the list of things we want
covered once the gateway config is reachable.
The StreamOptions model, ChatBody.stream_options, Usage.cost, and
AnthropicMessagesResponse.id existed only for that test, so they go with it.
Seven live e2e tests covering cost-tracking regressions that currently ship
unnoticed: cache-write tokens billed at the cache-creation rate (#34046),
per-component cost_breakdown on the spend row (#31686), cache reads billed at
the cache-read discount on streamed calls (#34812), cache tokens surviving the
anthropic-messages to Responses bridge (#34957), priority-tier rates applied to
input, output and reasoning (#35923, #35925), the per-component response cost
headers summing to the total (#36965), and cost injected into the final usage
frame of an /openai passthrough stream (#36503).
Every test registers its own deployment with a distinct custom rate per
component, so a component billed at the wrong rate cannot pass. The shared
helpers in cost_rows.py encode the one thing the two surfaces disagree on: the
spend row's input_cost is gross of cache while the response's cost-input header
is net of it.
* fix(auth): resolve bare model names against wildcard deployments in model access groups
* test(e2e): cover model access group permission checks on keys and teams
Greptile flagged that lazy per-slug pool initialization could double-build
under concurrent replay calls, splitting consumption across a discarded
pool. Pools are now built once at ReplaySource construction and per-key
consumption is a single atomic deque pop, with a barrier-synchronized
regression test that fails 10/10 under the lazy-init mutant
Replay previously matched interactions by transport verb and path in
recorded order, so a request whose body drifted from the recording
silently replayed the stale response, and reordering two independent
calls broke replay even though both were recorded. Match keys are now
canonical: fixture_canonical.py strips volatile headers and credential
fields, replaces unique markers, generated ids, uuids, and timestamps
with fixed placeholders, sorts object keys, and hashes what remains, so
a key is stable across runs and machines while any real content drift
is a hard ReplayMiss naming the computed key, the closest recorded key
with its file, and a content diff, with no fallthrough to a live call.
Matching is order-independent across distinct keys and FIFO within one
key. Recording now also redacts credential body and form fields (not
just auth headers) so provider keys never land in bundles.
Resolves LIT-5741
The strict-denial pin only asserted a 401, so any unrelated 401 (a bad key, a deleted key) would have kept it green while tag routing silently broke. The harness now keeps the 401 response body, the way it already does for 429s, and the pin asserts the tag-routing denial message.
* test(ui): repoint the e2e locators at the post-antd form controls
Nine Playwright specs went red after the tremor and antd removals, none of
them because the product broke. The specs selected on markup those libraries
owned: tremor's TextInput stamped data-testid="base-input", the antd toast
facade rendered .ant-notification, and the team member modal's email field was
an .ant-select. Removing the libraries deleted those hooks silently.
Repoint each onto a user-facing locator that survives the next migration:
getByLabel for the key name, the MCP tool argument and the two cache pricing
fields, getByRole("combobox") for the team member email search, and the toast
container for the add-model success message.
The pricing fields needed a source change to be reachable at all. antd's
Form.Item used to assign the field name as the input id and tie the label to
it; the react-hook-form rewrite renders FieldLabel with no htmlFor and lets
FormField generate an opaque control id, so both cache inputs lost their
accessible name and could only be told apart by placeholder, which they share.
Pin the id back to the field name and point the label at it.
* fix(ui): let FormField own the pricing field label instead of hand-rendering one
The previous commit gave the cache cost inputs an accessible name by rendering
a FieldLabel with htmlFor next to the FormField. FieldLabel forwards Label's
props, and Label only accepts children, so next build failed type checking.
FormField already renders a label wired to the control id it generates, and
FormField.test.tsx covers that association, so passing label through is both
type-safe and less markup. The read-only branch keeps its plain FieldLabel,
which has no control to point at.
Key and team router_settings set enable_tag_filtering on the request kwargs,
and get_deployments_for_tag already treats that as authoritative, but
_select_pre_routing_strategy only consulted the router-wide flag, so tagged
auto-router markers still captured untagged requests from keys that enabled
filtering. The e2e auto-router module now enables tag filtering through
key-level router_settings instead of flipping /config/update module-wide,
which was denying concurrently running tagged requests from other suites on
the shared per-build CI proxy.
E2E_FIXTURE_MODE selects the transport every e2e client is built on: live
(default, unchanged behavior), record (pass through to the live proxy while
writing every interaction to a fixture bundle), or replay (serve every
interaction from the bundle with no proxy and no provider spend). Both new
transports fulfil the existing Transport protocol, so no test changes shape.
A bundle is a directory with a manifest (record timestamp, harness version,
format version) and one JSON file per interaction, grouped per test in call
order. Replay against a manifest older than seven days hard-fails at
collection time naming the bundle age. Record always wipes and never reads
the previous bundle, refusing to wipe a directory that is not a bundle.
Auth header values are redacted on write; uploads store a sha256 digest.
unique_marker() becomes deterministic per test in record/replay modes so a
replay run regenerates exactly the requests the record run sent.
Content-based match keys, streaming chunk fidelity, and provider-scoping are
follow-ups (LIT-5741, LIT-5742, LIT-5745).
The tool_search x bedrock_invoke cell only ever probed the first turn, so
nothing in the suite has sent a server_tool_use block back to a provider.
Every turn of a real Claude Code session after the first carries the
server_tool_use and tool_search_tool_result blocks the previous turn
produced, and that path was uncovered.
Adds probe_tool_search_multiturn, which takes the real assistant turn
back, answers any client-side tool_use with the id the model actually
emitted, and replays the whole thing as history with the tools still
declared. The assertion refuses to go green unless both server-tool
blocks made it into the replayed history, so a first turn truncated at
max_tokens reads as a failure instead of a vacuous pass.
The replay assertion's red paths never run in a green cell, so they get
markerless harness tests of their own alongside the existing
_builder_unit_tests tree.
No production code.
The log details drawer moved off Ant Design in 03d2b16bc, so its section
header renders lucide ChevronUp/ChevronDown rather than antd's UpOutlined
and DownOutlined. The collapse test still waited on .anticon-up and
.anticon-down, which no longer exist anywhere under view_logs, so it
failed on every run and burned all three attempts identically.
Point the three assertions at .lucide-chevron-up and .lucide-chevron-down,
matching how the dashboard's other suites address lucide icons.
This cell needs the websearch_interception callback and a declared search
backend, both listed in its own module docstring. The ephemeral e2e stack
ships neither, so the request falls through to the bedrock transformation
and takes the by-design 400 that tells you to enable interception.
The cell has never been green here: the error path merged about an hour and
a half before the cell did, and the last full suite to pass predates the
cell entirely. Skip it with the reason recorded so the run reports honestly
instead of carrying a permanent red, and unskip once the stack ships the
config the docstring already spells out.
Both providers reworded the error strings these two cells pinned, so the
suite went red without any behavior changing. Anthropic's auth error is now
"API key is invalid." rather than "invalid x-api-key", and OpenAI rejects an
empty upload with "This model does not support the format you provided.",
which names neither "file" nor "audio".
Assert the durable shape instead. The otel cell pins the machine-readable
authentication_error type plus a non-empty message, and the embedded JSON
still has to parse, which is what proves the attribute survived untruncated.
The transcription cell pins that the 400 relays the provider's own rejection
and is typed as a client input error, so a regression that swallows the
provider reason or returns a 500 still fails.
The playground, logs drawer and AI Hub modal moved off antd, so the specs
that reached for .ant-select, .ant-drawer-content, .ant-modal and
.ant-radio-button-wrapper no longer match anything and time out.
Address the same controls through their accessible role and name instead,
which holds across the component library swap and reads closer to what a
user does.
Rebuilds the thirteen form-free components under common_components on
the in-repo shadcn layer, so they inherit the dashboard's design tokens
instead of styling themselves through Ant Design and Tremor.
SearchSelect and the three dropdowns that wrap it now forward an
optional input id, so an antd Form.Item label still resolves to its
control. The e2e steps that reached into antd's Select and Modal
internals now go through the test id, role and data-slot.
Rebuilds public_model_hub, MakeSkillPublicForm, ModelSelect and the
guardrail LogViewer on the in-repo shadcn layer, so they inherit the
dashboard's design tokens instead of styling themselves through Ant
Design and Tremor.
Public prop signatures are unchanged, so no caller moves. The two
teams e2e steps that reached into antd's Select internals now drive
the combobox through its test id, role and data-slot instead.
Every case in TestAccessControl asserted that something was refused. A gateway
that denied the allow-listed model too would have passed all of them, so the
suite could not tell "denied correctly" from "broken outright".
Adds the positive half: a key allow-listed for gemini-2.5-flash can call it and
gets back a real completion rather than a 200-wrapped error.
Also tightens the unknown-model case. It accepted any valid JSON, so a bare
"{}" or even "null" satisfied it. It now requires the OpenAI-shaped error
envelope with a message a client can actually surface, parsed through a typed
model instead of json.loads.
* test(e2e-ui): cover the Playground, Logs and Usage manual-QA flows
These three pages carried no e2e coverage, so the manual QA checklist was the
only thing standing behind them.
Playground: sends a chat from the UI for both configured models, and for both
virtual-key sources (the logged-in session, and a key pasted into the panel).
This is the only spec that drives the dashboard's own LLM call path rather than
an admin CRUD endpoint.
Logs: a request the proxy actually served appears in the table, its drawer
expands to the real request and response bodies, both copy to the clipboard,
the Input card collapses, the JSON view exposes Request/Response, and the End
User filter narrows the table to one customer.
Usage: traffic billed to a virtual key reaches Top Virtual Keys, the card
toggles between table and chart, and the key opens its key-info panel.
Router settings: the existing spec proved the UI can record a fallback; the new
one proves the fallback is honoured, by pointing a model at an unreachable
upstream and asserting the reply comes back anyway. It asserts the un-fallen-back
call fails first, so a quietly-working primary cannot fake a pass.
Supporting changes:
- helpers/traffic.ts generates the traffic these pages render, rather than
seeding rows no code produced. Its two wait helpers exist because the Logs
and Usage pages read different stores: spend logs are flushed on a timer, and
the Usage page reads a background rollup *and* fetches once on mount, so
waiting on the DOM there can never converge.
- helpers/playground.ts holds the playground controls, now shared with the
fallback spec. Everything is scoped to the visible copy of the config panel,
which is rendered twice for the docked and collapsed layouts.
- run_e2e.sh gains E2E_KEEP_ALIVE=1, which brings the stack up and blocks so a
spec can be re-run against it without paying for a UI rebuild each iteration.
Verified with the full suite on a fresh stack: 89 passed, 0 failed, 5 skipped.
* test(e2e-ui): cover listing and calling MCP tools
Covers the two MCP manual-QA items the create-only spec cannot reach:
opening a server's tool list, and calling a tool and seeing its result.
Both need an MCP server that actually answers, so this points at DeepWiki's
public MCP server -- Streamable HTTP, auth None, so there is no credential to
hold and nothing to leak from a public repo. The call is made by the proxy,
not the browser; nothing in the e2e chart restricts that egress.
The external dependency is real and is left visible: an upstream outage turns
these red rather than auto-skipping, because a spec that skips itself on
connection trouble also skips when the proxy's MCP client is what broke.
E2E_SKIP_EXTERNAL_MCP=1 is the explicit opt-out.
Not yet executed against a live stack.
* test(e2e-ui): verify key mutations round-trip instead of trusting the toast
The recurring customer report is a form that says "Saved!" and then either
no-ops or clobbers an unrelated field. A toast-only assertion passes in both
cases, and outside three specs that is all this suite checks.
Adds helpers/roundTrip.ts, factoring out the idiom clearCustomPricing,
credentials and routerSettings already use: capture the outgoing request body,
then read the resource back through the management API.
Applies it to the keys spec:
- create: the key is readable from /key/list and owns a team_id, rather than
trusting a table row rendered from the create response the UI already held
- update limits: TPM/RPM are on the wire AND persisted, and the key's models
and team are unchanged -- bumping one field wiping another is the reported
failure mode (PR #34452), not a hypothetical
- delete: the key is gone from /key/list, not merely toasted as deleted
- regenerate: the stored token actually changed
/key/list shape is per KeyListResponseObject in litellm/proxy/_types.py.
Not yet executed: ports 4000/8090 are held by a parallel run.
* test(e2e-ui): let the local harness run on non-default ports
Two checkouts cannot run run_e2e.sh at the same time: it hardcodes 4000/5432/
8090, so the second aborts on "port 4000 is in use" and the only way forward is
to stop someone else's stack.
PROXY_PORT / POSTGRES_PORT / MOCK_LLM_PORT now override those, defaulting to the
historical values so an unset environment behaves exactly as before -- CI, the
CircleCI job and the chart's sidecar all keep working untouched.
Two details that would otherwise make a relocated stack fail confusingly:
- the suite resolves its target from E2E_UI_BASE_URL, which defaults to :4000
independently, so the run would build and boot correctly and then test
whatever was on the default port. run_e2e.sh now derives it.
- the mock server binds its port in server.py, so moving it needs MOCK_LLM_PORT
there too. Its HOST stays loopback-only: 127.0.0.1:8090 from inside the
proxy's own pod is the contract the e2e chart's sidecar is written against.
* test(e2e-ui): cover MCP server edit and delete, verified via the API
mcpServers.spec.ts only ever creates a server, and creation is the one MCP
operation nobody has complained about. The reports are all on the other side:
an alias rename that needs three or four saves to take, a delete that needs two
attempts. Both produce a success toast on the failing attempt, so a toast-only
assertion cannot tell them from working software.
Rename asserts the new alias and the target server_id are on the PUT, then
polls /v1/mcp/server until the stored alias matches -- one save has to be
enough. Delete asserts the server is really gone from the list.
Points at an unreachable URL: these exercise litellm's persistence, never the
upstream, so a live MCP server would add a network dependency for nothing.
mcpTools.spec.ts is where a real upstream is needed.
Both pass against a local stack, as do the mcpTools specs from 5e189e9b1a.
Neither reproduced the reported failures on this build -- they guard, they did
not catch.
* test(e2e-ui): verify team create, invite and delete against the API
Three team mutations stopped at a toast, and one of those toasts is matched as
loosely as /success/i -- almost any notification satisfied it.
- create: the team is readable from /team/list and kept the models chosen in
the modal, rather than trusting the UI's own "Team created"
- invite: the invited address really appears in members_with_roles, which is
the point of the flow
- delete: the team is gone from /team/list. The existing assertion was that
the row vanished, which is the client dropping it from local state and
happens whether or not the delete reached the database.
Shapes read off a live proxy: /team/list is a bare array; /team/info nests the
record under team_info. All 6 tests pass locally.
* test(e2e-ui): verify team-admin member and key mutations against the API
The team-admin flows stopped at a success toast. A member add that lands on
the wrong team, a remove that takes out the wrong row, and a key that comes
back unscoped all produce the same toast as the working case, so the existing
assertions could not tell them apart.
Each mutation now pins what went on the wire and reads the result back:
member add/remove assert team_id and the member identifier on the request,
then poll /team/info's roster; the team key asserts team_id on /key/generate
and reads /key/list back to confirm the key is owned by the admin's own team
rather than orphaned.
* test(e2e-ui): verify model add and limit edits against the stored deployment
The Models specs checked the rendered result: the TPM/RPM edit asserted the
new numbers were visible in view mode, and the two add flows asserted a row
showed up in the table. Both render from state the UI already holds, so a save
the backend dropped and a save it kept look the same.
Each mutation now pins the request and reads the deployment back. The limits
edit also asserts the fields it did not touch -- upstream model and team
ownership -- are unchanged, because handleModelUpdate rebuilds and PATCHes the
whole litellm_params blob, which is how an unrelated field gets clobbered by a
save that reports success.
The two add flows assert model_name, the routed model and custom_llm_provider
on the wire and in storage; a deployment that loses its provider looks correct
in the table and is unroutable.
The Team-BYOK test is unchanged -- it is skipped without a license, so any
change to it would be unverified.
* test(e2e-ui): delete the MCP servers these specs create
MCP servers outlive the test that made them, the MCP page contacts every
server it lists, and most of the ones these specs create point at an
unreachable host. They accumulate, and each one makes navigateToPage's
networkidle wait a little slower to settle.
Measured on a local stack: with eleven leaked servers the whole MCP suite
failed on a 30s navigation timeout, including specs that leaked nothing.
Deleting the leftovers made all five pass. With per-test cleanup added, a
run from a clean slate leaves zero behind and takes 20s instead of 1m24s.
mcpServers.spec.ts carried a note that no teardown was needed because the
runner brings up a fresh database each time. That holds for CI and is why
this went unnoticed; it does not hold for a local stack that is reused.
* test(e2e-ui): say why the team-model setup call failed
The setup that creates a team-scoped model asserted a bare `ok()`, so a
failure read "expected true, received false" and pointed at the UI. The call
is enterprise-gated -- creating a model with model_info.team_id returns 403
without LITELLM_LICENSE -- and that is invisible from the old message. It now
carries the status and body, which names the cause immediately.
The api_base also pointed at the mock's default port rather than the one the
harness started; nothing in the test calls the model, but the two should not
disagree.
* test(e2e-ui): add a model through the UI and serve traffic with it
Every existing Add Model test stops at "the row appears in the table",
which a deployment that cannot serve a single request also does. The
manual-QA item this replaces is the whole loop: fill the form, pass Test
Connect, add it, confirm it works.
The new test ends by calling the model it just created. That is the only
assertion that rules out a dropped api_base, a mangled provider prefix,
or a name the router never registers -- all of which look identical in
the UI.
No provider credential is involved. OpenAI-Compatible is the provider
whose form exposes API Base, so the deployment points at the harness's
own mock LLM. The mock speaks the OpenAI wire format, so Test Connect
performs a real completion against a real endpoint and really succeeds.
Also adds teardown for the deployment it creates. A local run throws its
database away, but the deployed stack does not, and a leaked deployment
shows up in every later Models table and /v2/model/info readback.
Both new assertions were mutation-tested: pointing the traffic poll at a
name that was never created fails the test, and the wire assertion fails
when the typed name is not what reaches /model/new.
* test(e2e-ui): print the proxy log when the proxy dies on its own
In E2E_KEEP_ALIVE=1 mode the harness blocks until the proxy pid goes away, then
printed a bare "Proxy exited." and fell straight into cleanup, which rm -f's the
log. The proxy has now exited by itself twice, minutes after a run had finished,
leaving nothing to look at. Both startup failure paths already tail -n 100 the log
before giving up, so this was the one death that stayed silent
Dump the same 100 lines before exiting. A normal Ctrl-C teardown still deletes the
log and prints nothing, which is why INT and TERM now exit instead of running
cleanup and falling back into the wait loop: under the single trap a SIGTERM
deleted the log, resumed the loop, and would then report "tail: no such file",
besides running cleanup twice
* test(e2e-ui): split the log-drawer copy assertions off the expand test
The copy assertions need `navigator.clipboard`, which the browser only
exposes in a secure context. Locally the suite runs against
http://127.0.0.1 and localhost is trustworthy, so it is there. In CI the
run pod is pointed at a plain-HTTP cluster DNS name, where it is
undefined -- and InputCard.handleCopy calls writeText unguarded, so the
click throws before MessageManager.success and no toast ever renders.
That failed all three attempts of litellm-e2e-ui build 10.
Measured rather than inferred: on http://127.0.0.1:4100
isSecureContext/typeof navigator.clipboard are true/"object", and on a
DNS name resolving to that same address they are false/"undefined",
which reproduces the CI failure exactly.
Splitting keeps the drawer-rendering coverage running everywhere and
confines the skip to the part the browser has actually switched off. The
copy assertions still run in full wherever the origin is trustworthy.
The underlying product behaviour is left alone deliberately: any
deployment served over plain HTTP on a hostname has a copy button that
throws and gives no feedback, and that deserves its own fix rather than
being papered over from a test.
* test(e2e-ui): cut the added comments back to what the code cannot say itself
Greptile flagged the helper commentary, and it was right: CLAUDE.md says not to
write comments unless they explain very complex business logic, and much of what
was added here narrated ordinary test setup and motivation instead.
Trims 310 comment lines across the 14 files this branch touched. Kept only the
notes that record something unrecoverable from the code: why the request
listener is armed before the click, why a locator walks up the DOM, why an
assertion exists beyond the toast. Pre-existing comments are left alone.
No behaviour change. The only non-comment hunk is a prettier reformat.
* test(e2e): cover google-native generateContent framing and prometheus queue time
Adds live coverage for three shipped regressions that had none, all reached
through surfaces a customer drives from Google SDKs and operator dashboards.
The managed google-native route (`/v1beta/models/{model}:generateContent`) had
no harness support at all, so EndpointsClient gains generate_content and
stream_generate_content plus the request body models, and a new suite asserts
the two contracts that broke there: the response carries
x-litellm-response-cost so SDK traffic reconciles against spend (LIT-4076), and
the stream relays single-prefixed SSE frames with no OpenAI [DONE] terminator.
A doubled `data:` prefix, a leaked bytes literal, or the [DONE] sentinel each
fail the stream test; [DONE] absence is only asserted once real content has
arrived, because a first-chunk upstream error legitimately falls back to the
OpenAI error shape and does emit it.
The prometheus test pins litellm_request_queue_time_seconds to an actual
observation on our own key's series rather than to the family merely existing,
which is the distinction the original regression turned on: the histogram stayed
registered while nothing was ever written to it (LIT-2034).
Each assertion was mutation-checked against the live proxy; inverting the
[DONE] expectation, the cost-header expectation, or the metric name fails the
corresponding test.
* refactor(e2e): simplify google native coverage
* test(e2e): add failing reproducers for two open gateway bugs
Both tests assert the behavior a customer expects and both are red today. They
are reproducers, not regressions: the product is wrong, not the tests.
Native passthrough returns almost none of the operational headers the managed
route does. A /gemini/ generateContent call comes back with three x-litellm-*
headers and no x-ratelimit-* at all, against sixteen and four on
/v1beta/models/{m}:generateContent for the same prompt, and critically it omits
x-litellm-response-cost. Customers front provider-native traffic through this
route and read those headers to reconcile spend and pace themselves, so native
traffic is currently invisible to the tooling that covers every other route.
/budget/update rejects any model_max_budget with a 500. The reported symptom was
model ids containing dots, and that reproduces (prisma raises "Unexpected
`-5.2[FloatValue]` Expected `:`" because the key is interpolated into a GraphQL
query unquoted, so glm-5.2 lexes as an identifier followed by a float), but the
plain name gpt4o fails too, on a separate "model_max_budget should be of any of
the following types: Json" type mismatch at budget_management_endpoints.py:173.
Omitting the field returns 200. The test drives both names so the failure says
whether per-model budgets are broken outright or only for punctuated ids; today
it stops on the plain name, which is the wider bug.
* test(e2e): add reproducer for unenforced end-user per-model rate limits
model_max_budget accepts an rpm_limit alongside the spend cap, and /budget/new
stores it: the create response echoes {"gemini-2.5-flash": {"rpm_limit": 1,
"max_budget": 100.0, "budget_duration": "1d"}}. Attach that budget to an end
user, drive three calls as that user, and all three return 200. The limit is
accepted, persisted, and then ignored.
The same shape already works when the budget hangs off a key, which is what
makes this quietly dangerous: the API gives every indication the cap is in
force. A customer using it to hold one end user to a slow rate on a shared key
gets no throttling at all.
Harness additions this needs: ModelBudgetEntry carries the rpm_limit/tpm_limit
the route already accepts, BudgetNewBody and create_budget carry
model_max_budget, and create_customer can attach an existing budget_id rather
than only an inline max_budget.
Red today, for the reason in the assertion message.
* test(e2e): tighten model_max_budget reproducers and drop in-loop closure
Trim the reproducer docstrings to the contract they assert, keeping the
failure messages that document each red-by-design bug. Replace the nested
per-model closure in the /budget/update test with a module-level predicate
and a per-model helper so nothing closes over a loop variable, and fix the
import order the merge left unsorted.
* test(e2e): skip the three reproducers while their gateway bugs stay open
The passthrough header contract, /budget/update model_max_budget, and
end-user per-model rpm enforcement reproducers all still fail against
staging by design. Skip each with the product gap named so the combined
suite can gate merges on green while the collector keeps reporting the
cells as uncovered.
* test(e2e): validate model budget response contracts
* refactor(e2e): unify model budget schema
* refactor(e2e): reuse shared model budget type
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
The otel trace tests asserted that a streamed call produces exactly one gen-AI
span. The proxy opens one gen-AI span per upstream attempt, so a call the
router retried carries an error span for every failed attempt beside the one
that answered, and the assertion fails on a request that succeeded.
Select the served attempt instead: drop spans whose otel.status_code is ERROR,
require exactly one survivor, and run the TTFT and streaming-flag assertions
against it. That keeps what these assertions exist for, a split trace or a
stream logged as two served spans, while tolerating a retry.
Only the failed attempt lacks TTFT, so the old code also had a second failure
mode: when the first span happened to be the error one, the test reported the
attribute as missing rather than as belonging to a different attempt.
test_span_selection.py covers the selection itself against Jaeger-shaped
payloads and carries no e2e marker, since reproducing a first-attempt failure
live is not something a test can arrange.
Claude Code drives Opus 4.7 with thinking {"type": "adaptive"} plus
output_config {"effort": "max"}. The anthropic-to-openai adapter
forwarded thinking verbatim for Claude models but dropped output_config,
and Bedrock Converse streams zero reasoningContent blocks for adaptive
thinking without an effort tier. Forward the effort subset of
output_config for Bedrock targets, accept it in the converse supported
params, and map it with the model's effort ceiling applied. Re-enable
the skipped e2e compat cell that catches this
Bedrock InvokeModel rejects tool_search_tool_* tool types unless the
request body carries the tool-search-tool-2025-10-19 beta. The model
allowlist gating that beta omitted Haiku 4.5 (and Opus 4.7, supported
since launch per live verification), so every tool-search request on
those models got a Bedrock 400. Add both to the allowlist and re-enable
the e2e compat cell that caught it.
The mateo-berri PAT now arrives via systemd LoadCredential as a file
instead of the EnvironmentFile, so pytest, the proxy, and the
model-driven claude CLI never inherit it and a same-UID /proc read
cannot lift it. run_daily.sh reads the credential when present, still
accepts an exported GITHUB_TOKEN for manual runs, and dies up front
when publishing is enabled with neither. Full CLI sandboxing is
tracked in LIT-5420
The existing web_search cells drive Claude Code's client-side WebSearch
tool, which the CLI executes itself and feeds back as a tool_result. The
CLI never emits a web_search_20250305 definition, so those cells stayed
green while the Anthropic-managed server tool 400'd on Bedrock.
Add a cell that posts the server tool to a Bedrock deployment over
/v1/messages and asserts a web_search_tool_result block comes back, and
reword the compat row so it no longer reads as coverage of the server
tool. Model the server tool as a composed base shared with tool_search.
Resolves LIT-5391
Ports the daily cron VM publisher from the unmerged tests/claude_code
checkout so the automation runs the e2e suite from litellm_internal_staging.
Adds find_regressions to matrix_builder for the green to red auto-merge
gate, pins the cron venv to Python 3.12, and ships the systemd units, env
template, and runbook alongside
The suite already waits for a new model or agent to become servable before
handing it back, but that wait returns on the first successful read. Every
request opens a fresh connection (e2e_http calls requests.* with no Session), so
a load-balanced Service routes each one independently: one successful read proves
one replica converged, and the caller's next request re-rolls and can land on a
replica that has not reloaded yet.
At replicaCount: 2 this surfaced as 30 failures on a SHA that is green at 1
replica -- 400 "Invalid model name passed", 404 "Guardrail not found", "no
healthy deployments for this model", and a /model/info listing that contained
one of two models created moments apart.
Add PROPAGATION_TIMEOUT (default 15s, override E2E_PROPAGATION_TIMEOUT) and
settle_propagation(), sized off the proxy's proxy_config_reload_interval_seconds
(30s by default, 7s on the e2e stack) plus margin, and settle after every
control-plane create whose object the suite then uses:
- ProxyClient.create_model and A2AClient.register_agent, after their existing
polls -- the poll still fails loudly if the object never appears at all
- GuardrailsClient.register, which had no barrier; create_content_filter_guardrail
and create_bedrock_guardrail now route through it instead of POSTing directly
- the guardrail creates in mcp_client and logging_client
- the vertex passthrough model, whose body cannot go through create_model
Left alone: the /model/new calls that assert a 403 or read back a status code,
since they never use the model.
