Rebuilds the thirteen form-free components under common_components on
the in-repo shadcn layer, so they inherit the dashboard's design tokens
instead of styling themselves through Ant Design and Tremor.
SearchSelect and the three dropdowns that wrap it now forward an
optional input id, so an antd Form.Item label still resolves to its
control. The e2e steps that reached into antd's Select and Modal
internals now go through the test id, role and data-slot.
Rebuilds public_model_hub, MakeSkillPublicForm, ModelSelect and the
guardrail LogViewer on the in-repo shadcn layer, so they inherit the
dashboard's design tokens instead of styling themselves through Ant
Design and Tremor.
Public prop signatures are unchanged, so no caller moves. The two
teams e2e steps that reached into antd's Select internals now drive
the combobox through its test id, role and data-slot instead.
Every case in TestAccessControl asserted that something was refused. A gateway
that denied the allow-listed model too would have passed all of them, so the
suite could not tell "denied correctly" from "broken outright".
Adds the positive half: a key allow-listed for gemini-2.5-flash can call it and
gets back a real completion rather than a 200-wrapped error.
Also tightens the unknown-model case. It accepted any valid JSON, so a bare
"{}" or even "null" satisfied it. It now requires the OpenAI-shaped error
envelope with a message a client can actually surface, parsed through a typed
model instead of json.loads.
* test(e2e-ui): cover the Playground, Logs and Usage manual-QA flows
These three pages carried no e2e coverage, so the manual QA checklist was the
only thing standing behind them.
Playground: sends a chat from the UI for both configured models, and for both
virtual-key sources (the logged-in session, and a key pasted into the panel).
This is the only spec that drives the dashboard's own LLM call path rather than
an admin CRUD endpoint.
Logs: a request the proxy actually served appears in the table, its drawer
expands to the real request and response bodies, both copy to the clipboard,
the Input card collapses, the JSON view exposes Request/Response, and the End
User filter narrows the table to one customer.
Usage: traffic billed to a virtual key reaches Top Virtual Keys, the card
toggles between table and chart, and the key opens its key-info panel.
Router settings: the existing spec proved the UI can record a fallback; the new
one proves the fallback is honoured, by pointing a model at an unreachable
upstream and asserting the reply comes back anyway. It asserts the un-fallen-back
call fails first, so a quietly-working primary cannot fake a pass.
Supporting changes:
- helpers/traffic.ts generates the traffic these pages render, rather than
seeding rows no code produced. Its two wait helpers exist because the Logs
and Usage pages read different stores: spend logs are flushed on a timer, and
the Usage page reads a background rollup *and* fetches once on mount, so
waiting on the DOM there can never converge.
- helpers/playground.ts holds the playground controls, now shared with the
fallback spec. Everything is scoped to the visible copy of the config panel,
which is rendered twice for the docked and collapsed layouts.
- run_e2e.sh gains E2E_KEEP_ALIVE=1, which brings the stack up and blocks so a
spec can be re-run against it without paying for a UI rebuild each iteration.
Verified with the full suite on a fresh stack: 89 passed, 0 failed, 5 skipped.
* test(e2e-ui): cover listing and calling MCP tools
Covers the two MCP manual-QA items the create-only spec cannot reach:
opening a server's tool list, and calling a tool and seeing its result.
Both need an MCP server that actually answers, so this points at DeepWiki's
public MCP server -- Streamable HTTP, auth None, so there is no credential to
hold and nothing to leak from a public repo. The call is made by the proxy,
not the browser; nothing in the e2e chart restricts that egress.
The external dependency is real and is left visible: an upstream outage turns
these red rather than auto-skipping, because a spec that skips itself on
connection trouble also skips when the proxy's MCP client is what broke.
E2E_SKIP_EXTERNAL_MCP=1 is the explicit opt-out.
Not yet executed against a live stack.
* test(e2e-ui): verify key mutations round-trip instead of trusting the toast
The recurring customer report is a form that says "Saved!" and then either
no-ops or clobbers an unrelated field. A toast-only assertion passes in both
cases, and outside three specs that is all this suite checks.
Adds helpers/roundTrip.ts, factoring out the idiom clearCustomPricing,
credentials and routerSettings already use: capture the outgoing request body,
then read the resource back through the management API.
Applies it to the keys spec:
- create: the key is readable from /key/list and owns a team_id, rather than
trusting a table row rendered from the create response the UI already held
- update limits: TPM/RPM are on the wire AND persisted, and the key's models
and team are unchanged -- bumping one field wiping another is the reported
failure mode (PR #34452), not a hypothetical
- delete: the key is gone from /key/list, not merely toasted as deleted
- regenerate: the stored token actually changed
/key/list shape is per KeyListResponseObject in litellm/proxy/_types.py.
Not yet executed: ports 4000/8090 are held by a parallel run.
* test(e2e-ui): let the local harness run on non-default ports
Two checkouts cannot run run_e2e.sh at the same time: it hardcodes 4000/5432/
8090, so the second aborts on "port 4000 is in use" and the only way forward is
to stop someone else's stack.
PROXY_PORT / POSTGRES_PORT / MOCK_LLM_PORT now override those, defaulting to the
historical values so an unset environment behaves exactly as before -- CI, the
CircleCI job and the chart's sidecar all keep working untouched.
Two details that would otherwise make a relocated stack fail confusingly:
- the suite resolves its target from E2E_UI_BASE_URL, which defaults to :4000
independently, so the run would build and boot correctly and then test
whatever was on the default port. run_e2e.sh now derives it.
- the mock server binds its port in server.py, so moving it needs MOCK_LLM_PORT
there too. Its HOST stays loopback-only: 127.0.0.1:8090 from inside the
proxy's own pod is the contract the e2e chart's sidecar is written against.
* test(e2e-ui): cover MCP server edit and delete, verified via the API
mcpServers.spec.ts only ever creates a server, and creation is the one MCP
operation nobody has complained about. The reports are all on the other side:
an alias rename that needs three or four saves to take, a delete that needs two
attempts. Both produce a success toast on the failing attempt, so a toast-only
assertion cannot tell them from working software.
Rename asserts the new alias and the target server_id are on the PUT, then
polls /v1/mcp/server until the stored alias matches -- one save has to be
enough. Delete asserts the server is really gone from the list.
Points at an unreachable URL: these exercise litellm's persistence, never the
upstream, so a live MCP server would add a network dependency for nothing.
mcpTools.spec.ts is where a real upstream is needed.
Both pass against a local stack, as do the mcpTools specs from 5e189e9b1a.
Neither reproduced the reported failures on this build -- they guard, they did
not catch.
* test(e2e-ui): verify team create, invite and delete against the API
Three team mutations stopped at a toast, and one of those toasts is matched as
loosely as /success/i -- almost any notification satisfied it.
- create: the team is readable from /team/list and kept the models chosen in
the modal, rather than trusting the UI's own "Team created"
- invite: the invited address really appears in members_with_roles, which is
the point of the flow
- delete: the team is gone from /team/list. The existing assertion was that
the row vanished, which is the client dropping it from local state and
happens whether or not the delete reached the database.
Shapes read off a live proxy: /team/list is a bare array; /team/info nests the
record under team_info. All 6 tests pass locally.
* test(e2e-ui): verify team-admin member and key mutations against the API
The team-admin flows stopped at a success toast. A member add that lands on
the wrong team, a remove that takes out the wrong row, and a key that comes
back unscoped all produce the same toast as the working case, so the existing
assertions could not tell them apart.
Each mutation now pins what went on the wire and reads the result back:
member add/remove assert team_id and the member identifier on the request,
then poll /team/info's roster; the team key asserts team_id on /key/generate
and reads /key/list back to confirm the key is owned by the admin's own team
rather than orphaned.
* test(e2e-ui): verify model add and limit edits against the stored deployment
The Models specs checked the rendered result: the TPM/RPM edit asserted the
new numbers were visible in view mode, and the two add flows asserted a row
showed up in the table. Both render from state the UI already holds, so a save
the backend dropped and a save it kept look the same.
Each mutation now pins the request and reads the deployment back. The limits
edit also asserts the fields it did not touch -- upstream model and team
ownership -- are unchanged, because handleModelUpdate rebuilds and PATCHes the
whole litellm_params blob, which is how an unrelated field gets clobbered by a
save that reports success.
The two add flows assert model_name, the routed model and custom_llm_provider
on the wire and in storage; a deployment that loses its provider looks correct
in the table and is unroutable.
The Team-BYOK test is unchanged -- it is skipped without a license, so any
change to it would be unverified.
* test(e2e-ui): delete the MCP servers these specs create
MCP servers outlive the test that made them, the MCP page contacts every
server it lists, and most of the ones these specs create point at an
unreachable host. They accumulate, and each one makes navigateToPage's
networkidle wait a little slower to settle.
Measured on a local stack: with eleven leaked servers the whole MCP suite
failed on a 30s navigation timeout, including specs that leaked nothing.
Deleting the leftovers made all five pass. With per-test cleanup added, a
run from a clean slate leaves zero behind and takes 20s instead of 1m24s.
mcpServers.spec.ts carried a note that no teardown was needed because the
runner brings up a fresh database each time. That holds for CI and is why
this went unnoticed; it does not hold for a local stack that is reused.
* test(e2e-ui): say why the team-model setup call failed
The setup that creates a team-scoped model asserted a bare `ok()`, so a
failure read "expected true, received false" and pointed at the UI. The call
is enterprise-gated -- creating a model with model_info.team_id returns 403
without LITELLM_LICENSE -- and that is invisible from the old message. It now
carries the status and body, which names the cause immediately.
The api_base also pointed at the mock's default port rather than the one the
harness started; nothing in the test calls the model, but the two should not
disagree.
* test(e2e-ui): add a model through the UI and serve traffic with it
Every existing Add Model test stops at "the row appears in the table",
which a deployment that cannot serve a single request also does. The
manual-QA item this replaces is the whole loop: fill the form, pass Test
Connect, add it, confirm it works.
The new test ends by calling the model it just created. That is the only
assertion that rules out a dropped api_base, a mangled provider prefix,
or a name the router never registers -- all of which look identical in
the UI.
No provider credential is involved. OpenAI-Compatible is the provider
whose form exposes API Base, so the deployment points at the harness's
own mock LLM. The mock speaks the OpenAI wire format, so Test Connect
performs a real completion against a real endpoint and really succeeds.
Also adds teardown for the deployment it creates. A local run throws its
database away, but the deployed stack does not, and a leaked deployment
shows up in every later Models table and /v2/model/info readback.
Both new assertions were mutation-tested: pointing the traffic poll at a
name that was never created fails the test, and the wire assertion fails
when the typed name is not what reaches /model/new.
* test(e2e-ui): print the proxy log when the proxy dies on its own
In E2E_KEEP_ALIVE=1 mode the harness blocks until the proxy pid goes away, then
printed a bare "Proxy exited." and fell straight into cleanup, which rm -f's the
log. The proxy has now exited by itself twice, minutes after a run had finished,
leaving nothing to look at. Both startup failure paths already tail -n 100 the log
before giving up, so this was the one death that stayed silent
Dump the same 100 lines before exiting. A normal Ctrl-C teardown still deletes the
log and prints nothing, which is why INT and TERM now exit instead of running
cleanup and falling back into the wait loop: under the single trap a SIGTERM
deleted the log, resumed the loop, and would then report "tail: no such file",
besides running cleanup twice
* test(e2e-ui): split the log-drawer copy assertions off the expand test
The copy assertions need `navigator.clipboard`, which the browser only
exposes in a secure context. Locally the suite runs against
http://127.0.0.1 and localhost is trustworthy, so it is there. In CI the
run pod is pointed at a plain-HTTP cluster DNS name, where it is
undefined -- and InputCard.handleCopy calls writeText unguarded, so the
click throws before MessageManager.success and no toast ever renders.
That failed all three attempts of litellm-e2e-ui build 10.
Measured rather than inferred: on http://127.0.0.1:4100
isSecureContext/typeof navigator.clipboard are true/"object", and on a
DNS name resolving to that same address they are false/"undefined",
which reproduces the CI failure exactly.
Splitting keeps the drawer-rendering coverage running everywhere and
confines the skip to the part the browser has actually switched off. The
copy assertions still run in full wherever the origin is trustworthy.
The underlying product behaviour is left alone deliberately: any
deployment served over plain HTTP on a hostname has a copy button that
throws and gives no feedback, and that deserves its own fix rather than
being papered over from a test.
* test(e2e-ui): cut the added comments back to what the code cannot say itself
Greptile flagged the helper commentary, and it was right: CLAUDE.md says not to
write comments unless they explain very complex business logic, and much of what
was added here narrated ordinary test setup and motivation instead.
Trims 310 comment lines across the 14 files this branch touched. Kept only the
notes that record something unrecoverable from the code: why the request
listener is armed before the click, why a locator walks up the DOM, why an
assertion exists beyond the toast. Pre-existing comments are left alone.
No behaviour change. The only non-comment hunk is a prettier reformat.
* test(e2e): cover google-native generateContent framing and prometheus queue time
Adds live coverage for three shipped regressions that had none, all reached
through surfaces a customer drives from Google SDKs and operator dashboards.
The managed google-native route (`/v1beta/models/{model}:generateContent`) had
no harness support at all, so EndpointsClient gains generate_content and
stream_generate_content plus the request body models, and a new suite asserts
the two contracts that broke there: the response carries
x-litellm-response-cost so SDK traffic reconciles against spend (LIT-4076), and
the stream relays single-prefixed SSE frames with no OpenAI [DONE] terminator.
A doubled `data:` prefix, a leaked bytes literal, or the [DONE] sentinel each
fail the stream test; [DONE] absence is only asserted once real content has
arrived, because a first-chunk upstream error legitimately falls back to the
OpenAI error shape and does emit it.
The prometheus test pins litellm_request_queue_time_seconds to an actual
observation on our own key's series rather than to the family merely existing,
which is the distinction the original regression turned on: the histogram stayed
registered while nothing was ever written to it (LIT-2034).
Each assertion was mutation-checked against the live proxy; inverting the
[DONE] expectation, the cost-header expectation, or the metric name fails the
corresponding test.
* refactor(e2e): simplify google native coverage
* test(e2e): add failing reproducers for two open gateway bugs
Both tests assert the behavior a customer expects and both are red today. They
are reproducers, not regressions: the product is wrong, not the tests.
Native passthrough returns almost none of the operational headers the managed
route does. A /gemini/ generateContent call comes back with three x-litellm-*
headers and no x-ratelimit-* at all, against sixteen and four on
/v1beta/models/{m}:generateContent for the same prompt, and critically it omits
x-litellm-response-cost. Customers front provider-native traffic through this
route and read those headers to reconcile spend and pace themselves, so native
traffic is currently invisible to the tooling that covers every other route.
/budget/update rejects any model_max_budget with a 500. The reported symptom was
model ids containing dots, and that reproduces (prisma raises "Unexpected
`-5.2[FloatValue]` Expected `:`" because the key is interpolated into a GraphQL
query unquoted, so glm-5.2 lexes as an identifier followed by a float), but the
plain name gpt4o fails too, on a separate "model_max_budget should be of any of
the following types: Json" type mismatch at budget_management_endpoints.py:173.
Omitting the field returns 200. The test drives both names so the failure says
whether per-model budgets are broken outright or only for punctuated ids; today
it stops on the plain name, which is the wider bug.
* test(e2e): add reproducer for unenforced end-user per-model rate limits
model_max_budget accepts an rpm_limit alongside the spend cap, and /budget/new
stores it: the create response echoes {"gemini-2.5-flash": {"rpm_limit": 1,
"max_budget": 100.0, "budget_duration": "1d"}}. Attach that budget to an end
user, drive three calls as that user, and all three return 200. The limit is
accepted, persisted, and then ignored.
The same shape already works when the budget hangs off a key, which is what
makes this quietly dangerous: the API gives every indication the cap is in
force. A customer using it to hold one end user to a slow rate on a shared key
gets no throttling at all.
Harness additions this needs: ModelBudgetEntry carries the rpm_limit/tpm_limit
the route already accepts, BudgetNewBody and create_budget carry
model_max_budget, and create_customer can attach an existing budget_id rather
than only an inline max_budget.
Red today, for the reason in the assertion message.
* test(e2e): tighten model_max_budget reproducers and drop in-loop closure
Trim the reproducer docstrings to the contract they assert, keeping the
failure messages that document each red-by-design bug. Replace the nested
per-model closure in the /budget/update test with a module-level predicate
and a per-model helper so nothing closes over a loop variable, and fix the
import order the merge left unsorted.
* test(e2e): skip the three reproducers while their gateway bugs stay open
The passthrough header contract, /budget/update model_max_budget, and
end-user per-model rpm enforcement reproducers all still fail against
staging by design. Skip each with the product gap named so the combined
suite can gate merges on green while the collector keeps reporting the
cells as uncovered.
* test(e2e): validate model budget response contracts
* refactor(e2e): unify model budget schema
* refactor(e2e): reuse shared model budget type
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
The otel trace tests asserted that a streamed call produces exactly one gen-AI
span. The proxy opens one gen-AI span per upstream attempt, so a call the
router retried carries an error span for every failed attempt beside the one
that answered, and the assertion fails on a request that succeeded.
Select the served attempt instead: drop spans whose otel.status_code is ERROR,
require exactly one survivor, and run the TTFT and streaming-flag assertions
against it. That keeps what these assertions exist for, a split trace or a
stream logged as two served spans, while tolerating a retry.
Only the failed attempt lacks TTFT, so the old code also had a second failure
mode: when the first span happened to be the error one, the test reported the
attribute as missing rather than as belonging to a different attempt.
test_span_selection.py covers the selection itself against Jaeger-shaped
payloads and carries no e2e marker, since reproducing a first-attempt failure
live is not something a test can arrange.
Claude Code drives Opus 4.7 with thinking {"type": "adaptive"} plus
output_config {"effort": "max"}. The anthropic-to-openai adapter
forwarded thinking verbatim for Claude models but dropped output_config,
and Bedrock Converse streams zero reasoningContent blocks for adaptive
thinking without an effort tier. Forward the effort subset of
output_config for Bedrock targets, accept it in the converse supported
params, and map it with the model's effort ceiling applied. Re-enable
the skipped e2e compat cell that catches this
Bedrock InvokeModel rejects tool_search_tool_* tool types unless the
request body carries the tool-search-tool-2025-10-19 beta. The model
allowlist gating that beta omitted Haiku 4.5 (and Opus 4.7, supported
since launch per live verification), so every tool-search request on
those models got a Bedrock 400. Add both to the allowlist and re-enable
the e2e compat cell that caught it.
The mateo-berri PAT now arrives via systemd LoadCredential as a file
instead of the EnvironmentFile, so pytest, the proxy, and the
model-driven claude CLI never inherit it and a same-UID /proc read
cannot lift it. run_daily.sh reads the credential when present, still
accepts an exported GITHUB_TOKEN for manual runs, and dies up front
when publishing is enabled with neither. Full CLI sandboxing is
tracked in LIT-5420
The existing web_search cells drive Claude Code's client-side WebSearch
tool, which the CLI executes itself and feeds back as a tool_result. The
CLI never emits a web_search_20250305 definition, so those cells stayed
green while the Anthropic-managed server tool 400'd on Bedrock.
Add a cell that posts the server tool to a Bedrock deployment over
/v1/messages and asserts a web_search_tool_result block comes back, and
reword the compat row so it no longer reads as coverage of the server
tool. Model the server tool as a composed base shared with tool_search.
Resolves LIT-5391
Ports the daily cron VM publisher from the unmerged tests/claude_code
checkout so the automation runs the e2e suite from litellm_internal_staging.
Adds find_regressions to matrix_builder for the green to red auto-merge
gate, pins the cron venv to Python 3.12, and ships the systemd units, env
template, and runbook alongside
The suite already waits for a new model or agent to become servable before
handing it back, but that wait returns on the first successful read. Every
request opens a fresh connection (e2e_http calls requests.* with no Session), so
a load-balanced Service routes each one independently: one successful read proves
one replica converged, and the caller's next request re-rolls and can land on a
replica that has not reloaded yet.
At replicaCount: 2 this surfaced as 30 failures on a SHA that is green at 1
replica -- 400 "Invalid model name passed", 404 "Guardrail not found", "no
healthy deployments for this model", and a /model/info listing that contained
one of two models created moments apart.
Add PROPAGATION_TIMEOUT (default 15s, override E2E_PROPAGATION_TIMEOUT) and
settle_propagation(), sized off the proxy's proxy_config_reload_interval_seconds
(30s by default, 7s on the e2e stack) plus margin, and settle after every
control-plane create whose object the suite then uses:
- ProxyClient.create_model and A2AClient.register_agent, after their existing
polls -- the poll still fails loudly if the object never appears at all
- GuardrailsClient.register, which had no barrier; create_content_filter_guardrail
and create_bedrock_guardrail now route through it instead of POSTing directly
- the guardrail creates in mcp_client and logging_client
- the vertex passthrough model, whose body cannot go through create_model
Left alone: the /model/new calls that assert a 403 or read back a status code,
since they never use the model.
This reverts commit dcb4e5033c.
The suites landed without the proof-of-fix and QA runbook the PR body
itself flagged as outstanding, so the coverage they claim is unverified
against a live proxy
* test(e2e): retry provider-transient statuses at the transport with bounded backoff
The Anthropic passthrough cost test failed a full-suite run on a real 529
overloaded_error. Passthrough routes forward provider responses verbatim
and bypass the router's num_retries, so provider blips reach the harness
only on those paths. Following standard practice, the retry is scoped to
the dependency boundary instead of rerunning tests: only the enumerated
transient statuses (500/502/503/504/529, the set production SDKs retry by
default) are retried, with bounded exponential backoff and a printed line
per retry so flakiness stays visible in run logs.
429 is deliberately excluded: the quota suites assert the proxy's own
rate-limit and budget 429s, and a transport that absorbed them would break
those tests. Network errors and timeouts are not retried either, so a hang
surfaces as a hang. request_with_retry takes injected callables, and the
new harness tests pin the contract with protocol fakes, no monkeypatching
* test(e2e): narrow the transport retry to 529, the one status the proxy cannot emit
Greptile's review is right that status-only classification could absorb an
intermittently failing proxy: at the transport a 500/502/503/504 from the
proxy is indistinguishable from one it relayed, and the proxy is the system
under test. 529 is the only status litellm provably never originates
(Anthropic's overload signal, forwarded verbatim on passthrough) and the
only transient observed across the full-suite runs, so the set shrinks to
exactly that. The canary tests now also pin 500/502/503/504 as never
retried
The Locust throughput SLO test is a different testing category from
functional e2e (variance-driven, historically flaky, currently
skip-annotated against LIT-5119) and erodes trust in the suite as a
release gate; it comes out of the default collection along with its
exclusive plumbing (locustfile, load-mock registration fixtures,
run_chat_load). Re-implementation as its own pipeline is tracked in
LIT-5163. The weekly session-anomaly test never ran in the suite (opt-in
via E2E_WEEKLY_ANOMALY, driven by its own workflow) and stays, as do the
markerless aggregation unit tests.
The vllm passthrough test read-times-out (60s) against the shared
vllm-cpu backend in every run on the per-SHA e2e stack; it is removed
until LIT-5164 establishes whether that is backend capacity or a
passthrough defect. Its registry cells return to the gap list, which is
the honest state
* test(e2e): cover legacy text /completions endpoint
The /completions (and /v1/completions) text-completion route had zero e2e
coverage despite being the second-busiest endpoint in production; everything
'completions' in the suite was chat. Add a text-completion endpoint test that
registers an OpenAI instruct deployment, drives /v1/completions through the
gateway, and asserts real generated text. Adds text_completions() + the
completion request/result models to EndpointsClient, the 'completions' endpoint
to the coverage registry vocab, and the registry cell.
* test(e2e): assert /v1/completions choices shape, not just joined text
Assert the response carries a choices array and the first choice has real text,
so a malformed response (no choices) and a clean-but-empty completion are
distinct failures. Drop the unused text property / id / model fields (model only
what the test reads).
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
* fix(e2e): address greptile and veria review on vendor strategy suite
Store search tool keys as os.environ refs and resolve them in SearchAPIRouter.
Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells
* fix(e2e): drop search_api_router os.environ expansion from vendor suite
Keep the PR test-only. Search tools register without an api_key so the
proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a.
* test(e2e): drop search e2e suite from vendor strategy PR
Remove the /v1/search coverage file and its registry rows so this PR
no longer carries search endpoint testing.
Pure rename, no behavior change. create_mcp_server.tsx and its test move
to CreateMCPServer, the two importers and one stale e2e comment follow,
and the local/filename-pascal-case suppression drops now that the file
passes the rule on its own.
The rename is scoped to this one component rather than the whole
directory because three PRs are currently open against its snake_case
siblings; the rest can follow once those land.
The strict-priority e2e (added with the zero-increment limiter fix) can
never pass on stage: the proxy there does not run the
dynamic_rate_limiter_v3 callbacks + priority_reservation settings the
module requires, confirmed by zero limiter log lines across every
gateway and backend pod during the 2026-08-02 run. Config lives in the
infra repo; LIT-5118 tracks adding it.
The throughput SLO test failed the same run with 65.9% of requests dying
at the ELB as 502/503 before reaching a pod. The per-replica SLO rework
fixed the RPS-floor assertion but cannot help when stage idles at one
warm gateway replica; LIT-5119 tracks pre-scaling the fleet for the load
phase.
Both skips name their ticket, and the coverage registry returns the two
cells to the gap list while they are in place.
A single read of key_info.spend races the batched spend writer: deltas
earned before a reset flush to the DB up to ~60s later
(proxy_batch_write_at) and land on the row after the reset zeroed it.
The stage runs on Jul 30 and Aug 2 failed
test_key_budget_reset_at_advances_after_window exactly this way, with
spend back at the driven total while budget_reset_at had advanced and
calls flowed again.
Replace the single reads in rung 3 (spend zeroed after reset) and rung 4
(roomy window keeps spend) with _poll_key_spend, which re-reads to a 90s
deadline covering one full flush-plus-reset cycle. A reset that never
zeroes the row keeps spend pinned and still times out, so the regression
guard keeps its teeth.
A keyless internal user signing in to the Admin UI was redirected off the
post-login landing to /ui/connect, which renders nothing but the MCP apps panel,
so a plain gateway sign-in ended on an MCP OAuth surface the user never asked
for. The landing now renders the keys dashboard for every role. The key lookup
that existed only to make that routing decision goes with it, along with the
useKeys enabled flag it was the sole caller of and the role-hydration hold that
guarded its one-frame dashboard flash
The gateway DCR consent flow moves the other way. Its /authorize handed the
browser to /ui/chat/integrations, whose layout hard-blocks when enable_chat_ui
is off, which is the default, and client-side redirects to /ui/ without the
query string; that destroys the connect_flow handle and strands the MCP client
until the 600s flow cookie expires. It now lands on /ui/connect, which reads
connect_flow and connect_client, mounts the consent banner and puts the apps
panel in connect mode. /ui/chat/integrations keeps its connect-mode handling
this release so flows sealed before the deploy still finish
Resolves LIT-5104
Resolves LIT-4911
The floor was an absolute fleet number, so it asserted replicas x per-replica rate
and went red on how many gateway pods happened to be warm rather than on the request
path. The test now measures one replica first, with a short serial pass that only ever
occupies a single pod, and requires the concurrent phase to reach at least that rate.
A serial latency budget carries the request-path assertion the floor used to imply,
and both hold at one replica or seven.
Zero-error runs that "sustained 16.7 RPS" were queueing, not slow requests: the load
model is a mock_response deployment with no upstream, a single-worker replica serves
it in about 57ms, and 100 closed-loop users against 1/0.057 RPS of capacity sit at
6s each by Little's law.
The runner also kept locust's --json summary and threw away everything else, so a run
where 93% of requests failed said nothing about what they got. It now passes --csv,
reads the failure breakdown back, and reports locust's own generator-saturation
warnings, both folded into the assertion messages.
Resolves LIT-5054
The atomic check-and-increment path skipped any counter whose increment
was <= 0. The dynamic rate limiter always passes a zero token increment
pre-call because usage lands on the counters post-response, so on a model
configured with only tpm the limiter evaluated no counters at all: no
model-wide TPM cap and no priority reservation, in either generous or
strict mode. Regressed in dd57ae6691 when the pre-call flow moved off the
read-only should_rate_limit check, which did evaluate token limits.
Keep zero-increment counters in the payload so they act as a pure check
(current + 0 > limit), matching the pre-regression semantics in both the
Lua and in-memory paths. Adds unit regressions at the primitive and hook
level plus a live e2e covering the priority_generous/priority_strict
registry rows.
* feat(proxy): add a generic list contract for management/v1 entity lists
Paging, sorting, filtering and search for an entity collection, declared once
as a ListSpec and served by handle_list. The route injects a ListExecutor that
owns its table, so this module never imports Prisma.
The caller's scope is derived from the caller alone and ANDed with whatever
they filtered on, so a query parameter can only narrow what they may read.
This is the shared half of the budgets list; it lands here so the endpoint has
something to register against, and drops out when the framework arrives on its
own branch.
* feat(proxy): add GET /management/v1/budgets
The Budgets page reads /budget/list, which returns the whole table as a bare
array with no way to page, sort or filter it. A customer with enough budgets to
fill the page has no way to find one.
Registers LiteLLM_BudgetTable against the management/v1 list contract: sortable
on budget_id, max_budget, tpm_limit, rpm_limit and created_at, default order
newest-first with budget_id breaking ties, search on budget_id, and filters for
budget_duration, max_budget and created_at. budget_duration is deliberately not
sortable; the column holds "7d"/"30d" strings, so a lexicographic ORDER BY puts
"30d" ahead of "7d".
tpm_limit and rpm_limit are BigInt? in Prisma, so rows validate through a
pydantic model on the way out and serialize as JSON numbers.
A caller without admin view is refused 403 as a problem document rather than
served an empty page. /budget/list is untouched.
* fix(proxy): rework the budgets list onto the merged list contract
PR #35308 landed a different shape than this branch was written against: `where`
is a tuple of frozen predicates rather than a Prisma-shaped mapping, `ListSpec`
carries both the row and the wire type, and `where_sql` / `order_by_sql` render
for a raw-SQL executor. The budgets executor now queries through `query_raw` the
way the spend logs facet does, selecting only the columns it serves.
Also casts datetime binds in `where_sql`. They cross into the query engine as
JSON, so an uncast placeholder arrives as text and Postgres refuses
`timestamp >= text` outright; every `filter[created_at][gte|lte]` was answering
500. The cast reads the bind as an instant and drops it to naive UTC to match
Prisma's TIMESTAMP(3) column, the same one /spend/logs/ui applies.
* refactor(proxy): fold the predicate renderer instead of recursing
recursive_detector flags `_render_all`, and the flag is fair: it recursed once
per predicate, so the stack grew with the number of filters on the request for
no reason. Walking a predicate list is a running bind index, which is a fold.
`_render` still re-enters for `AnyOf`, but its clauses are plain comparisons
built by `?q=`, so that nesting is one level deep and no caller can drive it
deeper.
The SLO measures how many gateway replicas happen to be warm rather than the
request path. Clearing the floor needs roughly 5-7 replicas at ~10-14 RPS each;
stage idles at one and reactive HPA scale-up lands minutes into a ~3 minute
test.
It failed both of its assertions on consecutive days: 93.3% errors at an
inflated 264 RPS, where the failing requests never reached a pod and closed-loop
RPS rose because they failed fast, then 16.7 RPS with zero failures.
The covers marker and registry row stay put, so the cell returns to the gap
list rather than disappearing from the denominator.
All three send a `telemetry` object in the arguments to Datadog's
search_datadog_logs tool. Datadog tightened that tool's input schema to reject
unknown properties, so every call now fails validation with 'unexpected
additional properties ["telemetry"]' before the behavior each test exists to
prove is reached.
`telemetry` was never a documented Datadog parameter; the tests relied on the
server ignoring extra properties. The proxy transmitted exactly what the tests
supplied and surfaced the upstream error faithfully, so this is test-side.
The covers markers and registry rows stay put: the collector counts a cell as
covered only when a test pytest would actually run declares it, so skipping
hands all four cells back to the gap list where they belong.
The collector read @pytest.mark.covers off every collected item, and
collection does not evaluate skips, so a test carrying both a skip and a
covers marker reported its cell as covered while asserting nothing. 17
files under tests/e2e do exactly that, which inflated the headline from
290/434 to 311/434.
A cell now counts as covered only when at least one test pytest would
actually run declares it; a cell claimed by both a live and a skipped
test stays covered. Skip state comes from pytest's own evaluator, so
skip, skipif (bool and string conditions), and module-level pytestmark
resolve exactly as they do in the e2e run. Cells left uncovered this way
are listed under the headline and exported as skipped_markers (JSON) and
litellm_e2e_coverage_skipped_markers (Prometheus) so the gap surfaces
instead of disappearing; the Loki line contract is unchanged. A marker
on a skipped test that points outside the registry is still an orphan,
so --strict keeps its reach.
Because skipif resolves against the environment the collector runs in,
the number now depends on that environment; run it where the e2e suite
runs. A pytest.skip() call inside a test body remains invisible to a
static pass, which the module docstring and README both state.
The batch rate limiter counts input tokens by awaiting litellm.afile_content
with no timeout, so a slow Files API holds POST /v1/batches open past any
client deadline; stage saw 63.6s against the harness's 60s read timeout. The
test times out before reaching the unattributed-spend-row assertion it exists
to guard, so it reports an infrastructure hang rather than the contract.
Skipping keeps the signal honest until the fetch is bounded.
