create-release.yml is back to calling the chaos test through no mechanism at
all; it never called it. Also drops the E2E_REDIS_CHAOS opt-in gate itself:
the redis_chaos marker still exists for -m selection and is still excluded
from the per-PR selector by path (tests/e2e/(ui|claude_code|load)/), but the
test no longer needs an env var to run once its file is targeted.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Now that create-release.yml gates stable and RC releases on this test directly,
the weekly schedule is redundant: every release gets a run against its own
commit instead of whatever happened to be on the default branch that Saturday.
workflow_dispatch stays for manual runs.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Local runs measured 3.5 KB per request, so 10 KB keeps close to 3x headroom
while tightening from the earlier 12 KB.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Local runs measured p50 0.19s, p90 0.23s, p99 0.69s, so 2s/3s/5s left several
times that as slack. 1s/2s/3s keeps a comfortable margin while catching a
smaller regression than the looser ceilings would have.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Three local runs measured 1.33x-1.36x, so 2x is the tightest bound the data
supports and still catches a regression far smaller than 4x would. Noted in the
comment that this is the ceiling to loosen first if a weekly run trips it,
since core count shifts how much of baseline CPU is fixed per-request work.
Co-Authored-By: Claude Code <noreply@anthropic.com>
RSS moved 0.91x-1.40x across three identical local runs, so it stays loose at
2x rather than the arbitrary 1.5x carried over from the pre-padding-payload
calibration. CPU per request held steady at 1.33x-1.36x across the same runs,
so 4x replaces the looser 6x it inherited from stale numbers.
Co-Authored-By: Claude Code <noreply@anthropic.com>
A ratio against the healthy phase cannot bound either metric. Once the Redis
circuit breaker opens, a request skips Redis instead of waiting on its socket
timeout, so the chaos phase can measure cheaper than the baseline it is compared
against: local runs came in at 0.61x baseline p90 while a log-bytes ratio read
724x. Splitting Budget into RatioBudget and AbsoluteBudget lets RSS and CPU keep
the ratio they need, since both are machine-shaped, while latency and log volume
get the wall-clock ceiling a user actually cares about.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Pad the locust payload to tens of KB so per-request bookkeeping cost scales with
body size instead of hiding behind a 40-byte prompt. Turn on
use_redis_transaction_buffer in the chaos config and JSON_LOGS in the workflow so
the spend buffer, pod lock, and JSON-encoded breaker tracebacks are all part of
the measured chaos cost. Add a log-bytes-per-request budget alongside latency,
RSS, and CPU, reading the proxy's log file size at each phase split; its ceiling
is uncalibrated since no chaos run has measured it yet.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The Anthropic Messages route reaches the same Redis touchpoints and cost-tracking
callback through its own request path, so a failure-path regression there would not
surface from chat completions alone. Each simulated user now picks one endpoint round
robin and stays on it, and the per-endpoint split is asserted and reported so a run
that silently drove only one route fails instead of passing.
Co-Authored-By: Claude Code <noreply@anthropic.com>
CLIENT PAUSE ALL for the length of the chaos phase instead of CLIENT PAUSE WRITE, so every Redis touchpoint on the request path times out rather than just the writes. The pause is sized to the phase because it freezes the control connection too; teardown's CLIENT UNPAUSE is a safety net for a phase that overran
Latency, RSS and CPU are now budgeted as chaos-over-baseline ratios (p50/p90/p99 for latency and RSS, CPU seconds per request once) through a small phase_budget module, replacing the machine-shaped absolutes. The Redis timeout rate is reported but no longer asserted
The final /metrics scrape waits for litellm_deployment_failure_responses_total to stop moving, since that counter is bumped from the async logging queue and lagged the load generator by thousands of increments. The model group carries a unique marker so a deployment left behind by an aborted run cannot absorb this run's retries
Co-Authored-By: Claude Code <noreply@anthropic.com>
The sequential version sent one request at a time, so a Redis outage never
reached the concurrency where the failed-tracking alert body actually grows.
This drives the proxy with locust against one model group of three mock
deployments, two failing at order 1 and one serving at order 2, so every
request spends its retries on the failing pair and lands on the serving
deployment through the order-based fallback. Two phases, a healthy baseline
and a CLIENT PAUSE WRITE window, and every request must succeed in both.
Latency, RSS and CPU are reported as p50/p90/p99 per phase rather than
asserted on: RSS and CPU come from psutil on the proxy's process tree, since
a multi-worker proxy serves /metrics from the prometheus multiprocess
collector and that drops the process collector's series. Thresholds stay open
until weekly runs give real baselines.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Each checkpoint now samples until no new worker has answered for the settle
window, and the growth assertion refuses a worker set that changed between the
warm and after checkpoints instead of comparing only the intersection, so a
leaking worker reached by one checkpoint alone cannot drop out of the gate
On the release gate the e2e tests only see the nginx router, and the chart's
ingress sent /debug/memory/summary to the backend catch-all, so the RSS check
measured the backend pod instead of the gateway workers that serve the failing
requests. Render it as an Exact gateway path next to /test, name the host in the
summary response so workers behind one origin never collide on pid alone, and
key the harness readings by (origin, hostname, pid)
The failed-batch redesign left list_batches, BatchList, BatchListQuery and
the batch object's metadata and created_at fields with no caller, and they
duplicated the batches suite's own client. delete_user discarded its result,
so a user that outlived the class fixture went unnoticed; unwrap turns that
into a teardown error like delete_key already does.
Add an embeddings case with its own closed-port primary and mock backup (the fallback map in
the gateway config gains the pair; LiteLLMParamsBody.mock_response accepts the list an embedding
mock needs). Assert from /metrics that the proxy's resident memory grows by no more than 200 MB
across each case where the process collector reports it (Linux), that the router counted a
successful fallback for every request, and that every spend row is a success.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
The suite used to mint its own RS256 tokens from a stand-in issuer, which
could only ever prove the proxy agreed with the tests: the claims were
whatever the tests chose to sign. Every JWT bug worth catching lives in the
shape of what an identity provider really emits, so the suite now runs
against Keycloak (realm in idp_realm.json), provisions a group and a user per
test through its admin API, and signs in through the direct-access grant.
That changes what the tokens look like: sub is Keycloak's opaque user uuid
rather than a friendly name, groups arrives from a protocol mapper, aud is
the IdP's own audience, and the JWKS carries an encryption key beside the
signing key so the proxy has to select on kid. The expiry case now takes a
one-second token from a second client in the realm and waits for it to lapse
instead of forging a stale exp.
The proxy config the suite needs is unchanged. CI runs it against a Keycloak
deployed beside the ephemeral stack, which lives in the releaser repo.
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The e2e directive has every test create its deployments through the management API and delete
them on teardown. Drop the static model_list from the gateway config; the test now registers the
closed-port primary and the mock backup itself, and the fallback map stays in router_settings
where proxy-level config belongs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
A loopback Redis answers many commands inside the 1 ms socket timeout, so nothing guaranteed
the failure path ran. The test now holds the proxy's Redis in CLIENT PAUSE WRITE for its
duration, so every write the proxy sends, the spend counter increment included, outlives the
timeout, and lifts the pause in teardown. Reads stay live so the control connection can do that.
Enable the prometheus callback in the gateway config and assert from /metrics that the proxy
counted at least the breaker's five timeouts and that, during each case, it saw fresh timeouts,
a breaker transition, or an open breaker rejecting every call. The open breaker is the state a
customer's worker sits in, and cost tracking fails on every request either way.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
The retrieve writer and the CheckBatchCost poller are two different spend
writers, and the in-run failed batch only proves the first. Split the
key_attribution batch cell in two: retrieve_batch_cost_joins_retrieving_key,
which test_terminal_batch_cost_row_joins_the_retrieving_key claims, and
poller_batch_cost_joins_creating_key, which no test claims yet and so shows
up on the coverage dashboard as a P1 gap instead of hiding behind the
retrieve leg. The rationale records why one run cannot hand the poller a
completed batch on a stack that boots a fresh Postgres per build.
The batch list is served from LiteLLM_ManagedObjectTable whenever the managed
files hook is loaded, and the Buildkite e2e stacks bundle a fresh Postgres per
build, so a prior run's marker batch is never listed and the baton could only
ever pass vacuously. Each run now creates a batch OpenAI fails at validation
within seconds, retrieves it by its raw provider id with the same key until it
is failed, and asserts the {provider_batch_id}_batch_cost row that retrieve
writes joins the key's token hash and alias
The Responses path returns a mock for any mock_response string, so the InternalServerError
sentinel never failed there. Point the primary deployment's api_base at a closed port instead,
which fails every endpoint the same way, then parametrize the test over /chat/completions and
/v1/responses and register both on the coverage cell.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
The completed marker batch is now retrieved by its raw provider id with this
run's key, so the proxy prices it inline and the {provider_batch_id}_batch_cost
row must join that key's token and alias. The CheckBatchCost poller only bills
batches it created in the same database, which a stack booted fresh per run
never holds for a completed marker, so the old unified-id assertion had no row
to find. Every run also replays one callback log through
POST /v1/rust_control_plane/logs, the third spend writer, and asserts its row
joins the key like the eight request paths
It is a functional e2e test, not a load test, so give it its own workflow instead of a job
inside the load anomaly run. It keeps the Saturday 12:00 UTC cadence and manual dispatch, and
boots the timeout-config proxy with Postgres and Valkey services exactly as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
Add tests/e2e/router/test_redis_timeout_e2e.py against a proxy booted from
tests/e2e/gateway/redis_timeout_ci_config.yml: a real Redis with socket_timeout 0.001, so every
command times out and the circuit breaker opens, plus a primary deployment that always fails and
falls back to a healthy one, so every request carries retry breadcrumbs into cost tracking. The
test drives twenty chat requests through the proxy and asserts each answers within ten seconds,
the last third is no slower than the first, /health/liveliness stays fast, and every request
still reaches the spend log.
Gate it behind the redis_timeout marker and E2E_REDIS_TIMEOUT, exclude it from the per-PR
e2e-changed selector, register the reliability.circuit_breaker.redis_timeout.stays_responsive
cell, and run it as its own job in the weekly load anomaly workflow with a Postgres and Valkey
service. Against a v1.100.0 proxy the run wedges the worker: requests time out and liveliness
stops answering (LIT-6780).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
One aliased key owned by a user with an email drives chat, queued chat,
messages, responses, embeddings, the Gemini passthrough, a batch file upload,
and a batch create against a live proxy. Each row must carry api_key equal to
the key's LiteLLM_VerificationToken.token and the alias in metadata, and
/spend/logs?api_key= and /user/daily/activity must report the key with its
alias and email. Health-check rows must keep the literal service-account key,
and the batch cost row for a completed marker batch must join the key that
created it. A re-hashed api_key (the v1.99.0 regression fixed by #39568 and
#39572) now fails the Buildkite e2e stage naming the write path
Resolves MAT-180
* fix(model-management): honor an explicit null as a clear on model update
PATCH /model/{model_id}/update merged the patch with exclude_none and then
popped explicit nulls only for the mirrored pricing fields, so a null sent for
max_input_tokens, mode, supports_vision or any other key was dropped and a value
pinned by an earlier save could never be removed.
The route now follows JSON Merge Patch over both blobs: a key absent from the
body is unchanged, a key sent as null is removed from the stored row, and a key
sent with a value is set. Ownership and identity keys keep ignoring a null, as
do the fields the stored models require, since clearing one writes a row no
reload can rebuild. Mirrored pricing keys still clear from both blobs.
Clearing a price also needed the router to stop merging a deployment's cost-map
entry onto its previous registration, which left the old rate in place and kept
billing at a price the deployment no longer carried.
Adds a create, read, partial-update, clear, enforce, delete lifecycle e2e that
reads back on every replica, and a harness helper for that read-back.
* fix(router): keep a deployment id that names a real model from evicting its catalog entry
Deployments are keyed into litellm.model_cost alongside the built-in catalog, so
evicting a deployment's stale entry by id could take a real model's entry with it:
registering a deployment whose model_info.id is "gpt-4o" stripped that model's
pricing, context window and capability flags process-wide, for every other
deployment of it, until the next price-map reload.
Only evict an entry this registration owns. A colliding id keeps the previous
merge, which pollutes the catalog entry rather than emptying it.
Also pins the Admin UI round trip: the model edit form echoes the whole /model/info
row back on save, and that read reports every key the deployment never stored as an
explicit null, so the clear path has to leave those keys alone.
* fix(router): decide cost-map eviction by what this registrar created
The previous guard read a catalog entry off `litellm_provider`, so a deployment
that declares its own provider in model_info was treated as one and kept billing
at a price it no longer carried. It also only held for a single registration: a
second one under a colliding id saw the id the first merge left behind and
evicted the catalog entry anyway.
Track the cost-map keys this registrar creates instead. A key it created is
evicted before re-registration; one it did not is left to merge, which is what a
deployment id colliding with a catalog model name needs.
Also folds the required-fields comment into the docstring that already gives the
reason.
* fix(router): release a deployment's cost-map key when it is deleted
The ownership ledger only grew. A deleted deployment kept its claim, so if a
later catalog refresh started publishing a model under that same name, the next
registration would treat the catalog entry as the deployment's own and evict it.
Deleting a deployment now gives the key back, which also stops the ledger
growing for the life of the process.
* fix(router): hold a cost-map key while another live router still serves it
The claim is process-wide but the release was per-deletion, so with two routers
serving one deployment id, the first deletion put the survivor back on merging
and the price it had just cleared would keep billing.
Release the key only once no live router still serves that id.
* fix(router): register a router in the live set when it gains a deployment
_live_routers was only joined when a router was constructed with a model_list,
but a router built empty is populated through add_deployment, and the empty
branch exists for exactly that. Such a router was invisible to the live-router
scan, so deleting the deployment from another router released the shared
cost-map key while it was still serving that id.
Joining the set where a deployment enters the list covers every path, and it
also lets a price reload rebuild what a dynamically built router serves.
* fix(e2e): read the stored model row from the control plane, not each gateway
The lifecycle suite polled /model/info on every URL in PROXY_REPLICA_URLS. Those
URLs are the stack's gateways, and gateway/routes/allowlist.py trims them to the
LLM data-plane surface, so /model/info answers only on the backend and 404s on
every replica. All five tests failed at their first read-back in CI while passing
against a monolith, where one process serves both planes.
The stored row has one answer behind it, so it is read through the shared
transport, which routes control-plane paths to the backend. What every gateway
must agree on is which models it serves, so the create and delete steps poll
/v1/models per replica instead, a route the gateway does serve.
read_back_everywhere now rejects a control-plane path outright rather than
timing out on it.
Two things surfaced behind that. /public/ was missing from the transport's
control-plane prefixes, so model_cost_map() was routed to a gateway and 404'd,
and the billing steps needed a data-plane wait: a PATCH lands on the backend and
each gateway picks it up on its own config reload, measured here at 12-24s, so
they now drive calls until the new rate reaches the spend row and let the
deadline fail them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1S92J8gSxxKVe1JBzxWBF
* test(models): keep polling outcomes immutable and document shared ownership
* test: validate opaque stream IDs and hide log-reader credentials
* test: isolate auto-router scenarios and clean partial setup
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e
PUT /v1/mcp/toolset dumped its payload with exclude_none, so a field sent as
null looked exactly like one the caller left out and the stored value
survived. An admin could not clear a toolset's description: the save reported
success and the old text came straight back. It now dumps with exclude_unset,
so absent keeps and null clears, which is what PUT /v1/mcp/server already did.
A null tools list clears the selection to empty, and a null toolset_name is
ignored because a toolset always has a name.
Adds create, read, partial-update, clear and delete e2e coverage for MCP
servers and toolsets, with every read-back polled on every replica so an edit
that lands on one replica and not another fails the test, plus an enforcement
test proving a key granted a toolset lists exactly that toolset's tools
against the real Datadog upstream.
* fix(e2e): refuse a read-back that no replica serves
A read-back over an empty replica mapping satisfied every predicate and
returned as if it had converged, so it would have asserted nothing and
passed. No wiring can produce that today, since the replica list always
falls back to at least one URL, but a helper whose whole job is proving a
write reached every replica should not have a shape that passes vacuously.
* fix(mcp): keep a null tools list a no-op on toolset update
Treating a null tools list as a clear meant an existing client that sends
tools=null during a partial update, meaning "leave the selection alone",
silently lost every tool the toolset grants. That is a permission surface,
so the quiet version of it is the worst version.
A toolset always has a tool list, the same way it always has a name, so a
null on either is now a no-op. Emptying the selection is an explicit [],
which cannot be confused with a field the caller left out, and which is
what the dashboard already sends.
* fix(e2e): keep MCP admin routes on the data plane
/v1/mcp/* is a lazily mounted feature, so a gateway registers it on the first
matching request, which happens after the startup route trim that drops
management endpoints. Routing it to the control plane therefore sent every MCP
call to the one backend process: the new lifecycle read-backs proved a single
process rather than every replica, and mcp_client's await_registered barrier
waited on a registry that does not serve the tools/list call it guards, so the
existing MCP suites polled a gateway that had not synced yet until poll_timeout
Verified against a two-gateway split stack (backend on 4001, gateways on 4010
and 4011, one postgres): both gateways answer /v1/mcp/server and /v1/mcp/toolset,
and each served 6 server reads and 7 toolset reads over the run
* fix(e2e): grant the toolset by the tool's own name, not the wire name
tools/list serves a tool as <prefix><tool_name>, but a toolset grants by the
tool's own name: resolve_toolset_permissions reads toolset.tools[].tool_name
straight through, and the prefix is added on the way out. The test built the
toolset from the names tools/list reported, so the grant matched nothing, the
scoped key listed no tools, and await_tools ran out its whole poll_timeout
before failing
Measure the prefix off search_datadog_logs, whose own name is known, rather than
guessing it from the alias, since the proxy can be configured to prefix with a
short server id instead. The expectation compared against tools/list stays in
wire names; only what the toolset stores crosses back
* test(mcp): build immutable lifecycle updates and replica results
* test: validate opaque stream IDs and hide log-reader credentials
* test: isolate auto-router scenarios and clean partial setup
* test: honor Datadog search rate-limit reset headers
* test: share the Datadog read-back deadline across retries
* test: preserve captured MCP toolset update fields
* test(e2e/ui): cover key budget window, non-admin model scope edit, and key blocking
Three Playwright specs for the Virtual Keys flows customers hit most, each
reading its result back through /key/info and /v1/chat/completions rather
than trusting the toast:
- a monthly spend cap and reset window set through Edit Settings, surviving
a reload, with clearing the window leaving the cap in place
- a team member narrowing their own team key's models, and the proxy
refusing the model they dropped
- blocking a key from its detail page, then unblocking it
Each test owns the key it edits and deletes it on teardown, so retries and
--repeat-each never run out of fixtures.
* test(e2e/ui): tighten virtual key specs from review feedback
Replace the mutable suite-level key state with a Playwright fixture, so the
alias and token are never reassigned and cleanup stays tied to the test.
Assert /key/delete succeeded instead of discarding the response, so a failed
cleanup surfaces rather than leaving rows behind.
Drop the explanatory JSDoc the repo's comment policy disallows, keeping only
the one line explaining why Date.now() alone is not unique enough.
Type the master-key POST helper against a real guard instead of casting to
Record<string, any>.
Assert the unblocked key is served with a 200, not just the response text,
and that clearing the reset window also clears budget_reset_at.
* test(ui): assert the team response through Playwright
* test(e2e/ui): cover team-scoped model visibility, re-editing litellm params, and model health checks
Three Models and Endpoints flows had no end-to-end coverage, and all three
keep coming back as bug reports.
modelsByTeam walks an internal user through the Current team control and
asserts the table lists exactly what each team grants. It creates one
deployment that belongs to no team, proves that deployment is visible under
Personal, then proves it is absent under both seeded teams, so an empty
table cannot pass the same assertions.
editLitellmParams adds a temperature and a custom pair to a deployment,
saves, then re-edits the temperature and drops the custom pair. It checks
both update request bodies, polls the stored deployment until the new
temperature is there, reloads the page to confirm the second save is what
renders, and sends one chat completion to prove the deployment still serves.
modelHealthStatus runs the health check on a reachable deployment and on one
pointed at a dead port, asserts the healthy and unhealthy cells and the two
detail dialogs, and reloads to confirm both statuses are stored.
Every deployment these specs create carries a unique name and is deleted in
afterEach, including on the failure path.
* test(e2e/ui): find health rows across every page of the health table
The health table pages server-side at 50 rows with no search box, so on a
proxy carrying more deployments than that the two deployments the spec
creates can land on a later page and the lookup finds nothing.
Row lookups now walk the pages, using the table's own page indicator to
know when to advance and when to wrap back to the first page.
* test(e2e/ui): build the created deployment ids without mutating the array
* test(ui): scope model deployments to Playwright fixtures
