The Locust throughput SLO test is a different testing category from
functional e2e (variance-driven, historically flaky, currently
skip-annotated against LIT-5119) and erodes trust in the suite as a
release gate; it comes out of the default collection along with its
exclusive plumbing (locustfile, load-mock registration fixtures,
run_chat_load). Re-implementation as its own pipeline is tracked in
LIT-5163. The weekly session-anomaly test never ran in the suite (opt-in
via E2E_WEEKLY_ANOMALY, driven by its own workflow) and stays, as do the
markerless aggregation unit tests.
The vllm passthrough test read-times-out (60s) against the shared
vllm-cpu backend in every run on the per-SHA e2e stack; it is removed
until LIT-5164 establishes whether that is backend capacity or a
passthrough defect. Its registry cells return to the gap list, which is
the honest state
The strict-priority e2e (added with the zero-increment limiter fix) can
never pass on stage: the proxy there does not run the
dynamic_rate_limiter_v3 callbacks + priority_reservation settings the
module requires, confirmed by zero limiter log lines across every
gateway and backend pod during the 2026-08-02 run. Config lives in the
infra repo; LIT-5118 tracks adding it.
The throughput SLO test failed the same run with 65.9% of requests dying
at the ELB as 502/503 before reaching a pod. The per-replica SLO rework
fixed the RPS-floor assertion but cannot help when stage idles at one
warm gateway replica; LIT-5119 tracks pre-scaling the fleet for the load
phase.
Both skips name their ticket, and the coverage registry returns the two
cells to the gap list while they are in place.
The floor was an absolute fleet number, so it asserted replicas x per-replica rate
and went red on how many gateway pods happened to be warm rather than on the request
path. The test now measures one replica first, with a short serial pass that only ever
occupies a single pod, and requires the concurrent phase to reach at least that rate.
A serial latency budget carries the request-path assertion the floor used to imply,
and both hold at one replica or seven.
Zero-error runs that "sustained 16.7 RPS" were queueing, not slow requests: the load
model is a mock_response deployment with no upstream, a single-worker replica serves
it in about 57ms, and 100 closed-loop users against 1/0.057 RPS of capacity sit at
6s each by Little's law.
The runner also kept locust's --json summary and threw away everything else, so a run
where 93% of requests failed said nothing about what they got. It now passes --csv,
reads the failure breakdown back, and reports locust's own generator-saturation
warnings, both folded into the assertion messages.
Resolves LIT-5054
The SLO measures how many gateway replicas happen to be warm rather than the
request path. Clearing the floor needs roughly 5-7 replicas at ~10-14 RPS each;
stage idles at one and reactive HPA scale-up lands minutes into a ~3 minute
test.
It failed both of its assertions on consecutive days: 93.3% errors at an
inflated 264 RPS, where the failing requests never reached a pod and closed-loop
RPS rose because they failed fast, then 16.7 RPS with zero failures.
The covers marker and registry row stay put, so the cell returns to the gap
list rather than disappearing from the denominator.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
* test(e2e): harden stage flakes for batches, UI, and MCP
Unique batch model names avoid load-balancing onto stale azure-batch
deployments that still pointed at the retired gpt-4.1-mini-batch, which
only the managed/unified path was hitting. Retry batch retrieve on 500
and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite
when the compose-only mcp-upstream is unreachable on stage k8s
* test(e2e): cover Datadog remote MCP via search_datadog_logs
Register the regional Datadog MCP endpoint with DD-API-KEY /
DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is
not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*,
assert the proxy shipped it, list tools, call search_datadog_logs for the
marker, and delete the server on teardown. Math-upstream key-access tests
only skip when that compose service is unreachable
* test(e2e): drop compose math MCP upstream; use Datadog only
Key-access denial and happy-path MCP e2e both register the real regional
Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers.
Remove the mcp-upstream compose service and FastMCP add/multiply fixture
* docs(e2e): require real Datadog MCP for all mcp suite tests
Document that tests/e2e/mcp must register via datadog_mcp helpers against
mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams
* chore: restore mcp_e2e_upstream_server.py
Keep the FastMCP fixture file; e2e no longer wires it in compose, but the
module itself is not part of the Datadog-only cleanup
* fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load
pytest on the host never inherited compose env_file keys, so DD_API_KEY
stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the
dynamically loaded datadog_reader module in sys.modules so dataclasses
do not crash under Python 3.12
* test(e2e/batches): harden azure/vertex unified lifecycle flakes
Put the provider deployment name in every JSONL body so Azure does not
depend on a perfect model rewrite. Retry create/retrieve/cancel on
transient statuses with backoff. Drop cancel assertions for azure and
vertex (registry only has a shared basic cell; create+retrieve prove
routing, cancel stays best-effort cleanup)
* test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED
Post-login client redirects abort the first /ui/api-keys/ goto on stage.
Wait off /ui/login after cookie set, then accept the page once Create New
Key is visible even if goto raised ERR_ABORTED
* test(e2e): drop flaky key models dropdown Playwright suite
API management e2e already covers key generate/update persistence. The
UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and
no unique product signal. Remove the suite and unused browser fixtures
* test(e2e/batches): fail clearly when OPENAI/AZURE provider is missing
Replace bare next() over PROVIDERS with _model_for that raises ValueError
naming the missing provider and the known list, instead of StopIteration
* fix(e2e): migrate load suite from e2e_gateway to ProxyClient
Stage collection failed with ModuleNotFoundError: e2e_gateway after the
Gateway rename. Wire load/conftest and LoadClient to the shared
ProxyClient fixture like every other suite
* fix(e2e): drop duplicate datadog_mcp_url and CLAUDE section after merge
CodSpeed benchmarks the SDK with no IO, so it can't catch regressions that
only appear under real concurrent load through the full proxy stack (auth,
routing, logging, spend, Postgres, Redis). This adds a Locust load test under
tests/e2e/load that drives concurrent POST /chat/completions traffic against a
mock deployment (litellm_params.mock_response), so the measured throughput
reflects proxy overhead rather than a provider's latency, and asserts an
aggregate RPS SLO with a failure-ratio guard. The test is marked load and the
parent conftest sorts load-marked items last so it never perturbs
latency-sensitive suites. Covers reliability.perf.throughput.under_slo.
