The e2e harness exists to prove product features end to end against a live
proxy. The prior Hard Rule carved out an exception for "tests that cover the
harness itself" and pointed at coverage_registry/test_collector.py, which in
practice invited unit tests of harness helpers to be staged alongside e2e
work. That is the wrong tool: harness logic that is worth locking down does
not need a mock-driven unit test living under tests/e2e.
Drop the carve-out. The Hard Rule now reads that no unit tests of any kind
belong under tests/e2e, and the passing mention of unmarked harness coverage
in the transport section is removed so the doc no longer contradicts itself.
coverage_registry/test_collector.py still exists on disk and is left in place
for now; whether to relocate or remove it is a separate decision.
create-release.yml is back to calling the chaos test through no mechanism at
all; it never called it. Also drops the E2E_REDIS_CHAOS opt-in gate itself:
the redis_chaos marker still exists for -m selection and is still excluded
from the per-PR selector by path (tests/e2e/(ui|claude_code|load)/), but the
test no longer needs an env var to run once its file is targeted.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Now that create-release.yml gates stable and RC releases on this test directly,
the weekly schedule is redundant: every release gets a run against its own
commit instead of whatever happened to be on the default branch that Saturday.
workflow_dispatch stays for manual runs.
Co-Authored-By: Claude Code <noreply@anthropic.com>
A ratio against the healthy phase cannot bound either metric. Once the Redis
circuit breaker opens, a request skips Redis instead of waiting on its socket
timeout, so the chaos phase can measure cheaper than the baseline it is compared
against: local runs came in at 0.61x baseline p90 while a log-bytes ratio read
724x. Splitting Budget into RatioBudget and AbsoluteBudget lets RSS and CPU keep
the ratio they need, since both are machine-shaped, while latency and log volume
get the wall-clock ceiling a user actually cares about.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Pad the locust payload to tens of KB so per-request bookkeeping cost scales with
body size instead of hiding behind a 40-byte prompt. Turn on
use_redis_transaction_buffer in the chaos config and JSON_LOGS in the workflow so
the spend buffer, pod lock, and JSON-encoded breaker tracebacks are all part of
the measured chaos cost. Add a log-bytes-per-request budget alongside latency,
RSS, and CPU, reading the proxy's log file size at each phase split; its ceiling
is uncalibrated since no chaos run has measured it yet.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The Anthropic Messages route reaches the same Redis touchpoints and cost-tracking
callback through its own request path, so a failure-path regression there would not
surface from chat completions alone. Each simulated user now picks one endpoint round
robin and stays on it, and the per-endpoint split is asserted and reported so a run
that silently drove only one route fails instead of passing.
Co-Authored-By: Claude Code <noreply@anthropic.com>
CLIENT PAUSE ALL for the length of the chaos phase instead of CLIENT PAUSE WRITE, so every Redis touchpoint on the request path times out rather than just the writes. The pause is sized to the phase because it freezes the control connection too; teardown's CLIENT UNPAUSE is a safety net for a phase that overran
Latency, RSS and CPU are now budgeted as chaos-over-baseline ratios (p50/p90/p99 for latency and RSS, CPU seconds per request once) through a small phase_budget module, replacing the machine-shaped absolutes. The Redis timeout rate is reported but no longer asserted
The final /metrics scrape waits for litellm_deployment_failure_responses_total to stop moving, since that counter is bumped from the async logging queue and lagged the load generator by thousands of increments. The model group carries a unique marker so a deployment left behind by an aborted run cannot absorb this run's retries
Co-Authored-By: Claude Code <noreply@anthropic.com>
The sequential version sent one request at a time, so a Redis outage never
reached the concurrency where the failed-tracking alert body actually grows.
This drives the proxy with locust against one model group of three mock
deployments, two failing at order 1 and one serving at order 2, so every
request spends its retries on the failing pair and lands on the serving
deployment through the order-based fallback. Two phases, a healthy baseline
and a CLIENT PAUSE WRITE window, and every request must succeed in both.
Latency, RSS and CPU are reported as p50/p90/p99 per phase rather than
asserted on: RSS and CPU come from psutil on the proxy's process tree, since
a multi-worker proxy serves /metrics from the prometheus multiprocess
collector and that drops the process collector's series. Thresholds stay open
until weekly runs give real baselines.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The suite used to mint its own RS256 tokens from a stand-in issuer, which
could only ever prove the proxy agreed with the tests: the claims were
whatever the tests chose to sign. Every JWT bug worth catching lives in the
shape of what an identity provider really emits, so the suite now runs
against Keycloak (realm in idp_realm.json), provisions a group and a user per
test through its admin API, and signs in through the direct-access grant.
That changes what the tokens look like: sub is Keycloak's opaque user uuid
rather than a friendly name, groups arrives from a protocol mapper, aud is
the IdP's own audience, and the JWKS carries an encryption key beside the
signing key so the proxy has to select on kid. The expiry case now takes a
one-second token from a second client in the realm and waits for it to lapse
instead of forging a stale exp.
The proxy config the suite needs is unchanged. CI runs it against a Keycloak
deployed beside the ephemeral stack, which lives in the releaser repo.
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The retrieve writer and the CheckBatchCost poller are two different spend
writers, and the in-run failed batch only proves the first. Split the
key_attribution batch cell in two: retrieve_batch_cost_joins_retrieving_key,
which test_terminal_batch_cost_row_joins_the_retrieving_key claims, and
poller_batch_cost_joins_creating_key, which no test claims yet and so shows
up on the coverage dashboard as a P1 gap instead of hiding behind the
retrieve leg. The rationale records why one run cannot hand the poller a
completed batch on a stack that boots a fresh Postgres per build.
It is a functional e2e test, not a load test, so give it its own workflow instead of a job
inside the load anomaly run. It keeps the Saturday 12:00 UTC cadence and manual dispatch, and
boots the timeout-config proxy with Postgres and Valkey services exactly as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
Add tests/e2e/router/test_redis_timeout_e2e.py against a proxy booted from
tests/e2e/gateway/redis_timeout_ci_config.yml: a real Redis with socket_timeout 0.001, so every
command times out and the circuit breaker opens, plus a primary deployment that always fails and
falls back to a healthy one, so every request carries retry breadcrumbs into cost tracking. The
test drives twenty chat requests through the proxy and asserts each answers within ten seconds,
the last third is no slower than the first, /health/liveliness stays fast, and every request
still reaches the spend log.
Gate it behind the redis_timeout marker and E2E_REDIS_TIMEOUT, exclude it from the per-PR
e2e-changed selector, register the reliability.circuit_breaker.redis_timeout.stays_responsive
cell, and run it as its own job in the weekly load anomaly workflow with a Postgres and Valkey
service. Against a v1.100.0 proxy the run wedges the worker: requests time out and liveliness
stops answering (LIT-6780).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
One aliased key owned by a user with an email drives chat, queued chat,
messages, responses, embeddings, the Gemini passthrough, a batch file upload,
and a batch create against a live proxy. Each row must carry api_key equal to
the key's LiteLLM_VerificationToken.token and the alias in metadata, and
/spend/logs?api_key= and /user/daily/activity must report the key with its
alias and email. Health-check rows must keep the literal service-account key,
and the batch cost row for a completed marker batch must join the key that
created it. A re-hashed api_key (the v1.99.0 regression fixed by #39568 and
#39572) now fails the Buildkite e2e stage naming the write path
Resolves MAT-180
Add tests/e2e/jwt_issuer.py, a test-only RS256 issuer that serves a JWKS
document and signs arbitrary claims over an open loopback-only /token
endpoint, so e2e tests can mint tokens without ever holding a signing key.
One key per process keeps the proxy's cached JWKS valid for the whole run.
Add the first five live JWT auth tests in tests/e2e/other: a valid token for
an existing team is accepted and its spend row carries the claimed team and
user, a tampered signature and an expired token are refused with 401, a
token naming a team that does not exist is refused with 403, and a plain
sk- virtual key keeps working with enable_jwt_auth on. Harness unit tests
cover the issuer itself. Team and user create/delete land on the shared
ProxyClient (warn-only teardown, /user/delete typed as the int it returns)
instead of a fourth per-suite copy.
Document the issuer command, the E2E_JWT_ISSUER_PORT convention (default
4190), JWT_PUBLIC_KEY_URL and the litellm_jwtauth config block in
tests/e2e/CONTRIBUTING.md, and register the new cells in other.yaml.
The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.
The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.
Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
Four cases in tests/e2e/quota_management/budgets, driving real OpenAI calls
through a group whose shared pool is drained to exhaustion: the spender key
stays blocked, a key that spent nothing of its own is blocked by the same
pool, a sibling group with no budget keeps serving, and the budget read
reports the spend drawn against the group.
Adds set/get/delete access group budget to BudgetClient and the four
matching rows to the coverage registry.
Adds a scheduled GitHub Actions lane on top of the merged record/replay
transport. A Saturday cron records the `replayable` e2e tests against the
real providers and publishes the fixture bundle as a private
`e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull
that artifact by its pinned digest, verify the checksum before extracting,
and replay it with provider credentials set to bogus values, so a run that
ever reached a real provider fails instead of passing.
An egress sentinel pins the provider hostnames to a local sink for the whole
replay job and counts every connection that reaches them; the job asserts
that count is zero, so hermeticity is proven by measurement. A red Saturday
publishes no bundle, so the next weekday finds nothing fresh and fails loudly
rather than replaying a week-old recording, and the transport's seven-day
freshness gate hard-fails any bundle that has drifted too far. The lane also
runs on demand from the Actions tab with a record/replay `mode` input.
Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic
test now counts to twenty so its recorded response banks several content
deltas, matching the assertion that the stream arrives incrementally.
The record/replay harness stored a streamed provider response as one
buffered body, so a replayed stream arrived coalesced and the
/v1/messages streaming test could not be edge-wired. Keep each SSE
transfer chunk in the bundle in the order the provider sent it (a new
streamed response shape at BUNDLE_FORMAT_VERSION 4) so replay reproduces
the provider's split points, the recorded usage chunk keeps its
position, and a mid-stream upstream error replays as the same
mid-stream error rather than a clean body.
Resolves LIT-5742
Adversarial review of the new multipart keying turned up collisions where
two different provider requests computed the same replay key, which is the
dangerous failure for a replay harness: the second request silently gets the
first one's response instead of missing loudly.
- a part counts as an upload when it has a filename or declares its own
content type, and the declared content type joins the identity, so two
uploads of the same bytes under the same field no longer collapse
- the uploaded parts contribute a JSON list of [field, filename, type]
triples instead of a "field:filename" string, so a separator inside a
filename can no longer impersonate a field boundary
- repeated field names get a "name[n]" suffix with a literal "[" doubled
first, so a repeated field and a literally indexed one stay distinct
- a field value that is not UTF-8 is stored as a base64 sha256 digest;
base64 rather than hex because the canonicalizer rewrites 64-character
hex runs to <sha256> and folded every binary value onto one key
- a field whose name reads as a credential is stored as <secret>. This
stays key-preserving because the key is recomputed from the stored
request rather than saved beside it, so the live request carrying the
real value still matches its redacted fixture
- the uploaded byte length leaves the key. The canonicalizer absorbs
timestamp and id drift inside a file, and that drift moves the count,
so keeping it there made re-records miss
Also stops a lookalike parameter such as "xboundary=" from being read as
the multipart boundary, and gives the OpenAI batch backend model a single
constant instead of three copies of the literal.
BUNDLE_FORMAT_VERSION goes to 3 because all of this moves recorded keys.
A bundle recorded under the old rules now fails naming both versions
instead of missing on every call.
Chat completions, embeddings, the non-streaming /v1/messages tests, and the
OpenAI batch deployment now register through the provider edge, so
E2E_FIXTURE_MODE=record captures their provider calls and replay serves them
back offline. None of them was wired before, so record was a silent no-op over
these suites and replay quietly went live instead of using the bundle
Multipart uploads now key on their parsed parts: every ordinary form field,
plus the field name, filename, content digest, and length of each file part.
The boundary is envelope rather than content, so it stays out of the digest
instead of changing the key on every run. A body that does not parse as its
declared envelope still has the boundary normalized away before hashing, so
the fallback is at least stable, and it records a name that says why
Binary uploads hash byte for byte. Canonicalizing them first meant decoding
with errors="replace", which collapsed every invalid byte to one U+FFFD and
gave two different PDFs of the same length the same key
Bundles stay out of the repo: they hold verbatim provider response bodies and
expire seven days after recording. Publishing them for CI is LIT-5748, and
streaming fidelity is LIT-5742
Replaces the test-side fixture transport with an in-process provider-edge
HTTP server the proxy's deployments point their api_base at. Record forwards
provider calls verbatim and writes them to the bundle; replay answers them
from the bundle with zero provider calls while key auth, routing, cost
calculation, and spend-log writes still execute against the live proxy and
database. Drift comes back as HTTP 599 naming the computed and closest
recorded keys. Request headers are never stored and responses are kept
byte-identical between modes from the proxy's side of the socket.
Replay previously matched interactions by transport verb and path in
recorded order, so a request whose body drifted from the recording
silently replayed the stale response, and reordering two independent
calls broke replay even though both were recorded. Match keys are now
canonical: fixture_canonical.py strips volatile headers and credential
fields, replaces unique markers, generated ids, uuids, and timestamps
with fixed placeholders, sorts object keys, and hashes what remains, so
a key is stable across runs and machines while any real content drift
is a hard ReplayMiss naming the computed key, the closest recorded key
with its file, and a content diff, with no fallthrough to a live call.
Matching is order-independent across distinct keys and FIFO within one
key. Recording now also redacts credential body and form fields (not
just auth headers) so provider keys never land in bundles.
Resolves LIT-5741
E2E_FIXTURE_MODE selects the transport every e2e client is built on: live
(default, unchanged behavior), record (pass through to the live proxy while
writing every interaction to a fixture bundle), or replay (serve every
interaction from the bundle with no proxy and no provider spend). Both new
transports fulfil the existing Transport protocol, so no test changes shape.
A bundle is a directory with a manifest (record timestamp, harness version,
format version) and one JSON file per interaction, grouped per test in call
order. Replay against a manifest older than seven days hard-fails at
collection time naming the bundle age. Record always wipes and never reads
the previous bundle, refusing to wipe a directory that is not a bundle.
Auth header values are redacted on write; uploads store a sha256 digest.
unique_marker() becomes deterministic per test in record/replay modes so a
replay run regenerates exactly the requests the record run sent.
Content-based match keys, streaming chunk fidelity, and provider-scoping are
follow-ups (LIT-5741, LIT-5742, LIT-5745).
The Locust throughput SLO test is a different testing category from
functional e2e (variance-driven, historically flaky, currently
skip-annotated against LIT-5119) and erodes trust in the suite as a
release gate; it comes out of the default collection along with its
exclusive plumbing (locustfile, load-mock registration fixtures,
run_chat_load). Re-implementation as its own pipeline is tracked in
LIT-5163. The weekly session-anomaly test never ran in the suite (opt-in
via E2E_WEEKLY_ANOMALY, driven by its own workflow) and stays, as do the
markerless aggregation unit tests.
The vllm passthrough test read-times-out (60s) against the shared
vllm-cpu backend in every run on the per-SHA e2e stack; it is removed
until LIT-5164 establishes whether that is backend capacity or a
passthrough defect. Its registry cells return to the gap list, which is
the honest state
The collector read @pytest.mark.covers off every collected item, and
collection does not evaluate skips, so a test carrying both a skip and a
covers marker reported its cell as covered while asserting nothing. 17
files under tests/e2e do exactly that, which inflated the headline from
290/434 to 311/434.
A cell now counts as covered only when at least one test pytest would
actually run declares it; a cell claimed by both a live and a skipped
test stays covered. Skip state comes from pytest's own evaluator, so
skip, skipif (bool and string conditions), and module-level pytestmark
resolve exactly as they do in the e2e run. Cells left uncovered this way
are listed under the headline and exported as skipped_markers (JSON) and
litellm_e2e_coverage_skipped_markers (Prometheus) so the gap surfaces
instead of disappearing; the Loki line contract is unchanged. A marker
on a skipped test that points outside the registry is still an orphan,
so --strict keeps its reach.
Because skipif resolves against the environment the collector runs in,
the number now depends on that environment; run it where the e2e suite
runs. A pytest.skip() call inside a test body remains invisible to a
static pass, which the module docstring and README both state.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
* test(e2e): add other suite covering master-key auth and health lifecycle
Covers the other.* holding-pen cells that were uncovered: master-key
valid_allows/invalid_denied on the admin /user/list gate, and the
lifecycle probes liveness.ping, readiness.public_probe,
readiness.reports_db_status, and readiness_details.authenticated_diagnostics.
New tests/e2e/other/ suite on the shared ProxyClient; the health probes
send no auth header to prove the public routes need no credential, and the
details route is asserted to reject an anonymous caller while exposing
version/db diagnostics to the master key.
* test(e2e): cover block_code_execution and openai_moderation guardrails
Extends the guardrails suite with two built-in guardrails registered per
request (default_on=False, opted in via the chat body's guardrails selector)
so neither intercepts unrelated traffic on the shared proxy.
block_code_execution.pre_call.blocks: a python code block plus a run-this
request is intercepted with the canned content-blocked message and the model
never runs, while the same code block asked about with don't-run-it reaches
the model. Verified live.
openai_moderations.pre_call.blocks: a flagged prompt is rejected 400 naming
the moderation policy while a benign prompt passes. The guardrail calls
OpenAI's moderation API; verifying it needs an OpenAI key with moderation
quota (this account currently 429s the moderation endpoint).
Adds a shared create_backend_model helper and a generic register() plus
per-request guardrails/max_tokens on the client so more built-ins can reuse
the same path.
* test(e2e): cover presidio PII masking (pre_call + post_call)
Registers a presidio guardrail per request (default_on=False) with the
analyzer/anonymizer bases supplied in the registration params, so the test
controls its own dependency and needs no proxy restart.
presidio.pre_call.masks: a repeat-verbatim request comes back with the
<EMAIL_ADDRESS> placeholder and never the raw email, proving the prompt was
anonymized before the model saw it.
presidio.post_call.masks: with apply_to_output the model's own emitted email
is masked on the way out, so the caller never receives the raw value.
Both verified live against real presidio analyzer + anonymizer containers.
logging_only is intentionally not covered: /spend/logs exposes no prompt
messages to read back the masked log, and a logging_only run also masked the
response, contradicting its contract; noted in the module docstring for a
follow-up.
* test(e2e): cover presidio logging_only masking via OTEL read-back
Adds the third presidio cell, guardrail.presidio.logging_only.masks. The
logging_only contract (mask what is logged, do not block) is verified by
reading the request's gen-AI span back from the real OTEL destination: the
span's gen_ai.input.messages attribute carries the <EMAIL_ADDRESS> placeholder,
never the raw email, and the call itself is not blocked.
Reads the trace via the shared OtelReader, promoted from logging/ to the suite
root so both suites use it. The masked prompt is polled to a deadline because
logging_only masks the payload asynchronously and the span can briefly export
before the mask lands. Drops the throwaway chat_send in favor of the existing
transport.send for the call-id capture.
* fix(e2e): tolerate cross-pod guardrail sync delay in team-opt-out test
Stage runs multiple gateway pods behind the shared key. POST /guardrails
registers a new default-on guardrail in-process immediately only on the
pod that served the create call; every other pod picks it up on its next
periodic DB sync (proxy_server.py, every 30s), so the very next chat call
can race a pod that has not synced yet. Poll to a 40s deadline instead of
asserting on the first response, matching the existing pattern in
test_budget_reset_advances_e2e.py.
* test(e2e): cover a guardrail on the MCP tool-call path (content_filter pre_mcp_call)
Adds guardrail.litellm_content_filter.pre_mcp_call.blocks: against the real
Datadog MCP server, a content_filter guardrail configured mode=pre_mcp_call
blocks a banned keyword in an MCP tool call's arguments with HTTP 400 attributed
to the pre_mcp_call hook, and lets a clean argument reach the upstream server.
The guardrail attaches with default_on because per-key/request guardrail
selection is dropped from the synthetic MCP request the hook sees; the banned
keyword is unique per run so default_on only intercepts this test's own call.
mode must be pre_mcp_call - a pre_call config silently no-ops on tools/call
because the event type is rewritten for call_mcp_tool.
Drives the tool directly via /mcp-rest/tools/call for a deterministic check of
the same pre_mcp_call enforcement the OpenAI-SDK chat path hits when a model
invokes an MCP tool.
* fix(e2e): mid-conversation messages test uses client.proxy not client.gateway
EndpointsClient exposes .proxy after the Gateway->ProxyClient rename; the
mid-conversation system test still referenced .gateway, which fails the e2e
basedpyright gate. Aligns it with the rest of the harness.
* test(e2e): address review on the guardrail coverage
MCP tool-call guardrail: poll the banned call until the guardrail is enforced
instead of asserting on the first call, so the control-plane -> data-plane
guardrail sync cannot race the check into a false pass-through; add a repeat
banned call after enforcement to guard against a partial-propagation state.
OpenAI moderation: distinguish a moderation-endpoint 429 (rate limit / no
moderation quota) from a guardrail failure, so an account-capability gap reads
as such rather than as "did not block". Runs green with a moderation-capable key.
* test(e2e): close partial-propagation false-pass in MCP guardrail block test
The single post-block repeat call could be load-balanced back to the same
already-synced data-plane pod, so the test could pass while another pod still
lacked the guardrail and let the banned MCP call reach Datadog. Anchor a wait to
the guardrail create time (every pod is guaranteed to have DB-synced only after a
full ~30s sync interval), then require the banned call to stay blocked across
several attempts; a pass-through after that window is a real leak, not a race.
* test(e2e): drop xfail-style rate-limit branch from openai_moderation test
OpenAI's /v1/moderations is free and returns 200 with the env key (verified
directly), so the RateLimitedError branch mislabeled the failure: a 429 there is
insufficient_quota (no account billing), not throttling. The branch also only
printed a softer message before failing anyway, an xfail-in-disguise the e2e rules
forbid. A 429 now falls through and fails loudly with the full result.
Add a live spend-tracking e2e that drives a streaming anthropic-format
/v1/messages request through litellm's anthropic-messages -> OpenAI Responses
adapter and asserts the consumed stream writes exactly one SpendLogs row with
nonzero cost and token counts, attributed to the calling key under
custom_llm_provider openai and the /v1/messages call_type.
The deployment is a Responses-only OpenAI model (gpt-5.3-codex), so a served,
costed row proves the Responses path was taken; the chat-completions bridge
would have failed at OpenAI on an endpoint the model does not expose. Adds a
streaming /v1/messages method to the shared Gateway and the suite client, the
model to the inline compose config and driver-model registration, a coverage
registry row (quota_management.spend_tracking.messages_bridge.logs_cost), and
the matching variant vocab entry. The _summarize spend-row detail also gains
call_type and custom_llm_provider so a failed assertion prints the fields it
asserts on.
Resolves LIT-4546
* test(e2e): harden stage flakes for batches, UI, and MCP
Unique batch model names avoid load-balancing onto stale azure-batch
deployments that still pointed at the retired gpt-4.1-mini-batch, which
only the managed/unified path was hitting. Retry batch retrieve on 500
and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite
when the compose-only mcp-upstream is unreachable on stage k8s
* test(e2e): cover Datadog remote MCP via search_datadog_logs
Register the regional Datadog MCP endpoint with DD-API-KEY /
DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is
not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*,
assert the proxy shipped it, list tools, call search_datadog_logs for the
marker, and delete the server on teardown. Math-upstream key-access tests
only skip when that compose service is unreachable
* test(e2e): drop compose math MCP upstream; use Datadog only
Key-access denial and happy-path MCP e2e both register the real regional
Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers.
Remove the mcp-upstream compose service and FastMCP add/multiply fixture
* docs(e2e): require real Datadog MCP for all mcp suite tests
Document that tests/e2e/mcp must register via datadog_mcp helpers against
mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams
* chore: restore mcp_e2e_upstream_server.py
Keep the FastMCP fixture file; e2e no longer wires it in compose, but the
module itself is not part of the Datadog-only cleanup
* fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load
pytest on the host never inherited compose env_file keys, so DD_API_KEY
stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the
dynamically loaded datadog_reader module in sys.modules so dataclasses
do not crash under Python 3.12
* test(e2e/batches): harden azure/vertex unified lifecycle flakes
Put the provider deployment name in every JSONL body so Azure does not
depend on a perfect model rewrite. Retry create/retrieve/cancel on
transient statuses with backoff. Drop cancel assertions for azure and
vertex (registry only has a shared basic cell; create+retrieve prove
routing, cancel stays best-effort cleanup)
* test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED
Post-login client redirects abort the first /ui/api-keys/ goto on stage.
Wait off /ui/login after cookie set, then accept the page once Create New
Key is visible even if goto raised ERR_ABORTED
* test(e2e): drop flaky key models dropdown Playwright suite
API management e2e already covers key generate/update persistence. The
UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and
no unique product signal. Remove the suite and unused browser fixtures
* refactor(e2e): fold claude_code HTTP probes onto shared Gateway methods
Migrate tests/e2e/claude_code/http_probe.py off its own httpx client onto the
shared transport, and promote count_tokens and native anthropic messages to
first-class Gateway methods (Gateway.count_tokens / Gateway.messages) with typed
request/response models in the shared models.py so other suites reuse them.
The probes now take an injected Gateway and issue their request through the
shared count_tokens/messages methods, reusing the split control/data-plane
routing, timeout, and typed Result handling the rest of tests/e2e uses. The wire
shape is preserved: the pydantic bodies serialize byte-for-byte to what the old
httpx probes sent, and the anthropic-version header is carried by a small
AnthropicHeaders model. httpx is gone from the module.
* test(e2e): drop unit-level probe harness test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CodSpeed benchmarks the SDK with no IO, so it can't catch regressions that
only appear under real concurrent load through the full proxy stack (auth,
routing, logging, spend, Postgres, Redis). This adds a Locust load test under
tests/e2e/load that drives concurrent POST /chat/completions traffic against a
mock deployment (litellm_params.mock_response), so the measured throughput
reflects proxy overhead rather than a provider's latency, and asserts an
aggregate RPS SLO with a failure-ratio guard. The test is marked load and the
parent conftest sorts load-marked items last so it never perturbs
latency-sensitive suites. Covers reliability.perf.throughput.under_slo.
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
Add an e2e suite at tests/e2e/mcp/ that proves MCP authorization over the
api_key auth family. An admin registers an upstream MCP server through the
management API (POST /v1/mcp/server, persisted in the DB and picked up without
a restart) and queues its deletion. Two keys are created against that one
server: one granted access through object_permission.mcp_servers and one with
no MCP grant. The permitted key is a live control proving the upstream is
reachable and the tool is callable, so a denial on the ungranted key is an
authorization decision rather than a dead server. The denied key then sees
none of the server's tools on tools/list and is refused a tools/call with a
403 access_denied.
A deterministic self-hosted FastMCP upstream (add/multiply over
streamable-http) is added to the e2e compose stack so the suite runs offline
with a known tool set. KeyGenerateBody gains an optional typed
object_permission so the shared gateway can create a key with an MCP grant.
The e2e docs claimed `e2e`-marked tests skip when no proxy answers the
liveness probe, but the harness has always hard-failed: conftest.py's
pytest_runtest_setup calls pytest.fail, its module docstring states
"hard failures only ... never skip", and logging/conftest.py forbids
skipping outright. Align the docs to the code so the single most
important contract reads the same everywhere; a dead proxy turns a run
red instead of being silently skipped and mistaken for a pass. The
per-suite conftest docstrings that described the shared hook as a
"proxy liveness skip" are corrected to "liveness gate" for the same
reason.
Also scope the no-unit-tests hard rule to what it means: never
substitute a unit test for e2e feature coverage, while explicitly
allowing tests that cover the harness itself (e.g.
coverage_registry/test_collector.py), which carry no e2e marker and
run whether or not a proxy is up.
No product code and no harness logic changed.
Resolves LIT-4554
