* test(e2e): add a HashiCorp Vault secret manager lane
key_management_system had no end-to-end coverage: the Rust crates and the
Python unit tests all run against mocked managers. This adds a secret_manager
suite that drives a proxy configured with hashicorp_vault against a real Vault.
The tests seed a fresh secret name per test with the runner's OPENAI_API_KEY and
register a deployment pointing at os.environ/<name>. The proxy's env never holds
that name, so get_secret's os.environ fallback cannot mask a broken manager, and
a bogus value in Vault must come back as the provider's 401. Virtual keys are
checked written to and removed from Vault under prefix_for_stored_virtual_keys.
The setting is global to the proxy, so the lane has its own config and the
secret_manager_vault opt-in marker, and stays out of the per-PR selector.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): make the secret manager suite backend-agnostic
One marker and opt-in (secret_manager / E2E_SECRET_MANAGER=<system>) pick the
backend from secret_backends.BACKENDS. The tests reach the manager through a
SecretStore protocol, and each backend contributes a secret_store_<system>.py
module, a registry entry, and gateway/secret_manager_<system>_ci_config.yml.
requires_capability deselects tests a backend cannot support (CyberArk does
not delete), and test_secret_backends.py checks every lane config against its
backend without a live stack.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): add a CyberArk Conjur secret manager lane
Adds cyberark as the second secret_manager backend: a Conjur store over its
REST API (policy-declared variables, raw-text values, policy-patch teardown),
its lane config, and a registry entry without deletes_stored_keys, since the
proxy's CyberArk delete answers not_supported and Conjur keeps the key.
secret_manager/backend.sh up|down <system> boots any backend in Docker and
writes proxy.env and tests.env, so every lane runs the same way; the registry
test checks the script boots exactly the registered backends. e2e_http gains
send_text_external for APIs that speak raw text rather than JSON.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(e2e): give the secret manager suite a client with .proxy and address review
The shared resources fixture reads client.proxy, so a bare ProxyClient errored every
live test at setup. backend.sh now writes its env under a per-user directory with
umask 077, the markerless unit tests are gone per tests/e2e/AGENTS.md, and routine
comments are trimmed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): hold every worker under an idle RSS budget before any traffic
The harness reads /debug/memory/summary on every replica once at collection
time, right after the readiness gate and before this pytest process sends any
traffic, and the memory suite's first test fails when any worker idles past
E2E_MEMORY_IDLE_RSS_BUDGET_MB (768 MB by default) or gives no reading at all.
A v1.100.x worker with a database idled at 836-886 MB where v1.101.0rc1 idled
at 544 MB on the same database: prisma-client-py's default recursive type
depth generated 91k TypedDict classes that v1.101.0's recursive_type_depth = -1
cut to 19k. The budget starts at the rc1 reading plus headroom.
* test(e2e): read idle RSS only when the idle budget test is selected
Gate the collection-time /debug/memory/summary read on a selected test using
the idle_rss fixture and skip it under --collect-only, so sessions that never
run the idle budget test pay no round trip. Drop the markerless unit test file
the e2e guide bans and assert live that every configured replica was measured
* test(e2e): take the idle RSS read after collection settles
Read every replica's RSS from a tryfirst pytest_collection_finish hook so -k
and -m deselection has already run, and only when a selected test still asks
for the idle_rss fixture and the run is not --collect-only
* test(e2e): record the heaviest idle RSS reading as junit properties
* test(e2e): attach the idle RSS properties from the harness's setup hook
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): assert OTLP HTTP TLS behavior against a real TLS sink
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): assert rejected exports by outcome, not by exception type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): honor SSL_CERT_FILE and ssl_verify in the v2 OTLP HTTP exporters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): hoist otlp_tls imports and type the TLS sink fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): gate the OTLP TLS export test behind an otel_tls opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(e2e): drop CONTRIBUTING.md edit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* test(e2e): assert a cooldown reaches a sibling replica within the 1s Redis read interval
* test(e2e): skip the sibling replica cooldown cell when one gateway URL is named
* test(e2e): collapse repeated gateway addresses so the sibling cooldown cell skips instead of erroring
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(otel v2): map completions, images, speech, transcription and moderation output onto the generation output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(redaction): redact text completion choices in the standard logging payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): compare decoded generation output text and follow the live moderation verdict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel v2): compare logged byte counts with the received media and move e2e schemas into models.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): keep the otel_v2 Langfuse output e2e file out of the stage-mirror gate it cannot run in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Buildkite ephemeral stack runs the gateway in another pod, so it cannot reach the pytest host's provider edge. The GitHub changed-e2e lane runs gateways on the runner and sets E2E_PROVIDER_EDGE_HOST_REACHABLE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A deployment registered while a module- or class-scoped fixture is being set up
was bound to whichever test asked for the fixture first, so every later test in
the module shared that partition. A session-scoped fixture is set up by every
xdist worker, so its deployment could never have one owner at all.
The e2e conftest now wraps pytest_fixture_setup and records the node the fixture
is scoped to: registrations made during a module or class fixture's setup carry
that node's slug, and a session- or package-scoped one has no owner and stays
live. The registration seam test moves from tests/e2e to the cache harness tests
beside the rest of the attribution coverage.
The cache edge keyed every recording on its own process's PYTEST_CURRENT_TEST.
Under xdist that names whatever test the serving worker is in, which is
unrelated to the caller: the proxy is a separate pod, and the Claude Code compat
matrix registered its shared aliases from every worker, each pointing at that
worker's edge, so the router spread one worker's calls across all eight edges.
Builds 234 and 235 of litellm-e2e, same commit, credited the same Bedrock
request to unrelated tests 92% of the time, and Bedrock never converged past a
~20% hit rate while OpenAI, whose deployments are per test, sat at 90%.
A deployment registered from inside a test now carries its test's slug in the
edge URL it is pointed at, `{edge}/{mount}/t/{slug}`, and the edge reads that
segment off every request before forwarding. A request without one is forwarded
live and never cached, and the edge no longer falls back to process state. The
compat aliases are registered with provider_live=True and stay on their real
provider path: no single test owns them, and the matrix exists to prove the real
CLI against real providers.
It drives the real CLI for several seconds. The edge stamps every
upstream call with PYTEST_CURRENT_TEST, a process-global that names
whichever test the worker is in when the call arrives rather than the one
that made it, so a test that holds a worker that long collects other
tests' in-flight calls. Build 234's key report credits this test with 20
Bedrock and 7 Anthropic misses, and it makes no provider call at all.
Those misattributed calls take the wrong test id into the cache key and
write recordings under it, so the test was polluting the shared corpus it
exists to protect.
Deselected unless E2E_CLI_DETERMINISM is set, the same opt-in shape the
managed-files, prompt-caching and redis-chaos markers already use. The
attribution bug itself is older than this branch and is reported, not
fixed here.
Each route now has litellm/rust_bridge/<route>/{entrypoints,callbacks}.py and a
public dispatch module (litellm/chat_completions/dispatch.py,
litellm/responses/dispatch.py, litellm/messages/dispatch.py) that binds the
public call to the legacy Python signature, builds a frozen request, and asks
the runtime to pick Rust or Python from the catalog. The legacy implementations
stay in litellm/main.py, litellm/responses/main.py and the anthropic messages
handler, and litellm/__init__.py re-exports the dispatch names over them the
same way it already does for ocr
The per-handler shims in rust_bridge/chat_completions/native.py and
rust_bridge/messages/native.py are removed along with their call sites in the
anthropic and bedrock chat handlers and the http handler. The exception
mapping that every callbacks module repeated moves to rust_bridge/failures.py
and the signature binding helpers to rust_bridge/public_call.py
create-release.yml is back to calling the chaos test through no mechanism at
all; it never called it. Also drops the E2E_REDIS_CHAOS opt-in gate itself:
the redis_chaos marker still exists for -m selection and is still excluded
from the per-PR selector by path (tests/e2e/(ui|claude_code|load)/), but the
test no longer needs an env var to run once its file is targeted.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The sequential version sent one request at a time, so a Redis outage never
reached the concurrency where the failed-tracking alert body actually grows.
This drives the proxy with locust against one model group of three mock
deployments, two failing at order 1 and one serving at order 2, so every
request spends its retries on the failing pair and lands on the serving
deployment through the order-based fallback. Two phases, a healthy baseline
and a CLIENT PAUSE WRITE window, and every request must succeed in both.
Latency, RSS and CPU are reported as p50/p90/p99 per phase rather than
asserted on: RSS and CPU come from psutil on the proxy's process tree, since
a multi-worker proxy serves /metrics from the prometheus multiprocess
collector and that drops the process collector's series. Thresholds stay open
until weekly runs give real baselines.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Add tests/e2e/router/test_redis_timeout_e2e.py against a proxy booted from
tests/e2e/gateway/redis_timeout_ci_config.yml: a real Redis with socket_timeout 0.001, so every
command times out and the circuit breaker opens, plus a primary deployment that always fails and
falls back to a healthy one, so every request carries retry breadcrumbs into cost tracking. The
test drives twenty chat requests through the proxy and asserts each answers within ten seconds,
the last third is no slower than the first, /health/liveliness stays fast, and every request
still reaches the spend log.
Gate it behind the redis_timeout marker and E2E_REDIS_TIMEOUT, exclude it from the per-PR
e2e-changed selector, register the reliability.circuit_breaker.redis_timeout.stays_responsive
cell, and run it as its own job in the weekly load anomaly workflow with a Postgres and Valkey
service. Against a v1.100.0 proxy the run wedges the worker: requests time out and liveliness
stops answering (LIT-6780).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
The changed-tests workflow overrode the suite's `--reruns 1` with `--reruns 0`, so a
transport blip failed a pass that pytest.ini already scopes to network errors and
5xx responses. Pass 2 of run 33692484803 also went red 15s after a model write with
"no healthy deployments": the barrier only polled /v1/models through nginx, which
proves one gateway converged, and the next request rolled the other. The stack now
exports LITELLM_PROXY_REPLICA_URLS, the barrier polls every replica with the full
budget before settling, and up.sh refuses to boot without DD_API_KEY, since the
gateway config enables the datadog callback on every run
The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.
The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.
Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
The prompt-cache affinity test needs the prompt_caching pre-call check on the
proxy, which the CI stack does not carry until project-releaser #223 lands, so
it now sits behind a prompt_caching_stack marker that is deselected unless
E2E_PROMPT_CACHING_STACK is set, the same shape as managed_files. The
per-directory deselection hooks for weekly and managed_files move into the
parent conftest as one OPT_IN_MARKERS table, so the collector counts a gated
cell only where its env var is set (31/36 today, 32/36 with #223).
The simple-shuffle cell asked for three picks, which a shuffle ignoring the
weights passes one time in eight; it now asks for ten.
* test(e2e): send no-cache on every cacheable request body, opt in only where a hit is the assertion
The e2e proxy runs with the response cache on, so any test that re-sends an
identical chat, messages, responses, completions, embeddings or rerank body
reads back a redis copy of an earlier call instead of reaching the provider.
Five tests in the last week failed that way. Default cache: {"no-cache": true}
on those request models and pass cache=None only in the two tests whose
assertion is the cache hit itself.
* test(e2e): give image edits and OCR a 180s client timeout
Both routes wait on providers that can legitimately take longer than the
60s transport-wide request timeout (gpt-image edits, Azure Document
Intelligence), and a client-side read timeout there fails a green request.
post/upload now accept a per-call timeout like get already does; only those
two call sites use it.
* test(e2e): rerun once on network errors and upstream 5xx only
Assertion failures still fail on the first attempt; only an outcome whose
error string carries the e2e_http network kind or a 5xx status gets one
more try. Test Engine records every attempt, so the flake rate stays
visible while a single provider blip no longer reds the rc run.
* test(e2e): let the reseed burst survive one upstream failure and print why
The burst is the precondition, not the property: one 5xx among six
concurrent calls still leaves five workers racing the cold counter, which
is what the reseed assertion measures. Two or more failures still abort,
and the failing bodies are now in the message instead of only the status
codes.
* test(e2e): keep polling Jaeger through a transient query failure
poll_traces_for_call already waits up to POLL_TIMEOUT for spans to land,
but a single refused connection to the query API failed the test on the
spot. Jaeger restarted twice during today's gate runs (19:05 and 19:41
UTC, each under a minute) and took ten and three otel tests with it while
the same tests passed on the rc build minutes later. A network failure
now counts as not-yet inside the same deadline; if Jaeger is still
unreachable when the deadline passes the test fails with that error, and
any non-network failure still fails immediately.
The websocket routes under /openai_passthrough and /openai had no e2e
coverage, so nothing catches the regression from issue #36088, where both
prefixes carried HTTP routes only and refused every upgrade with a 403
before a socket ever existed.
Two tests cover it. The realtime one opens /openai_passthrough/v1/realtime
and asserts OpenAI's own session.created frame comes back, which proves the
route is registered and relayed upstream. The responses one asserts
/openai/v1/responses accepts the upgrade, since a responses.connect socket
waits for the client to speak first and has no opening frame to check.
A refused upgrade is an HTTP response rather than a close frame, so both
assert on the handshake. ws_base_url moves into e2e_config now that a
second suite needs it
Replaces the test-side fixture transport with an in-process provider-edge
HTTP server the proxy's deployments point their api_base at. Record forwards
provider calls verbatim and writes them to the bundle; replay answers them
from the bundle with zero provider calls while key auth, routing, cost
calculation, and spend-log writes still execute against the live proxy and
database. Drift comes back as HTTP 599 naming the computed and closest
recorded keys. Request headers are never stored and responses are kept
byte-identical between modes from the proxy's side of the socket.
E2E_FIXTURE_MODE selects the transport every e2e client is built on: live
(default, unchanged behavior), record (pass through to the live proxy while
writing every interaction to a fixture bundle), or replay (serve every
interaction from the bundle with no proxy and no provider spend). Both new
transports fulfil the existing Transport protocol, so no test changes shape.
A bundle is a directory with a manifest (record timestamp, harness version,
format version) and one JSON file per interaction, grouped per test in call
order. Replay against a manifest older than seven days hard-fails at
collection time naming the bundle age. Record always wipes and never reads
the previous bundle, refusing to wipe a directory that is not a bundle.
Auth header values are redacted on write; uploads store a sha256 digest.
unique_marker() becomes deterministic per test in record/replay modes so a
replay run regenerates exactly the requests the record run sent.
Content-based match keys, streaming chunk fidelity, and provider-scoping are
follow-ups (LIT-5741, LIT-5742, LIT-5745).
The suite already waits for a new model or agent to become servable before
handing it back, but that wait returns on the first successful read. Every
request opens a fresh connection (e2e_http calls requests.* with no Session), so
a load-balanced Service routes each one independently: one successful read proves
one replica converged, and the caller's next request re-rolls and can land on a
replica that has not reloaded yet.
At replicaCount: 2 this surfaced as 30 failures on a SHA that is green at 1
replica -- 400 "Invalid model name passed", 404 "Guardrail not found", "no
healthy deployments for this model", and a /model/info listing that contained
one of two models created moments apart.
Add PROPAGATION_TIMEOUT (default 15s, override E2E_PROPAGATION_TIMEOUT) and
settle_propagation(), sized off the proxy's proxy_config_reload_interval_seconds
(30s by default, 7s on the e2e stack) plus margin, and settle after every
control-plane create whose object the suite then uses:
- ProxyClient.create_model and A2AClient.register_agent, after their existing
polls -- the poll still fails loudly if the object never appears at all
- GuardrailsClient.register, which had no barrier; create_content_filter_guardrail
and create_bedrock_guardrail now route through it instead of POSTing directly
- the guardrail creates in mcp_client and logging_client
- the vertex passthrough model, whose body cannot go through create_model
Left alone: the /model/new calls that assert a 403 or read back a status code,
since they never use the model.
The floor was an absolute fleet number, so it asserted replicas x per-replica rate
and went red on how many gateway pods happened to be warm rather than on the request
path. The test now measures one replica first, with a short serial pass that only ever
occupies a single pod, and requires the concurrent phase to reach at least that rate.
A serial latency budget carries the request-path assertion the floor used to imply,
and both hold at one replica or seven.
Zero-error runs that "sustained 16.7 RPS" were queueing, not slow requests: the load
model is a mock_response deployment with no upstream, a single-worker replica serves
it in about 57ms, and 100 closed-loop users against 1/0.057 RPS of capacity sit at
6s each by Little's law.
The runner also kept locust's --json summary and threw away everything else, so a run
where 93% of requests failed said nothing about what they got. It now passes --csv,
reads the failure breakdown back, and reports locust's own generator-saturation
warnings, both folded into the assertion messages.
Resolves LIT-5054
* fix(e2e): stop the cache-settings test from persisting a degraded Redis config
TestCacheSettings.test_update_persists_cache_backend_to_get read the live cache
settings and wrote them back, intending a no-op. Its capture modelled only
type/host/port, so on a TLS cluster the write-back silently dropped `ssl` and
`redis_startup_nodes`.
That is not recoverable on its own. `/cache/settings` persists what it receives
into LiteLLM_CacheConfig, that row outranks the YAML `cache_params`, and
init_cache_settings_in_db re-applies it on a timer, so a restart does not clear
it. The proxy ends up driving a TLS-only cluster endpoint as a plaintext
standalone node and every Redis call blocks to socket timeout.
On the affected deployment that took out rate limiting entirely (the v3 limiter
is a Lua script on Redis with no DB fallback), Redis-only budget levels (tag,
per-model, team-member, per-window), spend tracking, `ResetBudgetJob` (which
self-starved at 54 skipped runs per 15 min), and `ProxyConfig.add_deployment`,
whose last statement syncs guardrails and never ran. 60 of 72 failures in one
run traced back here.
The settings blob is now round-tripped verbatim via a RootModel over an
exhaustive value union, so a subset cannot be written. Two guards make a
regression fail loudly at this test instead of silently downstream:
- refuse to write when GET reports redis_type=cluster but omits
redis_startup_nodes, which is the exact precondition for persisting a
downgrade. GET resolves the stored row overlaid with REDIS_* env and never
reads YAML, so a cluster configured only in YAML cannot round-trip here
- compare /cache/ping before and after, so a write that breaks connectivity
fails this test rather than every suite that follows
The underlying product defect is filed as LIT-4816: GET cannot express the
effective config, and a partial POST is allowed to downgrade transport. This
change only stops the suite from triggering it; the Admin UI can still do so.
basedpyright clean (0 errors) under the e2e gate.
* fix(e2e): scope the bedrock guardrail per request and send OpenAI's current token param
Two failures that had nothing to do with the guardrail or route under test.
create_bedrock_guardrail registered with default_on=True, which applies the
guardrail to every request the proxy serves. The upstream ApplyGuardrail call was
answering 403, and that came back to unrelated traffic as
`403 Bedrock guardrail request failed`, failing three a2a tests and a passthrough
headers test alongside the bedrock one. The harness already supports the
per-request `guardrails` selector, so the guardrail is now registered opted out of
default_on and selected by the test that wants it. A broken upstream guardrail
fails its own test instead of whatever else is running.
Note this only contains the blast radius; the 403 itself still needs the
bedrock:ApplyGuardrail permission (or a valid guardrail identifier) on the
deployment, so test_bedrock_pre_call_blocks_harmful_prompt can still fail on its
own until that is sorted.
The OpenAI passthrough body sent `max_tokens`, which newer models reject with
"Unsupported parameter: 'max_tokens' is not supported with this model. Use
'max_completion_tokens' instead." Passthrough forwards the body untranslated, so
drop_params does not apply and the body has to satisfy OpenAI's contract
directly. vllm_chat keeps max_tokens, which vLLM accepts.
basedpyright clean (0 errors) under the e2e gate.
* fix(e2e): drop the pinned a2a api_key that broke every message/send
#34512 pinned `api_key="os.environ/ANTHROPIC_API_KEY"` on the a2a bridge agent.
The a2a bridge forwards the agent's litellm_params straight into
litellm.acompletion() without expanding "os.environ/" indirection, so that literal
string was sent upstream as x-api-key and every message/send failed with
`AnthropicException - {"type":"authentication_error","message":"invalid x-api-key"}`.
Omitting api_key restores the normal provider resolution: litellm reads
ANTHROPIC_API_KEY from the proxy's own environment for this provider, which is what
the agent-owner flow depends on and what the suite did before #34512.
Verified against a live proxy, same agent shape each time:
api_key omitted -> message/send 200
api_key "os.environ/ANTHROPIC_API_KEY" -> message/send 500 invalid x-api-key
api_key <literal key> -> message/send 200
and the key itself is valid (direct call to api.anthropic.com returns 200), so this
was indirection that never got expanded rather than a bad credential.
This accounts for four failures (test_semver_protocol_version_registers_and_serves,
test_message_send_runs_completion_bridge, test_pinned_v0_3_serves_flat_message_shape,
test_pinned_v1_0_serves_nested_message_shape). They were previously reported as
`403 Bedrock guardrail request failed`, because a default_on Bedrock guardrail
short-circuited the request before it ever reached the bridge and hid this.
The bridge silently ignoring "os.environ/" in agent params is a product defect in
its own right, filed separately; anyone configuring an agent credential that way
through the UI hits the same wall.
basedpyright clean (0 errors) under the e2e gate.
* test(e2e): make the load suite less aggressive against a shared proxy
750 users at spawn rate 50 saturated the request path hard enough to distort the
latency-sensitive suites sharing the same proxy, and it spends real provider money
at that rate. Drop to 200 users at spawn rate 20.
The RPS floor moves with the user count rather than staying put, so the assertion
keeps its meaning instead of becoming a formality: 355 RPS over 750 users is
~0.47 RPS/user, and 90 over 200 holds that same per-user expectation with a
similar pass margin. A request-path regression still trips it.
All four knobs stay env-overridable (E2E_LOAD_USERS, E2E_LOAD_SPAWN_RATE,
E2E_LOAD_DURATION_SECONDS, E2E_LOAD_MIN_RPS) for a deliberate load run.
Note the recorded failure for this test was "no requests completed in 60s", which
was the gateway wedged on unreachable Redis rather than a throughput regression;
this change is about not perturbing its neighbours, not about that failure.
* fix(e2e): make the reasoning-tokens assertion exercise a request that reasons
test_openai_chat_reasoning_reports_reasoning_tokens asked "A train travels 60 miles
in 1.5 hours. What is its average speed in mph?" at reasoning_effort="low", then
asserted reasoning_tokens > 0. The model answers that directly without reasoning, so
0 is correct behavior and the assertion was testing the model's discretion rather
than litellm's reporting.
Verified against a live proxy on a dedicated openai/gpt-5.6 deployment, matching how
the test provisions its model:
reasoning_effort=low, one-step arithmetic -> reasoning_tokens=0
reasoning_effort=high, the prompt used here -> reasoning_tokens=114
Raised to high effort with a prompt that requires a proof plus a search, so the
field under test is actually populated and the assertion fails only if litellm stops
surfacing it.
While confirming this I also checked prompt caching, which needed no change:
cached_tokens comes back 3615 of 3618 prompt tokens on a repeated large prefix
against a dedicated deployment. An earlier reading of 0 was an artifact of probing a
fan-out alias whose requests land on different deployments, not a caching defect.
* test(e2e): skip the files-list test while LIT-4820 is open
GET /v1/files does not include a just-uploaded file. The upload returns 200 and
GET /v1/files/{id} resolves it, but the listing never contains it: the returned set
stays fixed at 27 entries whose newest created_at is roughly ten hours older than
the upload, on both the managed (/v1/files?model=) and provider-scoped
(/openai/v1/files) routes. Polled for 40s, so not an eventual-consistency window.
Filed as LIT-4820. Skipping keeps a known, ticketed product bug from holding the
suite red and masking a new regression somewhere else in the same test.
The assertion is left exactly as it was on purpose. It encodes the contract we
actually want, that a file retrievable by id is also enumerable, and anything that
lists files (a UI picker, cleanup tooling that lists then deletes and would
therefore leak provider-side files) depends on it. Relaxing it to get green would
delete the signal. The skip reason says so and links the ticket, and the ticket
records that removing this marker is part of its definition of done.
Matches the existing pattern in this file, where test_unified_file_and_batch_create
skips with a reason citing LIT-3266.
While skipped, the registry cell llm.files.openai.list.nonstream.works has no
passing covering test, so files-list coverage reports as uncovered rather than
passing, which is the honest state.
* fix(e2e): parse Sentinel node lists in the cache-settings model
The value union covered scalar lists and lists of mappings, but not lists of
lists. `redis_startup_nodes` holds host/port mappings while `sentinel_nodes` holds
positional pairs (CACHE_SETTINGS_FIELDS documents `[['localhost', 26379]]`), so on
a Sentinel deployment pydantic rejected the response:
sentinel_nodes.list[dict[str,...]].1
Input should be a valid dictionary [input_value=['localhost', 26380]]
The round-trip test reads GET /cache/settings before it writes anything, so that
rejection failed the test at the read, before any assertion ran. A Sentinel
deployment would have looked like a broken cache-settings route rather than a
model too narrow to parse a documented shape.
A list element may now be a scalar, a list or a mapping, which covers both node
shapes without special-casing either and tolerates a heterogeneous list instead of
rejecting the whole response.
Adds TestCacheSettingsModel, harness-level with no `e2e` marker so it runs without
a proxy, covering all four backend shapes (cluster mappings, sentinel pairs, plain
node, url mode with a null discrete field) plus transport() key selection.
Confirmed it fails on the previous union and passes on this one:
old union -> 1 failed, 4 passed (the sentinel case)
new union -> 5 passed
* test(e2e): remove the cache-settings round-trip test
The test could not fail for the thing it claimed to test, and could break the
deployment it ran against. Both halves of that are worth stating.
It read the live settings, wrote back identical values, and asserted the read-back
matched. If POST /cache/settings were a complete no-op that returned 200 and touched
nothing, GET would still return the values read a moment earlier and the test would
pass. It verified that GET is stable, not that the route persists anything.
Against that, /cache/settings persists what it receives into LiteLLM_CacheConfig,
that row outranks YAML cache_params, and init_cache_settings_in_db re-applies it on a
timer. A write that omits ssl or redis_startup_nodes converts a TLS cluster into a
plaintext standalone client and every later Redis call blocks to socket timeout. On
2026-07-25 that failed 60 of 72 tests in one run: rate limiting stopped enforcing,
Redis-only budgets admitted billable over-budget spend, ResetBudgetJob self-starved,
and guardrail sync never ran.
Guarding the previous shape was not sufficient. Writing the blob verbatim plus a
cluster precondition and a /cache/ping check narrowed the hazard but did not remove
it, because GET cannot express the effective config: it resolves the stored row
overlaid with REDIS_* env and never reads YAML. On a fresh deploy it cannot see
YAML's ssl to echo back, so a TLS non-cluster deployment could still have a row
written that drops it. No round-trip through this route is safe on a shared proxy.
Removed with the models and helpers it owned, and TestCacheSettingsModel with them
since it existed only to protect that parsing.
The registry row mgmt.cache_settings.update.happy_path stays, now carrying the
rationale for why it is deliberately uncovered and what a safe test would require
(an isolated proxy, or LIT-4816 fixed so a partial write cannot downgrade
transport). Coverage therefore reports this cell as a gap, which is the honest
state. Collector passes --strict; the module still collects 11 tests.
require_env hard-failed a test (and, for the shared litellm-ops secret, drove
piling every provider credential into one blob) whenever an optional cred was
absent. Most call sites either read a value the test actually uses or just
gated on the runner's env for a key the gateway consumes.
Read os.environ directly where the test uses the value; drop the presence-only
gates so those cases run against the proxy instead of pre-failing on the
runner's environment. Removes the require_env helper from e2e_config.
A gateway whose native extension is unavailable falls back to the Python
implementation without raising, so it answers /v1/messages normally and the
only difference on the wire is the absent x-litellm-rust header. Nothing in
the suite read that header, so a Rust deployment that had stopped running
Rust produced a fully green e2e run.
Assert the marker on the streamed Messages assertions when E2E_EXPECT_RUST is
set. It stays opt-in because the same suite image also runs against the
standard gateway, which has no Rust path and must keep passing; the two
deployments are already separate Applications, so this is one value on the
Rust instance rather than branching inside the tests.
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping
Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly.
* test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse
Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL,
TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path,
Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm
chat, and Nova Sonic realtime. Registry cells updated for the new markers.
* test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes
Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and
Langfuse driver models now use Anthropic haiku so local runs stay green when
Gemini daily quota is exhausted.
* test(e2e): drop Langfuse spend suite; feature is being deprecated
Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the
langfuse-only conftest driver/credentials fixtures.
* test(e2e): fold provider/batch feature tests into their endpoint suites
Keep the e2e layout endpoint- and suite-scoped instead of one file per
provider or feature
Move the virtual-key auth case into access_control/test_access_control_e2e.py
as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone
test_virtual_key_auth_e2e.py
Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock
assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py.
The hosted_vllm batch case is skipped for now since it needs a live vLLM server
(HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the
gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant
Merge the cohere, gemini and hosted_vllm chat cases into
llm_translation/test_chat_completions_regression_e2e.py so /chat/completions
coverage lives in one endpoint file, and repoint the coverage_registry source
fields to the new homes
Move the shared CacheControl / TextBlock / RichMessage request blocks into the
root models.py (re-exported from endpoints_client) so quota_management can use
them without a cross-suite import, which also clears the basedpyright errors in
test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in
test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed
json.loads path
* test(e2e): address review feedback and re-home virtual-key coverage
Replace the tautological Bedrock assume-role batch id assertion (`startswith(...)
or batch.id`, always true) with a managed-id shape check, since the unified
target_model_names path re-encodes the id rather than returning a raw ARN
Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no
longer consume the key's sole request unit before batch create runs; the batch
create then clears the generic per-request limiter and the batch limiter is what
returns the "Batch rate limit exceeded" body the assertions check
Set exercised_on to [] on the pass-through header test; it drives a pass-through
endpoint, not /chat/completions
Move the virtual-key valid_allows / invalid_denied cells from other.yaml to
mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management,
and point its covers marker at the new ids
* test(e2e): harden stage flakes for batches, UI, and MCP
Unique batch model names avoid load-balancing onto stale azure-batch
deployments that still pointed at the retired gpt-4.1-mini-batch, which
only the managed/unified path was hitting. Retry batch retrieve on 500
and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite
when the compose-only mcp-upstream is unreachable on stage k8s
* test(e2e): cover Datadog remote MCP via search_datadog_logs
Register the regional Datadog MCP endpoint with DD-API-KEY /
DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is
not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*,
assert the proxy shipped it, list tools, call search_datadog_logs for the
marker, and delete the server on teardown. Math-upstream key-access tests
only skip when that compose service is unreachable
* test(e2e): drop compose math MCP upstream; use Datadog only
Key-access denial and happy-path MCP e2e both register the real regional
Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers.
Remove the mcp-upstream compose service and FastMCP add/multiply fixture
* docs(e2e): require real Datadog MCP for all mcp suite tests
Document that tests/e2e/mcp must register via datadog_mcp helpers against
mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams
* chore: restore mcp_e2e_upstream_server.py
Keep the FastMCP fixture file; e2e no longer wires it in compose, but the
module itself is not part of the Datadog-only cleanup
* fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load
pytest on the host never inherited compose env_file keys, so DD_API_KEY
stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the
dynamically loaded datadog_reader module in sys.modules so dataclasses
do not crash under Python 3.12
* test(e2e/batches): harden azure/vertex unified lifecycle flakes
Put the provider deployment name in every JSONL body so Azure does not
depend on a perfect model rewrite. Retry create/retrieve/cancel on
transient statuses with backoff. Drop cancel assertions for azure and
vertex (registry only has a shared basic cell; create+retrieve prove
routing, cancel stays best-effort cleanup)
* test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED
Post-login client redirects abort the first /ui/api-keys/ goto on stage.
Wait off /ui/login after cookie set, then accept the page once Create New
Key is visible even if goto raised ERR_ABORTED
* test(e2e): drop flaky key models dropdown Playwright suite
API management e2e already covers key generate/update persistence. The
UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and
no unique product signal. Remove the suite and unused browser fixtures
CodSpeed benchmarks the SDK with no IO, so it can't catch regressions that
only appear under real concurrent load through the full proxy stack (auth,
routing, logging, spend, Postgres, Redis). This adds a Locust load test under
tests/e2e/load that drives concurrent POST /chat/completions traffic against a
mock deployment (litellm_params.mock_response), so the measured throughput
reflects proxy overhead rather than a provider's latency, and asserts an
aggregate RPS SLO with a failure-ratio guard. The test is marked load and the
parent conftest sorts load-marked items last so it never perturbs
latency-sensitive suites. Covers reliability.perf.throughput.under_slo.
* fix(e2e): make the datadog read-back find what DataDog actually indexes
Live verification of the merged #33604 against real DataDog (us5) exposed
three read-back defects that the local-sink tests could never see; all
three fixes are verified against the real API:
- Marker search: DataDog consumes the shipped JSON message into the
event's attributes and leaves the indexed message EMPTY, so the
full-text '"marker"' query matched nothing and every test failed with
zero events. The query is now '*:*marker*', which scans all attributes
(the marker sits in messages.content); verified to return exactly the
event for the call.
- Rate limit: the Logs Search API budget is 2 requests per 10s org-wide
(x-ratelimit-name logs_public_search_api). Polling at POLL_INTERVAL=5s
sat exactly at the limit and the reader hard-failed on the first 429.
Searches now pace at DD_SEARCH_INTERVAL (10s default) and a 429 backs
off and retries up to 5 times; only non-429 failures stay hard fails.
- Envelope status: DataDog re-derives the indexed event status from the
parsed payload's status attribute ('success') and normalizes it to its
OK severity, so the assertion expects 'ok', not the shipped 'info'.
Live run: chat_completions and responses pass every assertion including
the exact response-cost cross-check; messages red-pins the LIT-4447
duplicate for real (one call -> two sync-sweep copies + one async batch
copy, same request id, confirmed in proxy debug logs). The duplicate is
race-dependent, so the pin flickers until #33589 lands.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): datadog log delivery for streamed chat, messages, and responses
Rewritten from the dd-sink version (original #33566) to judge delivery on
what real DataDog ingested, matching the merged #33604 conversion: the
dd_logs reader searches events back through the Logs Search API and the
assertions validate the indexed envelope (source:litellm tag, ok status)
and the StandardLoggingPayload fields under the event's attributes.
Each streamed test drives one STREAMED call per route, asserts the stream
actually streamed (event-stream content type, >0 chunks, no upstream error
event), then pins exactly one DataDog event whose payload records
stream=true, the aggregated token count, and a response_cost equal to the
/spend/logs row for the call - a stream's headers ship before its cost
exists, so the spend row is the cross-check anchor, and the spend row and
DataDog event must also agree on total_tokens.
Coverage registry: adds logging.datadog.stream.exports_metric exercised on
chat_completions, messages, and responses.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update test_datadog_log_e2e.py
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): read datadog log delivery back from the real datadog api (#33604)
* test(e2e): read datadog log delivery back from the real datadog api
* test(e2e): compare datadog-read cost with math.isclose, not bit-equality
The response_cost now round-trips through DataDog's attribute indexing
pipeline, whose float serialization is not guaranteed to preserve the
exact bit pattern the proxy shipped. rel_tol=1e-9 (equal to 9 significant
digits) still fails on any real cost discrepancy while tolerating
representation drift. Addresses the Greptile P2 on this PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): widen the duplicate-settle window to 30s for real DataDog
Against the local sink one poll interval (5s) after the first hit was
enough to catch a same-call duplicate, because both events arrived in the
same flush batch. Against real DataDog, ingestion jitter can make one
call's two events searchable tens of seconds apart, so a 5s settle could
let the LIT-4447 duplicate slip past the exactly-one assertion. The reader
now keeps re-reading for DD_SETTLE_SECONDS (default 30s, env-overridable
via E2E_DD_SETTLE_SECONDS) after the first event appears, returning early
only when a duplicate is already visible - more waiting cannot clear it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(e2e): point UI tests at dashboard service; register complexity router
Stage gateway 404s /ui; the Next.js dashboard is litellm-ui:3000. Drive
playwright against E2E_UI_BASE_URL and wait on login placeholders after
client render. Register complexity-smart-router via /model/new when the
proxy does not already list it so stage matches compose config
* docs(e2e): clarify E2E_UI_BASE_URL should be ALB when ingress splits UI
* docs(e2e): prefer single path-routing host for control plane and UI
CONTROL_PLANE and UI already default to PROXY_BASE_URL; clarify that
stage should set one ALB host rather than three endpoints
* fix(e2e): always capture complexity router model_id for teardown
Split /model/new from the data-plane wait so a propagation timeout still
deletes the control-plane registration (greptile orphan-model concern)
* fix(e2e): click exact Login button so SSO control is not matched
Playwright strict mode matched both Login and Login with SSO
* fix(router): score complexity by difficulty not request length
The LLM classifier prompt treated short wording as SIMPLE, so probes like
"Is P equal to NP?" stayed on the SIMPLE backend even though the classifier
ran. Judge intellectual difficulty so short hard questions route higher
* fix(e2e): open key edit via Key ID and wait for team models
Key Alias text is not the row open control on the virtual keys table;
KeyInfoView opens from the Key ID button in that row. Also wait for a
real team model in the edit Models dropdown so we do not race the async
availableModels fetch that only has All Team Models on first paint
* fix(e2e): keep settled DD events on empty search; bump mcp for OSV
Do not let a transient empty DataDog search wipe events already seen in
the settle window (Greptile P1). Make the logs-search from window
env-overridable via E2E_DD_SEARCH_FROM (Greptile P2). Prefer the mono
Key ID button when opening key edit. Bump mcp 1.26.0 -> 1.28.1 so OSV
clears the three high GHSA findings on the staging PR
* revert: drop mcp lock bump from e2e staging PR
OSV mcp upgrade is unrelated to the e2e fixes; leave the dep pin alone
---------
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): datadog log delivery for successful chat, messages, and responses
Covers logging.datadog.success.exports_metric on all three routes: one
successful non-streaming call must reach the DataDog logs intake as exactly
one log event whose StandardLoggingPayload message carries the model group,
real token counts, and a response cost equal to the x-litellm-response-cost
header of the same response. Delivery is judged at the intake: the compose
stack gains a dd-sink service recording every batch the datadog callback
ships via the DD_BASE_URL testing override, and a typed reader replays it.
Writing these caught a live product bug: /v1/messages double-logs every
success (two byte-identical events per call), filed as LIT-4447; the messages
test tolerates byte-identical duplicates of the one event until it lands,
while a second differing event still fails
* test(e2e): address review findings on the datadog delivery suite
Consolidates the fresh-key first_ok helper into logging_client now that the
otel PR it mirrored has merged (both test files use the shared copy), moves
intake batch parsing into a helper so no path can leave the batch unbound,
and gives the sink's /health endpoint a truthful text/plain content type
* test(e2e): tolerate same-logical-event duplicates by call id, not byte identity
A clean LIT-4447 repro showed the duplicated payload is built twice and can
mint a fresh synthetic completion id per emission, arriving as two separate
intake POSTs with the same litellm_call_id and identical substantive fields.
Byte-identity was therefore a flaky criterion; duplicates now qualify only
when they share the call id, call type, model group, tokens, and cost, and a
second differing event still fails
* test(e2e): assert the scenario strictly; the messages test is the LIT-4447 regression pin
Per review direction the tests now assert exactly what the scenario promises:
exactly one DataDog log event per successful call, on every route. The
/v1/messages test therefore fails on current code against the known
double-log (LIT-4447) and is its regression pin; it goes green when the fix
lands. The duplicate-tolerance machinery is removed
* Simplify docstrings for DataDog log tests
Removed redundant phrasing about cost cross-checking in docstrings.
* Update test_datadog_log_e2e.py
