/vllm/batches and /vllm/files are high-volume passthrough routes with no e2e
coverage. They ride litellm's generic /vllm/{endpoint} forwarder, so the test
uploads a JSONL through /vllm/v1/files and creates a batch through
/vllm/v1/batches (BatchClient with provider=vllm), asserting the forwarded file
and batch objects come back. Lives next to TestHostedVllmBatch and is skip-marked
for the same reason: no live vLLM server (HOSTED_VLLM_API_BASE) in the e2e env.
Adds the two llm-translation registry cells.
Enterprise MCP users mint virtual keys against tool access groups rather than
explicit server ids. Nothing exercised that end to end.
Registers the upstream MCP server tagged with a server-side access group
(mcp_access_groups), then mints one key granted that group and one granted a
different group. Asserts the granted key sees the tagged server's tools on
tools/list and the other key sees none, so access-group scoping can't leak the
server across the boundary.
Adds mcp_access_groups support to the e2e MCP client (server registration, key
generation, ObjectPermission) and the registry cell
mcp.list_tools.api_key.access_group_scoped.
The a2a completion-bridge tests registered the agent with only
custom_llm_provider and model, so the bridge's litellm.acompletion had no
api_key and relied on the gateway resolving ANTHROPIC_API_KEY from its ambient
env. When that env var is absent, POST /a2a/{id} returns 500 with
"Missing Anthropic API Key" and every message/send test (completion bridge,
pinned v0.3/v1.0 message shapes, semver serves) fails while the
register/discovery/rejection tests still pass.
Give A2ABridgeParams an optional api_key and register the bridge agent with
api_key="os.environ/ANTHROPIC_API_KEY", matching how the rest of the suite
wires anthropic-backed models (e.g. the ratelimit redis tests). The agent now
carries its provider key explicitly instead of depending on ambient gateway env.
* fix(proxy): restore atomic user upsert when adding team members
Parallel /team/new calls naming the same not-yet-existing member were
returning 500 "Unique constraint failed on the fields: (`user_id`)".
The upsert in add_new_member passed an empty update branch. Prisma only
compiles an upsert down to a single INSERT ... ON CONFLICT when that branch
writes something; with an empty one it emits SELECT-then-INSERT instead, so
concurrent requests all read "no such user" and all insert. Postgres
statement logs confirm it: the empty form logs BEGIN/SELECT/INSERT/COMMIT,
the non-empty form logs INSERT ... ON CONFLICT ("user_id") DO UPDATE SET.
Re-state user_id in the update branch as a no-op so the native upsert path
comes back. The teams append stays in the filtered update below it, so an
already-existing member still cannot pick up a duplicate team id.
tests/test_team.py::test_team_new failed 9 of 15 runs against a live proxy
before this and 0 of 15 after. The existing unit test asserted only that
upsert had been called on a mock, so it passed either way; it now pins the
shape of both branches and fails when the update branch goes back to empty.
* test: point the live codex tests at gpt-5.3-codex
OpenAI deprecated gpt-5.2-codex, so test_openai_codex and
test_openai_codex_stream started failing against the live API with
model_not_found. gpt-5.3-codex is the current codex model; both tests pass
on it. The remaining gpt-5.2-codex references in the suite are mocked
transformation tests and are unaffected.
* test(e2e): update models page specs for the shared DataTable
The DataTable migration in #34363 changed three things the models page
specs were pinned to, and five tests went red.
Row click no longer opens the detail view; the Model ID cell owns that
now, so both specs click its `model-id-<id>` test id instead of the row.
The search box placeholder switched from an ASCII "..." to a real
ellipsis, so the specs use getByPlaceholder with a substring instead of
an exact attribute match that punctuation can break again. The results
count moved from `models-results-count` ("Showing 1 - 50 of 137 results")
to the shared pagination's `pagination-range` ("Showing 1-50 of 137").
The Team-BYOK test also filtered rows on the team alias, which the Team
ID column has never rendered in either the old or the new table; it
filters on the team id now, which is what the column actually shows and
what the assertion's own comment intends.
Verified against a local proxy serving a fresh build with the seeded
e2e postgres and mock upstream: all five failing tests pass, and the
full suite is 82 passed / 4 skipped at CI parity (workers=1).
require_env hard-failed a test (and, for the shared litellm-ops secret, drove
piling every provider credential into one blob) whenever an optional cred was
absent. Most call sites either read a value the test actually uses or just
gated on the runner's env for a key the gateway consumes.
Read os.environ directly where the test uses the value; drop the presence-only
gates so those cases run against the proxy instead of pre-failing on the
runner's environment. Removes the require_env helper from e2e_config.
* test(e2e): cover key max_budget blocks on personal, team, and team-member keys
* refactor(e2e): convert budget enforcement cases to the resources-fixture pattern
The E2ECase class pattern existed only in this file; every other suite uses
plain pytest tests with the resources fixture. Rewrites the nine cases as two
spec classes and removes the now-dead E2ECase protocol and run_case driver
from lifecycle.py
A gateway whose native extension is unavailable falls back to the Python
implementation without raising, so it answers /v1/messages normally and the
only difference on the wire is the absent x-litellm-rust header. Nothing in
the suite read that header, so a Rust deployment that had stopped running
Rust produced a fully green e2e run.
Assert the marker on the streamed Messages assertions when E2E_EXPECT_RUST is
set. It stays opt-in because the same suite image also runs against the
standard gateway, which has no Rust path and must keep passing; the two
deployments are already separate Applications, so this is one value on the
Rust instance rather than branching inside the tests.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
* test(e2e): cover customer chat/messages cost + streaming paths
Fills five uncovered P0 registry cells matching the customer's confirmed stack
(OpenAI SDK, Bedrock, /v1/messages) and their per-request cost dependency:
- /v1/messages logs cost that matches the x-litellm-response-cost header (LIT-4076)
- OpenAI /chat/completions streams real content, and a non-streamed call is costed
- Bedrock Converse /chat/completions returns real content non-streamed and streamed
The streaming checks aggregate delta content and parse every chunk as JSON, so a
clean-but-empty stream or a truncated chunk fails instead of passing on a bare 200.
* test(e2e): add tool-use coverage for openai, bedrock converse, anthropic responses
Function-calling regression guards on the paths the customer's agentic SDK usage
exercises: OpenAI and Bedrock Converse /chat/completions, and Anthropic
/v1/responses. The model is forced to call a weather tool and the test asserts the
returned tool call names the function and carries JSON-parseable arguments with the
expected field, so a dropped tool_call or malformed argument JSON fails instead of
passing on a bare 200. Adds a minimal tool_calls field to the response OutMessage.
* test(e2e): cover bedrock converse responses + thinking
Adds llm.responses.bedrock_converse.basic/tool_use and
llm.chat_completions.bedrock_converse.thinking. The thinking test enables extended
thinking and requires reasoning_content plus a real answer, so a path that drops
the reasoning block fails rather than passing.
* test(e2e): cover bedrock embeddings + openai structured output and reasoning
Bedrock Titan embeddings return a real vector; OpenAI structured output must yield
schema-conforming JSON with the correct extracted values (age==42, not just valid
JSON); an OpenAI reasoning call must report reasoning tokens, so a non-reasoning
fallback fails. Adds response_format to ChatBody and reasoning-token details to Usage.
* test(e2e): cover vision + streaming tool calls on openai and bedrock converse
Vision on both providers must describe the image (not just 200); the streamed
OpenAI tool call is reassembled from its fragments and its argument JSON parsed, so
a stream that never completes the call or splits its JSON fails. Extends ChatMessage
content to a typed text/image union.
* test(e2e): cover openai prompt caching hit on repeated large prefix
A repeated large-prefix prompt must report cached prompt tokens on the second call,
so a cache regression that stops reusing the prefix (and silently re-bills full
input) fails here.
* test(e2e): cover openai audio speech + bedrock rerank and image generation
Marks the OpenAI TTS cell and adds Bedrock Titan rerank (top_n honored, scored) and
Bedrock Titan image generation (returns b64/url), the customer's non-chat AWS
surfaces.
* test(e2e): cover end-user (customer) create persistence
mgmt.end_user.new.happy_path: create an end-user via /customer/new and confirm
/customer/info reports it, the end-user-identity surface the customer relies on for
per-customer controls. Adds customer models + management-client methods.
* test(e2e): enforce key model allow-list on the passthrough route
other.auth.passthrough.model_allowlist_enforced: a key scoped to gemini must be
denied a claude call through the anthropic passthrough route (403), so custom-auth
scoping is not bypassable by going through passthrough instead of /chat/completions.
* test(e2e): address Greptile - assert stream data events, correlate messages spend by key
- streaming: assert len(stream_events) > 1 instead of chunks > 1, since chunks
counts the terminal data: [DONE] marker and would pass a single content event
- messages cost: correlate the spend row by the unique scoped key rather than the
Anthropic response id, which need not equal the proxy spend-log request_id
* fix(e2e): reference client.proxy in mid-conversation native providers test
EndpointsClient exposes the shared ProxyClient as .proxy and has never had a
.gateway attribute, so these two calls raised AttributeError at runtime and
failed the tests/e2e basedpyright zero-error gate for any PR touching e2e
files. Introduced in 23b5b7d199.
* test(e2e): cover 12 non-core LLM coverage registry cells
Raises Non-Core LLMs registry coverage from 24/50 to 36/50 (overall 51.9%
to 54.8%). Four cells were already asserted by existing tests and only
gain their covers marker (openai embeddings, openai image generation,
openai TTS, cohere rerank); one is dual-marked onto the existing
spend-tracking embeddings test rather than duplicated.
New tests: bedrock and vertex embeddings, streaming TTS (asserts chunked
transfer encoding so a buffered body cannot pass), audio transcriptions
via the realtime suite's wav fixture, moderations flag/pass pair, and
files list/retrieve in the batches suite.
Harness: e2e_http.upload generalized to any form model with a
file_content_type override (batches path unchanged), new stream_binary
primitive + BinaryStream for binary chunked responses, transcribe and
moderations client methods, file retrieve/list client methods.
* fix(e2e): close streamed TTS response on error paths and surface the error body
With stream=True a non-2xx response returned with the body unread, keeping
the socket checked out until garbage collection; the sibling
_streaming_outcome already consumes resp.text on error. The response now
closes on every path and BinaryStream carries a bounded error_body so a
failed stream call is triageable.
* test(e2e): assert streamed TTS response carries no content-length
httpbin.org is an external dependency prone to transient 503s (caused the
stage failure); its echo-body assertion also doesn't exercise a real LLM
provider. Point the custom pass-through endpoint at the real Anthropic
Messages API instead. Anthropic doesn't echo headers back, but it gates
real behavior on two of them, which is enough to prove forwarding: a
static x-api-key configured on the endpoint (never supplied by the caller)
must reach upstream or the call 401s, and an invalid x-pass-anthropic-version
sent by the caller must reach upstream with the prefix stripped, which
Anthropic echoes verbatim in its 400 body. Verified live against a local
proxy and the real Anthropic API: valid version returns a real completion,
invalid version returns the exact marker in the 400 body.
Adds get_external to e2e_http.py for absolute third-party GETs (no proxy base url or auth, same Result classification) and rewires fetch_agent_card through it, dropping the urllib.request escape hatch. Creates tests/code_coverage_tests/check_e2e_no_raw_requests.py, the checker tests/e2e/CLAUDE.md already referenced, and wires it into the code-quality workflow so raw HTTP client imports outside the transport fail CI; pre-existing uses (root conftest liveness probe, claude_code version resolver) are grandfathered and exception-type-only imports stay allowed.
* test(e2e): add Azure AI Foundry and Anthropic /v1/messages coverage
* test(e2e): add Azure AI Foundry + Anthropic messages coverage for the Rust bridge
* test(e2e): guard against an empty SSE stream in the Azure Foundry tool-use streaming case
* test(e2e): add other suite covering master-key auth and health lifecycle
Covers the other.* holding-pen cells that were uncovered: master-key
valid_allows/invalid_denied on the admin /user/list gate, and the
lifecycle probes liveness.ping, readiness.public_probe,
readiness.reports_db_status, and readiness_details.authenticated_diagnostics.
New tests/e2e/other/ suite on the shared ProxyClient; the health probes
send no auth header to prove the public routes need no credential, and the
details route is asserted to reject an anonymous caller while exposing
version/db diagnostics to the master key.
* test(e2e): cover block_code_execution and openai_moderation guardrails
Extends the guardrails suite with two built-in guardrails registered per
request (default_on=False, opted in via the chat body's guardrails selector)
so neither intercepts unrelated traffic on the shared proxy.
block_code_execution.pre_call.blocks: a python code block plus a run-this
request is intercepted with the canned content-blocked message and the model
never runs, while the same code block asked about with don't-run-it reaches
the model. Verified live.
openai_moderations.pre_call.blocks: a flagged prompt is rejected 400 naming
the moderation policy while a benign prompt passes. The guardrail calls
OpenAI's moderation API; verifying it needs an OpenAI key with moderation
quota (this account currently 429s the moderation endpoint).
Adds a shared create_backend_model helper and a generic register() plus
per-request guardrails/max_tokens on the client so more built-ins can reuse
the same path.
* test(e2e): cover presidio PII masking (pre_call + post_call)
Registers a presidio guardrail per request (default_on=False) with the
analyzer/anonymizer bases supplied in the registration params, so the test
controls its own dependency and needs no proxy restart.
presidio.pre_call.masks: a repeat-verbatim request comes back with the
<EMAIL_ADDRESS> placeholder and never the raw email, proving the prompt was
anonymized before the model saw it.
presidio.post_call.masks: with apply_to_output the model's own emitted email
is masked on the way out, so the caller never receives the raw value.
Both verified live against real presidio analyzer + anonymizer containers.
logging_only is intentionally not covered: /spend/logs exposes no prompt
messages to read back the masked log, and a logging_only run also masked the
response, contradicting its contract; noted in the module docstring for a
follow-up.
* test(e2e): cover presidio logging_only masking via OTEL read-back
Adds the third presidio cell, guardrail.presidio.logging_only.masks. The
logging_only contract (mask what is logged, do not block) is verified by
reading the request's gen-AI span back from the real OTEL destination: the
span's gen_ai.input.messages attribute carries the <EMAIL_ADDRESS> placeholder,
never the raw email, and the call itself is not blocked.
Reads the trace via the shared OtelReader, promoted from logging/ to the suite
root so both suites use it. The masked prompt is polled to a deadline because
logging_only masks the payload asynchronously and the span can briefly export
before the mask lands. Drops the throwaway chat_send in favor of the existing
transport.send for the call-id capture.
* fix(e2e): tolerate cross-pod guardrail sync delay in team-opt-out test
Stage runs multiple gateway pods behind the shared key. POST /guardrails
registers a new default-on guardrail in-process immediately only on the
pod that served the create call; every other pod picks it up on its next
periodic DB sync (proxy_server.py, every 30s), so the very next chat call
can race a pod that has not synced yet. Poll to a 40s deadline instead of
asserting on the first response, matching the existing pattern in
test_budget_reset_advances_e2e.py.
* test(e2e): cover a guardrail on the MCP tool-call path (content_filter pre_mcp_call)
Adds guardrail.litellm_content_filter.pre_mcp_call.blocks: against the real
Datadog MCP server, a content_filter guardrail configured mode=pre_mcp_call
blocks a banned keyword in an MCP tool call's arguments with HTTP 400 attributed
to the pre_mcp_call hook, and lets a clean argument reach the upstream server.
The guardrail attaches with default_on because per-key/request guardrail
selection is dropped from the synthetic MCP request the hook sees; the banned
keyword is unique per run so default_on only intercepts this test's own call.
mode must be pre_mcp_call - a pre_call config silently no-ops on tools/call
because the event type is rewritten for call_mcp_tool.
Drives the tool directly via /mcp-rest/tools/call for a deterministic check of
the same pre_mcp_call enforcement the OpenAI-SDK chat path hits when a model
invokes an MCP tool.
* fix(e2e): mid-conversation messages test uses client.proxy not client.gateway
EndpointsClient exposes .proxy after the Gateway->ProxyClient rename; the
mid-conversation system test still referenced .gateway, which fails the e2e
basedpyright gate. Aligns it with the rest of the harness.
* test(e2e): address review on the guardrail coverage
MCP tool-call guardrail: poll the banned call until the guardrail is enforced
instead of asserting on the first call, so the control-plane -> data-plane
guardrail sync cannot race the check into a false pass-through; add a repeat
banned call after enforcement to guard against a partial-propagation state.
OpenAI moderation: distinguish a moderation-endpoint 429 (rate limit / no
moderation quota) from a guardrail failure, so an account-capability gap reads
as such rather than as "did not block". Runs green with a moderation-capable key.
* test(e2e): close partial-propagation false-pass in MCP guardrail block test
The single post-block repeat call could be load-balanced back to the same
already-synced data-plane pod, so the test could pass while another pod still
lacked the guardrail and let the banned MCP call reach Datadog. Anchor a wait to
the guardrail create time (every pod is guaranteed to have DB-synced only after a
full ~30s sync interval), then require the banned call to stay blocked across
several attempts; a pass-through after that window is a real leak, not a race.
* test(e2e): drop xfail-style rate-limit branch from openai_moderation test
OpenAI's /v1/moderations is free and returns 200 with the env key (verified
directly), so the RateLimitedError branch mislabeled the failure: a 429 there is
insufficient_quota (no account billing), not throttling. The branch also only
printed a softer message before failing anyway, an xfail-in-disguise the e2e rules
forbid. A 429 now falls through and fails loudly with the full result.
The mid-conversation-system messages test still referenced the removed
client.gateway attribute, so the tests/e2e basedpyright gate reported 9
errors and went red on every e2e PR. EndpointsClient exposes .proxy, so
point the /v1/messages post helper at client.proxy.transport.
Ran the before/after proof live against Vertex Claude (global endpoint,
project vertex-check-481318): base transform 400s an unflagged model
(claude-opus-4-7) on a mid-conversation role:system reminder, the fix
hoists it to a 200, and a flagged model (claude-opus-4-8) keeps the
reminder in messages with cache_read held at 15615 across the reminder
turn. Flip both vertex.mid_conversation_system rows to fail_before_fix:
proven.
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted
Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400
Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag
Closes the Vertex/Azure gap from the customer RCA
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping
Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly.
* test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse
Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL,
TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path,
Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm
chat, and Nova Sonic realtime. Registry cells updated for the new markers.
* test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes
Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and
Langfuse driver models now use Anthropic haiku so local runs stay green when
Gemini daily quota is exhausted.
* test(e2e): drop Langfuse spend suite; feature is being deprecated
Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the
langfuse-only conftest driver/credentials fixtures.
* test(e2e): fold provider/batch feature tests into their endpoint suites
Keep the e2e layout endpoint- and suite-scoped instead of one file per
provider or feature
Move the virtual-key auth case into access_control/test_access_control_e2e.py
as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone
test_virtual_key_auth_e2e.py
Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock
assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py.
The hosted_vllm batch case is skipped for now since it needs a live vLLM server
(HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the
gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant
Merge the cohere, gemini and hosted_vllm chat cases into
llm_translation/test_chat_completions_regression_e2e.py so /chat/completions
coverage lives in one endpoint file, and repoint the coverage_registry source
fields to the new homes
Move the shared CacheControl / TextBlock / RichMessage request blocks into the
root models.py (re-exported from endpoints_client) so quota_management can use
them without a cross-suite import, which also clears the basedpyright errors in
test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in
test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed
json.loads path
* test(e2e): address review feedback and re-home virtual-key coverage
Replace the tautological Bedrock assume-role batch id assertion (`startswith(...)
or batch.id`, always true) with a managed-id shape check, since the unified
target_model_names path re-encodes the id rather than returning a raw ARN
Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no
longer consume the key's sole request unit before batch create runs; the batch
create then clears the generic per-request limiter and the batch limiter is what
returns the "Batch rate limit exceeded" body the assertions check
Set exercised_on to [] on the pass-through header test; it drives a pass-through
endpoint, not /chat/completions
Move the virtual-key valid_allows / invalid_denied cells from other.yaml to
mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management,
and point its covers marker at the new ids
* test(e2e): add reliability suite covering fallback, timeout, and cache behavior
* test(e2e): move reliability suite under router and drive it with real deployments
* test(e2e): make the router complexity fixture opt-in so reliability tests can coexist
