Adds live e2e coverage for the Bedrock combinations behind recent customer
incidents: llm_provider-* response-header forwarding on /chat/completions
(nonstream and stream), regional us.anthropic.* inference-profile ids over
the invoke route, and the Admin UI Test Connection probe for a
responses-mode Bedrock Mantle deployment. Registers the matching cells in
the coverage registry and publishes the provider x feature matrix table in
its README.
The websocket routes under /openai_passthrough and /openai had no e2e
coverage, so nothing catches the regression from issue #36088, where both
prefixes carried HTTP routes only and refused every upgrade with a 403
before a socket ever existed.
Two tests cover it. The realtime one opens /openai_passthrough/v1/realtime
and asserts OpenAI's own session.created frame comes back, which proves the
route is registered and relayed upstream. The responses one asserts
/openai/v1/responses accepts the upgrade, since a responses.connect socket
waits for the client to speak first and has no opening frame to check.
A refused upgrade is an HTTP response rather than a close frame, so both
assert on the handshake. ws_base_url moves into e2e_config now that a
second suite needs it
The passthrough tests and their coverage registry rows pointed at the internal
ticket id, which does not resolve for anyone following a link from
status.litellm.ai. Each test docstring and registry rationale now names the
GitHub issue it pins: #36086 for the two prefix routing cases, #36087 for the
file list cursors, #36523 for streamed Responses cost, and #36646 for
embeddings spend.
The final streaming usage frame only carries usage.cost when the proxy runs
with litellm_settings.include_cost_in_streaming_usage: true, and that flag is
readable only off the module-level litellm setting. There is no header, key,
or management route that turns it on per request, so a test cannot ask the
shared e2e proxy for it, and the proxy's config does not live in this repo.
The registry row stays as an uncovered gap with the reason recorded, rather
than being deleted, so the behavior is still on the list of things we want
covered once the gateway config is reachable.
The StreamOptions model, ChatBody.stream_options, Usage.cost, and
AnthropicMessagesResponse.id existed only for that test, so they go with it.
Five e2e tests over routes a customer drives through the gateway, each one
pinning a fix that currently has no live coverage.
The dedicated /openai_passthrough prefix used to be swallowed by the
provider-scoped /{provider}/v1/files and /{provider}/v1/batches routes, which
bound "openai_passthrough" as a provider name and failed inside the gateway
before ever reaching OpenAI. Two tests now upload a file and list batches
through that prefix and assert OpenAI's own objects come back.
Streamed /openai_passthrough/v1/responses and /openai_passthrough/v1/embeddings
are relayed to OpenAI but still have to be costed, since the customer budgets
against this traffic. Both used to land a row the gateway could not use: the
streamed responses call logged a zero-cost row under a random id, and
embeddings wrote no row at all. Each test now reconciles the logged spend and
token counts against the response the caller was actually served.
GET /v1/files narrowed its data to the caller's own rows but left first_id and
last_id addressing the shared provider account's page, handing any caller raw
provider file ids belonging to other tenants. The new test asserts both cursors
address rows in the page the caller can see.
ResourceManager.defer now accepts any callable rather than one returning None,
so a delete that answers with a response model can be deferred as-is.
On models without supports_mid_conversation_system, a system entry between
an assistant tool_use turn and the user tool_result turn became a user turn
in that position and the provider rejected the request ("tool_use ids were
found without tool_result blocks immediately after"). That run of entries
now goes right after the tool_result turn, where consecutive user turns
merge upstream. The converted turn also carries only role and content, as
the hoist did, so an entry with extra keys no longer 400s with "Extra
inputs are not permitted".
The e2e cache priming re-sends the identical first turn until its own cache
entry reads back before the reminder turn goes out, since Vertex can take a
few seconds to serve a freshly written entry.
Seven live e2e tests covering cost-tracking regressions that currently ship
unnoticed: cache-write tokens billed at the cache-creation rate (#34046),
per-component cost_breakdown on the spend row (#31686), cache reads billed at
the cache-read discount on streamed calls (#34812), cache tokens surviving the
anthropic-messages to Responses bridge (#34957), priority-tier rates applied to
input, output and reasoning (#35923, #35925), the per-component response cost
headers summing to the total (#36965), and cost injected into the final usage
frame of an /openai passthrough stream (#36503).
Every test registers its own deployment with a distinct custom rate per
component, so a component billed at the wrong rate cannot pass. The shared
helpers in cost_rows.py encode the one thing the two surfaces disagree on: the
spend row's input_cost is gross of cache while the response's cost-input header
is net of it.
Adds e2e coverage for batches terminal state and cost write-back, failure
paths, per-backend file content downloads, and two-gateway routing (LIT-5730).
* fix(auth): resolve bare model names against wildcard deployments in model access groups
* test(e2e): cover model access group permission checks on keys and teams
The tool_search x bedrock_invoke cell only ever probed the first turn, so
nothing in the suite has sent a server_tool_use block back to a provider.
Every turn of a real Claude Code session after the first carries the
server_tool_use and tool_search_tool_result blocks the previous turn
produced, and that path was uncovered.
Adds probe_tool_search_multiturn, which takes the real assistant turn
back, answers any client-side tool_use with the id the model actually
emitted, and replays the whole thing as history with the tools still
declared. The assertion refuses to go green unless both server-tool
blocks made it into the replayed history, so a first turn truncated at
max_tokens reads as a failure instead of a vacuous pass.
The replay assertion's red paths never run in a green cell, so they get
markerless harness tests of their own alongside the existing
_builder_unit_tests tree.
No production code.
* test(e2e): cover google-native generateContent framing and prometheus queue time
Adds live coverage for three shipped regressions that had none, all reached
through surfaces a customer drives from Google SDKs and operator dashboards.
The managed google-native route (`/v1beta/models/{model}:generateContent`) had
no harness support at all, so EndpointsClient gains generate_content and
stream_generate_content plus the request body models, and a new suite asserts
the two contracts that broke there: the response carries
x-litellm-response-cost so SDK traffic reconciles against spend (LIT-4076), and
the stream relays single-prefixed SSE frames with no OpenAI [DONE] terminator.
A doubled `data:` prefix, a leaked bytes literal, or the [DONE] sentinel each
fail the stream test; [DONE] absence is only asserted once real content has
arrived, because a first-chunk upstream error legitimately falls back to the
OpenAI error shape and does emit it.
The prometheus test pins litellm_request_queue_time_seconds to an actual
observation on our own key's series rather than to the family merely existing,
which is the distinction the original regression turned on: the histogram stayed
registered while nothing was ever written to it (LIT-2034).
Each assertion was mutation-checked against the live proxy; inverting the
[DONE] expectation, the cost-header expectation, or the metric name fails the
corresponding test.
* refactor(e2e): simplify google native coverage
* test(e2e): add failing reproducers for two open gateway bugs
Both tests assert the behavior a customer expects and both are red today. They
are reproducers, not regressions: the product is wrong, not the tests.
Native passthrough returns almost none of the operational headers the managed
route does. A /gemini/ generateContent call comes back with three x-litellm-*
headers and no x-ratelimit-* at all, against sixteen and four on
/v1beta/models/{m}:generateContent for the same prompt, and critically it omits
x-litellm-response-cost. Customers front provider-native traffic through this
route and read those headers to reconcile spend and pace themselves, so native
traffic is currently invisible to the tooling that covers every other route.
/budget/update rejects any model_max_budget with a 500. The reported symptom was
model ids containing dots, and that reproduces (prisma raises "Unexpected
`-5.2[FloatValue]` Expected `:`" because the key is interpolated into a GraphQL
query unquoted, so glm-5.2 lexes as an identifier followed by a float), but the
plain name gpt4o fails too, on a separate "model_max_budget should be of any of
the following types: Json" type mismatch at budget_management_endpoints.py:173.
Omitting the field returns 200. The test drives both names so the failure says
whether per-model budgets are broken outright or only for punctuated ids; today
it stops on the plain name, which is the wider bug.
* test(e2e): add reproducer for unenforced end-user per-model rate limits
model_max_budget accepts an rpm_limit alongside the spend cap, and /budget/new
stores it: the create response echoes {"gemini-2.5-flash": {"rpm_limit": 1,
"max_budget": 100.0, "budget_duration": "1d"}}. Attach that budget to an end
user, drive three calls as that user, and all three return 200. The limit is
accepted, persisted, and then ignored.
The same shape already works when the budget hangs off a key, which is what
makes this quietly dangerous: the API gives every indication the cap is in
force. A customer using it to hold one end user to a slow rate on a shared key
gets no throttling at all.
Harness additions this needs: ModelBudgetEntry carries the rpm_limit/tpm_limit
the route already accepts, BudgetNewBody and create_budget carry
model_max_budget, and create_customer can attach an existing budget_id rather
than only an inline max_budget.
Red today, for the reason in the assertion message.
* test(e2e): tighten model_max_budget reproducers and drop in-loop closure
Trim the reproducer docstrings to the contract they assert, keeping the
failure messages that document each red-by-design bug. Replace the nested
per-model closure in the /budget/update test with a module-level predicate
and a per-model helper so nothing closes over a loop variable, and fix the
import order the merge left unsorted.
* test(e2e): skip the three reproducers while their gateway bugs stay open
The passthrough header contract, /budget/update model_max_budget, and
end-user per-model rpm enforcement reproducers all still fail against
staging by design. Skip each with the product gap named so the combined
suite can gate merges on green while the collector keeps reporting the
cells as uncovered.
* test(e2e): validate model budget response contracts
* refactor(e2e): unify model budget schema
* refactor(e2e): reuse shared model budget type
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
The existing web_search cells drive Claude Code's client-side WebSearch
tool, which the CLI executes itself and feeds back as a tool_result. The
CLI never emits a web_search_20250305 definition, so those cells stayed
green while the Anthropic-managed server tool 400'd on Bedrock.
Add a cell that posts the server tool to a Bedrock deployment over
/v1/messages and asserts a web_search_tool_result block comes back, and
reword the compat row so it no longer reads as coverage of the server
tool. Model the server tool as a composed base shared with tool_search.
Resolves LIT-5391
This reverts commit dcb4e5033c.
The suites landed without the proof-of-fix and QA runbook the PR body
itself flagged as outstanding, so the coverage they claim is unverified
against a live proxy
* test(e2e): cover legacy text /completions endpoint
The /completions (and /v1/completions) text-completion route had zero e2e
coverage despite being the second-busiest endpoint in production; everything
'completions' in the suite was chat. Add a text-completion endpoint test that
registers an OpenAI instruct deployment, drives /v1/completions through the
gateway, and asserts real generated text. Adds text_completions() + the
completion request/result models to EndpointsClient, the 'completions' endpoint
to the coverage registry vocab, and the registry cell.
* test(e2e): assert /v1/completions choices shape, not just joined text
Assert the response carries a choices array and the first choice has real text,
so a malformed response (no choices) and a clean-but-empty completion are
distinct failures. Drop the unused text property / id / model fields (model only
what the test reads).
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
* fix(e2e): address greptile and veria review on vendor strategy suite
Store search tool keys as os.environ refs and resolve them in SearchAPIRouter.
Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells
* fix(e2e): drop search_api_router os.environ expansion from vendor suite
Keep the PR test-only. Search tools register without an api_key so the
proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a.
* test(e2e): drop search e2e suite from vendor strategy PR
Remove the /v1/search coverage file and its registry rows so this PR
no longer carries search endpoint testing.
* feat(proxy): add a generic list contract for management/v1 entity lists
Paging, sorting, filtering and search for an entity collection, declared once
as a ListSpec and served by handle_list. The route injects a ListExecutor that
owns its table, so this module never imports Prisma.
The caller's scope is derived from the caller alone and ANDed with whatever
they filtered on, so a query parameter can only narrow what they may read.
This is the shared half of the budgets list; it lands here so the endpoint has
something to register against, and drops out when the framework arrives on its
own branch.
* feat(proxy): add GET /management/v1/budgets
The Budgets page reads /budget/list, which returns the whole table as a bare
array with no way to page, sort or filter it. A customer with enough budgets to
fill the page has no way to find one.
Registers LiteLLM_BudgetTable against the management/v1 list contract: sortable
on budget_id, max_budget, tpm_limit, rpm_limit and created_at, default order
newest-first with budget_id breaking ties, search on budget_id, and filters for
budget_duration, max_budget and created_at. budget_duration is deliberately not
sortable; the column holds "7d"/"30d" strings, so a lexicographic ORDER BY puts
"30d" ahead of "7d".
tpm_limit and rpm_limit are BigInt? in Prisma, so rows validate through a
pydantic model on the way out and serialize as JSON numbers.
A caller without admin view is refused 403 as a problem document rather than
served an empty page. /budget/list is untouched.
* fix(proxy): rework the budgets list onto the merged list contract
PR #35308 landed a different shape than this branch was written against: `where`
is a tuple of frozen predicates rather than a Prisma-shaped mapping, `ListSpec`
carries both the row and the wire type, and `where_sql` / `order_by_sql` render
for a raw-SQL executor. The budgets executor now queries through `query_raw` the
way the spend logs facet does, selecting only the columns it serves.
Also casts datetime binds in `where_sql`. They cross into the query engine as
JSON, so an uncast placeholder arrives as text and Postgres refuses
`timestamp >= text` outright; every `filter[created_at][gte|lte]` was answering
500. The cast reads the bind as an instant and drops it to naive UTC to match
Prisma's TIMESTAMP(3) column, the same one /spend/logs/ui applies.
* refactor(proxy): fold the predicate renderer instead of recursing
recursive_detector flags `_render_all`, and the flag is fair: it recursed once
per predicate, so the stack grew with the number of filters on the request for
no reason. Walking a predicate list is a running bind index, which is a fold.
`_render` still re-enters for `AnyOf`, but its clauses are plain comparisons
built by `?q=`, so that nesting is one level deep and no caller can drive it
deeper.
The collector read @pytest.mark.covers off every collected item, and
collection does not evaluate skips, so a test carrying both a skip and a
covers marker reported its cell as covered while asserting nothing. 17
files under tests/e2e do exactly that, which inflated the headline from
290/434 to 311/434.
A cell now counts as covered only when at least one test pytest would
actually run declares it; a cell claimed by both a live and a skipped
test stays covered. Skip state comes from pytest's own evaluator, so
skip, skipif (bool and string conditions), and module-level pytestmark
resolve exactly as they do in the e2e run. Cells left uncovered this way
are listed under the headline and exported as skipped_markers (JSON) and
litellm_e2e_coverage_skipped_markers (Prometheus) so the gap surfaces
instead of disappearing; the Loki line contract is unchanged. A marker
on a skipped test that points outside the registry is still an orphan,
so --strict keeps its reach.
Because skipif resolves against the environment the collector runs in,
the number now depends on that environment; run it where the e2e suite
runs. A pytest.skip() call inside a test body remains invisible to a
static pass, which the module docstring and README both state.
#35271 restored the hierarchy where a team-scoped key is governed by the
team and team-member budgets only; the owner's personal max_budget applies
to their personal keys. Three places in the e2e suite still encoded the
old direction and would fail against a proxy built from staging.
test_user_budget_enforced_across_all_their_keys asserted that the owner's
team-member key is refused once their personal budget is exhausted. It now
asserts only the personal keys are refused, and keeps the team key as the
control that must keep serving, which pins the restored direction instead
of leaving it unasserted. Renamed to match what it now covers.
test_team_member_key_user_budget_resets_after_window drove a team key to a
block off the owner's personal budget, so nothing can block it any more and
_drive_to_block could never succeed. Its premise is gone rather than moved,
so it is removed; the sibling personal-key test still covers
quota_management.budget.internal_user.resets_after_window.
The registry rationale for that row dropped its "and team-member keys"
clause for the same reason.
* fix(e2e): stop the cache-settings test from persisting a degraded Redis config
TestCacheSettings.test_update_persists_cache_backend_to_get read the live cache
settings and wrote them back, intending a no-op. Its capture modelled only
type/host/port, so on a TLS cluster the write-back silently dropped `ssl` and
`redis_startup_nodes`.
That is not recoverable on its own. `/cache/settings` persists what it receives
into LiteLLM_CacheConfig, that row outranks the YAML `cache_params`, and
init_cache_settings_in_db re-applies it on a timer, so a restart does not clear
it. The proxy ends up driving a TLS-only cluster endpoint as a plaintext
standalone node and every Redis call blocks to socket timeout.
On the affected deployment that took out rate limiting entirely (the v3 limiter
is a Lua script on Redis with no DB fallback), Redis-only budget levels (tag,
per-model, team-member, per-window), spend tracking, `ResetBudgetJob` (which
self-starved at 54 skipped runs per 15 min), and `ProxyConfig.add_deployment`,
whose last statement syncs guardrails and never ran. 60 of 72 failures in one
run traced back here.
The settings blob is now round-tripped verbatim via a RootModel over an
exhaustive value union, so a subset cannot be written. Two guards make a
regression fail loudly at this test instead of silently downstream:
- refuse to write when GET reports redis_type=cluster but omits
redis_startup_nodes, which is the exact precondition for persisting a
downgrade. GET resolves the stored row overlaid with REDIS_* env and never
reads YAML, so a cluster configured only in YAML cannot round-trip here
- compare /cache/ping before and after, so a write that breaks connectivity
fails this test rather than every suite that follows
The underlying product defect is filed as LIT-4816: GET cannot express the
effective config, and a partial POST is allowed to downgrade transport. This
change only stops the suite from triggering it; the Admin UI can still do so.
basedpyright clean (0 errors) under the e2e gate.
* fix(e2e): scope the bedrock guardrail per request and send OpenAI's current token param
Two failures that had nothing to do with the guardrail or route under test.
create_bedrock_guardrail registered with default_on=True, which applies the
guardrail to every request the proxy serves. The upstream ApplyGuardrail call was
answering 403, and that came back to unrelated traffic as
`403 Bedrock guardrail request failed`, failing three a2a tests and a passthrough
headers test alongside the bedrock one. The harness already supports the
per-request `guardrails` selector, so the guardrail is now registered opted out of
default_on and selected by the test that wants it. A broken upstream guardrail
fails its own test instead of whatever else is running.
Note this only contains the blast radius; the 403 itself still needs the
bedrock:ApplyGuardrail permission (or a valid guardrail identifier) on the
deployment, so test_bedrock_pre_call_blocks_harmful_prompt can still fail on its
own until that is sorted.
The OpenAI passthrough body sent `max_tokens`, which newer models reject with
"Unsupported parameter: 'max_tokens' is not supported with this model. Use
'max_completion_tokens' instead." Passthrough forwards the body untranslated, so
drop_params does not apply and the body has to satisfy OpenAI's contract
directly. vllm_chat keeps max_tokens, which vLLM accepts.
basedpyright clean (0 errors) under the e2e gate.
* fix(e2e): drop the pinned a2a api_key that broke every message/send
#34512 pinned `api_key="os.environ/ANTHROPIC_API_KEY"` on the a2a bridge agent.
The a2a bridge forwards the agent's litellm_params straight into
litellm.acompletion() without expanding "os.environ/" indirection, so that literal
string was sent upstream as x-api-key and every message/send failed with
`AnthropicException - {"type":"authentication_error","message":"invalid x-api-key"}`.
Omitting api_key restores the normal provider resolution: litellm reads
ANTHROPIC_API_KEY from the proxy's own environment for this provider, which is what
the agent-owner flow depends on and what the suite did before #34512.
Verified against a live proxy, same agent shape each time:
api_key omitted -> message/send 200
api_key "os.environ/ANTHROPIC_API_KEY" -> message/send 500 invalid x-api-key
api_key <literal key> -> message/send 200
and the key itself is valid (direct call to api.anthropic.com returns 200), so this
was indirection that never got expanded rather than a bad credential.
This accounts for four failures (test_semver_protocol_version_registers_and_serves,
test_message_send_runs_completion_bridge, test_pinned_v0_3_serves_flat_message_shape,
test_pinned_v1_0_serves_nested_message_shape). They were previously reported as
`403 Bedrock guardrail request failed`, because a default_on Bedrock guardrail
short-circuited the request before it ever reached the bridge and hid this.
The bridge silently ignoring "os.environ/" in agent params is a product defect in
its own right, filed separately; anyone configuring an agent credential that way
through the UI hits the same wall.
basedpyright clean (0 errors) under the e2e gate.
* test(e2e): make the load suite less aggressive against a shared proxy
750 users at spawn rate 50 saturated the request path hard enough to distort the
latency-sensitive suites sharing the same proxy, and it spends real provider money
at that rate. Drop to 200 users at spawn rate 20.
The RPS floor moves with the user count rather than staying put, so the assertion
keeps its meaning instead of becoming a formality: 355 RPS over 750 users is
~0.47 RPS/user, and 90 over 200 holds that same per-user expectation with a
similar pass margin. A request-path regression still trips it.
All four knobs stay env-overridable (E2E_LOAD_USERS, E2E_LOAD_SPAWN_RATE,
E2E_LOAD_DURATION_SECONDS, E2E_LOAD_MIN_RPS) for a deliberate load run.
Note the recorded failure for this test was "no requests completed in 60s", which
was the gateway wedged on unreachable Redis rather than a throughput regression;
this change is about not perturbing its neighbours, not about that failure.
* fix(e2e): make the reasoning-tokens assertion exercise a request that reasons
test_openai_chat_reasoning_reports_reasoning_tokens asked "A train travels 60 miles
in 1.5 hours. What is its average speed in mph?" at reasoning_effort="low", then
asserted reasoning_tokens > 0. The model answers that directly without reasoning, so
0 is correct behavior and the assertion was testing the model's discretion rather
than litellm's reporting.
Verified against a live proxy on a dedicated openai/gpt-5.6 deployment, matching how
the test provisions its model:
reasoning_effort=low, one-step arithmetic -> reasoning_tokens=0
reasoning_effort=high, the prompt used here -> reasoning_tokens=114
Raised to high effort with a prompt that requires a proof plus a search, so the
field under test is actually populated and the assertion fails only if litellm stops
surfacing it.
While confirming this I also checked prompt caching, which needed no change:
cached_tokens comes back 3615 of 3618 prompt tokens on a repeated large prefix
against a dedicated deployment. An earlier reading of 0 was an artifact of probing a
fan-out alias whose requests land on different deployments, not a caching defect.
* test(e2e): skip the files-list test while LIT-4820 is open
GET /v1/files does not include a just-uploaded file. The upload returns 200 and
GET /v1/files/{id} resolves it, but the listing never contains it: the returned set
stays fixed at 27 entries whose newest created_at is roughly ten hours older than
the upload, on both the managed (/v1/files?model=) and provider-scoped
(/openai/v1/files) routes. Polled for 40s, so not an eventual-consistency window.
Filed as LIT-4820. Skipping keeps a known, ticketed product bug from holding the
suite red and masking a new regression somewhere else in the same test.
The assertion is left exactly as it was on purpose. It encodes the contract we
actually want, that a file retrievable by id is also enumerable, and anything that
lists files (a UI picker, cleanup tooling that lists then deletes and would
therefore leak provider-side files) depends on it. Relaxing it to get green would
delete the signal. The skip reason says so and links the ticket, and the ticket
records that removing this marker is part of its definition of done.
Matches the existing pattern in this file, where test_unified_file_and_batch_create
skips with a reason citing LIT-3266.
While skipped, the registry cell llm.files.openai.list.nonstream.works has no
passing covering test, so files-list coverage reports as uncovered rather than
passing, which is the honest state.
* fix(e2e): parse Sentinel node lists in the cache-settings model
The value union covered scalar lists and lists of mappings, but not lists of
lists. `redis_startup_nodes` holds host/port mappings while `sentinel_nodes` holds
positional pairs (CACHE_SETTINGS_FIELDS documents `[['localhost', 26379]]`), so on
a Sentinel deployment pydantic rejected the response:
sentinel_nodes.list[dict[str,...]].1
Input should be a valid dictionary [input_value=['localhost', 26380]]
The round-trip test reads GET /cache/settings before it writes anything, so that
rejection failed the test at the read, before any assertion ran. A Sentinel
deployment would have looked like a broken cache-settings route rather than a
model too narrow to parse a documented shape.
A list element may now be a scalar, a list or a mapping, which covers both node
shapes without special-casing either and tolerates a heterogeneous list instead of
rejecting the whole response.
Adds TestCacheSettingsModel, harness-level with no `e2e` marker so it runs without
a proxy, covering all four backend shapes (cluster mappings, sentinel pairs, plain
node, url mode with a null discrete field) plus transport() key selection.
Confirmed it fails on the previous union and passes on this one:
old union -> 1 failed, 4 passed (the sentinel case)
new union -> 5 passed
* test(e2e): remove the cache-settings round-trip test
The test could not fail for the thing it claimed to test, and could break the
deployment it ran against. Both halves of that are worth stating.
It read the live settings, wrote back identical values, and asserted the read-back
matched. If POST /cache/settings were a complete no-op that returned 200 and touched
nothing, GET would still return the values read a moment earlier and the test would
pass. It verified that GET is stable, not that the route persists anything.
Against that, /cache/settings persists what it receives into LiteLLM_CacheConfig,
that row outranks YAML cache_params, and init_cache_settings_in_db re-applies it on a
timer. A write that omits ssl or redis_startup_nodes converts a TLS cluster into a
plaintext standalone client and every later Redis call blocks to socket timeout. On
2026-07-25 that failed 60 of 72 tests in one run: rate limiting stopped enforcing,
Redis-only budgets admitted billable over-budget spend, ResetBudgetJob self-starved,
and guardrail sync never ran.
Guarding the previous shape was not sufficient. Writing the blob verbatim plus a
cluster precondition and a /cache/ping check narrowed the hazard but did not remove
it, because GET cannot express the effective config: it resolves the stored row
overlaid with REDIS_* env and never reads YAML. On a fresh deploy it cannot see
YAML's ssl to echo back, so a TLS non-cluster deployment could still have a row
written that drops it. No round-trip through this route is safe on a shared proxy.
Removed with the models and helpers it owned, and TestCacheSettingsModel with them
since it existed only to protect that parsing.
The registry row mgmt.cache_settings.update.happy_path stays, now carrying the
rationale for why it is deliberately uncovered and what a safe test would require
(an isolated proxy, or LIT-4816 fixed so a partial write cannot downgrade
transport). Coverage therefore reports this cell as a gap, which is the honest
state. Collector passes --strict; the module still collects 11 tests.
The /openai/{endpoint} passthrough forwards a raw OpenAI-format request to
api.openai.com (or OPENAI_API_BASE) with the proxy's OPENAI_API_KEY swapped in,
and still logs a costed pass_through_endpoint SpendLogs row. Nothing exercised
that path end to end.
Adds a live /openai/v1/chat/completions passthrough test that asserts a 2xx
completion and a costed row with custom_llm_provider=openai, mirroring the
gemini and anthropic passthrough cost tests, and registers
llm.chat_completions.openai.passthrough.nonstream.cost_logged.
/images/edits is a distinct native route from /images/generations: a multipart
request with the source image sent as the 'image' part plus an edit prompt, not
a JSON body. Nothing exercised it end to end.
Adds a live test that registers an OpenAI image model, sends a small generated
PNG plus an edit prompt to /v1/images/edits, and asserts an image comes back
(b64 or url). Generalizes the multipart transport helper with a file_field
argument (default 'file') so the image part can be named 'image', adds an
image_edit client method, the images_edits endpoint to the coverage schema, and
the llm.images_edits.openai.basic.nonstream.works cell.
/vllm/batches and /vllm/files are high-volume passthrough routes with no e2e
coverage. They ride litellm's generic /vllm/{endpoint} forwarder, so the test
uploads a JSONL through /vllm/v1/files and creates a batch through
/vllm/v1/batches (BatchClient with provider=vllm), asserting the forwarded file
and batch objects come back. Lives next to TestHostedVllmBatch and is skip-marked
for the same reason: no live vLLM server (HOSTED_VLLM_API_BASE) in the e2e env.
Adds the two llm-translation registry cells.
Enterprise MCP users mint virtual keys against tool access groups rather than
explicit server ids. Nothing exercised that end to end.
Registers the upstream MCP server tagged with a server-side access group
(mcp_access_groups), then mints one key granted that group and one granted a
different group. Asserts the granted key sees the tagged server's tools on
tools/list and the other key sees none, so access-group scoping can't leak the
server across the boundary.
Adds mcp_access_groups support to the e2e MCP client (server registration, key
generation, ObjectPermission) and the registry cell
mcp.list_tools.api_key.access_group_scoped.
* test(e2e): add other suite covering master-key auth and health lifecycle
Covers the other.* holding-pen cells that were uncovered: master-key
valid_allows/invalid_denied on the admin /user/list gate, and the
lifecycle probes liveness.ping, readiness.public_probe,
readiness.reports_db_status, and readiness_details.authenticated_diagnostics.
New tests/e2e/other/ suite on the shared ProxyClient; the health probes
send no auth header to prove the public routes need no credential, and the
details route is asserted to reject an anonymous caller while exposing
version/db diagnostics to the master key.
* test(e2e): cover block_code_execution and openai_moderation guardrails
Extends the guardrails suite with two built-in guardrails registered per
request (default_on=False, opted in via the chat body's guardrails selector)
so neither intercepts unrelated traffic on the shared proxy.
block_code_execution.pre_call.blocks: a python code block plus a run-this
request is intercepted with the canned content-blocked message and the model
never runs, while the same code block asked about with don't-run-it reaches
the model. Verified live.
openai_moderations.pre_call.blocks: a flagged prompt is rejected 400 naming
the moderation policy while a benign prompt passes. The guardrail calls
OpenAI's moderation API; verifying it needs an OpenAI key with moderation
quota (this account currently 429s the moderation endpoint).
Adds a shared create_backend_model helper and a generic register() plus
per-request guardrails/max_tokens on the client so more built-ins can reuse
the same path.
* test(e2e): cover presidio PII masking (pre_call + post_call)
Registers a presidio guardrail per request (default_on=False) with the
analyzer/anonymizer bases supplied in the registration params, so the test
controls its own dependency and needs no proxy restart.
presidio.pre_call.masks: a repeat-verbatim request comes back with the
<EMAIL_ADDRESS> placeholder and never the raw email, proving the prompt was
anonymized before the model saw it.
presidio.post_call.masks: with apply_to_output the model's own emitted email
is masked on the way out, so the caller never receives the raw value.
Both verified live against real presidio analyzer + anonymizer containers.
logging_only is intentionally not covered: /spend/logs exposes no prompt
messages to read back the masked log, and a logging_only run also masked the
response, contradicting its contract; noted in the module docstring for a
follow-up.
* test(e2e): cover presidio logging_only masking via OTEL read-back
Adds the third presidio cell, guardrail.presidio.logging_only.masks. The
logging_only contract (mask what is logged, do not block) is verified by
reading the request's gen-AI span back from the real OTEL destination: the
span's gen_ai.input.messages attribute carries the <EMAIL_ADDRESS> placeholder,
never the raw email, and the call itself is not blocked.
Reads the trace via the shared OtelReader, promoted from logging/ to the suite
root so both suites use it. The masked prompt is polled to a deadline because
logging_only masks the payload asynchronously and the span can briefly export
before the mask lands. Drops the throwaway chat_send in favor of the existing
transport.send for the call-id capture.
* fix(e2e): tolerate cross-pod guardrail sync delay in team-opt-out test
Stage runs multiple gateway pods behind the shared key. POST /guardrails
registers a new default-on guardrail in-process immediately only on the
pod that served the create call; every other pod picks it up on its next
periodic DB sync (proxy_server.py, every 30s), so the very next chat call
can race a pod that has not synced yet. Poll to a 40s deadline instead of
asserting on the first response, matching the existing pattern in
test_budget_reset_advances_e2e.py.
* test(e2e): cover a guardrail on the MCP tool-call path (content_filter pre_mcp_call)
Adds guardrail.litellm_content_filter.pre_mcp_call.blocks: against the real
Datadog MCP server, a content_filter guardrail configured mode=pre_mcp_call
blocks a banned keyword in an MCP tool call's arguments with HTTP 400 attributed
to the pre_mcp_call hook, and lets a clean argument reach the upstream server.
The guardrail attaches with default_on because per-key/request guardrail
selection is dropped from the synthetic MCP request the hook sees; the banned
keyword is unique per run so default_on only intercepts this test's own call.
mode must be pre_mcp_call - a pre_call config silently no-ops on tools/call
because the event type is rewritten for call_mcp_tool.
Drives the tool directly via /mcp-rest/tools/call for a deterministic check of
the same pre_mcp_call enforcement the OpenAI-SDK chat path hits when a model
invokes an MCP tool.
* fix(e2e): mid-conversation messages test uses client.proxy not client.gateway
EndpointsClient exposes .proxy after the Gateway->ProxyClient rename; the
mid-conversation system test still referenced .gateway, which fails the e2e
basedpyright gate. Aligns it with the rest of the harness.
* test(e2e): address review on the guardrail coverage
MCP tool-call guardrail: poll the banned call until the guardrail is enforced
instead of asserting on the first call, so the control-plane -> data-plane
guardrail sync cannot race the check into a false pass-through; add a repeat
banned call after enforcement to guard against a partial-propagation state.
OpenAI moderation: distinguish a moderation-endpoint 429 (rate limit / no
moderation quota) from a guardrail failure, so an account-capability gap reads
as such rather than as "did not block". Runs green with a moderation-capable key.
* test(e2e): close partial-propagation false-pass in MCP guardrail block test
The single post-block repeat call could be load-balanced back to the same
already-synced data-plane pod, so the test could pass while another pod still
lacked the guardrail and let the banned MCP call reach Datadog. Anchor a wait to
the guardrail create time (every pod is guaranteed to have DB-synced only after a
full ~30s sync interval), then require the banned call to stay blocked across
several attempts; a pass-through after that window is a real leak, not a race.
* test(e2e): drop xfail-style rate-limit branch from openai_moderation test
OpenAI's /v1/moderations is free and returns 200 with the env key (verified
directly), so the RateLimitedError branch mislabeled the failure: a 429 there is
insufficient_quota (no account billing), not throttling. The branch also only
printed a softer message before failing anyway, an xfail-in-disguise the e2e rules
forbid. A 429 now falls through and fails loudly with the full result.
