litellm/tests/e2e
yuneng-jiang 4b3355bdc6
test(e2e): prove the virtual key lifecycle on every gateway replica (#40023)
* test(e2e): prove the virtual key lifecycle on every replica

Walks one virtual key through create, read, partial update, clear, enforce
and delete against a live proxy and database, reading every write back on
every gateway replica.

The management suite already had single write-then-read tests for keys, but
none of them proved that a partial /key/update leaves the untouched fields
alone, that an explicit null clears a field, or that a write is visible on
more than the one gateway that took it.

Adds read_back_everywhere to the shared ProxyClient: it polls a GET path on
every URL in PROXY_REPLICA_URLS until each replica's parsed body satisfies
the caller's predicate, and fails naming the replica that never converged.
The CLEAR sentinel in the e2e models makes an explicit JSON null expressible
in a body the transport otherwise strips of None fields.

Documents /key/update's merge patch semantics on the endpoint docstring.

* test(e2e): prove key revocation and field preservation on every replica

Applies the findings from an adversarial review of the first commit.

The delete step only checked that chat was refused on the gateway that took
the write, so it would have passed while a sibling gateway kept serving the
deleted key. It now serves one call from every replica first, so each has the
key cached and the delete has something to revoke everywhere, then polls every
replica for the refusal.

The file also carried its own poll loop that tested the deadline before
attempting, so it gave up one attempt early and skipped the attempt landing
exactly on the deadline. It now shares the harness helper, which is generic
over the polled value rather than over a parsed body, so the same loop covers
both the info read-back and the chat refusal.

The model the enforcement step registers now carries a unique marker in its
alias, matching every other deployment this suite creates, so concurrent runs
never share one model group.

The docstring sentence claimed an explicit null clears any field. It does not:
the metadata-backed fields merge into stored metadata, where a null is a silent
no-op, and only the key's own columns clear. Regenerating the dashboard types
picks up the corrected text.

* fix(e2e): delete a deployment that never becomes servable

Registering a model posts /model/new and then waits for every replica to list
it. When that wait timed out the deployment already existed in the database but
its id had never been returned, so no caller could delete it and the row
outlived the run. It is now deleted before the failure propagates.

Found by review on the key lifecycle suite, whose module fixture registers a
deployment this way, but every caller of the shared helper had the same
exposure.

* docs(e2e): drop the duplicated notes from the lifecycle docstrings

The delete method restated what the warm-up helper already explains, and the
module restated the merge patch rule that the endpoint and the request model
both document.
2026-09-07 11:30:46 -07:00
..
a2a test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
access_control test(e2e): require a 200 inside the regenerate grace window and drop the helper docstrings 2026-09-05 11:53:22 -07:00
batches test(e2e/batches): run the list assertion when a batch completes before cancel 2026-09-04 18:50:47 -07:00
claude_code fix(e2e): wait for every gateway before using a new model and keep the network rerun 2026-09-05 16:10:40 -07:00
coverage_registry test(e2e): prove the virtual key lifecycle on every gateway replica (#40023) 2026-09-07 11:30:46 -07:00
gateway ci(e2e): declare the embedding model the access_control canary calls 2026-09-05 18:58:07 -07:00
guardrails test(e2e): cover presidio post_call, tool_permission, and weave logging cells (#39279) 2026-09-05 13:03:28 -07:00
llm_translation Merge pull request #39953 from BerriAI/litellm_/litellm-e2e-flaky-test-2159ae 2026-09-05 16:04:45 -07:00
load test(e2e): move load/perf testing out of the main suite and drop the vllm passthrough test (#35820) 2026-08-04 14:12:25 -07:00
logging test(e2e): cover presidio post_call, tool_permission, and weave logging cells (#39279) 2026-09-05 13:03:28 -07:00
management test(e2e): prove the virtual key lifecycle on every gateway replica (#40023) 2026-09-07 11:30:46 -07:00
mcp test(e2e): drop coverage registry cell for the alias-grant test 2026-09-01 09:03:47 -07:00
other test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149) 2026-07-21 14:06:29 -07:00
quota_management test(e2e): cover model access group budgets against a live proxy 2026-08-29 15:46:49 -07:00
router fix(router): skip the refusing deployment when retrying a non-transient error 2026-09-05 22:25:13 -07:00
ui test(e2e/ui): cover member role and budget edits, member permission delegation, and team guardrail removal (#40042) 2026-09-07 11:18:16 -07:00
CLAUDE.md test(e2e): judge /v1/messages streaming on the clock, not on the provider's delta count 2026-09-05 14:44:21 -07:00
conftest.py ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
CONTRIBUTING.md fix(e2e-changed): keep the gate off suites the stack cannot run 2026-09-05 21:03:50 -07:00
e2e_config.py chore: merge litellm_internal_staging into litellm_/e2e-test-performance-7d53be 2026-09-05 16:15:11 -07:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py test(e2e): drop the explanatory sentence from the StreamingResponse docstring 2026-09-05 14:53:12 -07:00
fixture_bundle.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
fixture_canonical.py test(e2e): key multipart uploads by structured part identity 2026-08-21 19:32:02 -07:00
fixture_mode.py feat(e2e): move record/replay to the provider edge (LIT-5745) 2026-08-19 18:39:15 -07:00
junit_properties.py Refuse a source path carrying a colon 2026-09-01 16:02:54 -07:00
lifecycle.py test(e2e): pin openai_passthrough routing, cost logging, and file list isolation 2026-08-20 03:03:35 -07:00
models.py test(e2e): prove the virtual key lifecycle on every gateway replica (#40023) 2026-09-07 11:30:46 -07:00
otel_client.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00
provider_edge.py fix(e2e): record a streamed chunk only after its downstream write lands 2026-08-24 13:26:58 -07:00
proxy_client.py test(e2e): prove the virtual key lifecycle on every gateway replica (#40023) 2026-09-07 11:30:46 -07:00
pytest.ini ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
test_e2e_http.py test(e2e): annotate new stream-timing locals as Final and trim the docstrings 2026-09-05 14:49:56 -07:00
test_fixture_bundle.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
test_fixture_canonical.py test(e2e): pin query params and multipart form fields as replay match-key identity 2026-08-20 15:59:34 -04:00
test_fixture_mode.py feat(e2e): move record/replay to the provider edge (LIT-5745) 2026-08-19 18:39:15 -07:00
test_junit_properties.py test(e2e): read JUnit properties off the real collected pytest Item 2026-09-01 19:12:07 -07:00
test_provider_edge.py fix(e2e): record a streamed chunk only after its downstream write lands 2026-08-24 13:26:58 -07:00
test_proxy_client.py test(e2e): prove the virtual key lifecycle on every gateway replica (#40023) 2026-09-07 11:30:46 -07:00
transport.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00