litellm/tests/e2e
yuneng-jiang 36bd7f1138
fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022)
* fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e

PUT /v1/mcp/toolset dumped its payload with exclude_none, so a field sent as
null looked exactly like one the caller left out and the stored value
survived. An admin could not clear a toolset's description: the save reported
success and the old text came straight back. It now dumps with exclude_unset,
so absent keeps and null clears, which is what PUT /v1/mcp/server already did.
A null tools list clears the selection to empty, and a null toolset_name is
ignored because a toolset always has a name.

Adds create, read, partial-update, clear and delete e2e coverage for MCP
servers and toolsets, with every read-back polled on every replica so an edit
that lands on one replica and not another fails the test, plus an enforcement
test proving a key granted a toolset lists exactly that toolset's tools
against the real Datadog upstream.

* fix(e2e): refuse a read-back that no replica serves

A read-back over an empty replica mapping satisfied every predicate and
returned as if it had converged, so it would have asserted nothing and
passed. No wiring can produce that today, since the replica list always
falls back to at least one URL, but a helper whose whole job is proving a
write reached every replica should not have a shape that passes vacuously.

* fix(mcp): keep a null tools list a no-op on toolset update

Treating a null tools list as a clear meant an existing client that sends
tools=null during a partial update, meaning "leave the selection alone",
silently lost every tool the toolset grants. That is a permission surface,
so the quiet version of it is the worst version.

A toolset always has a tool list, the same way it always has a name, so a
null on either is now a no-op. Emptying the selection is an explicit [],
which cannot be confused with a field the caller left out, and which is
what the dashboard already sends.

* fix(e2e): keep MCP admin routes on the data plane

/v1/mcp/* is a lazily mounted feature, so a gateway registers it on the first
matching request, which happens after the startup route trim that drops
management endpoints. Routing it to the control plane therefore sent every MCP
call to the one backend process: the new lifecycle read-backs proved a single
process rather than every replica, and mcp_client's await_registered barrier
waited on a registry that does not serve the tools/list call it guards, so the
existing MCP suites polled a gateway that had not synced yet until poll_timeout

Verified against a two-gateway split stack (backend on 4001, gateways on 4010
and 4011, one postgres): both gateways answer /v1/mcp/server and /v1/mcp/toolset,
and each served 6 server reads and 7 toolset reads over the run

* fix(e2e): grant the toolset by the tool's own name, not the wire name

tools/list serves a tool as <prefix><tool_name>, but a toolset grants by the
tool's own name: resolve_toolset_permissions reads toolset.tools[].tool_name
straight through, and the prefix is added on the way out. The test built the
toolset from the names tools/list reported, so the grant matched nothing, the
scoped key listed no tools, and await_tools ran out its whole poll_timeout
before failing

Measure the prefix off search_datadog_logs, whose own name is known, rather than
guessing it from the alias, since the proxy can be configured to prefix with a
short server id instead. The expectation compared against tools/list stays in
wire names; only what the toolset stores crosses back

* test(mcp): build immutable lifecycle updates and replica results

* test: validate opaque stream IDs and hide log-reader credentials

* test: isolate auto-router scenarios and clean partial setup

* test: honor Datadog search rate-limit reset headers

* test: share the Datadog read-back deadline across retries

* test: preserve captured MCP toolset update fields
2026-09-08 22:50:13 -07:00
..
a2a test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
access_control test(e2e): require a 200 inside the regenerate grace window and drop the helper docstrings 2026-09-05 11:53:22 -07:00
batches test(e2e/batches): run the list assertion when a batch completes before cancel 2026-09-04 18:50:47 -07:00
claude_code fix(e2e): wait for every gateway before using a new model and keep the network rerun 2026-09-05 16:10:40 -07:00
coverage_registry fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
gateway ci(e2e): declare the embedding model the access_control canary calls 2026-09-05 18:58:07 -07:00
guardrails test(guardrails): document audit prompt storage prerequisite 2026-09-07 11:43:32 -07:00
llm_translation fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
load test(e2e): move load/perf testing out of the main suite and drop the vllm passthrough test (#35820) 2026-08-04 14:12:25 -07:00
logging fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
management fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
mcp fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
other test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149) 2026-07-21 14:06:29 -07:00
quota_management test(e2e): cover model access group budgets against a live proxy 2026-08-29 15:46:49 -07:00
router fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
ui test(e2e/ui): cover key budget window, non-admin model scope edit, and key blocking (#40027) 2026-09-08 22:49:54 -07:00
CLAUDE.md test(e2e): judge /v1/messages streaming on the clock, not on the provider's delta count 2026-09-05 14:44:21 -07:00
conftest.py ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
CONTRIBUTING.md test(guardrails): document audit prompt storage prerequisite 2026-09-07 11:43:32 -07:00
e2e_config.py chore: merge litellm_internal_staging into litellm_/e2e-test-performance-7d53be 2026-09-05 16:15:11 -07:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
fixture_bundle.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
fixture_canonical.py test(e2e): key multipart uploads by structured part identity 2026-08-21 19:32:02 -07:00
fixture_mode.py feat(e2e): move record/replay to the provider edge (LIT-5745) 2026-08-19 18:39:15 -07:00
junit_properties.py Refuse a source path carrying a colon 2026-09-01 16:02:54 -07:00
lifecycle.py test(e2e): pin openai_passthrough routing, cost logging, and file list isolation 2026-08-20 03:03:35 -07:00
models.py fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
otel_client.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00
provider_edge.py fix(e2e): record a streamed chunk only after its downstream write lands 2026-08-24 13:26:58 -07:00
proxy_client.py fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
pytest.ini ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
test_e2e_http.py fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
test_fixture_bundle.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
test_fixture_canonical.py test(e2e): pin query params and multipart form fields as replay match-key identity 2026-08-20 15:59:34 -04:00
test_fixture_mode.py feat(e2e): move record/replay to the provider edge (LIT-5745) 2026-08-19 18:39:15 -07:00
test_junit_properties.py test(e2e): read JUnit properties off the real collected pytest Item 2026-09-01 19:12:07 -07:00
test_provider_edge.py fix(e2e): record a streamed chunk only after its downstream write lands 2026-08-24 13:26:58 -07:00
test_proxy_client.py fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e (#40022) 2026-09-08 22:50:13 -07:00
transport.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00