litellm/tests/e2e
devin-ai-integration[bot] e26a6450c8
test(integration): add MCP gateway coverage wave 1 with a dedicated mcp shard and proxy coverage artifact (#42711)
* test(integration): drop the contracts.json manifest and the covers requirement

Groups live as a GROUPS literal in run.py, the browser expectations move next to the
browser tests, and the runner fails only on pytest failure, collection errors or a
selected file that collects zero tests. The covers marker stays registered for the
existing tests but is no longer checked. The mcp directory gets its own group

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): run mcp as its own shard with xdist and a peer proxy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): INTEGRATION_COVERAGE=1 runs the proxy under coverage for the MCP modules

The mcp shard sets it. The proxy and its peer start under coverage run in parallel mode,
get SIGTERM after the tests so coverage flushes, and the combined text and HTML reports
land in the suite results that CircleCI already stores as artifacts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): let the test proxy flush coverage when uvicorn re-raises SIGTERM

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add SSE, stdio, scripted, OpenAPI and OAuth 2.1 MCP peer doubles

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add MCP transport and access-control matrices

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add MCP credential and OAuth flow coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add MCP LLM endpoint, accounting, guardrail, resilience and lifecycle coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): stop the same-URL grant test from counting a late initialize as a leaked call and satisfy the test-tree lint

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): assert the REST denied-server listing is refused or empty

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): pin the REST denied-server listing to 403 access_denied

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-23 07:48:46 -07:00
..
a2a test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
access_control Merge remote-tracking branch 'github/main' into litellm_rust_bridge_declarative_route_catalog 2026-09-17 11:08:08 -07:00
batches fix(bedrock): treat blank AWS_S3_* env vars as unset for batch jobs (#42528) 2026-09-22 17:55:22 -07:00
claude_code chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job (#42608) 2026-09-22 19:43:31 -07:00
coverage_registry test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
gateway test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
guardrails fix(policy_engine): keep inherited parent guardrails when a child policy condition misses (#42548) 2026-09-22 23:27:38 -07:00
llm_translation test(e2e): tolerate provider-side flakes on five full-suite cells (#42628) 2026-09-22 20:12:14 -07:00
load chore: merge litellm_internal_staging into litellm_e2e_reliability_module_cells 2026-09-12 13:05:02 -07:00
logging fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters (#42106) 2026-09-22 20:50:53 +00:00
management fix(proxy): write key deleted audit logs for cascade and alias key deletions (#42446) 2026-09-22 11:52:08 -07:00
mcp test(e2e): restore LIT-3467 implementation for rework 2026-09-19 16:21:53 -07:00
migrations test(migrations): close the gaps the upgrade assertions left open 2026-09-21 13:15:46 -07:00
other fix(jwt): accept a team alias in x-litellm-team-id (#42445) 2026-09-22 02:22:00 -07:00
quota_management feat(logging): add normalized_error cluster key to error_information (#41715) 2026-09-22 15:56:50 -07:00
router test(e2e): tolerate provider-side flakes on five full-suite cells (#42628) 2026-09-22 20:12:14 -07:00
secret_manager test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
ui test(integration): add MCP gateway coverage wave 1 with a dedicated mcp shard and proxy coverage artifact (#42711) 2026-09-23 07:48:46 -07:00
AGENTS.md test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
conftest.py test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
CONTRIBUTING.md test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
e2e_config.py test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py test(e2e): tolerate provider-side flakes on five full-suite cells (#42628) 2026-09-22 20:12:14 -07:00
fixture_bundle.py test: add strict stateless provider replay identity 2026-09-14 16:55:43 -07:00
fixture_canonical.py feat(e2e): key the provider cache per test and mount Bedrock behind it 2026-09-16 02:15:46 -07:00
fixture_mode.py fix(e2e): own a shared fixture's deployment by the fixture's node, not the first test 2026-09-16 17:35:05 -07:00
fixture_profile.py test: preserve strict replay numeric spelling 2026-09-14 17:11:56 -07:00
idp.py test(e2e): restore LIT-3467 implementation for rework 2026-09-19 16:21:53 -07:00
idp_realm.json test(e2e): harden JWT fixtures and cover management lifecycles 2026-09-11 16:36:12 -07:00
junit_properties.py test: bind management E2E callers and isolate JWT actors 2026-09-12 13:29:04 -07:00
lifecycle.py fix(e2e): clean up batch files reliably and expire Azure inputs 2026-09-07 12:15:06 -07:00
memory_readings.py test(e2e): hold every worker under an idle RSS budget before any traffic (#42552) 2026-09-22 14:47:14 -07:00
models.py feat(logging): add normalized_error cluster key to error_information (#41715) 2026-09-22 15:56:50 -07:00
otel_client.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00
PROVIDER_CACHE.md fix(e2e): own a shared fixture's deployment by the fixture's node, not the first test 2026-09-16 17:35:05 -07:00
provider_cache.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
provider_cache_redis.py chore(e2e): report the key components behind a mount that never converges 2026-09-16 15:01:08 -07:00
provider_cache_routing.py revert(e2e): unmount Gemini, its api_base means two things 2026-09-16 08:37:25 -07:00
provider_edge.py fix(responses): forward safety_identifier through the chat completion bridge 2026-09-21 04:09:59 +00:00
provider_edge_bedrock.py feat(e2e): cache the responses and embeddings endpoints behind the edge 2026-09-16 02:34:09 -07:00
proxy_client.py test(e2e): hold every worker under an idle RSS budget before any traffic (#42552) 2026-09-22 14:47:14 -07:00
pytest.ini test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
stack_lock.py test(e2e): run the memory cell alone on the shared stack (#42518) 2026-09-22 15:13:37 -07:00
test_e2e_http.py test: bind management E2E callers and isolate JWT actors 2026-09-12 13:29:04 -07:00
test_fixture_bundle.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
test_fixture_canonical.py test(e2e): pin query params and multipart form fields as replay match-key identity 2026-08-20 15:59:34 -04:00
test_fixture_mode.py feat(e2e): move record/replay to the provider edge (LIT-5745) 2026-08-19 18:39:15 -07:00
test_idp.py test: enforce isolated actors and stop OIDC process groups 2026-09-12 13:49:49 -07:00
test_junit_properties.py test(e2e): read JUnit properties off the real collected pytest Item 2026-09-01 19:12:07 -07:00
test_provider_edge.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
test_proxy_client.py test(e2e): assert a cooldown reaches a sibling replica within the 1s Redis read interval (#42422) 2026-09-22 12:42:41 -07:00
test_stack_lock.py test(e2e): run the memory cell alone on the shared stack (#42518) 2026-09-22 15:13:37 -07:00
transport.py fix(e2e): route credential, cost map, and UI login calls to the control plane (#42506) 2026-09-22 12:59:56 -07:00