litellm/tests/e2e
yujonglee 630c4624f6
test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503)
* test(e2e): add a HashiCorp Vault secret manager lane

key_management_system had no end-to-end coverage: the Rust crates and the
Python unit tests all run against mocked managers. This adds a secret_manager
suite that drives a proxy configured with hashicorp_vault against a real Vault.

The tests seed a fresh secret name per test with the runner's OPENAI_API_KEY and
register a deployment pointing at os.environ/<name>. The proxy's env never holds
that name, so get_secret's os.environ fallback cannot mask a broken manager, and
a bogus value in Vault must come back as the provider's 401. Virtual keys are
checked written to and removed from Vault under prefix_for_stored_virtual_keys.

The setting is global to the proxy, so the lane has its own config and the
secret_manager_vault opt-in marker, and stays out of the per-PR selector.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(e2e): make the secret manager suite backend-agnostic

One marker and opt-in (secret_manager / E2E_SECRET_MANAGER=<system>) pick the
backend from secret_backends.BACKENDS. The tests reach the manager through a
SecretStore protocol, and each backend contributes a secret_store_<system>.py
module, a registry entry, and gateway/secret_manager_<system>_ci_config.yml.
requires_capability deselects tests a backend cannot support (CyberArk does
not delete), and test_secret_backends.py checks every lane config against its
backend without a live stack.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(e2e): add a CyberArk Conjur secret manager lane

Adds cyberark as the second secret_manager backend: a Conjur store over its
REST API (policy-declared variables, raw-text values, policy-patch teardown),
its lane config, and a registry entry without deletes_stored_keys, since the
proxy's CyberArk delete answers not_supported and Conjur keeps the key.

secret_manager/backend.sh up|down <system> boots any backend in Docker and
writes proxy.env and tests.env, so every lane runs the same way; the registry
test checks the script boots exactly the registered backends. e2e_http gains
send_text_external for APIs that speak raw text rather than JSON.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(e2e): give the secret manager suite a client with .proxy and address review

The shared resources fixture reads client.proxy, so a bare ProxyClient errored every
live test at setup. backend.sh now writes its env under a per-user directory with
umask 077, the markerless unit tests are gone per tests/e2e/AGENTS.md, and routine
comments are trimmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 18:02:32 -07:00
..
a2a test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
access_control Merge remote-tracking branch 'github/main' into litellm_rust_bridge_declarative_route_catalog 2026-09-17 11:08:08 -07:00
batches fix(bedrock): treat blank AWS_S3_* env vars as unset for batch jobs (#42528) 2026-09-22 17:55:22 -07:00
claude_code chore: consolidate CLAUDE.md into AGENTS.md 2026-09-19 02:30:35 +00:00
coverage_registry test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
gateway test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
guardrails feat(proxy): opt-in include_guardrail_response returns guardrail_information in the response (#42327) 2026-09-22 12:43:30 -07:00
llm_translation fix(e2e): route credential, cost map, and UI login calls to the control plane (#42506) 2026-09-22 12:59:56 -07:00
load chore: merge litellm_internal_staging into litellm_e2e_reliability_module_cells 2026-09-12 13:05:02 -07:00
logging fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters (#42106) 2026-09-22 20:50:53 +00:00
management fix(proxy): write key deleted audit logs for cascade and alias key deletions (#42446) 2026-09-22 11:52:08 -07:00
mcp test(e2e): restore LIT-3467 implementation for rework 2026-09-19 16:21:53 -07:00
migrations test(migrations): close the gaps the upgrade assertions left open 2026-09-21 13:15:46 -07:00
other fix(jwt): accept a team alias in x-litellm-team-id (#42445) 2026-09-22 02:22:00 -07:00
quota_management feat(logging): add normalized_error cluster key to error_information (#41715) 2026-09-22 15:56:50 -07:00
router test(e2e): run the memory cell alone on the shared stack (#42518) 2026-09-22 15:13:37 -07:00
secret_manager test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
ui test(e2e-ui): check the MCP Tools tab against the upstream's own tools/list (#42397) 2026-09-21 20:35:30 -07:00
AGENTS.md test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
conftest.py test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
CONTRIBUTING.md test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
e2e_config.py test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
fixture_bundle.py test: add strict stateless provider replay identity 2026-09-14 16:55:43 -07:00
fixture_canonical.py feat(e2e): key the provider cache per test and mount Bedrock behind it 2026-09-16 02:15:46 -07:00
fixture_mode.py fix(e2e): own a shared fixture's deployment by the fixture's node, not the first test 2026-09-16 17:35:05 -07:00
fixture_profile.py test: preserve strict replay numeric spelling 2026-09-14 17:11:56 -07:00
idp.py test(e2e): restore LIT-3467 implementation for rework 2026-09-19 16:21:53 -07:00
idp_realm.json test(e2e): harden JWT fixtures and cover management lifecycles 2026-09-11 16:36:12 -07:00
junit_properties.py test: bind management E2E callers and isolate JWT actors 2026-09-12 13:29:04 -07:00
lifecycle.py fix(e2e): clean up batch files reliably and expire Azure inputs 2026-09-07 12:15:06 -07:00
memory_readings.py test(e2e): hold every worker under an idle RSS budget before any traffic (#42552) 2026-09-22 14:47:14 -07:00
models.py feat(logging): add normalized_error cluster key to error_information (#41715) 2026-09-22 15:56:50 -07:00
otel_client.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00
PROVIDER_CACHE.md fix(e2e): own a shared fixture's deployment by the fixture's node, not the first test 2026-09-16 17:35:05 -07:00
provider_cache.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
provider_cache_redis.py chore(e2e): report the key components behind a mount that never converges 2026-09-16 15:01:08 -07:00
provider_cache_routing.py revert(e2e): unmount Gemini, its api_base means two things 2026-09-16 08:37:25 -07:00
provider_edge.py fix(responses): forward safety_identifier through the chat completion bridge 2026-09-21 04:09:59 +00:00
provider_edge_bedrock.py feat(e2e): cache the responses and embeddings endpoints behind the edge 2026-09-16 02:34:09 -07:00
proxy_client.py test(e2e): hold every worker under an idle RSS budget before any traffic (#42552) 2026-09-22 14:47:14 -07:00
pytest.ini test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
stack_lock.py test(e2e): run the memory cell alone on the shared stack (#42518) 2026-09-22 15:13:37 -07:00
test_e2e_http.py test: bind management E2E callers and isolate JWT actors 2026-09-12 13:29:04 -07:00
test_fixture_bundle.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
test_fixture_canonical.py test(e2e): pin query params and multipart form fields as replay match-key identity 2026-08-20 15:59:34 -04:00
test_fixture_mode.py feat(e2e): move record/replay to the provider edge (LIT-5745) 2026-08-19 18:39:15 -07:00
test_idp.py test: enforce isolated actors and stop OIDC process groups 2026-09-12 13:49:49 -07:00
test_junit_properties.py test(e2e): read JUnit properties off the real collected pytest Item 2026-09-01 19:12:07 -07:00
test_provider_edge.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
test_proxy_client.py test(e2e): assert a cooldown reaches a sibling replica within the 1s Redis read interval (#42422) 2026-09-22 12:42:41 -07:00
test_stack_lock.py test(e2e): run the memory cell alone on the shared stack (#42518) 2026-09-22 15:13:37 -07:00
transport.py fix(e2e): route credential, cost map, and UI login calls to the control plane (#42506) 2026-09-22 12:59:56 -07:00