litellm/.github/e2e-stack
yujonglee 630c4624f6
test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503)
* test(e2e): add a HashiCorp Vault secret manager lane

key_management_system had no end-to-end coverage: the Rust crates and the
Python unit tests all run against mocked managers. This adds a secret_manager
suite that drives a proxy configured with hashicorp_vault against a real Vault.

The tests seed a fresh secret name per test with the runner's OPENAI_API_KEY and
register a deployment pointing at os.environ/<name>. The proxy's env never holds
that name, so get_secret's os.environ fallback cannot mask a broken manager, and
a bogus value in Vault must come back as the provider's 401. Virtual keys are
checked written to and removed from Vault under prefix_for_stored_virtual_keys.

The setting is global to the proxy, so the lane has its own config and the
secret_manager_vault opt-in marker, and stays out of the per-PR selector.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(e2e): make the secret manager suite backend-agnostic

One marker and opt-in (secret_manager / E2E_SECRET_MANAGER=<system>) pick the
backend from secret_backends.BACKENDS. The tests reach the manager through a
SecretStore protocol, and each backend contributes a secret_store_<system>.py
module, a registry entry, and gateway/secret_manager_<system>_ci_config.yml.
requires_capability deselects tests a backend cannot support (CyberArk does
not delete), and test_secret_backends.py checks every lane config against its
backend without a live stack.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(e2e): add a CyberArk Conjur secret manager lane

Adds cyberark as the second secret_manager backend: a Conjur store over its
REST API (policy-declared variables, raw-text values, policy-patch teardown),
its lane config, and a registry entry without deletes_stored_keys, since the
proxy's CyberArk delete answers not_supported and Conjur keeps the key.

secret_manager/backend.sh up|down <system> boots any backend in Docker and
writes proxy.env and tests.env, so every lane runs the same way; the registry
test checks the script boots exactly the registered backends. e2e_http gains
send_text_external for APIs that speak raw text rather than JSON.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(e2e): give the secret manager suite a client with .proxy and address review

The shared resources fixture reads client.proxy, so a bare ProxyClient errored every
live test at setup. backend.sh now writes its env under a per-user directory with
umask 077, the markerless unit tests are gone per tests/e2e/AGENTS.md, and routine
comments are trimmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 18:02:32 -07:00
..
assert_tests_ran.py test(e2e): complete OAuth triggers and preserve failure diagnostics 2026-09-19 17:03:38 -07:00
down.sh test(e2e): start persistent Keycloak in the changed-test runner 2026-09-11 17:09:35 -07:00
oidc-profile.sh test: bind management E2E callers and isolate JWT actors 2026-09-12 13:29:04 -07:00
redact_output.py ci(e2e): give string_leaves a single trailing return 2026-09-21 11:02:28 -07:00
secrets_to_env.py fix(e2e-stack): print add-mask lines only under GitHub Actions (#42423) 2026-09-21 22:42:13 -07:00
select_tests.py test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
start-idp.sh test(e2e): start persistent Keycloak in the changed-test runner 2026-09-11 17:09:35 -07:00
up.sh fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters (#42106) 2026-09-22 20:50:53 +00:00