mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-24 00:52:24 +00:00
630c4624f6
* test(e2e): add a HashiCorp Vault secret manager lane key_management_system had no end-to-end coverage: the Rust crates and the Python unit tests all run against mocked managers. This adds a secret_manager suite that drives a proxy configured with hashicorp_vault against a real Vault. The tests seed a fresh secret name per test with the runner's OPENAI_API_KEY and register a deployment pointing at os.environ/<name>. The proxy's env never holds that name, so get_secret's os.environ fallback cannot mask a broken manager, and a bogus value in Vault must come back as the provider's 401. Virtual keys are checked written to and removed from Vault under prefix_for_stored_virtual_keys. The setting is global to the proxy, so the lane has its own config and the secret_manager_vault opt-in marker, and stays out of the per-PR selector. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(e2e): make the secret manager suite backend-agnostic One marker and opt-in (secret_manager / E2E_SECRET_MANAGER=<system>) pick the backend from secret_backends.BACKENDS. The tests reach the manager through a SecretStore protocol, and each backend contributes a secret_store_<system>.py module, a registry entry, and gateway/secret_manager_<system>_ci_config.yml. requires_capability deselects tests a backend cannot support (CyberArk does not delete), and test_secret_backends.py checks every lane config against its backend without a live stack. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(e2e): add a CyberArk Conjur secret manager lane Adds cyberark as the second secret_manager backend: a Conjur store over its REST API (policy-declared variables, raw-text values, policy-patch teardown), its lane config, and a registry entry without deletes_stored_keys, since the proxy's CyberArk delete answers not_supported and Conjur keeps the key. secret_manager/backend.sh up|down <system> boots any backend in Docker and writes proxy.env and tests.env, so every lane runs the same way; the registry test checks the script boots exactly the registered backends. e2e_http gains send_text_external for APIs that speak raw text rather than JSON. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(e2e): give the secret manager suite a client with .proxy and address review The shared resources fixture reads client.proxy, so a bare ProxyClient errored every live test at setup. backend.sh now writes its env under a per-user directory with umask 077, the markerless unit tests are gone per tests/e2e/AGENTS.md, and routine comments are trimmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
20 lines
2.3 KiB
INI
20 lines
2.3 KiB
INI
[pytest]
|
|
# Config when any e2e suite under tests/e2e/ is run directly, e.g.
|
|
# uv run pytest tests/e2e/quota_management/spend_tracking/ -v
|
|
# The e2e marker is also registered in conftest.py for runs rooted elsewhere.
|
|
addopts = --strict-markers --strict-config --reruns 1 --only-rerun "kind='network'" --only-rerun "status_code=5[0-9][0-9]"
|
|
markers =
|
|
e2e: live test that requires a running proxy and real provider keys
|
|
replayable: edge-wired test whose provider traffic replays from a fixture bundle, so it makes zero provider calls in replay mode; the record/replay CI lane selects it with -m replayable
|
|
load: heavy throughput/load test; collected last so it never perturbs latency-sensitive suites
|
|
weekly: real-provider anomaly load test that spends real money; deselected unless E2E_WEEKLY_ANOMALY is set
|
|
managed_files: needs a proxy running with require_managed_files enabled; deselected unless E2E_MANAGED_FILES_STACK is set
|
|
prompt_caching_stack: needs a proxy running with router_settings.optional_pre_call_checks including prompt_caching; deselected unless E2E_PROMPT_CACHING_STACK is set
|
|
cli_determinism: drives the real claude CLI for several seconds; deselected unless E2E_CLI_DETERMINISM is set
|
|
redis_chaos: load test that pauses the proxy's Redis outright mid-run; needs a proxy booted from gateway/redis_chaos_ci_config.yml on the same host, and is deselected unless E2E_REDIS_CHAOS is set
|
|
quiet_stack: measures the proxy itself, so it runs while no other test on this host is hitting the stack; every other test waits for it to finish
|
|
mcp_oauth_live: real Linear OAuth consent via a captured browser session; deselected unless E2E_MCP_OAUTH_LIVE is set
|
|
provider_edge_host: routes provider traffic through the pytest host's edge in every fixture mode, so the gateway must reach the pytest host; deselected unless E2E_PROVIDER_EDGE_HOST_REACHABLE is set
|
|
otel_v2: needs a proxy running with LITELLM_OTEL_V2=true; deselected unless E2E_OTEL_V2 is set
|
|
otel_tls: needs a stack whose gateway exports OTLP over TLS signed by the CA in SSL_CERT_FILE; deselected unless E2E_OTEL_EXPORTER_ENDPOINT is set
|
|
secret_manager: needs a proxy booted from gateway/secret_manager_<system>_ci_config.yml against that live secret manager; deselected unless E2E_SECRET_MANAGER names the backend (see secret_manager/secret_backends.py)
|