* test(e2e): add a HashiCorp Vault secret manager lane
key_management_system had no end-to-end coverage: the Rust crates and the
Python unit tests all run against mocked managers. This adds a secret_manager
suite that drives a proxy configured with hashicorp_vault against a real Vault.
The tests seed a fresh secret name per test with the runner's OPENAI_API_KEY and
register a deployment pointing at os.environ/<name>. The proxy's env never holds
that name, so get_secret's os.environ fallback cannot mask a broken manager, and
a bogus value in Vault must come back as the provider's 401. Virtual keys are
checked written to and removed from Vault under prefix_for_stored_virtual_keys.
The setting is global to the proxy, so the lane has its own config and the
secret_manager_vault opt-in marker, and stays out of the per-PR selector.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): make the secret manager suite backend-agnostic
One marker and opt-in (secret_manager / E2E_SECRET_MANAGER=<system>) pick the
backend from secret_backends.BACKENDS. The tests reach the manager through a
SecretStore protocol, and each backend contributes a secret_store_<system>.py
module, a registry entry, and gateway/secret_manager_<system>_ci_config.yml.
requires_capability deselects tests a backend cannot support (CyberArk does
not delete), and test_secret_backends.py checks every lane config against its
backend without a live stack.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): add a CyberArk Conjur secret manager lane
Adds cyberark as the second secret_manager backend: a Conjur store over its
REST API (policy-declared variables, raw-text values, policy-patch teardown),
its lane config, and a registry entry without deletes_stored_keys, since the
proxy's CyberArk delete answers not_supported and Conjur keeps the key.
secret_manager/backend.sh up|down <system> boots any backend in Docker and
writes proxy.env and tests.env, so every lane runs the same way; the registry
test checks the script boots exactly the registered backends. e2e_http gains
send_text_external for APIs that speak raw text rather than JSON.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(e2e): give the secret manager suite a client with .proxy and address review
The shared resources fixture reads client.proxy, so a bare ProxyClient errored every
live test at setup. backend.sh now writes its env under a per-user directory with
umask 077, the markerless unit tests are gone per tests/e2e/AGENTS.md, and routine
comments are trimmed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): run the memory cell alone on the shared stack
* test(e2e): hold the stack lock for every collected test, marker or not
* test(e2e): prove the stack lock's reader sharing and writer preference across processes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): hold every worker under an idle RSS budget before any traffic
The harness reads /debug/memory/summary on every replica once at collection
time, right after the readiness gate and before this pytest process sends any
traffic, and the memory suite's first test fails when any worker idles past
E2E_MEMORY_IDLE_RSS_BUDGET_MB (768 MB by default) or gives no reading at all.
A v1.100.x worker with a database idled at 836-886 MB where v1.101.0rc1 idled
at 544 MB on the same database: prisma-client-py's default recursive type
depth generated 91k TypedDict classes that v1.101.0's recursive_type_depth = -1
cut to 19k. The budget starts at the rc1 reading plus headroom.
* test(e2e): read idle RSS only when the idle budget test is selected
Gate the collection-time /debug/memory/summary read on a selected test using
the idle_rss fixture and skip it under --collect-only, so sessions that never
run the idle budget test pay no round trip. Drop the markerless unit test file
the e2e guide bans and assert live that every configured replica was measured
* test(e2e): take the idle RSS read after collection settles
Read every replica's RSS from a tryfirst pytest_collection_finish hook so -k
and -m deselection has already run, and only when a selected test still asks
for the idle_rss fixture and the run is not --collect-only
* test(e2e): record the heaviest idle RSS reading as junit properties
* test(e2e): attach the idle RSS properties from the harness's setup hook
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): attribute provider and model info on pre-call rejected requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep pre-call rejections out of deployment cooldown and prometheus deployment state
Stamp model_info only into the logging metadata so the router's failure callbacks do not
count a key-level 429 or guardrail 403 against the deployment, treat a resolved plus an
unresolved deployment as ambiguous provider attribution, and stop the prometheus deployment
counters and deployment_state from treating a proxy-side reject as a selected deployment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): skip deployment attribution when the rejected body's model is not a string
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): bucket non-string request models as other instead of raising in failure hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve team deployments and treat guardrail rejects as proxy-side in failure attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): flag pre-routing rejects instead of matching exception names
Post-call GuardrailRaisedException failures kept their deployment labels on main but lost them
on this branch because every GuardrailRaisedException was treated as a pre-routing reject.
The proxy failure path now flags litellm_params with proxy_rejected_before_routing only when
it adds deployment attribution itself, and the Prometheus logger keys deployment selection
off that flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): key pre-routing reject flag off provider handoff, not caller metadata
Caller-supplied metadata.model_info (kept for keys allowed to override pricing) no longer
suppresses proxy_rejected_before_routing. The hook now checks the logging object's
first_api_call_start_time, which only the provider handoff sets, so Prometheus never
records a deployment failure for a request that was rejected before routing.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): poll for both served and rejected spend rows before asserting attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(websearch_interception): propagate parent session/trace ids into intercepted searches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(websearch_interception): let parent correlation win over configured search params and type test params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): bill an intercepted web search under the parent request session
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop unrelated reformatting from the websearch session harness change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): keep the websearch interception session suite out of the stage-mirror gate
The stage-mirror stack runs no websearch_interception callback or search tool, so the suite is deselected there and the changed-tests gate fails on a file that executed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): run the websearch interception session suite on the stage-mirror stack
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(e2e): leave CONTRIBUTING.md untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Resolves the six conflicts main accumulated against the SDK migration.
endpoints_client.py stays deleted (main only added the safety_identifier
kwarg to it), and its two new importers move onto the SDK clients: the
Bedrock safety_identifier forwarding test in test_responses_e2e.py now
sends its /v1/responses leg through the OpenAI SDK and its chat leg
through ProxyClient.chat, and test_messages_azure_foundry_e2e.py drops
the x-litellm-rust header check together with EXPECT_RUST, which main
removed from e2e_config in a84f68b6e3. proxy_client.py keeps both the
rerank and router-settings model imports. uv.lock is main's lock plus
the anthropic e2e-dev entry, and the pricing JSONs are taken verbatim
from main since this PR never meant to change them.
The Buildkite ephemeral stack runs the gateway in another pod, so it cannot reach the pytest host's provider edge. The GitHub changed-e2e lane runs gateways on the runner and sets E2E_PROVIDER_EDGE_HOST_REACHABLE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
