* test(e2e): hold every worker under an idle RSS budget before any traffic
The harness reads /debug/memory/summary on every replica once at collection
time, right after the readiness gate and before this pytest process sends any
traffic, and the memory suite's first test fails when any worker idles past
E2E_MEMORY_IDLE_RSS_BUDGET_MB (768 MB by default) or gives no reading at all.
A v1.100.x worker with a database idled at 836-886 MB where v1.101.0rc1 idled
at 544 MB on the same database: prisma-client-py's default recursive type
depth generated 91k TypedDict classes that v1.101.0's recursive_type_depth = -1
cut to 19k. The budget starts at the rc1 reading plus headroom.
* test(e2e): read idle RSS only when the idle budget test is selected
Gate the collection-time /debug/memory/summary read on a selected test using
the idle_rss fixture and skip it under --collect-only, so sessions that never
run the idle budget test pay no round trip. Drop the markerless unit test file
the e2e guide bans and assert live that every configured replica was measured
* test(e2e): take the idle RSS read after collection settles
Read every replica's RSS from a tryfirst pytest_collection_finish hook so -k
and -m deselection has already run, and only when a selected test still asks
for the idle_rss fixture and the run is not --collect-only
* test(e2e): record the heaviest idle RSS reading as junit properties
* test(e2e): attach the idle RSS properties from the harness's setup hook
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): attribute provider and model info on pre-call rejected requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep pre-call rejections out of deployment cooldown and prometheus deployment state
Stamp model_info only into the logging metadata so the router's failure callbacks do not
count a key-level 429 or guardrail 403 against the deployment, treat a resolved plus an
unresolved deployment as ambiguous provider attribution, and stop the prometheus deployment
counters and deployment_state from treating a proxy-side reject as a selected deployment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): skip deployment attribution when the rejected body's model is not a string
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): bucket non-string request models as other instead of raising in failure hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve team deployments and treat guardrail rejects as proxy-side in failure attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): flag pre-routing rejects instead of matching exception names
Post-call GuardrailRaisedException failures kept their deployment labels on main but lost them
on this branch because every GuardrailRaisedException was treated as a pre-routing reject.
The proxy failure path now flags litellm_params with proxy_rejected_before_routing only when
it adds deployment attribution itself, and the Prometheus logger keys deployment selection
off that flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): key pre-routing reject flag off provider handoff, not caller metadata
Caller-supplied metadata.model_info (kept for keys allowed to override pricing) no longer
suppresses proxy_rejected_before_routing. The hook now checks the logging object's
first_api_call_start_time, which only the provider handoff sets, so Prometheus never
records a deployment failure for a request that was rejected before routing.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): poll for both served and rejected spend rows before asserting attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(websearch_interception): propagate parent session/trace ids into intercepted searches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(websearch_interception): let parent correlation win over configured search params and type test params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): bill an intercepted web search under the parent request session
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop unrelated reformatting from the websearch session harness change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): keep the websearch interception session suite out of the stage-mirror gate
The stage-mirror stack runs no websearch_interception callback or search tool, so the suite is deselected there and the changed-tests gate fails on a file that executed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): run the websearch interception session suite on the stage-mirror stack
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(e2e): leave CONTRIBUTING.md untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Resolves the six conflicts main accumulated against the SDK migration.
endpoints_client.py stays deleted (main only added the safety_identifier
kwarg to it), and its two new importers move onto the SDK clients: the
Bedrock safety_identifier forwarding test in test_responses_e2e.py now
sends its /v1/responses leg through the OpenAI SDK and its chat leg
through ProxyClient.chat, and test_messages_azure_foundry_e2e.py drops
the x-litellm-rust header check together with EXPECT_RUST, which main
removed from e2e_config in a84f68b6e3. proxy_client.py keeps both the
rerank and router-settings model imports. uv.lock is main's lock plus
the anthropic e2e-dev entry, and the pricing JSONs are taken verbatim
from main since this PR never meant to change them.
The Buildkite ephemeral stack runs the gateway in another pod, so it cannot reach the pytest host's provider edge. The GitHub changed-e2e lane runs gateways on the runner and sets E2E_PROVIDER_EDGE_HOST_REACHABLE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
