litellm/tests/integration/routing
quietjules 0134008628
fix(responses): write one failure row per Responses WebSocket turn the deployment refuses (#50731)
* fix(responses): write one failure row per refused Responses WebSocket turn

A turn the deployment refuses on an open native or managed Responses WebSocket now writes its own spend
log failure row when it is refused, with its own request id, the turn's start time and request, the
socket's resolved model group and the websocket call type. Before, refused turns wrote nothing until the
socket closed, and then only one row for the whole socket, carrying the socket's open time and first frame.

Guardrail-blocked turns go through the same row builder, so their rows now carry the model group and call
type too. Turns refused or crashed in the turn check before a session-limit cancel also keep their rows.

The socket's close-time failure hooks still run once to settle the admission budget reservation, but when
the close failure is the first refused turn the spend logger skips the duplicate row.

* fix(responses): keep forged logging state out of refused-turn rows and wait for each row

A refused turn's failure row now leaves out the top-level fields the HTTP path strips from a client body, plus standard_callback_dynamic_params and litellm_params, so a frame cannot forge its own logging state or turn off message redaction. The relay now waits for each refused turn's row before it reads the next frame, and the write still finishes if the socket is cancelled meanwhile

* ci: start the unit workflow GitHub never created for the previous push

* fix(responses): keep the client's request tags on refused-turn rows

A refused turn's row now carries the key's tags plus the client's User-Agent tags, computed from the socket's request headers like its success rows, while the headers themselves stay out of the row

* fix(responses): keep a refused turn's frame from forging its spend row and store no blocked prompt

A refused turn's failure row now takes only the turn's model and call id from the frame, ignores frame metadata tags and model_group, and logs non-object metadata as empty instead of dropping the row. Guardrail-blocked frames no longer carry a stored request.

* fix(proxy): keep the socket's session id, guardrail records, and router-key strip on refused WebSocket turn rows

* fix(proxy): use the public router-reserved metadata field list, as main renamed it

* fix(responses): give refused WebSocket turn rows the socket's session id

When a socket's request carries no trace id, the router minted its own for the relayed turns, so success rows and refused-turn rows had different session ids. The endpoint now hands the router the socket's trace id

The refused-turn row test's session-limit case jumps the loop clock past the limit once the relay is parked, instead of racing a 50 ms timer

* test(responses): expect the parallel-slot refusal at the WebSocket handshake

Main now refuses a cached key that is over its limits before the request body is read, so a second socket on a key whose only parallel slot an open socket holds is refused at the upgrade instead of with a 429 frame. The test still checks that the refused turn's failure row keeps the slot held until the first socket closes

* fix(responses): take a refused WebSocket turn's agent and session fields from the socket

A frame's metadata.agent_id, billing_agent_id or session_id no longer lands on the refused turn's spend row. The row copies those fields from the socket, like its success rows

The integration test no longer pins the reopened socket's 1011 close, which is incidental to the held slot it checks

* test(responses): expect a 403 when a held parallel slot refuses a second WebSocket handshake

* test(responses): expect a held parallel slot to refuse a second socket's first turn with a 429 frame

Main's #51675 skips the headers-only limit check for a bodyless GET, and every WebSocket upgrade is one, so a second socket on a key whose only parallel slot is held now opens and its first turn gets a 429 max_parallel_requests frame and close 1008 instead of a 403 at the handshake

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 22:08:06 -08:00
..
_prompt_cache_pin_support.py fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) 2026-10-14 21:10:09 -07:00
_reasoning_effort_levels_support.py test(routing): audit #46601 (#48910) 2026-10-28 15:42:51 -07:00
either_role.json test(integration): add read-replica routing harness to the CircleCI integration suite (#42692) 2026-09-24 00:25:03 -07:00
test_advisor_failure_cooldown.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_anthropic_overloaded_cooldown_owned_proxy.py fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063) 2026-11-03 17:37:09 -08:00
test_audit_any_sweep_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_auto_router_management.py test(auto_router): verify total_spend in session integration responses (#51283) 2026-11-16 12:11:26 -08:00
test_azure_passthrough_spend_row.py fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733) 2026-10-14 12:40:48 -07:00
test_base_model_context_window_owned_proxy.py fix(router): let an exact base_model widen a deployment's context window (#46197) 2026-10-16 06:37:40 +00:00
test_batch_mode_deployments_wire.py fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order (#50971) 2026-11-10 18:35:19 -08:00
test_complexity_router_codex_compaction.py fix(complexity_router): keep the held user_turn route for Codex compaction requests (#48627) 2026-10-27 13:33:32 -07:00
test_complexity_router_databricks_classifier.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_complexity_router_llm_classifier.py refactor(types): replace Any with proven types in 7 files (#43844) 2026-10-02 02:47:47 -07:00
test_customer_org_and_key_model_limits_borrow_idle_capacity.py feat(dynamic_rate_limiter): customer organizations and keys borrow idle model capacity and split class shares (#51705) 2026-11-15 12:05:58 -08:00
test_declared_provider_bare_model_routes.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
test_deployment_mode_routing_wire.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_deployment_num_retries_generic_routes_wire.py fix(router): retry a /v1/messages stream the provider drops before the first content chunk (#44276) 2026-10-05 22:53:39 +00:00
test_dynamic_limiter_prometheus_metrics.py feat(dynamic_rate_limiter): customer organizations and keys borrow idle model capacity and split class shares (#51705) 2026-11-15 12:05:58 -08:00
test_dynamic_rate_limiter_borrower_regains_share.py feat(dynamic_rate_limiter): customer organizations and keys borrow idle model capacity and split class shares (#51705) 2026-11-15 12:05:58 -08:00
test_encrypted_content_affinity_no_peer.py fix(router): serve Responses turns from a sibling when the encrypted content origin has no boundary peer (#43015) 2026-09-24 17:31:43 -07:00
test_end_user_region_routing.py fix(router): drop the deployment rpm filter from enable_pre_call_checks (#48873) 2026-10-28 12:13:56 -07:00
test_enforce_model_rate_limits_daily_limits.py feat(router): add deployment rpd and tpd daily limits to enforce_model_rate_limits (#48875) 2026-10-28 15:52:44 -07:00
test_fallback_hop_foreign_encrypted_reasoning.py fix(router): keep encrypted reasoning when a fallback hops between same-key deployments with no api_base (#49396) 2026-11-02 05:47:04 -08:00
test_fallback_on_saturation.py feat(proxy): router_settings.fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team model_preferences (#51439) 2026-11-13 05:57:18 +00:00
test_fallback_rate_limit_check.py fix(router): re-check and reserve per-model rate limits on fallback targets (#50994) 2026-11-10 16:36:10 -08:00
test_fallback_retry_header_contracts.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_fallback_settles_serving_model.py fix(rate-limit): settle a fallback's tokens on the model group that served it (#51082) 2026-11-10 18:36:55 -08:00
test_include_fallback_errors_wire.py fix(proxy): send the streamed Responses fallback metadata as a typed Responses event (#50804) 2026-11-10 21:29:32 +00:00
test_key_max_parallel_requests.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_key_router_settings_image_generation.py fix(proxy): apply key and team router_settings to /v1/images/generations (#49286) 2026-10-30 12:38:21 -07:00
test_key_tpm_reservation.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_least_busy_concurrent_decisions.py fix(least-busy): reserve the picked deployment so a burst of requests spreads out (#47484) 2026-11-05 00:30:12 +00:00
test_messages_stream_retry_budget_sources_owned_proxy.py fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063) 2026-11-03 17:37:09 -08:00
test_model_capability_flags.py fix: keep model capability flags out of provider requests (#46647) 2026-10-15 14:30:10 +00:00
test_non_chat_model_endpoint_gate.py feat(router): opt in to non-chat deployment filtering (#46257) 2026-10-29 08:33:36 -07:00
test_observed_routing.py test: fix integration tests that fail on test order, races and intended changes (#49310) 2026-10-30 13:25:47 -07:00
test_otpm_default_reservation_fraction.py fix(router): cap the default OTPM reservation and reconcile token counters after each call (#49054) 2026-10-29 20:32:04 -07:00
test_payment_required_cooldown.py fix(router): keep a lone deployment warm under router allowed_fails unless the policy names the exception (#48644) 2026-10-27 15:12:53 -07:00
test_payment_required_mapping.py fix(exceptions): map upstream 402 to PaymentRequiredError and cool down 402 deployments (#44879) 2026-10-06 20:35:13 -07:00
test_pdf_document_pre_call_checks_owned_proxy.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_per_model_rate_limit_fallbacks.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_per_model_rate_limit_fallbacks_lifecycle.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_pre_call_response_format_filter.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_priority_class_share_light_team_not_starved.py fix(dynamic_rate_limiter): charge tokens used while the model was idle against a team's class share (#51952) 2026-11-15 16:49:47 -08:00
test_priority_class_share_splits_among_active_teams.py feat(dynamic_rate_limiter): customer organizations and keys borrow idle model capacity and split class shares (#51705) 2026-11-15 12:05:58 -08:00
test_priority_model_tpm_enforcement.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_priority_rate_limit_headers.py fix(dynamic_rate_limiter): keep counting and reporting priority requests after a priority pool's tokens run out (#52081) 2026-11-16 22:00:36 +00:00
test_priority_scheduler_queue_cleanup.py test(router): audit #47498 (#48799) 2026-10-28 04:36:50 -07:00
test_prompt_cache_pin_memory_routes.py fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) 2026-10-14 21:10:09 -07:00
test_prompt_cache_pin_ttl.py test(routing): expect /v1/responses to pin the served deployment for an hour after #50321 (#50474) 2026-11-07 00:08:05 +00:00
test_prompt_cache_pin_ttl_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_prompt_cache_pin_with_message_logging_off.py fix(router): write the prompt caching pin when turn_off_message_logging is on (#50276) 2026-11-05 15:59:57 -08:00
test_prompt_caching_affinity_tool_call_marks.py fix(router): pin /v1/responses requests for prompt caching affinity (#50321) 2026-11-06 12:56:54 -08:00
test_ptu_shares.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_ptu_shares_regex_timing.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_rate_limit_fallback_headers.py fix(proxy): check pre-call fallback access with the saved model_group_alias, never the one the client sent (#52077) 2026-11-16 22:02:42 +00:00
test_reasoning_effort_levels_chaos.py feat(router)!: apply each deployment's reasoning_effort_levels on every provider and route (#46601) 2026-10-20 04:34:55 +00:00
test_reasoning_effort_levels_wire.py fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) 2026-11-05 04:07:11 +00:00
test_redis_increment_with_floor.py ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) 2026-10-01 13:00:54 -07:00
test_redis_recovery.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_rerank_cancel_on_disconnect.py fix(proxy): honor cancel_on_disconnect on the rerank endpoints (#51252) 2026-11-14 03:59:22 -05:00
test_responses_stream_same_group_retry.py fix(responses): retry a stream that breaks before output in its own model group (#51953) 2026-11-15 12:19:58 -08:00
test_responses_websocket_turn_admission.py fix(responses): write one failure row per Responses WebSocket turn the deployment refuses (#50731) 2026-11-16 22:08:06 -08:00
test_router_strategies_count_async_requests_once.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_silent_model_shadow_wire.py test(router): audit #41368 (#48794) 2026-10-28 05:14:59 -07:00
test_stale_cost_map_boot.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_tag_routing_prefix.py fix(router)!: only prefixed caller tags route once tag_routing_prefix is set (#47428) 2026-10-20 00:12:28 -07:00
test_team_model_limit_borrows_idle_capacity.py feat(dynamic_rate_limiter): customer organizations and keys borrow idle model capacity and split class shares (#51705) 2026-11-15 12:05:58 -08:00
test_team_tag_routing.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_team_wildcard_precedence.py test(routing): a team's wildcard serves its keys before a proxy-wide wildcard of the same pattern (#48501) 2026-10-26 17:54:32 -07:00
test_usage_based_routing_redis_reads.py test(integration): repair tests that merged changes left stale (#50782) 2026-11-10 01:08:35 +00:00
test_user_config_routing.py feat(openai)!: honor per-deployment workload identity on non-chat routes (#39966) 2026-10-16 22:25:15 +00:00
test_video_poll_routing_wire.py fix(videos)!: keep model_id in proxy video ids so per-model keys resolve on status/content (#44614) 2026-10-20 05:42:59 +00:00
test_vllm_passthrough_spend_row.py fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733) 2026-10-14 12:40:48 -07:00
test_vllm_passthrough_stream_relay.py fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733) 2026-10-14 12:40:48 -07:00
test_vllm_passthrough_stream_usage_settings.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_web_search_deployment_filter_owned_proxy.py fix(router): drop supports_web_search false deployments for Anthropic web_search_* and Gemini googleSearch tools (#50275) 2026-11-06 01:58:17 +00:00
test_wildcard_model_listing.py fix(providers): route Azure Responses past dated api versions, count Grok reasoning tokens on azure_ai, refresh provider sets after register_model, fix gpt-6.1-sol Azure rows (#46377) 2026-10-16 15:23:46 -07:00