litellm/tests/unit/router_strategy
Mateo Wang d5ca88900c
fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063)
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages

Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.

An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.

* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping

* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame

The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529

* test(proxy): expect the top-level Anthropic envelope from count_tokens errors

count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.

* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets

* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status

An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.

* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status

* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload

* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases

* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status

* fix(proxy): walk nested mid-stream fallbacks without recursion

* fix(anthropic): preserve 503s and answer upstream 529s on messages routes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): use integration coverage for keepalive errors

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-03 17:37:09 -08:00
..
adaptive_router fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063) 2026-11-03 17:37:09 -08:00
complexity_router fix(router): restore four base behaviors the RoutingContext refactor changed (#49395) 2026-10-31 22:58:34 +00:00
__init__.py test: add __init__.py to intermediate tests/unit packages 2026-09-20 12:55:27 +00:00
test_auto_router.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_auto_router_model_naming.py feat(router): infer custom classifier tiers from model mappings (#48987) 2026-10-29 11:30:23 -07:00
test_auto_router_tuning_baseline.py refactor(router): fence the router surface and document the router-core boundary (#48839) 2026-10-29 00:01:52 +00:00
test_base_routing_strategy.py fix(router): unregister strategy selectors from global callbacks on discard (#45795) 2026-10-21 02:39:36 +00:00
test_baseline_request.py refactor(router): fence the router surface and document the router-core boundary (#48839) 2026-10-29 00:01:52 +00:00
test_budget_limiter.py fix(batches): retry confirmed pre-accounting failures (#47625) 2026-10-20 14:12:26 -07:00
test_budget_limiter_hotpath.py test(router_strategy): keep foreign asyncio task noise out of the budget limiter caplog asserts (#43375) 2026-10-29 06:22:06 -07:00
test_complexity_router.py feat(decisions): add Parallel AI web-grounded decisions (#49188) 2026-11-03 04:47:02 +00:00
test_complexity_tier_predictor.py test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
test_fuse_presets.py test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
test_lar1_routing.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_least_busy.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_litellm_encoder.py fix(router): restore four base behaviors the RoutingContext refactor changed (#49395) 2026-10-31 22:58:34 +00:00
test_llm_v2.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_lowest_cost.py fix(router): price cost-based routing through model_info.base_model (#47617) 2026-10-20 22:20:16 +00:00
test_lowest_latency.py fix(router): keep latency routing alive when latency samples are unusable (#48216) 2026-10-24 13:43:07 -07:00
test_lowest_tpm_rpm.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_lowest_tpm_rpm_v2.py fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks (#49831) 2026-11-03 22:06:59 +00:00
test_quality_router.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_router_routing_groups.py refactor(router): extract PythonCore behind RouterCore and pass the router its config (#49119) 2026-10-31 12:54:23 -07:00
test_router_routing_plugins.py test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
test_router_tag_regex_routing.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_router_tag_routing.py fix(router): restore four base behaviors the RoutingContext refactor changed (#49395) 2026-10-31 22:58:34 +00:00
test_savings_baseline.py refactor(router): route the router family through RoutingContext and close the outside surface (#49118) 2026-10-30 19:18:54 -07:00
test_simple_shuffle.py test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
test_stall_detector.py test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00