litellm/tests/unit/test_router
joshua-berri 3c3f266322
fix(router): a cancelled or retried attempt releases its io-token reservation and least-busy count (#50486)
* fix(router): a cancelled or retried attempt releases its io-token reservation and least-busy count

A routed attempt cancelled mid-call (client disconnect with cancel_on_disconnect) kept its ITPM/OTPM
reservation and its least-busy in-flight count, and retries sharing one logging object leaked the
count of every attempt after the first. Each async attempt now runs under its own attempt id, so the
attempt and its own callbacks release what it holds exactly once, in tasks a cancellation can't stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): settle an io-token reservation once even through a logged copy of its metadata

/v1/messages and /v1/responses stash the reservation in litellm_metadata, which the logging object
copies. A cancelled attempt refunded it, then the proxy's failure logging refunded the copy again.
Each stash now carries an id, and a reservation already settled under that id is not settled again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): keep spent tokens counted, refund a previous attempt's stash, and fit every call in flight

A request cancelled after its provider answered now keeps the tokens it spent counted instead of
refunding them. A retry refunds the reservation its previous attempt left behind even though it runs
under a new attempt. The least-busy reservation store holds twice the keys, since each call in flight
now keeps one more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): never evict a settled reservation id before its counters' window has passed

Settled ids lived in a capacity-bounded cache, so past 10,000 settlements in two minutes a late copy
of an evicted reservation was refunded a second time. They now leave only when their window expires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): start releases eagerly and keep attempt settlement out of router_utils

A release that never waits (an in-memory counter) now finishes before its caller goes on, as it did
when the caller awaited it inline, so a websocket turn's usage is counted by the time its logging is
flushed. Each reservation carries its own id, and the attempt settlement module moves to
litellm_core_utils, since litellm/utils.py may not import from router_utils.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): count only finished answers as spent, and declare the least-busy redis family

A stream's wrapper returning is not an answer: a stream cancelled before its first chunk now hands
back its whole reservation. Only a non-streaming result marks its attempt as answered. The module
level least-busy shared decrement carries its redis family, as every direct redis caller must.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:54:26 -08:00
..
core fix(router): record retries and fallback hops past a request's first logged failure (#50445) 2026-11-06 19:48:25 -08:00
__init__.py test: add __init__.py to every tests/unit directory phase 16 touches 2026-09-20 11:52:38 +00:00
test_enforce_model_rate_limits.py feat(router): daily_reset_time picks when deployment rpd and tpd daily limits start over (#48957) 2026-10-28 20:20:01 -07:00
test_io_token_rate_limits.py fix(router): a cancelled or retried attempt releases its io-token reservation and least-busy count (#50486) 2026-11-07 16:54:26 -08:00
test_router.py feat(router): add the Switchyard stage router as an auto router strategy (#50407) 2026-11-06 12:07:57 -08:00
test_router_callback_hook_sequence.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_router_endpoints.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
test_router_entry_points.py feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes (#49408) 2026-11-06 11:14:47 -08:00
test_router_facade.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
test_router_fallbacks.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
test_router_fuzz_regressions_rust_only.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
test_router_get_deployments.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
test_router_helper_utils.py feat(redis): serve the proxy's common Redis call with Rust stores (#50170) 2026-11-06 06:01:11 +00:00
test_router_host.py refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
test_router_max_parallel_requests.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
test_router_provider_endpoints.py feat(redis): serve the proxy's common Redis call with Rust stores (#50170) 2026-11-06 06:01:11 +00:00
test_router_retries.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
test_router_surface_check.py refactor(router): fence the router surface and document the router-core boundary (#48839) 2026-10-29 00:01:52 +00:00
test_router_timeout.py refactor(router): delete the python router core, the rust core is the only core (#50178) 2026-11-05 10:40:46 -08:00
test_router_vector_store_endpoints.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00