mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
3c3f266322
* fix(router): a cancelled or retried attempt releases its io-token reservation and least-busy count A routed attempt cancelled mid-call (client disconnect with cancel_on_disconnect) kept its ITPM/OTPM reservation and its least-busy in-flight count, and retries sharing one logging object leaked the count of every attempt after the first. Each async attempt now runs under its own attempt id, so the attempt and its own callbacks release what it holds exactly once, in tasks a cancellation can't stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): settle an io-token reservation once even through a logged copy of its metadata /v1/messages and /v1/responses stash the reservation in litellm_metadata, which the logging object copies. A cancelled attempt refunded it, then the proxy's failure logging refunded the copy again. Each stash now carries an id, and a reservation already settled under that id is not settled again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): keep spent tokens counted, refund a previous attempt's stash, and fit every call in flight A request cancelled after its provider answered now keeps the tokens it spent counted instead of refunding them. A retry refunds the reservation its previous attempt left behind even though it runs under a new attempt. The least-busy reservation store holds twice the keys, since each call in flight now keeps one more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): never evict a settled reservation id before its counters' window has passed Settled ids lived in a capacity-bounded cache, so past 10,000 settlements in two minutes a late copy of an evicted reservation was refunded a second time. They now leave only when their window expires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): start releases eagerly and keep attempt settlement out of router_utils A release that never waits (an in-memory counter) now finishes before its caller goes on, as it did when the caller awaited it inline, so a websocket turn's usage is counted by the time its logging is flushed. Each reservation carries its own id, and the attempt settlement module moves to litellm_core_utils, since litellm/utils.py may not import from router_utils. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): count only finished answers as spent, and declare the least-busy redis family A stream's wrapper returning is not an answer: a stream cancelled before its first chunk now hands back its whole reservation. Only a non-streaming result marks its attempt as answered. The module level least-busy shared decrement carries its redis family, as every direct redis caller must. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| adaptive_router | ||
| complexity_router | ||
| switchyard_router | ||
| __init__.py | ||
| test_auto_router.py | ||
| test_auto_router_model_naming.py | ||
| test_auto_router_tuning_baseline.py | ||
| test_base_routing_strategy.py | ||
| test_baseline_request.py | ||
| test_budget_limiter.py | ||
| test_budget_limiter_hotpath.py | ||
| test_complexity_router.py | ||
| test_complexity_tier_predictor.py | ||
| test_fuse_presets.py | ||
| test_lar1_routing.py | ||
| test_least_busy.py | ||
| test_litellm_encoder.py | ||
| test_llm_v2.py | ||
| test_lowest_cost.py | ||
| test_lowest_latency.py | ||
| test_lowest_tpm_rpm.py | ||
| test_lowest_tpm_rpm_v2.py | ||
| test_quality_router.py | ||
| test_router_routing_groups.py | ||
| test_router_routing_plugins.py | ||
| test_router_tag_regex_routing.py | ||
| test_router_tag_routing.py | ||
| test_savings_baseline.py | ||
| test_simple_shuffle.py | ||
| test_stall_detector.py | ||