mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-06 07:48:06 +00:00
49f4a59cc8
* fix(router): keep latency routing alive when latency samples are unusable latency-based-routing ranks deployments by the average of the cached samples and keeps everything within lowest_latency_buffer times the lowest. A negative or NaN sample breaks that invariant, because the minimum no longer satisfies its own bound. The candidate list then comes back empty and random.choice raises IndexError, which turns every request to the group into a 500 until the cache entry expires. Samples come from completion_start_time - start_time, and that difference is negative whenever the two timestamps invert. Production recorded samples around -600s from successful calls. Drop non-finite and negative samples before ranking, and fall back to every candidate if the window is empty anyway, for example with a negative buffer. Co-Authored-By: Claude Code <noreply@anthropic.com> Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com> * test(router): seed latency_normalized in the unusable TTFT fallback test Main reads latency_normalized since #47727, so seeding latency left both deployments ranked at zero and the test picked at random. Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): assert ranked latency after dropping unusable TTFT samples Greptile on #48216: the new tests passed with the sample filter removed because the empty-window fallback alone saved them. They now assert _latency_per_deployment so unfiltered negative or NaN samples fail deterministically. Also adds Final to ttft_samples and drops a comment that restated the code Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): drop Final on a loop-local in latency ranking basedpyright rejects Final on a variable assigned inside a loop, and the type_check_gate counts it as a new reportGeneralTypeIssues error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): type the request kwargs in the unusable TTFT tests Greptile on #48216: bare dict annotations. The tests now hold the metadata dict in a typed local and assert on it directly Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| adaptive_router | ||
| complexity_router | ||
| __init__.py | ||
| test_auto_router.py | ||
| test_base_routing_strategy.py | ||
| test_budget_limiter.py | ||
| test_budget_limiter_hotpath.py | ||
| test_complexity_router.py | ||
| test_complexity_tier_predictor.py | ||
| test_fuse_presets.py | ||
| test_lar1_routing.py | ||
| test_least_busy.py | ||
| test_litellm_encoder.py | ||
| test_llm_v2.py | ||
| test_lowest_cost.py | ||
| test_lowest_latency.py | ||
| test_lowest_tpm_rpm.py | ||
| test_lowest_tpm_rpm_v2.py | ||
| test_quality_router.py | ||
| test_router_routing_groups.py | ||
| test_router_routing_plugins.py | ||
| test_router_tag_regex_routing.py | ||
| test_router_tag_routing.py | ||
| test_savings_baseline.py | ||
| test_simple_shuffle.py | ||
| test_stall_detector.py | ||