mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-17 09:34:48 +00:00
* fix(router): re-check and reserve per-model rate limits on fallback targets A provider error fallback handed the caller the fallback model group without its per-model TPM/RPM limits ever being checked. The router gets a fallback_rate_limit_check hook next to fallback_budget_check, the proxy wires it to the v3 limiter, which reserves the target model's per-model descriptors before each fallback hop and skips targets that are over their limit. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): dedupe a repeated fallback target before the reserving rate limit check Both walks now consult the attempted set before the rate limit check, so a chain like ["B", "B"] reserves B once. The deprecated settings parity test also learns the new setting. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): a bidirectional fallback chain never re-reserves the requested model group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): match the surfaced failure in the fallback reservation tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): keep the admission stash untouched while building fallback descriptors Building the fallback target's descriptors could rewrite the request stash's PTU ceiling, which post-call settlement reads for the model the caller named. The fallback check now hides the stash while it builds, like the frame check. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| core | ||
| __init__.py | ||
| test_enforce_model_rate_limits.py | ||
| test_io_token_rate_limits.py | ||
| test_router.py | ||
| test_router_callback_hook_sequence.py | ||
| test_router_endpoints.py | ||
| test_router_entry_points.py | ||
| test_router_facade.py | ||
| test_router_fallbacks.py | ||
| test_router_fuzz_regressions_rust_only.py | ||
| test_router_get_deployments.py | ||
| test_router_helper_utils.py | ||
| test_router_host.py | ||
| test_router_max_parallel_requests.py | ||
| test_router_provider_endpoints.py | ||
| test_router_retries.py | ||
| test_router_surface_check.py | ||
| test_router_timeout.py | ||
| test_router_vector_store_endpoints.py | ||