mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
85a3e566f3
* fix(router)!: name the request size when usage-based routing refuses an over-tpm request usage-based-routing-v2 answered "No deployments available" with no usage shown when one request alone was over every deployment's tpm, and a single-deployment group held the caller for 3 x 60 seconds on retries that could never clear it. The router now raises RequestOverDeploymentTpmError, a RateLimitError that names the estimated tokens and the largest tpm limit, carries no retry-after, skips retries, and still reaches group fallbacks. * fix(router): count cooling deployments before refusing an over-tpm request, keep the Prometheus label The over-tpm refusal now also reads every deployment in the model group, so a request that fits a cooling deployment keeps the old retryable 429. RequestOverDeploymentTpmError keeps the RateLimitError exception_class label in Prometheus. * fix(router): count only cooling deployments the request can still reach, and keep the logged error class The over-tpm refusal now reads cooling deployments from the caller's pool, kept to the ones team, region, and tag routing would still send the request to, so team models and tag-filtered requests get the right answer. Logs, OTEL, and Datadog keep naming the refusal RateLimitError like Prometheus does, and a tpm written as a number string counts as that number * fix(router): keep the sync over-tpm refusal a ValueError and count every deployment the caller may use The sync Router raises SyncRequestOverDeploymentTpmError, which is still a ValueError, so existing except ValueError handlers keep catching it. The never-retryable refusal now counts every deployment in the caller's pool that region and tag routing allow, so a deployment left out of this pick by a budget, health check, order, or pin keeps the retryable 429 * fix(router): keep retry records and trace events naming the over-tpm refusal RateLimitError Fallback records in previous_models and the litellm.routing.retry trace event now read the error's logged class name, as the logging payload already did, so they keep RateLimitError for the over-tpm refusal as they did before this change --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| adaptive_router | ||
| complexity_router | ||
| switchyard_router | ||
| __init__.py | ||
| test_auto_router.py | ||
| test_auto_router_model_naming.py | ||
| test_auto_router_tuning_baseline.py | ||
| test_base_routing_strategy.py | ||
| test_baseline_request.py | ||
| test_budget_limiter.py | ||
| test_budget_limiter_hotpath.py | ||
| test_complexity_router.py | ||
| test_complexity_tier_predictor.py | ||
| test_fuse_presets.py | ||
| test_lar1_routing.py | ||
| test_least_busy.py | ||
| test_litellm_encoder.py | ||
| test_llm_v2.py | ||
| test_lowest_cost.py | ||
| test_lowest_latency.py | ||
| test_lowest_tpm_rpm.py | ||
| test_lowest_tpm_rpm_v2.py | ||
| test_quality_router.py | ||
| test_router_routing_groups.py | ||
| test_router_routing_plugins.py | ||
| test_router_tag_regex_routing.py | ||
| test_router_tag_routing.py | ||
| test_savings_baseline.py | ||
| test_simple_shuffle.py | ||
| test_stall_detector.py | ||