mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
85a3e566f3
* fix(router)!: name the request size when usage-based routing refuses an over-tpm request usage-based-routing-v2 answered "No deployments available" with no usage shown when one request alone was over every deployment's tpm, and a single-deployment group held the caller for 3 x 60 seconds on retries that could never clear it. The router now raises RequestOverDeploymentTpmError, a RateLimitError that names the estimated tokens and the largest tpm limit, carries no retry-after, skips retries, and still reaches group fallbacks. * fix(router): count cooling deployments before refusing an over-tpm request, keep the Prometheus label The over-tpm refusal now also reads every deployment in the model group, so a request that fits a cooling deployment keeps the old retryable 429. RequestOverDeploymentTpmError keeps the RateLimitError exception_class label in Prometheus. * fix(router): count only cooling deployments the request can still reach, and keep the logged error class The over-tpm refusal now reads cooling deployments from the caller's pool, kept to the ones team, region, and tag routing would still send the request to, so team models and tag-filtered requests get the right answer. Logs, OTEL, and Datadog keep naming the refusal RateLimitError like Prometheus does, and a tpm written as a number string counts as that number * fix(router): keep the sync over-tpm refusal a ValueError and count every deployment the caller may use The sync Router raises SyncRequestOverDeploymentTpmError, which is still a ValueError, so existing except ValueError handlers keep catching it. The never-retryable refusal now counts every deployment in the caller's pool that region and tag routing allow, so a deployment left out of this pick by a budget, health check, order, or pin keeps the retryable 429 * fix(router): keep retry records and trace events naming the over-tpm refusal RateLimitError Fallback records in previous_models and the litellm.routing.retry trace event now read the error's logged class name, as the logging payload already did, so they keep RateLimitError for the over-tpm refusal as they did before this change --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| core | ||
| __init__.py | ||
| test_enforce_model_rate_limits.py | ||
| test_io_token_rate_limits.py | ||
| test_router.py | ||
| test_router_callback_hook_sequence.py | ||
| test_router_endpoints.py | ||
| test_router_entry_points.py | ||
| test_router_facade.py | ||
| test_router_fallbacks.py | ||
| test_router_fuzz_regressions_rust_only.py | ||
| test_router_get_deployments.py | ||
| test_router_helper_utils.py | ||
| test_router_host.py | ||
| test_router_max_parallel_requests.py | ||
| test_router_provider_endpoints.py | ||
| test_router_retries.py | ||
| test_router_surface_check.py | ||
| test_router_timeout.py | ||
| test_router_vector_store_endpoints.py | ||