mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
172e16acde
* feat(proxy): fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team_metadata.model_preferences Team router_settings.fallbacks now fire on proxy-side rate limits: the pre-call fallback walk reads router settings Key > Team > Global instead of the key's column alone, and skips a fallback the caller may not call. A key or team with router_settings.fallback_on_saturation: true walks that same fallback list as soon as the requested model reaches saturation_threshold in dynamic_rate_limiter_v3, and goes back to the requested model when no fallback admits the request. Responses carry x-litellm-fallback-reason: rate_limit:model_saturation_fallback. Removes the team_metadata.model_preferences steering and the x-litellm-model-steered-from header from #51350, which did the same job through a second knob. * fix(proxy): strict bool for fallback_on_saturation, drop a stale import and settle the pre-call tests fallback_on_saturation only turns on for a real boolean true, so a string like "yes" in router_settings no longer enables it. The saturation unit test reads saturation through the limiter's cache, so it asserts the threshold was reached instead of an exact count. The response polling test now hands the pre-call path an awaitable router_settings lookup, as the pre-call fallback walk reads Key > Team settings before the first call. * fix(proxy): pre-call fallbacks pass the budget and access-group checks a direct request does A fallback the pre-call walk picks now has to clear every per-model budget layer and the budgets of the access groups that grant it, and its spend is charged to those groups instead of the requested model's. At saturation a fallback that fails for any reason counts as a miss and the request goes back to the requested model. The client body can no longer opt itself into the saturation walk through router_settings_override, only the saved key or team router_settings can. The integration test for a forbidden fallback waits for the preferred model's own 429 so it cannot pass before saturation. * refactor(proxy): one helper decides which refused fallbacks end the pre-call walk _pre_call_with_fallbacks had grown past the strict C901 ceiling, so the two except clauses fold into one and _fallback_refusal_ends_the_walk carries the rule: at saturation every refusal is a miss, otherwise only a non-rate-limit failure or a hard per-model 429 reaches the caller. * fix(proxy): keep the caller's request object through the pre-call fallback walk Copying the request to drop a client fallback_on_saturation opt-in handed the hooks a different dict than the one the caller holds, so header metadata written into one never reached the other (test_add_litellm_metadata_from_request_headers). drop_client_saturation_opt_in now edits the request in place. * fix(proxy): pre-call fallbacks keep the pricing and budget-reservation rules a direct request passes A fallback is skipped when unpriced models are blocked and it has no pricing, when a zero-cost request would land on a priced model, and when the budget auth reserved for the requested model does not fit it (higher per-token prices, or reserved against an access group that does not grant it). * refactor(proxy): read fallback prices through the router registry entry point router_surface_check only lets the router surface shrink, so the per-token price lookup goes through registry().group() instead of the deprecated get_model_group_info. * fix(proxy): a fallback's group budget check skips the groups the request already holds The auth-time reservation sits in the same counter the plain budget read uses, so a fallback in a group the request already reserved against was refused by its own hold once the group was within one estimate of max_budget. Groups without a hold keep the plain read. * test(dynamic_rate_limiter_v3): the throttle metric test from main uses fallback_on_saturation instead of the removed steering A saturation handoff carries no rate_limited_by reason, so it is never counted as a throttle. * fix(proxy): the saturation handoff only fires under the fallback walk _pre_call_with_fallbacks arms the request before the hooks run, so routes that call the hooks directly (batches, fine-tuning, polling, images) keep admitting or refusing an opted-in key on a saturated model as before instead of surfacing the handoff 429. * style(proxy): keep the cast-ok reason within the line length * fix(proxy): never invent a metadata bucket when arming the saturation fallback walk, take main's limiter metric tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): arm the saturation fallback walk in the request context, refuse fallbacks for agent-owned keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the saturation walk state out of the body, check the end user's allowlist on fallbacks, release a refused fallback's rate limit hold Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only load the end user for fallback checks when a database is connected Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): grant pre-call fallbacks on the model an alias serves and release the rate limited model's hold before the walk Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): check a pre-call fallback's per-model budgets on its listed name and its served group, as auth does Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): assert the listed fallback name's budget blocks an aliased fallback, not the order of budget reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): release the dynamic limiter's reservation when the fallback walk leaves a model, and replay an exhausted walk with the requested model The dynamic rate limiter admits through its own inner v3 limiter, so the walk's hold release never reached its team-share reservation. It now exposes async_post_call_failure_hook, which refunds that reservation when the proxy refuses a request after admission, and _release_pre_call_hold calls it next to the registered limiter. An exhausted saturation walk replayed with the already aliased model name, so chained model_alias_map entries moved the request on to a further alias. It now replays with the model argument the first attempt saw. The hold-release test drives the real v3 limiter with one slot per key instead of recording hook calls. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type the dynamic limiter's failure hook request dict for the type-discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _prompt_cache_pin_support.py | ||
| _reasoning_effort_levels_support.py | ||
| either_role.json | ||
| test_advisor_failure_cooldown.py | ||
| test_anthropic_overloaded_cooldown_owned_proxy.py | ||
| test_audit_any_sweep_chaos.py | ||
| test_auto_router_management.py | ||
| test_azure_passthrough_spend_row.py | ||
| test_base_model_context_window_owned_proxy.py | ||
| test_batch_mode_deployments_wire.py | ||
| test_complexity_router_codex_compaction.py | ||
| test_complexity_router_databricks_classifier.py | ||
| test_complexity_router_llm_classifier.py | ||
| test_declared_provider_bare_model_routes.py | ||
| test_deployment_mode_routing_wire.py | ||
| test_deployment_num_retries_generic_routes_wire.py | ||
| test_dynamic_limiter_prometheus_metrics.py | ||
| test_dynamic_rate_limiter_borrower_regains_share.py | ||
| test_encrypted_content_affinity_no_peer.py | ||
| test_end_user_region_routing.py | ||
| test_enforce_model_rate_limits_daily_limits.py | ||
| test_fallback_hop_foreign_encrypted_reasoning.py | ||
| test_fallback_on_saturation.py | ||
| test_fallback_rate_limit_check.py | ||
| test_fallback_retry_header_contracts.py | ||
| test_fallback_settles_serving_model.py | ||
| test_include_fallback_errors_wire.py | ||
| test_key_max_parallel_requests.py | ||
| test_key_router_settings_image_generation.py | ||
| test_key_tpm_reservation.py | ||
| test_least_busy_concurrent_decisions.py | ||
| test_messages_stream_retry_budget_sources_owned_proxy.py | ||
| test_model_capability_flags.py | ||
| test_non_chat_model_endpoint_gate.py | ||
| test_observed_routing.py | ||
| test_otpm_default_reservation_fraction.py | ||
| test_payment_required_cooldown.py | ||
| test_payment_required_mapping.py | ||
| test_pdf_document_pre_call_checks_owned_proxy.py | ||
| test_per_model_rate_limit_fallbacks.py | ||
| test_per_model_rate_limit_fallbacks_lifecycle.py | ||
| test_pre_call_response_format_filter.py | ||
| test_priority_class_share_splits_among_active_teams.py | ||
| test_priority_model_tpm_enforcement.py | ||
| test_priority_rate_limit_headers.py | ||
| test_priority_scheduler_queue_cleanup.py | ||
| test_prompt_cache_pin_memory_routes.py | ||
| test_prompt_cache_pin_ttl.py | ||
| test_prompt_cache_pin_ttl_chaos.py | ||
| test_prompt_cache_pin_with_message_logging_off.py | ||
| test_prompt_caching_affinity_tool_call_marks.py | ||
| test_ptu_shares.py | ||
| test_ptu_shares_regex_timing.py | ||
| test_rate_limit_fallback_headers.py | ||
| test_reasoning_effort_levels_chaos.py | ||
| test_reasoning_effort_levels_wire.py | ||
| test_redis_increment_with_floor.py | ||
| test_redis_recovery.py | ||
| test_responses_websocket_turn_admission.py | ||
| test_router_strategies_count_async_requests_once.py | ||
| test_silent_model_shadow_wire.py | ||
| test_stale_cost_map_boot.py | ||
| test_tag_routing_prefix.py | ||
| test_team_model_limit_borrows_idle_capacity.py | ||
| test_team_tag_routing.py | ||
| test_team_wildcard_precedence.py | ||
| test_usage_based_routing_redis_reads.py | ||
| test_user_config_routing.py | ||
| test_video_poll_routing_wire.py | ||
| test_vllm_passthrough_spend_row.py | ||
| test_vllm_passthrough_stream_relay.py | ||
| test_vllm_passthrough_stream_usage_settings.py | ||
| test_web_search_deployment_filter_owned_proxy.py | ||
| test_wildcard_model_listing.py | ||