mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-18 04:44:37 +00:00
43be1c2ae7
* fix(router): stop skipping rate-limit checks when token counting fails Router._pre_call_checks() caught any token_counter() exception and returned the full, unfiltered deployment list, bypassing the RPM, region, and invalid-params checks for every deployment - not just the context-window check that actually depends on the count. A content block token_counter() cannot parse (e.g. an invalid image_url detail value) silently disabled rate-limit enforcement for that request, with only a log line and no error surfaced to the caller. Now a counting failure only disables the context-window check; the other checks still run for every deployment as before. Input-token resolution is pulled into its own method and computed once before the loop, so nothing needs to track "did counting fail" with a mutable flag across iterations. * test(router): cover the async region check when token counting fails Replace the unit test of the private input-token resolver with a regression test through async_get_healthy_deployments, the path the proxy uses: an EU-pinned request whose content cannot be token-counted must not get the US deployment * test(router): cover the param filter when token counting fails and satisfy router coverage * fix(router): answer 400 when no deployment takes the requested param, honor allowlists and drop_params The pre-call param filter now reuses the router's param-acceptance helper, so allowed_openai_params, drop_params, and additional_drop_params keep a deployment instead of filtering it out. When every available deployment lacks a requested param, the router answers 400 UnsupportedParamsError right away instead of a retried 429. Deployment context windows are resolved once per call. * fix(router): keep the retried 429 when a cooling-down deployment takes the param * fix(router): steer to a deployment that takes the param before dropping it, keep context-window fallbacks * fix(router): steer by deployment capability, apply drop settings after region and RPM checks A deployment-level allowed_openai_params no longer counts as taking the param when another deployment does, the drop and allow settings now apply only when no deployment that passed the region, RPM, and context-window checks takes it, and an authenticating provider is checked by its declared model instead of always passing * test(router): make the github_copilot pre-call filter test hermetic in CI * test(router): cover pre-call region and response_format filters through the proxy * fix(router): judge the 400 against the request's own deployments, not a name lookup * fix(router): judge the 400 against the passed deployments when the group name lists none * fix(router): let a request's drop and allow settings replace the deployment's in the param check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _prompt_cache_pin_support.py | ||
| either_role.json | ||
| test_advisor_failure_cooldown.py | ||
| test_audit_any_sweep_chaos.py | ||
| test_auto_router_management.py | ||
| test_azure_passthrough_spend_row.py | ||
| test_base_model_context_window_owned_proxy.py | ||
| test_complexity_router_databricks_classifier.py | ||
| test_complexity_router_llm_classifier.py | ||
| test_declared_provider_bare_model_routes.py | ||
| test_deployment_num_retries_generic_routes_wire.py | ||
| test_encrypted_content_affinity_no_peer.py | ||
| test_end_user_region_routing.py | ||
| test_fallback_hop_foreign_encrypted_reasoning.py | ||
| test_fallback_retry_header_contracts.py | ||
| test_include_fallback_errors_wire.py | ||
| test_key_max_parallel_requests.py | ||
| test_key_tpm_reservation.py | ||
| test_messages_stream_retry_budget_sources_owned_proxy.py | ||
| test_model_capability_flags.py | ||
| test_observed_routing.py | ||
| test_payment_required_cooldown.py | ||
| test_payment_required_mapping.py | ||
| test_pdf_document_pre_call_checks_owned_proxy.py | ||
| test_per_model_rate_limit_fallbacks.py | ||
| test_per_model_rate_limit_fallbacks_lifecycle.py | ||
| test_pre_call_response_format_filter.py | ||
| test_priority_model_tpm_enforcement.py | ||
| test_priority_rate_limit_headers.py | ||
| test_priority_scheduler_queue_cleanup.py | ||
| test_prompt_cache_pin_memory_routes.py | ||
| test_prompt_cache_pin_ttl.py | ||
| test_prompt_cache_pin_ttl_chaos.py | ||
| test_prompt_caching_affinity_tool_call_marks.py | ||
| test_ptu_shares.py | ||
| test_ptu_shares_regex_timing.py | ||
| test_redis_increment_with_floor.py | ||
| test_redis_recovery.py | ||
| test_stale_cost_map_boot.py | ||
| test_team_tag_routing.py | ||
| test_usage_based_routing_redis_reads.py | ||
| test_user_config_routing.py | ||
| test_video_poll_routing_wire.py | ||
| test_vllm_passthrough_spend_row.py | ||
| test_vllm_passthrough_stream_relay.py | ||
| test_vllm_passthrough_stream_usage_settings.py | ||