mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
b5a72c2042
* fix(router): pin /v1/responses requests for prompt caching affinity * fix(router): pin previous_response_id turns and keep native tools in the Responses pin key A Responses turn that continues a stored response now goes to the deployment that answered it, unless that deployment already failed this request. Native Responses tools are hashed into the pin key as sent instead of converted, so two requests with different native tools no longer share a pin and a native request no longer logs a false dropped-tool warning. The pin keeps its cacheable deployment ids in a sorted tuple so loggers can serialize request metadata. * fix(router): leave a Responses request with a malformed tool to the provider * fix(router): keep the Responses prompt cache pin out of logged metadata The pre-call check now keeps the pin in a router-owned in-flight store and puts only a random token in litellm_metadata, so loggers that ship metadata never see the routing key or deployment ids. The store is shared across check instances, so turning prompt_caching off and on at runtime keeps pinning, and a retry drops its previous attempt's token. The skipped-input debug log now names only the error type. * fix(router): apply the prompt caching pin after tag, region and pre-call filters, and leave it on retries The prompt caching check no longer narrows the candidates inside the callback filters. It names the deployment it prefers, and the router core applies that preference at the pinned deployment step, after tag, region, pre-call and plugin filters, only when the deployment is still a candidate. A pin can no longer send a request to a deployment its tags do not allow. A retry skips the pin when the pinned deployment was already tried in this request, read from the retry breadcrumbs in both metadata buckets and the excluded and retry-skipped ids. When the retry count outruns the breadcrumbs the check stands down, so a pinned deployment that keeps failing is not retried on every attempt. * test(router): list the prompt caching check as an in-memory cache caller The check's in-flight Responses pins live in a process-local InMemoryCache, so it opens no Redis span and needs no key family, like the other in-memory callers * fix(router): pin a lone healthy deployment, free every in-flight Responses pin, and clear a reused metadata dict's retry records * test(router): lift test cooldowns through the cooldown store's evict_local * fix(router): key Responses tools like their chat twin and drop the pin token after success A Responses function tool now hashes only the function fields it sent, so a chat request and a Responses request with the same tool share one pin. The success event also removes the pin token from the caller's metadata dict. * test(router): cover untyped tool parameters and custom tool grammars in the Responses pin key --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _prompt_cache_pin_support.py | ||
| _reasoning_effort_levels_support.py | ||
| either_role.json | ||
| test_advisor_failure_cooldown.py | ||
| test_anthropic_overloaded_cooldown_owned_proxy.py | ||
| test_audit_any_sweep_chaos.py | ||
| test_auto_router_management.py | ||
| test_azure_passthrough_spend_row.py | ||
| test_base_model_context_window_owned_proxy.py | ||
| test_batch_mode_deployments_wire.py | ||
| test_complexity_router_codex_compaction.py | ||
| test_complexity_router_databricks_classifier.py | ||
| test_complexity_router_llm_classifier.py | ||
| test_declared_provider_bare_model_routes.py | ||
| test_deployment_mode_routing_wire.py | ||
| test_deployment_num_retries_generic_routes_wire.py | ||
| test_encrypted_content_affinity_no_peer.py | ||
| test_end_user_region_routing.py | ||
| test_enforce_model_rate_limits_daily_limits.py | ||
| test_fallback_hop_foreign_encrypted_reasoning.py | ||
| test_fallback_retry_header_contracts.py | ||
| test_include_fallback_errors_wire.py | ||
| test_key_max_parallel_requests.py | ||
| test_key_router_settings_image_generation.py | ||
| test_key_tpm_reservation.py | ||
| test_least_busy_concurrent_decisions.py | ||
| test_messages_stream_retry_budget_sources_owned_proxy.py | ||
| test_model_capability_flags.py | ||
| test_non_chat_model_endpoint_gate.py | ||
| test_observed_routing.py | ||
| test_otpm_default_reservation_fraction.py | ||
| test_payment_required_cooldown.py | ||
| test_payment_required_mapping.py | ||
| test_pdf_document_pre_call_checks_owned_proxy.py | ||
| test_per_model_rate_limit_fallbacks.py | ||
| test_per_model_rate_limit_fallbacks_lifecycle.py | ||
| test_pre_call_response_format_filter.py | ||
| test_priority_model_tpm_enforcement.py | ||
| test_priority_rate_limit_headers.py | ||
| test_priority_scheduler_queue_cleanup.py | ||
| test_prompt_cache_pin_memory_routes.py | ||
| test_prompt_cache_pin_ttl.py | ||
| test_prompt_cache_pin_ttl_chaos.py | ||
| test_prompt_cache_pin_with_message_logging_off.py | ||
| test_prompt_caching_affinity_tool_call_marks.py | ||
| test_ptu_shares.py | ||
| test_ptu_shares_regex_timing.py | ||
| test_reasoning_effort_levels_chaos.py | ||
| test_reasoning_effort_levels_wire.py | ||
| test_redis_increment_with_floor.py | ||
| test_redis_recovery.py | ||
| test_responses_websocket_turn_admission.py | ||
| test_router_strategies_count_async_requests_once.py | ||
| test_silent_model_shadow_wire.py | ||
| test_stale_cost_map_boot.py | ||
| test_tag_routing_prefix.py | ||
| test_team_tag_routing.py | ||
| test_team_wildcard_precedence.py | ||
| test_usage_based_routing_redis_reads.py | ||
| test_user_config_routing.py | ||
| test_video_poll_routing_wire.py | ||
| test_vllm_passthrough_spend_row.py | ||
| test_vllm_passthrough_stream_relay.py | ||
| test_vllm_passthrough_stream_usage_settings.py | ||
| test_web_search_deployment_filter_owned_proxy.py | ||
| test_wildcard_model_listing.py | ||