mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-08 22:21:35 +00:00
05943b47a3
* fix(router): cool down failed fallback deployments and correct cooldown TTL after Redis backfill A deployment that failed partway through a fallback chain (any attempt after the first) was silently exempt from cooldown, because the has_logged_async_failure dedup flag blocks the normal failure callback for every attempt past the first. _trigger_cooldown_for_failed_deployment now explicitly evaluates cooldown for that deployment when the dedup flag is set, using the same deployment-config > response-header > router-default precedence as the primary failure path, and skips advisor-orchestration failures. Deployment-ID resolution prefers the exception's stamped failed_deployment_id, now also set from the generic-API-call fallback path (rerank, embeddings, /v1/messages, etc.), falling back to metadata inspection for call paths that don't stamp it yet. CooldownCache also recomputes the remaining TTL when DualCache promotes a Redis entry into the in-memory layer: before this, a cooldown entry restored from Redis kept the in-memory layer's default 600s TTL regardless of the deployment's real cooldown_time, so a deployment could stay excluded from routing for up to 10 minutes after a much shorter cooldown had already expired. * fix(router): address Greptile review on the fallback-cooldown trigger Two P1 findings on PR #35104: - _trigger_cooldown_for_failed_deployment never incremented the deployment's per-minute failure counter before evaluating cooldown, so a fallback deployment's repeated retryable failures never accumulated toward the default percent-fail-rate threshold that _should_cooldown_deployment checks. - The metadata-bucket fallback (checking "metadata" before "litellm_metadata" for a deployment_model_name marker) could be fooled by a caller with permission to set metadata, since neither bucket's authorship can be determined without knowing the call's function_name. Removed it entirely; cooldown now requires the server-stamped failed_deployment_id, matching what the primary chat-completions path and the generic-API-call path (rerank, embeddings, /v1/messages, etc.) already set unconditionally. * fix(router): freeze the litellm_params fallback mapping to satisfy the type-discipline gate * fix(router): defer f-string interpolation in fallback-cooldown debug logs * fix(router): annotate cooldown-path locals with Final to satisfy the LIT010 budget * fix(router): don't cool down deployments for request-scoped 404s on generic API fallbacks * fix(router): stamp the dynamic client-side-credential deployment id, not the shared static one * fix(router): don't cool down deployments for a caller-supplied x-litellm-timeout * fix(router): stamp dynamic client-side-credential id in completion fallback paths too The generic-API-call helper already stamped the effective (dynamic-if-client-side-credential) deployment id on exceptions, but the regular _completion/_acompletion exception handlers still stamped the static shared deployment's id. A tenant using invalid forwarded credentials could generate repeated failures attributed to, and eventually cooling down, the shared deployment other tenants rely on. Extracted the stamping logic into one shared helper used by all three call sites (generic API, sync completion, async completion) so the fix and future changes to it stay in one place. * test(router): add direct-reference unit tests for the new stamping helper router_code_coverage.py's coverage gate flags _stamp_failed_deployment_id_with_effective_model_info as untested because it only sees the function invoked indirectly through _completion/_acompletion's exception handlers. Added two tests that call it directly, covering both the dynamic-id-present and static-fallback branches. * test(router): cover the timeout stamping branch and async active-cooldown append _acompletion's litellm.Timeout handler and async_get_active_cooldowns' happy path both lacked direct coverage despite their sibling branches (the generic Exception handler, the sync get_active_cooldowns) being tested. * test(router): remove duplicate cooldown-trigger and fallback-helper tests #34416 landed its own TestTriggerCooldownForFailedDeployment/ TestRunAsyncFallbackTriggersCooldown classes and test_ageneric_api_call_with_fallbacks_helper_stamps_failed_deployment_id covering the exact same scenarios as this branch's earlier flat-function tests, once its version of fallback_event_handlers.py was taken as-is during the last merge. Dropping the redundant copies. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_core_utils | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| old_proxy_tests/tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| scim_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_config.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_entrypoint.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_passthrough_endpoints.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.