mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
725c0c158f
* fix(proxy): readiness check returns 200 when database is unreachable _db_health_readiness_check() catches health_check() exceptions but never updates db_health_cache to "disconnected" and never re-raises. The caller health_readiness() always returns 200 with "db": "connected" hardcoded, regardless of actual DB state. In Kubernetes, this means pods with dead database connections stay in the Service endpoints and continue receiving traffic they cannot serve. Changes: - Set db_health_cache to "disconnected" and re-raise the exception on health_check failure so health_readiness() returns 503 - Use actual db_health_status["status"] in the response instead of hardcoding "db": "connected" - Reduce cache TTL from 2 minutes to 15 seconds. The 2-minute window is too wide for readiness probes (typically 10-15s intervals) and means a pod can report healthy for up to 2 minutes after the DB dies - Only serve cached results when status is "connected". The previous condition (status != "unknown") would also cache "disconnected" for 2 minutes, delaying recovery detection after a DB comes back * fix(proxy): add DB connection self-healing to readiness check When the Prisma query engine's internal TCP connection pool holds dead connections (caused by network blips, Cloud SQL proxy restarts, or node-level issues), health_check() fails with httpx.ConnectError. The engine never recovers on its own because nothing triggers a disconnect/connect cycle to restart the subprocess with fresh connections. This leaves pods permanently failing readiness checks until they are manually restarted, even after the underlying DB becomes reachable again. Add a reconnect attempt to _db_health_readiness_check() when health_check() fails: 1. disconnect() - kills the query engine subprocess and closes all connections (has built-in backoff retry: 3 tries, 10s max) 2. connect() - starts a new engine with fresh TCP connections (has built-in backoff retry: 3 tries, 10s max) 3. health_check() - verifies the new connection works (has built-in backoff retry: 3 tries, 10s max) If reconnect succeeds, the pod immediately returns to service (200). If it fails, the original exception is re-raised (503). Reconnect attempts are rate-limited by probe frequency (~10-15s), so a permanently unreachable DB gets one attempt per cycle with no retry loops. This uses the same disconnect/connect mechanism that PrismaWrapper.recreate_prisma_client() uses for IAM token refresh, and aligns with the community-documented pattern for Prisma connection recovery in long-running processes (prisma/prisma#24718, #27024). * Add poetry lock and modify test_health_endpoints * Address allow_requests_on_db_unavailable regression * Address comments * resolve greptile issue * Restore accidentally deleted UI HTML files These were removed in an earlier commit but still exist on main. Restoring to keep the PR diff clean. * Guard reconnect with is_database_transport_error Only attempt disconnect/connect/health_check cycle for transport-level failures (unreachable DB, dropped connection). Data-layer errors like UniqueViolationError indicate the DB is reachable, so reconnecting would be pointless churn. * Address greptile's comments * Fix module alias after rebase and add adversarial test coverage - Unify module alias to _health_endpoints_module after rebase conflict - Add test for non-transport error with flag on (exercises is_database_transport_error guard) - Add test for disconnect() failure during reconnect cycle - Split non-transport error test into flag-off (re-raises) and flag-on (skips reconnect) variants * Remove stale UI HTML files reintroduced during rebase |
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| agent_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| client | ||
| common_utils | ||
| db | ||
| discovery_endpoints | ||
| experimental/mcp_server | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| middleware | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| public_endpoints | ||
| rag_endpoints | ||
| response_api_endpoints | ||
| spend_tracking | ||
| test_configs | ||
| ui_crud_endpoints | ||
| vector_store_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test_aiohttp_cleanup_closed.py | ||
| test_api_key_masking_in_errors.py | ||
| test_audio_speech_prometheus_hooks.py | ||
| test_batch_expiry.py | ||
| test_batch_metadata_none_fix.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_common_request_processing.py | ||
| test_custom_proxy.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_health_check_functions.py | ||
| test_health_check_max_tokens.py | ||
| test_litellm_pre_call_utils.py | ||
| test_model_dump_with_preserved_fields.py | ||
| test_model_id_header_propagation.py | ||
| test_prometheus_cleanup.py | ||
| test_proxy_cli.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_pyroscope.py | ||
| test_response_model_sanitization.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||
| test_tools_allowlist_enforcement.py | ||
| test_update_llm_router_resilience.py | ||