litellm/tests/test_litellm/proxy/health_endpoints
ishaan-berri 9aee0da7d8
fix: /health/readiness 503 loop when DB is unreachable (#26134)
* fix: /health/readiness returns 503 when DB is unreachable due to handle_db_exception re-raising

handle_db_exception() re-raises the Prisma exception inside _db_health_readiness_check's
except block, which propagates out to health_readiness() and gets wrapped in a 503.
The health endpoint never reached the reconnect path and the service never recovered.

Fix:
- Remove handle_db_exception() call from _db_health_readiness_check — that helper is
  for API request handlers (allow_requests_on_db_unavailable flag), not health checks
- Replace raw disconnect()+connect() with attempt_db_reconnect(), which uses the proper
  lock, cooldown, escalation, and heavy-reconnect (recreate_prisma_client) machinery

* test: update health readiness tests for handle_db_exception removal

- Remove tests that expected handle_db_exception to re-raise (old buggy behaviour)
- Remove tests asserting disconnect()/connect() calls (replaced by attempt_db_reconnect)
- Add regression tests covering the 503 loop fix:
  - transport errors never raise (ClientNotConnectedError, httpx.ConnectError, etc.)
  - reconnect success path returns 'connected'
  - reconnect failure path returns 'disconnected' without raising
  - non-transport errors return 'disconnected', skip reconnect

---------

Co-authored-by: yuneng-jiang <yuneng@berri.ai>
2026-04-20 15:29:43 -07:00
..
test_health_endpoints.py fix: /health/readiness 503 loop when DB is unreachable (#26134) 2026-04-20 15:29:43 -07:00