Commit graph

7 commits

Author SHA1 Message Date
Yassin Kortam
c3a8962c00
fix(proxy): only treat a recoverable database outage as grounds to serve without one (#35864)
`is_database_connection_error` answered True for any `PrismaError` it did not
recognize, on the reasoning that an unclassified failure might be an outage and
the safer default was to keep serving. That default is inverted for faults that
never resolve. A query engine that is missing or version-skewed, a malformed
generated query, or a misused transaction all satisfied the predicate, so with
`allow_requests_on_db_unavailable` enabled the proxy would absorb one, boot
clean, and keep issuing fallback identities for as long as the process ran.

The predicate is now an allowlist: the httpx transport errors, prisma's
`EngineConnectionError`, and a `no_db_connection` ProxyException. That is what a
real outage produces, since the query engine is a local HTTP server and an
unreachable database surfaces as a transport failure against it, so the
high-availability path is unchanged. Anything unrecognized is now treated as
permanent and surfaces instead of being absorbed.

Deciding whether to serve without a database and deciding what to tell the
caller are different questions, so they no longer share a predicate.
`is_database_infrastructure_error` keeps the previous broad behavior and now
backs the reporting and recovery paths: service-unavailable classification, the
access-group endpoint's status mapping, and the health watchdog's reconnect
trigger. Their behavior is unchanged. Without that split, a permanently faulted
engine would have started reporting as an authentication failure, sending an
operator after a credential problem that does not exist.
2026-08-05 14:15:13 -07:00
Tin Chi Lo
61c7e706dd fix(mcp): classify get_user_object's wrapped DB outage across the exception chain
get_user_object catches every DB failure in a broad except and re-raises a bare ValueError (litellm/proxy/auth/auth_checks.py), so a real outage and a missing user look identical and the original error survives only as __context__. The dcr_bridge admission path keyed its 503-vs-401 decision on the exception type, so a transient outage during a user-subject reload surfaced as a 401 rather than a retryable 503, and the regression test injected a raw ConnectionError, a shape get_user_object never produces, so it passed on a fiction

Add PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain, which walks __cause__/__context__ (bounded and cycle-safe) the PEP 3134 way, and route _raise_503_if_db_unavailable through it. Move the user's object_permission resolution inside the single classified try so an outage there is a 503 too, never an opaque 500. Pin get_user_object's wrapping with a contract test that drives the real function, and drive the reload tests with that same faithful shape so a chain-blind regression fails them
2026-07-13 13:25:41 -07:00
Yassin Kortam
52dc15adfe
fix(proxy): isolate poison spend-log rows so one bad record can't drop the whole batch (#31705)
update_spend_logs flushes the queue with a single create_many per batch, so one
row carrying bytes Postgres refuses (a residual NUL byte is the canonical case)
fails the entire insert and drops every good spend log alongside it. PR #29515
strips NUL bytes from the JSON columns, but the scalar string columns (end_user,
model, session_id, ...) still flow through unsanitized, so a poisoned row can
still reach the write and take a batch of up to 1000 good rows down with it.

On a genuine data-layer rejection the batch is now bisected so the good rows
still persist and only the offending row is dropped and logged with its
request_id. The classification lives in PrismaDBExceptionHandler.is_prisma_data_error
(matched by exact type so systemic subclasses like a missing table are not
mistaken for a single poison row), which keeps prisma an in-function import and
litellm.proxy.utils importable without the proxy extra. Transport failures,
including the "can't reach database server" outage that prisma mislabels as a
DataError, are re-raised unchanged so the existing connection-retry path still
runs and a transient outage never turns into silent per-row data loss.

The bisection carries a per-batch isolation budget so an authenticated caller
flooding poisoned rows cannot amplify one failed bulk insert into ~2N failed
inserts and N log lines; once the budget is spent the still-failing remainder
is dropped wholesale under a single log line.

Resolves LIT-4103
2026-06-30 12:21:14 -07:00
Yassin Kortam
da9d64b4de
fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures (#29986) 2026-06-10 23:48:11 +00:00
Ishaan Jaff
a939fa37ae
fix(proxy): restore broad is_database_connection_error; add is_database_transport_error for reconnect (#21796)
Any PrismaError should be treated as a DB connection error for the
allow_requests_on_db_unavailable feature and 503 responses. The narrow
keyword-based check is now in is_database_transport_error, which is
what the reconnect logic in auth_checks.py should use.

Fixes test_delete_access_group_503_on_db_connection_error and
test_handle_authentication_error_db_unavailable failures caused by
PR #21706 narrowing is_database_connection_error.
2026-02-21 12:04:00 -08:00
Ishaan Jaff
e0129710c8
fix(proxy): self-heal Prisma connection for auth and runtime (#21706)
* fix(proxy): add prisma reconnect primitive and db watchdog

* fix(proxy): start and stop prisma watchdog in lifecycle

* fix(auth): retry key lookup once after prisma reconnect

* test(proxy): add prisma self-heal watchdog coverage

* test(auth): cover reconnect-once behavior for key lookup

* refactor(auth): extract db reconnect helper and remove inline import

* fix(proxy): apply reconnect cooldown after attempt and add auth timeout path

* fix(auth): bound reconnect latency on key lookup path

* test(auth): assert reconnect timeout argument in key lookup

* test(proxy): verify reconnect cooldown timestamp set after attempt

* fix(proxy): harden prisma reconnect cycle semantics

* test(proxy): cover watchdog reconnect + timeout budget

* fix(proxy): bound watchdog probe and reconnect paths

* test(proxy): cover watchdog timeout and probe behavior

* fix(proxy): narrow prisma db connection error classification

* fix(proxy): add auth reconnect lock timeout budget

* fix(auth): pass lock timeout for db reconnect retries

* test(proxy): cover narrow prisma connection error detection

* test(proxy): add reconnect lock-timeout behavior coverage

* test(auth): assert reconnect lock timeout argument

* fix(proxy): avoid lock leak race in reconnect lock timeout path

* test(proxy): cover reconnect lock-timeout race cleanup
2026-02-20 18:11:36 -08:00
Krish Dholakia
ef42461c1e
Litellm fix GitHub action testing (#11163)
* test: add __init__.py files

* refactor: rename test folder to avoid naming conflict

* test: update workflows

* test: update tests

* test: update imports

* test: update tests

* test: remove unused import

* ci(test-litellm.yml): add pytest retry to github workflow

* test: fix test
2025-05-26 14:41:42 -07:00
Renamed from tests/litellm/proxy/db/test_exception_handler.py (Browse further)