Commit graph

5 commits

Author SHA1 Message Date
Yassin Kortam
52dc15adfe
fix(proxy): isolate poison spend-log rows so one bad record can't drop the whole batch (#31705)
update_spend_logs flushes the queue with a single create_many per batch, so one
row carrying bytes Postgres refuses (a residual NUL byte is the canonical case)
fails the entire insert and drops every good spend log alongside it. PR #29515
strips NUL bytes from the JSON columns, but the scalar string columns (end_user,
model, session_id, ...) still flow through unsanitized, so a poisoned row can
still reach the write and take a batch of up to 1000 good rows down with it.

On a genuine data-layer rejection the batch is now bisected so the good rows
still persist and only the offending row is dropped and logged with its
request_id. The classification lives in PrismaDBExceptionHandler.is_prisma_data_error
(matched by exact type so systemic subclasses like a missing table are not
mistaken for a single poison row), which keeps prisma an in-function import and
litellm.proxy.utils importable without the proxy extra. Transport failures,
including the "can't reach database server" outage that prisma mislabels as a
DataError, are re-raised unchanged so the existing connection-retry path still
runs and a transient outage never turns into silent per-row data loss.

The bisection carries a per-batch isolation budget so an authenticated caller
flooding poisoned rows cannot amplify one failed bulk insert into ~2N failed
inserts and N log lines; once the budget is spent the still-failing remainder
is dropped wholesale under a single log line.

Resolves LIT-4103
2026-06-30 12:21:14 -07:00
Yassin Kortam
da9d64b4de
fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures (#29986) 2026-06-10 23:48:11 +00:00
Ishaan Jaff
a939fa37ae
fix(proxy): restore broad is_database_connection_error; add is_database_transport_error for reconnect (#21796)
Any PrismaError should be treated as a DB connection error for the
allow_requests_on_db_unavailable feature and 503 responses. The narrow
keyword-based check is now in is_database_transport_error, which is
what the reconnect logic in auth_checks.py should use.

Fixes test_delete_access_group_503_on_db_connection_error and
test_handle_authentication_error_db_unavailable failures caused by
PR #21706 narrowing is_database_connection_error.
2026-02-21 12:04:00 -08:00
Ishaan Jaff
e0129710c8
fix(proxy): self-heal Prisma connection for auth and runtime (#21706)
* fix(proxy): add prisma reconnect primitive and db watchdog

* fix(proxy): start and stop prisma watchdog in lifecycle

* fix(auth): retry key lookup once after prisma reconnect

* test(proxy): add prisma self-heal watchdog coverage

* test(auth): cover reconnect-once behavior for key lookup

* refactor(auth): extract db reconnect helper and remove inline import

* fix(proxy): apply reconnect cooldown after attempt and add auth timeout path

* fix(auth): bound reconnect latency on key lookup path

* test(auth): assert reconnect timeout argument in key lookup

* test(proxy): verify reconnect cooldown timestamp set after attempt

* fix(proxy): harden prisma reconnect cycle semantics

* test(proxy): cover watchdog reconnect + timeout budget

* fix(proxy): bound watchdog probe and reconnect paths

* test(proxy): cover watchdog timeout and probe behavior

* fix(proxy): narrow prisma db connection error classification

* fix(proxy): add auth reconnect lock timeout budget

* fix(auth): pass lock timeout for db reconnect retries

* test(proxy): cover narrow prisma connection error detection

* test(proxy): add reconnect lock-timeout behavior coverage

* test(auth): assert reconnect lock timeout argument

* fix(proxy): avoid lock leak race in reconnect lock timeout path

* test(proxy): cover reconnect lock-timeout race cleanup
2026-02-20 18:11:36 -08:00
Krish Dholakia
ef42461c1e
Litellm fix GitHub action testing (#11163)
* test: add __init__.py files

* refactor: rename test folder to avoid naming conflict

* test: update workflows

* test: update tests

* test: update imports

* test: update tests

* test: remove unused import

* ci(test-litellm.yml): add pytest retry to github workflow

* test: fix test
2025-05-26 14:41:42 -07:00
Renamed from tests/litellm/proxy/db/test_exception_handler.py (Browse further)