mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-06 07:48:06 +00:00
b080c59e78
* fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows Postgres refuses every new connection with SQLSTATE 57P03 while it starts up, shuts down or replays WAL, and prisma surfaces that as a base DataError with no code. The spend-log writer took it for a poison row, bisected the batch down to single rows and dropped each one. Spend logs now retry and requeue on 57P03 and on server-terminated sessions (57P01, 57P02, 57P05), tool and model usage rollups requeue when the write provably never reached the server, the tool registry requeues its rows, and auth answers 503 instead of 401 while the auth database restarts. Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com> * fix(proxy): requeue tool discovery rows only when the write never reached Postgres, keep the auth fallback to 57P03 A terminated session (57P01, 57P02, 57P05) may have committed the tool upsert, and the upsert adds call_count on conflict, so replaying it counts the calls twice. The non-Redis tool discovery flush now uses the same never-sent rule as the rollups, which also stops a permanently rejected row (54000) from coming back every tick. One terminated session does not prove the database is down, so it no longer opens the allow_requests_on_db_unavailable fallback, it still answers 503 * fix(proxy): a full server (53300, P2024) also never took the write, so counters and tool rows are sent again Postgres refuses the connection before any statement runs when it is out of slots, and a prisma pool timeout never reaches the server either, so those failures are as safe to replay as 57P03 and P1001 --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com> |
||
|---|---|---|
| .. | ||
| db_transaction_queue | ||
| mcp_server | ||
| __init__.py | ||
| conftest.py | ||
| fake_prisma_engine.py | ||
| test_autorouter_session_rollup.py | ||
| test_budget_window_spend_writer.py | ||
| test_check_migration.py | ||
| test_create_views.py | ||
| test_daily_spend_bulk_upsert.py | ||
| test_db_lookup_gate.py | ||
| test_db_span.py | ||
| test_db_spend_update_writer.py | ||
| test_db_url_settings.py | ||
| test_exception_handler.py | ||
| test_exception_handler_reconnect_retry.py | ||
| test_gateway_request_tracking.py | ||
| test_health_check_latest.py | ||
| test_log_db_metrics.py | ||
| test_log_db_metrics_service_spans.py | ||
| test_master_key_migration.py | ||
| test_model_access_group_spend.py | ||
| test_model_insights_tasks.py | ||
| test_model_usage_rollup.py | ||
| test_object_permission_repository.py | ||
| test_pgbouncer.py | ||
| test_prisma_client.py | ||
| test_prisma_planned_engine_restart.py | ||
| test_prisma_query_span.py | ||
| test_prisma_self_heal.py | ||
| test_proxy_worker_heartbeat.py | ||
| test_query_engine_reaper.py | ||
| test_rds_iam_token_expiry.py | ||
| test_replica_identity.py | ||
| test_request_error_tracking.py | ||
| test_routing_prisma_wrapper.py | ||
| test_shadow_eval_funnel.py | ||
| test_spend_counter_reseed.py | ||
| test_spend_log_batching.py | ||
| test_spend_log_tool_index.py | ||
| test_token_auth.py | ||
| test_tool_registry_writer.py | ||
| test_tool_table_bulk_upsert.py | ||
| test_update_daily_tag_spend.py | ||