devin-ai-integration[bot] e135a199ad fix(proxy): fail parked DB lookups at a deadline and flip readiness while they stall (#42654) ... * fix(proxy): fail parked DB lookups at a deadline and flip readiness while they stall Under a load burst with a slow authentication database every request parked inside the pod with no deadline while /health/readiness kept answering 200 (its own ping gets a fresh connection), so the load balancer kept sending traffic until the pod hit its memory limit, and the parked requests completed against the provider minutes after every client had hung up Every pre-request read (key, team, user, end user, budget, membership, organization, object permission, jwt mapping, project, proxy budget, spend counter reseed) now runs under one deadline, PROXY_DB_LOOKUP_DEADLINE_SECONDS (default 10 s). A lookup that hits it fails the request with the existing 503 "authentication database is temporarily unreachable" answer, honours allow_requests_on_db_unavailable, and never triggers the transport reconnect (the transport is fine, the query is slow), which is what turned the repro's stall into "too many clients". Writes stay unbounded A deadline hit marks the pod stalled for PROXY_DB_LOOKUP_STALL_WINDOW_SECONDS (default 30 s, 0 disables), during which /health/readiness answers 503 with "db": "stalled" behind the same fail-open gate, so the pod leaves rotation before it fills its memory. The existing litellm_in_flight_requests gauge already exposes the parked set on /metrics The deadline is enforced on the wall clock: bounded_db_lookup waits on the lookup task with asyncio.wait and raises DBLookupDeadlineExceeded when the deadline passes even if the lookup absorbs its cancellation, where asyncio.wait_for on 3.12+ would sit on the cancelled task for as long as it takes The failure spend-log row no longer re-runs the key and team lookups when the failure itself is a database connection or deadline error, so a request that hit the deadline is answered after one deadline instead of two * fix(proxy): bound the spend counter gate wait and narrow the stalled lookup shortcut Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the global spend lookup on the prisma client handle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>		2026-09-24 10:09:49 -05:00
..
_experimental/mcp_server	fix(mcp): reject duplicate MCP server names and aliases (#42791)	2026-09-23 17:50:27 -07:00
a2a	fix(a2a): reject malformed protocolVersion suffixes while keeping semver prereleases	2026-07-21 13:24:58 -07:00
agent_endpoints	fix(mcp): keep tool attribution on guardrail-blocked REST calls (#42790)	2026-09-23 15:40:56 -07:00
analytics_endpoints	fix(proxy): attribute gate-rejected requests to their endpoint in cache analytics (#40824)	2026-09-12 16:03:38 -07:00
anthropic_endpoints	feat(proxy): opt-in litellm_call_id in JSON error bodies (#42391)	2026-09-21 19:17:18 -07:00
auth	fix(proxy): fail parked DB lookups at a deadline and flip readiness while they stall (#42654)	2026-09-24 10:09:49 -05:00
batches_endpoints	chore: merge main into fix/batch-retrieve-model-group	2026-09-19 16:25:39 -07:00
client	test: deflake fuzzy picker, breached-password HIBP, and MCP stdio timeout tests (rolling deflake 2026-09-22) (#42125)	2026-09-23 08:44:59 -07:00
common_utils	fix(proxy): list key and team model aliases in GET /v1/models (#42908)	2026-09-24 06:41:13 -07:00
config_resolvers	test: drop two inert type: ignore comments	2026-09-19 14:22:19 -07:00
container_endpoints	fix(containers): page upstream until a non-admin container list fills its limit	2026-09-02 21:26:34 -07:00
credential_endpoints	fix(credentials): answer 409 on a name collision, let PATCH resolve values from model_id	2026-09-15 10:41:43 -07:00
db	fix(proxy): fail parked DB lookups at a deadline and flip readiness while they stall (#42654)	2026-09-24 10:09:49 -05:00
discovery_endpoints	chore(proxy): drop explanatory docstrings from credentials hint helper and tests	2026-09-14 19:29:30 +00:00
enterprise_billing	feat(proxy): push-based OTLP billable-request metering for enterprise deployments (#31592)	2026-07-15 12:12:52 -07:00
experimental/mcp_server	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
fine_tuning_endpoints	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
google_endpoints	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
guardrails	fix(presidio): stream non-Anthropic raw SSE through the post_call hook unbuffered (#42777)	2026-09-23 22:08:23 -07:00
health_endpoints	fix(proxy): fail parked DB lookups at a deadline and flip readiness while they stall (#42654)	2026-09-24 10:09:49 -05:00
hooks	fix(proxy): fail parked DB lookups at a deadline and flip readiness while they stall (#42654)	2026-09-24 10:09:49 -05:00
image_endpoints	Merge remote-tracking branch 'origin/main' into litellm_fix_image_edits_bracketed_alias	2026-09-17 14:49:54 -07:00
list_api	feat: page the public model hub table off /public/v1/model_hub, keeping every filter (#39691)	2026-09-03 22:36:16 -07:00
logging_endpoints	fix(usage): recover aliases for v1.99 double-hashed spend keys	2026-09-03 14:21:25 +00:00
management_endpoints	fix(proxy): gate disable_global_guardrails on keys and teams to proxy admins (#42699)	2026-09-23 18:03:02 -07:00
management_helpers	feat(ui): simplify auto-router setup and clarify feature limits (#42625)	2026-09-22 18:03:32 -07:00
memory	feat(proxy): add a search param to key, memory, audit, and spend log listings	2026-09-03 15:20:13 -07:00
middleware	fix(proxy): release unclaimed budget reservations at request end (#42304)	2026-09-21 19:51:12 -07:00
ocr_endpoints	fix(ocr): validate body req_format in the proxy endpoint and run its tests in CI	2026-08-17 18:29:35 +00:00
openai_files_endpoint	fix(files): keep an explicit target_storage on its old path and refuse litellm_db as a caller choice	2026-09-19 13:20:02 -07:00
pass_through_endpoints	fix(passthrough): log upstream 4xx/5xx error bodies and carry them into the failure hook (#42695)	2026-09-23 23:42:24 -07:00
policy_engine	fix(policy_engine): keep inherited parent guardrails when a child policy condition misses (#42548)	2026-09-22 23:27:38 -07:00
prompts	fix(prompts): accept a string prompt_version and carry the viewed environment into code snippets	2026-08-29 12:55:34 -07:00
proxy_server	feat(secrets): route secret resolution through native Rust backends (#42619)	2026-09-23 08:24:57 -07:00
public_endpoints	feat(ui): simplify auto-router setup and clarify feature limits (#42625)	2026-09-22 18:03:32 -07:00
rag_endpoints	test(rag): drop the docstrings from the registered-store ingest tests	2026-09-19 04:10:49 -07:00
realtime_endpoints	refactor(proxy): replace configurable model access denied message with a fixed clean client message	2026-09-16 01:44:36 +00:00
rerank_endpoints	fix(proxy): carry litellm_call_id through endpoint specific error logs and failure responses	2026-09-16 02:29:26 +00:00
response_api_endpoints	fix(proxy): document request body and response schemas for the Responses API in OpenAPI (#42802)	2026-09-24 01:28:34 +00:00
shutdown	refactor(proxy): inject scheduled job shutdown timeouts	2026-09-21 19:37:28 +00:00
spend_tracking	fix(spend): return 400 from /spend/calculate for a model with no pricing row (#42497)	2026-09-22 15:44:00 -07:00
test_configs
types_utils	feat(proxy): unified custom_key_policy hook for key generate, update and regenerate	2026-09-12 16:09:24 -07:00
ui_crud_endpoints	fix: enforce disable_custom_api_keys from general_settings (#42437)	2026-09-22 13:28:31 -07:00
utils	fix(proxy): keep deployment labels on cache-hit post_call guardrail rejections (#42780)	2026-09-23 22:48:04 -07:00
vector_store_endpoints	fix(vector_stores): keep config-defined vector stores listed and read-only (#42574)	2026-09-23 04:02:16 +00:00
vector_store_files_endpoints	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
video_endpoints	fix(videos): forward uploaded source file on /v1/videos/edits to the provider	2026-08-24 15:34:21 -07:00
__init__.py
conftest.py	test(proxy): isolate the agent read-through singleton between unknown-agent tests	2026-09-22 01:34:31 +00:00
test__types.py	feat(auth): breached password detection, self-service change-password and forced password reset	2026-09-21 18:48:35 +00:00
test_aiohttp_cleanup_closed.py
test_aiohttp_session_recovery.py
test_api_key_masking_in_errors.py
test_audio_speech_prometheus_hooks.py	fix(proxy): match /v1/audio/speech content-type to the returned audio format	2026-08-29 13:44:43 -07:00
test_batch_expiry.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_batch_metadata_none_fix.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_batch_retrieve_bedrock.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_batch_x_litellm_model_encoding.py	fix(proxy): apply model grants to unified file and batch ids on batch routes	2026-09-10 18:10:18 -04:00
test_blocked_response_usage.py	fix(guardrails): remove the module-global translation mapping that leaked between tests	2026-09-03 03:28:52 -07:00
test_budget_reservation.py	feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174)	2026-09-22 04:41:11 +00:00
test_bug_report_config.py	feat(proxy): admin-only /debug/report sharing the bug report environment (#42440)	2026-09-22 12:05:23 -07:00
test_caching_routes.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_chat_completion_metadata.py
test_claude_code_marketplace.py	test: run the 30 test files stranded in the second mirror (#37595)	2026-08-20 10:59:43 -07:00
test_collector.py	feat(proxy): offload spend tracking to a pod-local collector sidecar (#40545)	2026-09-10 17:14:13 -07:00
test_common_request_processing.py	fix(shadow-eval): replay approved pre-call guardrail snapshots (#42774)	2026-09-23 17:43:54 -07:00
test_component_allowlists.py	test(proxy): make two proxy-infra tests independent of sibling-test state (#42581)	2026-09-22 14:58:42 -07:00
test_conftest.py	test(proxy): stop monkeypatch.undo re-planting fixture-mocked prisma_client	2026-08-13 20:11:06 -07:00
test_cors_config.py
test_custom_proxy.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_dynamic_mcp_route.py	fix(proxy): serve aggregate MCP endpoint on bare /mcp instead of 307-redirecting (#34845)	2026-08-14 17:04:32 -07:00
test_empty_model_list.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_enforce_user_param.py	test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769)	2026-08-20 20:24:49 -07:00
test_fallback_management_endpoints.py
test_fastapi_offline_routes.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_filter_models_by_team_access_group.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_health_check_functions.py	fix(health): skip background health check DB writes when the latest-row read fails	2026-09-14 23:03:30 +00:00
test_health_check_max_tokens.py	feat(auto-router): integrate JEV context and usage accounting	2026-09-18 21:38:15 +00:00
test_init_litellm_callbacks.py	test: run the 30 test files stranded in the second mirror (#37595)	2026-08-20 10:59:43 -07:00
test_langfuse_passthrough_security.py
test_lazy_openapi_snapshot.py	refactor(proxy): type the snapshot fragments and wrap a long test line	2026-08-26 14:39:51 -07:00
test_litellm_pre_call_utils.py	fix(shadow-eval): replay approved pre-call guardrail snapshots (#42774)	2026-09-23 17:43:54 -07:00
test_max_budget_env_var.py
test_mcp_asgi_response.py	fix(mcp): surface upstream 401 for token-forwarding MCP servers (#27847)	2026-05-13 12:03:36 -07:00
test_model_based_routing_files_batches.py	test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510)	2026-06-16 10:20:59 -07:00
test_model_deprecations_endpoint.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_model_dump_with_preserved_fields.py
test_model_id_header_propagation.py	feat(proxy): add x-litellm-model-name response header with deployment model string (#33698)	2026-07-17 20:29:42 -07:00
test_model_info_default_limits.py	perf(proxy): serialize /model/info listing once with orjson	2026-09-14 19:47:51 +00:00
test_model_level_guardrails.py	fix(mcp): keep tool attribution on guardrail-blocked REST calls (#42790)	2026-09-23 15:40:56 -07:00
test_model_list_aliases.py	fix(proxy): list key and team model aliases in GET /v1/models (#42908)	2026-09-24 06:41:13 -07:00
test_model_list_discoverable.py	feat(proxy): honor model_info.discoverable on the model listing endpoints (#42825)	2026-09-23 17:51:16 -07:00
test_model_list_healthy_only.py	perf(proxy): serialize /model/info listing once with orjson	2026-09-14 19:47:51 +00:00
test_modify_response_streaming_passthrough.py	fix(proxy): use e.request_data for logging_obj in ModifyResponseException streaming passthrough (#30800)	2026-06-18 23:29:08 -07:00
test_native_compaction.py	fix(gateway): expose /api/event_logging/batch on the gateway allowlist (#42572)	2026-09-22 14:09:40 -07:00
test_openai_ws_passthrough_routes.py	fix(proxy): keep the token's team model list in the websocket passthrough gate without a database	2026-09-04 19:05:35 -07:00
test_openapi_schema_validation.py	fix(proxy): preserve HTTP operations when injecting WebSocket stubs into OpenAPI schema	2026-05-06 00:28:42 +02:00
test_plugin_routes.py	refactor(proxy): resolve config and DB settings precedence in one SettingsStore	2026-09-17 23:36:27 -07:00
test_pointfive_dashboard_config.py	feat(pointfive): add pointfive to the dashboard logging integrations	2026-09-10 14:02:35 +03:00
test_pointfive_ui_callback.py	feat(pointfive): list pointfive in the proxy callback registry	2026-09-10 14:02:35 +03:00
test_pricing_field_strip.py	fix(proxy): fold litellm_metadata into metadata on chat routes so tag routing sees merged tags	2026-08-29 00:42:43 -07:00
test_prisma_engine_watchdog.py	fix(proxy): recreate the Prisma client when the writer session turns read-only (#40610)	2026-09-10 13:53:42 -07:00
test_prisma_migration.py	fix(proxy): run migrations through python -m prisma when the prisma console script is not on PATH	2026-09-09 18:17:12 -07:00
test_prometheus_cleanup.py	feat(deploy): metrics sidecar and separate metrics port in Helm and Terraform (#40163)	2026-09-08 13:31:07 -07:00
test_prometheus_metrics_server.py	feat(deploy): metrics sidecar and separate metrics port in Helm and Terraform (#40163)	2026-09-08 13:31:07 -07:00
test_provider_url_destination_guard.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_proxy_cli.py	fix(proxy_cli): import proxy_server once on script-style boot (#42584)	2026-09-22 14:29:50 -07:00
test_proxy_logging_hook_detection.py	fix(guardrails): store the masked output in spend logs when Presidio masks the response (#42441)	2026-09-22 01:19:22 -07:00
test_proxy_server.py	feat(proxy): honor model_info.discoverable on the model listing endpoints (#42825)	2026-09-23 17:51:16 -07:00
test_proxy_types.py	fix(router): honor team and key provider weights	2026-09-14 23:31:52 -07:00
test_proxy_utils.py	feat(errors): prefilled GitHub issue link on unmapped internal errors (#42065)	2026-09-21 22:30:41 -07:00
test_pyroscope.py	Implement normalize_nonempty_secret_str function to trim whitespace from secrets and treat empty values as unset. Update proxy_server to use this function for Grafana credentials. Enhance tests to validate the new normalization behavior.	2026-05-04 18:17:31 +00:00
test_read_model_list.py	feat: add minimal rust router + axum ai-gateway calling router.realtime (2/2) (#31135)	2026-06-23 19:16:34 -07:00
test_redis_auth_cache_flag.py	fix(proxy): share per-model budget counters across replicas through the spend counter cache (#39375)	2026-09-02 12:40:59 -07:00
test_response_model_sanitization.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_route_a2a_models.py	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)	2026-08-22 09:25:58 -07:00
test_route_llm_request.py	test(proxy): isolate the agent read-through singleton between unknown-agent tests	2026-09-22 01:34:31 +00:00
test_route_priority.py	perf(proxy): register liveness and core inference routes first (#40687)	2026-09-11 17:10:38 +00:00
test_sensitive_route_auth.py
test_shared_health_check.py	fix(proxy): derive auto-router health from its underlying models (#38174)	2026-08-26 16:41:54 -07:00
test_spend_log_cleanup.py	Merge pull request #41213 from BerriAI/litellm_spend_log_cleanup_cancel_outcome	2026-09-21 17:03:25 -07:00
test_swagger_chat_completions.py
test_team_member_update.py	fix(team): schedule membership audit writes after commit and lock the roster on role updates	2026-09-19 22:33:22 +00:00
test_team_org_move.py	test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769)	2026-08-20 20:24:49 -07:00
test_tools_allowlist_enforcement.py	fix(anthropic): close hybrid tool-name allowlist gap and keep native tools through guardrails	2026-08-26 20:37:11 -07:00
test_update_llm_router_resilience.py	fix(proxy): keep the no-model_list guard to absent reads so model_list: [] still evicts	2026-09-16 23:14:52 +00:00