mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
d98522b6f6
* feat(proxy): share database connections across workers with an in-container pgbouncer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): parse pgbouncer options iteratively to satisfy the recursion gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): refuse pgbouncer with token db auth and retry failed pooler restarts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): build pgbouncer 1.25.2 from a pinned source archive and verify pooler replacements The public Wolfi repository only carries pgbouncer 1.24.1-r3, which the image scan rejects (CVE-2026-6664, CVE-2026-6665, CVE-2026-6666, CVE-2025-12819). All three images now compile the checksummed 1.25.2 release in a builder stage. The supervisor now waits for a replacement pooler to listen before treating it as recovered, ends and retries one that never does, and takes the same lock for stop() and spawn so no replacement can be started after shutdown began. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): refuse to start pgbouncer on a loopback port another process already owns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): count pgbouncer ready only once its own unix socket answers, not any listener on the port Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): refuse pgbouncer older than 1.19, whose unix socket cannot vouch for the tcp port Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(helm,terraform): expose the in-container pgbouncer pool for the componentized gateway Add database.connectionPool to helm/litellm and gateway_connection_pool_* to terraform/litellm/aws so the componentized gateway can receive the LITELLM_PGBOUNCER_* env the classic image already honours. Both reject the pool under IAM or Entra token auth at render/plan time: the pooler holds one static database password for the life of the pod or task. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(gateway): launch the componentized gateway image through a pgbouncer-aware supervisor (#40592) The componentized gateway image started uvicorn directly, so the in-container PgBouncer never ran for it: every worker opened its own Prisma pool to the database. It also passed no keep-alive timeout, so behind a load balancer with a 60s idle timeout uvicorn's 5s default closed idle connections first and the balancer returned 502s on scale-out gateway.launch assembles DATABASE_URL, starts PgBouncer once per pod when LITELLM_PGBOUNCER_ENABLED is set, hands the workers the loopback URL and then runs uvicorn on gateway.main:app with KEEPALIVE_TIMEOUT as --timeout-keep-alive. The image builds PgBouncer 1.25.2 from a checksummed tarball, copies the compiled Rust extension into the /app source tree it imports from (it was only in site-packages, which PYTHONPATH=/app shadows) and asserts the native bridge loads. The app user is added to stats_users so operators can read the PgBouncer console with the application credentials The supervisor returns the pooled URL instead of writing into the mapping it was handed, a database user whose name PgBouncer would split into several stats_users entries is refused before the config is written, and the launcher tests drive main() with an injected serve callable Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(terraform): describe the gateway.launch pooler entrypoint in the aws module README Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): run pgbouncer exit hooks only in the parent and copy the CA into the runtime dir Gunicorn workers inherit the parent's atexit table, so a recycled worker (max_requests) stopped the shared pooler and removed its runtime dir, then hung in the inherited Popen lock. The hooks now no-op unless os.getpid() is the process that started PgBouncer A verified TLS upstream named the operator's CA bundle directly, which is often a 0600 root-owned file that nobody (the user PgBouncer drops to) cannot read, so every server connection failed with "failed to load CA". The bundle is copied into the runtime dir next to the ini and chowned with it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(terraform): run the gateway through gateway.launch and add the gcp connection-pool variables Cloud Run and ECS overrode the image command with uvicorn gateway.main:app, which skips the supervisor that starts the in-container PgBouncer, so LITELLM_PGBOUNCER_ENABLED was inert on both stacks. Both now exec python -m gateway.launch (under ddtrace-run when USE_DDTRACE is set), and the gcp module gains gateway_connection_pool_enabled / gateway_pool_max_db_connections / gateway_pool_max_client_conn wired to the gateway service only The test_launch password_env fixture now restores DATABASE_URL even when it was unset: monkeypatch.delenv records nothing for an absent var, so main() left postgresql://...@db.internal in the xdist worker's environ and the key-rotation e2e test in the same proxy-infra shard stopped skipping and tried to reach db.internal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(pgbouncer): keep channel_binding and gssencmode off the loopback URL Prisma would demand TLS channel binding from a pooler that only speaks plain TCP on 127.0.0.1. Also pass the request the marketplace test started needing after #40518 landed on top of #40496 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| a2a | ||
| agent_endpoints | ||
| analytics_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| batches_endpoints | ||
| client | ||
| common_utils | ||
| config_resolvers | ||
| container_endpoints | ||
| credential_endpoints | ||
| db | ||
| discovery_endpoints | ||
| enterprise_billing | ||
| experimental/mcp_server | ||
| fine_tuning_endpoints | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| list_api | ||
| logging_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| memory | ||
| middleware | ||
| ocr_endpoints | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| proxy_server | ||
| public_endpoints | ||
| rag_endpoints | ||
| realtime_endpoints | ||
| rerank_endpoints | ||
| response_api_endpoints | ||
| shutdown | ||
| spend_tracking | ||
| test_configs | ||
| types_utils | ||
| ui_crud_endpoints | ||
| utils | ||
| vector_store_endpoints | ||
| vector_store_files_endpoints | ||
| video_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test__types.py | ||
| test_aiohttp_cleanup_closed.py | ||
| test_aiohttp_session_recovery.py | ||
| test_api_key_masking_in_errors.py | ||
| test_audio_speech_prometheus_hooks.py | ||
| test_batch_expiry.py | ||
| test_batch_metadata_none_fix.py | ||
| test_batch_retrieve_bedrock.py | ||
| test_batch_x_litellm_model_encoding.py | ||
| test_blocked_response_usage.py | ||
| test_budget_reservation.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_claude_code_marketplace.py | ||
| test_common_request_processing.py | ||
| test_component_allowlists.py | ||
| test_conftest.py | ||
| test_cors_config.py | ||
| test_custom_proxy.py | ||
| test_dynamic_mcp_route.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_filter_models_by_team_access_group.py | ||
| test_health_check_functions.py | ||
| test_health_check_max_tokens.py | ||
| test_init_litellm_callbacks.py | ||
| test_langfuse_passthrough_security.py | ||
| test_lazy_openapi_snapshot.py | ||
| test_litellm_pre_call_utils.py | ||
| test_max_budget_env_var.py | ||
| test_mcp_asgi_response.py | ||
| test_model_based_routing_files_batches.py | ||
| test_model_deprecations_endpoint.py | ||
| test_model_dump_with_preserved_fields.py | ||
| test_model_id_header_propagation.py | ||
| test_model_info_default_limits.py | ||
| test_model_level_guardrails.py | ||
| test_model_list_healthy_only.py | ||
| test_modify_response_streaming_passthrough.py | ||
| test_openai_ws_passthrough_routes.py | ||
| test_openapi_schema_validation.py | ||
| test_plugin_routes.py | ||
| test_pointfive_dashboard_config.py | ||
| test_pointfive_ui_callback.py | ||
| test_pricing_field_strip.py | ||
| test_prisma_engine_watchdog.py | ||
| test_prisma_migration.py | ||
| test_prometheus_cleanup.py | ||
| test_prometheus_metrics_server.py | ||
| test_provider_url_destination_guard.py | ||
| test_proxy_cli.py | ||
| test_proxy_logging_hook_detection.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_pyroscope.py | ||
| test_read_model_list.py | ||
| test_redis_auth_cache_flag.py | ||
| test_response_model_sanitization.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_sensitive_route_auth.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||
| test_team_org_move.py | ||
| test_tools_allowlist_enforcement.py | ||
| test_update_llm_router_resilience.py | ||