A call admitted before the breaker opened could finish while the breaker was
HALF_OPEN and close it before the designated probe reported, so Redis traffic
resumed on a stale answer. The admission now records whether the call is the
probe and only the probe's success closes a half-open breaker.
The cron job lock manager also logged an error every cycle the open breaker
refused its Redis call, one line per job per pod. That refusal is now a debug
line like every other guarded call, while real Redis errors still log at error
* feat(proxy): share database connections across workers with an in-container pgbouncer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): parse pgbouncer options iteratively to satisfy the recursion gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse pgbouncer with token db auth and retry failed pooler restarts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): build pgbouncer 1.25.2 from a pinned source archive and verify pooler replacements
The public Wolfi repository only carries pgbouncer 1.24.1-r3, which the image
scan rejects (CVE-2026-6664, CVE-2026-6665, CVE-2026-6666, CVE-2025-12819).
All three images now compile the checksummed 1.25.2 release in a builder stage.
The supervisor now waits for a replacement pooler to listen before treating it
as recovered, ends and retries one that never does, and takes the same lock for
stop() and spawn so no replacement can be started after shutdown began.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse to start pgbouncer on a loopback port another process already owns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count pgbouncer ready only once its own unix socket answers, not any listener on the port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse pgbouncer older than 1.19, whose unix socket cannot vouch for the tcp port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(helm,terraform): expose the in-container pgbouncer pool for the componentized gateway
Add database.connectionPool to helm/litellm and gateway_connection_pool_* to
terraform/litellm/aws so the componentized gateway can receive the
LITELLM_PGBOUNCER_* env the classic image already honours. Both reject the
pool under IAM or Entra token auth at render/plan time: the pooler holds one
static database password for the life of the pod or task.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(gateway): launch the componentized gateway image through a pgbouncer-aware supervisor (#40592)
The componentized gateway image started uvicorn directly, so the in-container
PgBouncer never ran for it: every worker opened its own Prisma pool to the
database. It also passed no keep-alive timeout, so behind a load balancer with
a 60s idle timeout uvicorn's 5s default closed idle connections first and the
balancer returned 502s on scale-out
gateway.launch assembles DATABASE_URL, starts PgBouncer once per pod when
LITELLM_PGBOUNCER_ENABLED is set, hands the workers the loopback URL and then
runs uvicorn on gateway.main:app with KEEPALIVE_TIMEOUT as --timeout-keep-alive.
The image builds PgBouncer 1.25.2 from a checksummed tarball, copies the
compiled Rust extension into the /app source tree it imports from (it was only
in site-packages, which PYTHONPATH=/app shadows) and asserts the native bridge
loads. The app user is added to stats_users so operators can read the PgBouncer
console with the application credentials
The supervisor returns the pooled URL instead of writing into the mapping it
was handed, a database user whose name PgBouncer would split into several
stats_users entries is refused before the config is written, and the launcher
tests drive main() with an injected serve callable
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(terraform): describe the gateway.launch pooler entrypoint in the aws module README
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run pgbouncer exit hooks only in the parent and copy the CA into the runtime dir
Gunicorn workers inherit the parent's atexit table, so a recycled worker (max_requests) stopped the shared pooler and removed its runtime dir, then hung in the inherited Popen lock. The hooks now no-op unless os.getpid() is the process that started PgBouncer
A verified TLS upstream named the operator's CA bundle directly, which is often a 0600 root-owned file that nobody (the user PgBouncer drops to) cannot read, so every server connection failed with "failed to load CA". The bundle is copied into the runtime dir next to the ini and chowned with it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(terraform): run the gateway through gateway.launch and add the gcp connection-pool variables
Cloud Run and ECS overrode the image command with uvicorn gateway.main:app, which skips the supervisor that starts the in-container PgBouncer, so LITELLM_PGBOUNCER_ENABLED was inert on both stacks. Both now exec python -m gateway.launch (under ddtrace-run when USE_DDTRACE is set), and the gcp module gains gateway_connection_pool_enabled / gateway_pool_max_db_connections / gateway_pool_max_client_conn wired to the gateway service only
The test_launch password_env fixture now restores DATABASE_URL even when it was unset: monkeypatch.delenv records nothing for an absent var, so main() left postgresql://...@db.internal in the xdist worker's environ and the key-rotation e2e test in the same proxy-infra shard stopped skipping and tried to reach db.internal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pgbouncer): keep channel_binding and gssencmode off the loopback URL
Prisma would demand TLS channel binding from a pooler that only speaks
plain TCP on 127.0.0.1. Also pass the request the marketplace test
started needing after #40518 landed on top of #40496
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The writer health probe only ran SELECT 1, which a read-only Postgres
session answers fine, so a pooled connection left pointing at a demoted
primary kept failing every write with SQLSTATE 25006 until the pod was
restarted. Probe transaction_read_only instead, treat a 25006 on the
request path as a signal to recreate the client, and back off
exponentially while the database as a whole stays read-only so a replica
or an in-progress failover does not get its engine killed every cycle.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound concurrent key and spend-counter DB lookups to stop prisma pool thrash
A cache-miss burst fanned every get_key_object DB fallback and SpendCounterReseed
point lookup into the prisma query-engine httpx pool at once. httpcore's request
assignment is O(queued x connections) per event, so the event loop spent most of its
time in pool bookkeeping and the logging worker's 20s wait_for tripped. Callers now
wait on a small shared semaphore (PROXY_DB_LOOKUP_MAX_CONCURRENCY, default 25)
instead of queueing inside httpcore
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the in-flight counting prisma fake
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop module docstring from db_lookup_gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the in-flight counting table fake
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Each proxy worker flushed one Prisma upsert per active (date, category, route)
bucket every interval, so the Postgres primary saw workers x routes statements
per interval across the deployment. A flush now builds a single multi-row
INSERT ... ON CONFLICT DO UPDATE, and with use_redis_transaction_buffer on the
workers push snapshots to a Redis list that one lease-holding pod folds and
commits, so the whole deployment costs one statement per interval. The leader
keeps popping until the list is empty so a deployment wider than the dequeue
cap cannot build a backlog, and rows that fail both the commit and the Redis
re-queue fall back to the leader's own accumulator instead of being lost.
Resolves LIT-7371
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): default max_idle_connection_lifetime on componentized DB URLs
DatabaseURLSettings.apply_to_env() now appends max_idle_connection_lifetime=60
(or DATABASE_MAX_IDLE_CONNECTION_LIFETIME) to DATABASE_URL and DIRECT_URL before
the reader inherits the writer's connection params, so the gateway, backend and
migrations entrypoints get the same idle-connection reaping as the classic CLI.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep DB URL connection params across IAM/Entra token refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger proxy-infra after process-tree test flake
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): kill the whole prisma process group when a boot migration command times out
* fix(proxy): keep boot alive when the installed proxy extras has no prisma runner
* fix(spend_logs): keep partition DDL transactions alive for their statement timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format changed files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): avoid dict-literal kwargs and keep cast-ok on the cast line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): cast at the call site instead of widening PrismaClient.tx
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend_logs): require partition tx timeout to strictly exceed statement bound
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
SPEND_LOGS_URL only diverts spend logs when db_writer_client is set, and nothing in the proxy ever assigns that global, so the queued copy was only ever skipped as a duplicate by the local insert.
disable_spend_logs has to keep meaning that no request gets logged, and the row
that makes a batch chargeable exactly once is the one row it cannot drop, so with
logging off that row now carries only what tells the retrieves apart. SPEND_LOGS_URL
deployments get their copy back too: the claim writes straight to this table, so the
row is queued as well when an external writer is the one that takes the spend logs.
The takeover of a $0 row an older proxy left behind used to charge the batch when
the update could not reach the database. That leaves the row still reading $0, so
every later retrieve finds the same row and charges the batch again, which is the
repeat charging this PR exists to stop. The retrieve that does take the row over
is the one that charges, and a batch nobody retrieves again after that failure is
never charged, the same as one whose proxy died inside the write window.
A proxy without this fix wrote the batch's cost row on every poll while the batch
was still running, so that row reads $0 and the insert that claims the charge has
nowhere to land. The retrieve that charges the batch now writes its own payload
over that row under a where clause that still names spend 0.0, so exactly one
retrieve takes it over and every later one reads the charge and charges nothing
A proxy running the old code wrote <batch id>_batch_cost at $0 every time it polled a batch that was still running, so after an upgrade the claim found that row and read it as proof the batch had already been charged. Only a row that recorded a charge counts now, which leaves those $0 rows, and any row a client planted under the batch id, to be charged over
disable_spend_logs skipped the claim entirely, so under that setting every retrieve of a finished batch charged again. The claim now runs either way and writes the one row per batch that makes the charge exactly once, while the per-request logs stay off
The cost callback used to look for an existing `<batch id>_batch_cost` row before charging a
completed batch, which left a window where concurrent retrieves on any instance all charged the
key, and it would honor a row any request had written under that id. The spend update writer now
inserts the batch cost row itself with `create_many(skip_duplicates=True)` and only the retrieve
whose insert lands charges the key, team, and user. An existing row only takes the charge when it
is a successful `aretrieve_batch` row, so a client-chosen `x-litellm-call-id` on another endpoint
cannot suppress billing. Batch cost rows no longer get their own immediate flush path
`batch_cost_is_final` now treats the proxy's normalized `complete` status like `completed`, which
the enterprise batch cost poller relies on when it decides whether a completed batch is safe to
retire. Tests build that status with `model_copy` since the OpenAI `Batch` model rejects it
The `test-quality-ok` markers sit on the `patch(` lines the gate keys on, and the logging tests no
longer wrap the priced retrieve in `contextlib.suppress`
Every retrieve of a batch through the proxy shares one spend row, the batch id
plus the batch cost suffix, and spend log inserts skip duplicates. A poll that
landed while the batch was still validating or in progress wrote that row at
$0 and no later retrieve could overwrite it, and every completed retrieve after
the first added the cost to the key, team, and user counters again with no new
row to show for it.
The cost callback now writes nothing for a batch retrieve until the batch is
final, releasing the poll's budget reservation instead, and once it is final it
charges only when no spend row for that batch is queued for flush or already
stored. Batch cost rows are flushed to the database right away so a second
instance sees them, and the logger prices a batch only once it is final, which
also covers a failed batch that never produced an output file.
The reset job evicts the cached end-user object only from its own worker's
in-memory cache (plus Redis), so every other uvicorn worker and replica keeps
the pre-reset spend for up to user_api_key_cache_ttl (60s by default). Those
workers pass that stale spend as fallback_spend, and since the authoritative
floor read returned None for spend:end_user: keys, get_current_spend handed
the stale value straight back and the end user kept getting 429 after the
rollover on every worker but the one that ran the reset.
The floor read now consults LiteLLM_EndUserTable.spend for end-user counters,
the same way keys, teams, users, and orgs already read their rows. It runs only
when the shared counter sits below the cached spend (a reset or a Redis
restart) and stays behind the existing 5s in-process marker, so the normal
request path still does no DB read. Cold end-user counters keep seeding from
the cached object rather than the row, so from_db is unchanged for them.
* fix(proxy/db): translate libpq sslrootcert and verify-* into Prisma's strict TLS params
Prisma silently drops sslrootcert and treats sslmode=verify-ca/verify-full as
prefer, so a DATABASE_URL copied from the RDS docs connected over TLS without
checking the server certificate. The URL handed to Prisma (writer, DIRECT_URL,
read replica, componentized entrypoints) now carries sslmode=require,
sslcert=<bundle> and sslaccept=strict instead.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy/db): ruff format translate_libpq_ssl_params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
`PrismaClient.spend_log_flush_requested` was an `asyncio.Event` built at
import time, so it bound to whichever event loop first awaited it and every
later loop got `RuntimeError: ... is bound to a different event loop` out of
`_wait_for_spend_log_flush_request`. The queue monitor's blanket `except
Exception` swallowed that into its error logger, so the flush silently never
happened and the row sat in the worker's queue until the next poll.
The monitor now creates its own Event inside the loop that awaits it and
hands it to the client, and `request_spend_log_flush` signals through the
client instead of the class. A request that arrives before the monitor is
running is dropped and loses nothing, because the monitor reads the queue on
its first pass before it ever waits.
In CI this showed up as the proxy-endpoints shard flaking on
test_monitor_spend_logs_queue_flushes_as_soon_as_one_is_requested whenever
--dist=loadscope put the health-endpoint tests, which boot a proxy TestClient
and start a monitor, on the same worker ahead of the spend-log tests.
Every Prisma CLI call now goes through one runner that starts the command in
its own session and SIGKILLs the process group on timeout, so the Node process
and the Rust schema engine die together with the Python wrapper instead of
being reparented to pid 1, where they kept applying migrations after the proxy
had given up and held the Prisma advisory lock against every retry and every
later boot. Tests that faked subprocess.run now fake the runner, and the fake
Prisma CLI in the migration tests forks a grandchild that must not outlive a
timed-out migrate deploy.
Permanent Prisma/query-engine faults keep the 503 status and no_db_connection type but stop claiming the database is temporarily unreachable. A permanent fault anywhere in the exception chain outranks the transport error that surfaced it. MCP bridge and DCR flows gain a faulted resolution state with matching wording. Resolves LIT-5208
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The batch start told the seed which LiteLLM_SpendLogs rows were its own, but
using it as a hard cutoff also dropped rows another pod had already persisted.
Those rows are only repaid by that pod's own increment, so if it died first the
window row stayed permanently under the recorded spend.
The seed now reads both sums in one scan and takes off this batch's own spend,
flooring at the pre-batch total for the case where its log rows have not landed
yet. Redis payloads keep an empty request_ids so a leader from before the field
was dropped can still merge what it pops during a rolling deploy.
Claude-Session: https://claude.ai/code/session_01QvQzYztinxj8ZuD5YxbVdL
The one-time seed for a budget window row subtracted the batch's own
LiteLLM_SpendLogs rows by request_id, and request_id is the client's
x-litellm-call-id whenever the response carries no id of its own. Carrying
that set through the queue meant an unbounded, client-controlled aggregate
that the commit-failure requeue kept alive across retries.
Every log row at or after a batch's earliest start is owed by an increment
that still reaches the row, so summing only rows before it needs nothing
from the request. That drops request_ids end to end and closes the
cross-pod double count the id list could not see.
The LLM classifier's cost was recorded on the routing decision but never
reached any savings surface: per-request autorouter_savings stayed gross
and the session rollup recorded only the served request's spend, so
/auto_router/benchmarks overstated savings and understated routed spend.
Net the classifier cost into the savings figure at its one computation
owner and fold it into the rollup turn's spend, keeping
baseline_spend = spend + saved_spend. The response header's numeric
guard now shares the same reader.
Fixes#38816
Budget enforcement trusts a current LiteLLM_BudgetWindowSpend row without
reconciling it against LiteLLM_SpendLogs, so an increment dropped after a
failed commit let the entity spend past its window limit after the next
counter reseed. Failed increments now go back on the in-memory queue, or
back to the Redis buffer, and retry on the next scheduler tick like every
other spend category.
A model access group could gate which models a caller reaches but never how
much that group of callers could spend in total. Capping a shared pool meant
setting a per-entity budget on every key by hand, which caps each key
separately and still leaves no way to read what the group cost.
Spend is attributed to a group only when the group's name appears on an
allowlist the caller was granted (key, team, team-member scope, project or
org) and that group serves the requested model. Asking for a model that
merely belongs to a group attributes nothing, because nothing about the
caller named the group. Levels are unioned rather than ranked, so a team
granted "*" whose member is scoped to one group still counts as gated by
that group.
Enforcement runs on both paths tags already use: a reservation counter on
the pre-call path and a read-time max_budget check inside the existing
concurrent budget gather, so the ceiling still holds under
disable_budget_reservation.
Adds LiteLLM_ModelAccessGroupBudgetTable, which is the only place a group is
ever a row: the groups themselves stay free-text strings in
model_info.access_groups, so a row exists only once someone gives that group
a budget. GET, PUT and DELETE /access_group/{name}/budget manage it, and
/access_group/{name}/info now carries the spend and budget alongside the
models.
