Commit graph

61 commits

Author SHA1 Message Date
Kerry Lu
4bc157e281
feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations (#48961)
* feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations

* feat(proxy): roll back each migration of the dry run on its own and baseline-check databases without a ledger

* test(proxy): cover the dry run CLI and PrismaManager wrapper without a database

* fix(proxy): refuse the no-ledger dry run for a custom migration folder like boot does, and probe the rolled-back indexes directly

* ci(codecov): stop carrying forward the roi-postgres and redis-compat flags

Their carried-forward sessions were measured against older proxy_cli.py revisions (633 and 689 line maps against 710 today) and mark docstring lines of the merged file as missed, which failed codecov/patch on this PR at 70% while every unit shard reports the lines hit

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 09:39:26 -07:00
Kerry Lu
5667ac8497
feat(proxy): refuse migrations that existing rows would make fail, before the first one runs (#48493)
* feat(proxy): refuse migrations that existing rows would make fail, before the first one runs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(migrations): start the serving-proxy preflight case from a database without the api_key index

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): type the preflight count rows without casts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): ruff format the db preflight flag

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy-extras): keep the extras files at their existing formatting

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy-extras): type the ledger query and the reader pid so the type gate holds

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): declare the preflight report type where the extras package may be absent

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): suppress the optional-extras unknown on the preflight report variables

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): suppress the optional-extras unknown on the preflight import

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): return a typed preflight report when the extras package is unknown

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): sort the preflight import block

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): mark the preflight report binding Final

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(proxy): size the whole btree entry in the row check and name the flag for what it does

The index check now lays out each row's entry the way index_form_tuple does (tuple header, null bitmap, every indexed column at its alignment, rounded to 8) and compares it with BTMaxItemSize for the server's block_size, so a key that passes octet_length > 2704 but still fails the (api_key, startTime) index is caught. --db_preflight becomes --check_migrations_against_existing_data, db_preflight.py becomes migration_data_checks.py and the timeout env var LITELLM_MIGRATION_DATA_CHECK_TIMEOUT

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 12:14:42 -07:00
Yassin Kortam
1ac9cb2a1d
fix(proxy-extras): accept unencoded DB passwords in the v2 migration lock and redact its errors (#46592)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-15 14:47:23 -07:00
Yassin Kortam
2310daef7e
fix(proxy-extras): run prisma migrate deploy under the pinned coordinator lock instead of Prisma's session advisory lock (#46368)
Through a transaction-mode pooler Prisma takes pg_advisory_lock(72707369) on one
backend and releases it on another, so the first backend keeps the lock while idle
and the next boot waits on it. Migrations now run with
PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK=1 while LiteLLM holds the coordinator lock and
Prisma's key as transaction-level locks on one pinned connection, so nothing can
outlive the migration

Resolves LIT-9571

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 15:10:27 -07:00
Yassin Kortam
ae9c208b8d
fix(proxy-extras): bound migration lock waits with SET LOCAL lock_timeout in the migration SQL (#46329)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 20:51:03 -07:00
Yassin Kortam
cf1284f3d5
fix(proxy-extras): bound migration DDL lock waits with lock_timeout and retry (#45389)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-10 12:52:52 -07:00
devin-ai-integration[bot]
9a2c9e6099
fix(proxy-extras): name the database error when the Lens rename check cannot run (#45341)
* fix(proxy-extras): name the database error when the Lens rename check cannot run

The Lens rename check runs before prisma db push and raised a generic
"Cannot verify Lens data safety" RuntimeError on any psycopg error, with
the cause only in the chained exception. proxy_cli prints the message and
exits, so a Postgres at its connection limit read as a Lens problem.

The raised message now ends with the redacted database error text, and
the connection opener is an injected parameter so the unit test drives
the check without a database.

* test(proxy-extras): drop the Lens check test class docstring

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-08 04:18:25 -07:00
devin-ai-integration[bot]
984b4134be
fix(proxy-extras): log v1 migration failures at ERROR so LITELLM_LOG=ERROR shows them (#44202)
* fix(proxy-extras): log v1 migration failures at ERROR so LITELLM_LOG=ERROR shows them

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy-extras): reuse litellm secret redaction and mask configured DB passwords exactly

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy-extras): name the password alternation in _redact_credentials

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy-extras): wrap v1 migration ERROR lines at 120 columns

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy-extras): integration cells for v1 migration ERROR logging and password redaction

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy-extras): integration cells for component DB env vars, JSON logs, migration Job and v2 resolver

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy-extras): give every v1 migration integration cell the 900s timeout

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy-extras): bind the recovery forwarder before migrating so the retry cannot race it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy-extras): drop the slow P3005 integration cell and bound migration subprocesses

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy-extras): keep command repr and tolerate non-sequence cmd in migration error logs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy-extras): double the subprocess boundary in the cmd=None retry test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
2026-10-03 22:45:16 -07:00
devin-ai-integration[bot]
8b11b682f4
fix(proxy-extras): retry P3009 when a peer already recovered the named migration row (#44283)
* fix(proxy-extras): retry P3009 when a peer already recovered the named migration row

* fix(proxy-extras): pin P3009 recovery locals as Final and cover a recovered sibling row

* test(proxy-extras): stop the P3009 ledger fakes shadowing the partition detector cursor

* fix(proxy-extras): annotate the migration ledger reader locals as Final

---------

Co-authored-by: yuneng <yuneng@berri.ai>
2026-10-02 18:39:06 -07:00
devin-ai-integration[bot]
4758fce91a
fix(proxy-extras): hand libpq a root cert, not Prisma's sslcert, when the migration job builds indexes (#44203)
The migration job runs DatabaseURLSettings.apply_to_env(), which rewrites
DATABASE_SSLMODE=verify-full plus DATABASE_SSLROOTCERT into Prisma's TLS
dialect: sslmode=require&sslcert=<CA>&sslaccept=strict. The request-log
index build then hands that same URL to psycopg, and libpq reads sslcert
as a client certificate, failing with "certificate present, but not
private key file" on every verify-full deployment since #43948.

_strip_prisma_query_params now undoes the Prisma dialect before psycopg
sees the URL. sslaccept=strict (or any value Prisma treats as strict)
becomes sslrootcert=<CA> plus sslmode=verify-full whatever sslmode said,
since strict verifies chain and hostname and libpq only does that in
verify-full; sslmode=disable stays off. Without strict, Prisma verifies
nothing, so the CA is dropped and sslmode is kept as is. A URL that also
carries sslkey is libpq's own client-certificate form and is left alone.

Resolves LIT-9169

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-02 11:21:23 -07:00
devin-ai-integration[bot]
c66c8288c3
fix(proxy-extras): build the SpendLogs indexes in the migration job instead of in migrations (#43948)
The two SpendLogs index migrations shipped in v1.103.0 each break one table shape: the plain CREATE INDEX holds a SHARE lock on a large unpartitioned table and the CONCURRENTLY one fails with 0A000 on a partitioned parent. Both files are now inert and the indexes are built by a table-driven, shape-aware step after migrate deploy: CONCURRENTLY on a plain table, ON ONLY the parent plus per-partition CONCURRENTLY and ATTACH PARTITION on a partitioned one. The migration job builds them synchronously and exits non-zero on failure; a serving proxy that ran migrate deploy itself builds them in the background off the readiness path. A valid index of the same definition under another name is renamed and reused, an invalid one is rebuilt, and extra copies are reported with their DROP INDEX statement instead of being dropped. The migration checker rejects any CREATE INDEX on LiteLLM_SpendLogs or LiteLLM_ErrorLogs in future migrations

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-01 14:12:22 -07:00
moe-berri
259c166ef6
refactor(lens)!: rename internal engine code and API (#44034)
* chore(lens): remove deployment screenshots

* refactor(lens)!: rename internal engine package and API

* fix(lens): pin worker image for renamed API

* test(lens): cover fresh and populated rename migrations

* fix(lens): protect db-push upgrades and restore routing and CI

* fix(lens): resolve migration tables across schemas and include database driver
2026-10-01 11:02:14 -07:00
Yuneng Jiang
a37f0b4f54
test: isolate migration CI selection and exercise resolver boundaries 2026-09-12 18:49:58 -07:00
Yuneng Jiang
94032014df
fix(proxy): coordinate v2 migration startup and add container regression CI 2026-09-12 18:24:59 -07:00
yassin
6a30467948 chore: merge litellm_internal_staging into litellm_fix_invalid_index_after_migration_deadlock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-08 17:07:34 +00:00
Mateo Wang
c053fac7b7
Merge pull request #39466 from BerriAI/litellm_prisma_timeout_killpg
fix(proxy-extras): kill the whole Prisma process group when a command times out
2026-09-03 14:36:31 -07:00
mateo-berri
8572544b44 refactor(proxy-extras): pull the migrate deploy recovery branches into a budget helper
_setup_database_v2 decided the next attempt budget inline in eight
branches, each rebinding budget before continuing. The branches now live
in _budget_after_deploy_failure, which returns the budget the next pass
runs under, and the two identical idempotent-recovery blocks share
_mark_migration_applied. The loop backs off whenever a pass spent an
attempt, which is the same set of paths that slept before.
2026-09-03 00:20:46 -07:00
mateo-berri
7b8cc0319e fix(proxy-extras): only spend a migrate-deploy attempt when a pass made no progress
The v2 migration resolver gave `prisma migrate deploy` four attempts, and
every recovery path ended in a bare `continue`, so each one burned an attempt.
A database first brought up with `--use_prisma_db_push` has a full schema and
no migrations ledger, so the baseline spent attempt one and the first three
migrations whose objects already existed spent the rest. The proxy then exited
before binding its port, and that database could never be moved onto the
resolver.

The retry budget now counts only attempts that got nowhere. Creating the
baseline, and each migration newly marked applied, leaves the budget alone, so
a push-created database works through its pre-existing objects one pass at a
time. Timeouts, deadlock rollbacks, advisory-lock waits, and a repeat of a
recovery that already ran still spend an attempt, so a run that stops making
progress gives up exactly as before.
2026-09-02 23:56:09 -07:00
mateo-berri
105f99cea2 fix(proxy-extras): kill the whole Prisma process group when a command times out
Every Prisma CLI call now goes through one runner that starts the command in
its own session and SIGKILLs the process group on timeout, so the Node process
and the Rust schema engine die together with the Python wrapper instead of
being reparented to pid 1, where they kept applying migrations after the proxy
had given up and held the Prisma advisory lock against every retry and every
later boot. Tests that faked subprocess.run now fake the runner, and the fake
Prisma CLI in the migration tests forks a grandchild that must not outlive a
timed-out migrate deploy.
2026-09-02 18:29:55 -07:00
mateo-berri
095030ba96 chore: merge litellm_internal_staging into litellm_fix_invalid_index_after_migration_deadlock 2026-09-02 17:09:30 -07:00
mateo-berri
bfdaedf51b fix(proxy-extras): take the repair schema from DATABASE_URL and deploy the real-migration test on a fresh database 2026-09-02 16:23:50 -07:00
mateo-berri
979c3cb1aa fix(proxy-extras): retry when the advisory lock wait times out behind a concurrent migrate deploy 2026-09-02 15:29:10 -07:00
mateo-berri
2d64020d43 fix(proxy-extras): run the index repair over DIRECT_URL and exercise it through a real migrate deploy 2026-09-02 15:24:42 -07:00
yassin
4705dc6325 fix: retry P3009 when the deadlocked ledger row was already rolled back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-02 20:34:54 +00:00
mateo-berri
55c7872496 fix(proxy-extras): rebuild indexes left INVALID by a migration deadlock
Two replicas racing prisma migrate deploy can deadlock, and the loser dies
mid CREATE INDEX CONCURRENTLY, leaving the index INVALID. The retried
migration's IF NOT EXISTS then skips it, so the planner never uses it.

After migrations succeed, look for INVALID indexes on LiteLLM tables and
have one replica (advisory try-lock) REINDEX INDEX CONCURRENTLY each of
them, dropping _ccnew/_ccold leftovers of an interrupted rebuild instead.
The repair never blocks startup: a failed rebuild is logged and retried
on the next boot. Also encode DATABASE_URL query values with quote instead
of quote_plus so options=-c%20... reaches psycopg intact.
2026-09-02 12:57:06 -07:00
yassin
aaf59abb1f chore: merge litellm_internal_staging into litellm_fix_v2_resolver_migration_deadlock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-02 19:53:03 +00:00
mateo-berri
c83b4a1d19 fix(proxy-extras): honor a raised command timeout for migrate deploy and name the right knob on db push timeouts 2026-09-02 12:10:54 -07:00
mateo-berri
0c53961445 fix(proxy-extras): give prisma migrate deploy its own timeout budget 2026-09-02 10:44:51 -07:00
mateo-berri
5192b2162c fix(proxy-extras): schema-qualify the _prisma_migrations logs lookup for non-public Prisma schemas 2026-09-01 14:04:12 -07:00
mateo-berri
692f3b513c fix(proxy-extras): recover the v2 migration resolver from concurrent migrate deploy deadlocks
Two instances racing prisma migrate deploy on one database deadlock on
CREATE INDEX CONCURRENTLY: the victim gets P3018 with 40P01 and the
survivor then sees the failed ledger row as P3009. Both were treated as
unrecoverable, so neither instance came up.

Roll the deadlocked migration's ledger row back and retry the deploy on
P3018, consult the failed row's logs in _prisma_migrations to do the
same on P3009, and retry a deadlock reported without a Prisma error
code. Genuinely broken migrations still fail fast.
2026-09-01 13:43:01 -07:00
Yuneng Jiang
fd72a39b1b
revert: default the proxy back to the v1 migration resolver
This reverts merge commit 2b1bd20834 (#31125)

Two CircleCI jobs on the staging-to-main promotion went red the moment
that PR landed. proxy_multi_instance_tests boots two proxies against one
database, and both now race the same migration:

  Error: P3018 A migration failed to apply
  Database error code: 40P01, deadlock detected
  Process 73 waits for ShareLock on virtual transaction 4/11;
  blocked by process 75. Process 75 waits for ExclusiveLock on
  advisory lock [16384,0,72707369,1]; blocked by process 73

Neither proxy comes up, so the job times out after 300s waiting on
localhost:4000. The same wait took 36.5s on the last green run

Timeline: #31125 merged at 18:46:14Z and the failing run started at
18:49:59Z. The merge commit is not an ancestor of the last green
revision (194a3cc) and is an ancestor of the first failing one
(01de2837)

The v2 resolver was meant to avoid exactly this class of contention, so
the deadlock looks like a bug in it rather than a reason to abandon it.
Putting the default back to v1 buys time to fix it without holding up
the release
2026-09-01 12:56:08 -07:00
yuneng-jiang
2b1bd20834
Merge pull request #31125 from BerriAI/litellm_/stoic-jones-7de871
feat(proxy): default to the v2 migration resolver, keep v1 as an opt-out
2026-09-01 11:46:13 -07:00
Yassin Kortam
e34f43328c
fix(proxy): ship psycopg so partitioned SpendLogs detection actually runs (#38994)
ProxyExtrasDBManager.spend_logs_is_partitioned() (#38452) silently returns
False when psycopg can't be imported, and psycopg was never added to the
extra_proxy install, so every production image lacks it. Schema
reconciliation then generates the unfiltered primary-key rewrite against a
genuinely partitioned LiteLLM_SpendLogs and Postgres rejects it, exactly the
failure the fix was meant to prevent. Ships psycopg via extra_proxy and logs
a warning when it's still missing instead of failing silently.
2026-08-31 17:33:12 -07:00
Yuneng Jiang
2495673d01
fix(proxy-extras): stop a db push timeout crashing the migration job
subprocess.run leaves stderr as bytes on TimeoutExpired even under
text=True, unlike CalledProcessError. Classifying both in one handler
meant a real `prisma db push` timeout died on a TypeError, which
proxy_cli.py's `except RuntimeError` does not catch, so the migrations
Job container ended on an unhandled traceback instead of a clean exit.

Give the timeout its own handler and retry it, matching what the migrate
deploy loop beside it already does. That puts a fallthrough back into the
loop, so the trailing raise removed in the previous commit is reachable
again and comes back with it.

Also drop a comment restating why the resolver cases exist and widen the
db push test's docstring, which had stopped describing what it covers.
2026-08-28 02:14:57 -07:00
Yuneng Jiang
6b3e30e660
fix(proxy-extras): bound the db push retries in the reachable branch
The retry loop already raises on the final attempt, so the raise that
followed the loop could never run. Drop it and cover the exhaustion path
with a test that pins the attempt count and keeps the prisma error in the
message, which is the only thing that tells an operator why the boot
stopped.
2026-08-28 01:56:15 -07:00
Yuneng Jiang
7b36bfb967
fix(proxy-extras): retry transient db push failures, drop a vacuous test
`prisma db push` under v2 raised on the first failure while v1 retried it four
times, so making v2 the default silently cost --use_prisma_db_push its
retries. It now uses the same transient classification as migrate deploy.

The classifier moves onto ProxyExtrasDBManager next to _is_permission_error
and _is_idempotent_error, which do the same kind of stderr matching.

Replaces a test that claimed to pin the transient classification but fed it a
P3009 stderr, which an earlier branch catches, so it passed even when the
classifier was mutated to treat everything as transient. The replacement uses
an unclassified error and fails on that mutant. Drops a v1 test that duplicated
test_v1_default_still_calls_resolve_all_migrations.
2026-08-28 01:31:44 -07:00
Yuneng Jiang
4f6fd85ab1
feat(proxy): default to the v2 migration resolver, keep v1 as an opt-out
The v2 resolver skips the diff-and-force recovery that caused schema
thrashing when two LiteLLM versions contend for one database during a
rolling deploy. The standalone migration Job already defaulted to v2; this
aligns the proxy-server path.

v1 stays reachable two ways: --use_legacy_migration_resolver on the CLI, and
USE_V2_MIGRATION_RESOLVER=false for containerised deploys, where
prisma_migration.py calls run_server with a fixed argv and the env var is the
only route in. --use_v2_migration_resolver still parses, so existing commands
do not die on an unknown option.

Because v2 fails fast where v1 retried every failed deploy, a database that is
not accepting connections yet, or another instance holding the migration
advisory lock, would now kill a boot that used to ride it out. Those two
failures are retried, with Prisma's stderr logged each round, and still raise
once the attempts are spent.

Moves the resolver tests from litellm-proxy-extras/tests, which no CI job
runs, into tests/litellm-proxy-extras, and repoints the dedicated Postgres
CircleCI job at the legacy path so v1 keeps real-DB and proxy-boot coverage.
2026-08-28 01:10:05 -07:00
Devin AI
d990f24b9d chore(techdebt): type new signatures and drop slop comments from the last 24h
Removes restating comments added with the Teams alerting destination and the
lazy OpenAPI snapshot refactor, types three signatures that shipped untyped or
with bare dict, and ratchets the strict and basedpyright budgets down.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-28 07:59:32 +00:00
devin-ai-integration[bot]
fe87b187c6
fix: keep schema reconciliation from fighting a partitioned LiteLLM_SpendLogs (#38452)
* fix: keep schema reconciliation from fighting a partitioned LiteLLM_SpendLogs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: scope partitioned SpendLogs detection to Prisma's target schema

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: default partition detection to Prisma's public schema, not current_schema()

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-27 12:52:52 -07:00
Yassin Kortam
0659738b3e
fix(migrations): recover from an interrupted Prisma toolchain install (#35832)
The Prisma CLI is a Node program that installs a private Node runtime on its
first invocation. That one-time install shared the 60s budget that bounds each
migration command, so on a slow or cold machine it was killed before it could
finish. Prisma then decides whether to reinstall by testing the cache
directory for existence alone, and a killed install leaves that directory
behind, so every later attempt skipped the install and failed on a node binary
that was never written. The existing four-attempt retry loop could not help:
each attempt hit the same missing binary, which turned a slow start into a
container that never migrated again.

Migrations now prepare the toolchain as its own step under its own budget, and
a cache directory that exists without a node binary is deleted first so an
interrupted install reinstalls instead of persisting. Both budgets are
overridable, LITELLM_PRISMA_BOOTSTRAP_TIMEOUT for the install and
LITELLM_PRISMA_COMMAND_TIMEOUT for each Prisma command, and every previously
hardcoded timeout now goes through one helper rather than thirteen literals.
The per-command default stays at 60s.

An override is only honoured when it parses as a finite positive number.
Infinity and NaN parse as floats and survive a plain positivity check, and
subprocess treats either as no deadline at all, so a value like `inf` or a
fat-fingered `1e400` would have silently disabled the timeout it was meant to
configure.
2026-08-05 10:14:46 -07:00
Yassin Kortam
87c2e03af8
feat(db): opt-in REPLICA IDENTITY FULL after prisma migrations (#35267)
Logical replication consumers need FULL replica identity to reconstruct the
old row of an UPDATE or DELETE, and prisma leaves every table it creates at
the postgres default. Operators had to re-apply the setting by hand after
each migration run.

Setting LITELLM_SET_REPLICA_IDENTITY_FULL now re-asserts it on every LiteLLM
table at the end of a successful migration run, through the prisma CLI so the
dependency-free proxy-extras package stays that way. Tables that are already
FULL are skipped, foreign tables in the same schema are left alone, and a
database that refuses the ALTER is reported rather than failing the run.

Resolves LIT-3022
2026-07-30 15:45:40 -07:00
Yuneng Jiang
2b8b9502d9
[Fix] v2 resolver: swallow non-connection DB errors; wrap resolve failures
Addresses two further Greptile findings:

- `_warn_if_db_ahead_of_head` only caught `psycopg.OperationalError`.
  Non-connection DB errors (e.g. `InsufficientPrivilege` / 42501 if the
  runtime DB user lacks SELECT on `_prisma_migrations`) would propagate
  uncaught and crash startup — contradicting the docstring's
  "informational only, never blocks" guarantee. Widen the catch to
  `psycopg.DatabaseError` so all DB-layer errors are swallowed.

- In the P3009 and P3018 idempotent-recovery paths, the call to
  `_resolve_specific_migration(name)` was not wrapped in its own
  try/except. Being inside an active `except CalledProcessError`
  handler, a new `CalledProcessError` from the resolve call would NOT
  re-enter the same handler — it would propagate out as
  `CalledProcessError`, past `proxy_cli.py`'s `except RuntimeError`,
  crashing startup with an unhandled traceback instead of the intended
  clean `sys.exit(2)`. Wrap both call sites to convert to RuntimeError.

Adds unit tests for both behaviors.
2026-04-21 15:53:07 -07:00
Yuneng Jiang
9049f37864
[Fix] v2 migration resolver: address Greptile review findings
- Open the psycopg connection in `_warn_if_db_ahead_of_head` with
  autocommit=True. Without it, psycopg3's `with conn` calls COMMIT on
  clean exit, which fails after the `UndefinedTable` (fresh-DB) branch
  left the transaction in an aborted state — crashing first-run startups.

- Wrap the v2 `prisma db push` path in try/except and raise RuntimeError
  on CalledProcessError/TimeoutExpired. Otherwise these propagate past
  proxy_cli.py's `except RuntimeError` as unhandled tracebacks.

- Reword the loop-exhaustion error to cover the non-timeout exit path
  (repeated P3005/P3009/P3018 idempotent-recovery `continue`s), not
  just persistent timeouts.

Adds a unit test for the db_push error wrapping.
2026-04-21 15:34:24 -07:00
Yuneng Jiang
a16c00e22c
[Feature] Proxy: opt-in v2 migration resolver (--use_v2_migration_resolver)
Default behavior (v1) is unchanged. Users who have seen schema thrashing
during rolling deploys can opt into the v2 resolver with
`--use_v2_migration_resolver`.

Why v2 is safer:
- Runs `prisma migrate deploy` only.
- Recovers from P3005 (baseline) and idempotent P3009/P3018 errors, same
  as v1.
- Never calls `_resolve_all_migrations`, which generates a schema diff
  between the live DB and the shipped schema.prisma and applies it via
  `prisma db execute`. That path bypassed every migration's SQL and was
  the root cause of thrashing when two LiteLLM versions contended for
  the same DB.
- Logs a non-blocking warning when the DB has migrations applied that
  are newer than anything this build ships (ahead-of-HEAD). It does not
  refuse to start — many users have unusual ledger state from past
  thrashing, and blocking startup would be a breaking change.

Also prints a message on startup when the default (v1) resolver is in
use, pointing operators at the opt-in flag.

Adds unit tests covering the v2 fail-fast paths, the stripping of
Prisma-specific query params from DATABASE_URL (needed for psycopg),
the timestamp helpers, and pins the default: v1 still invokes
`_resolve_all_migrations`, v2 must not.
2026-04-21 14:20:35 -07:00
Ishaan Jaffer
e6a20af646
fix(proxy-extras): skip post-deploy sanity check when no migrations pending
When prisma migrate deploy reports 'No pending migrations to apply' the DB
already matches schema — running _resolve_all_migrations (migrate diff +
prisma db execute) adds 25+ seconds unnecessarily, causing the proxy to
miss the 90-second startup timeout in test_litellm_proxy_server_config_no_general_settings.
2026-04-17 15:59:41 -07:00
Ishaan Jaffer
33175a8ee7
fix(proxy-extras): fall back to prisma db execute when migrate diff fails on pooler URL
When DIRECT_URL is not set and DATABASE_URL is a Neon pooler URL, prisma migrate diff
fails (pooler doesn't support extended query protocol for schema introspection). Previously
_resolve_all_migrations returned early without applying any migrations, leaving the
budget_limits column missing and causing test_auth_callback_new_user to fail.

Now falls back to running each migration SQL file via prisma db execute --file, which
works with pooler URLs and is safe to re-run due to IF NOT EXISTS guards.
2026-04-17 15:38:48 -07:00
Ishaan Jaffer
33a2cee4af
fix(proxy-extras): use DIRECT_URL for prisma migrate diff, tempfile for diff dir 2026-04-17 15:17:15 -07:00
Ishaan Jaffer
7c47bbd226
fix(migration): run schema sanity check after P3009/P3018 idempotent migration recovery 2026-04-17 15:01:10 -07:00
Ishaan Jaffer
e8461b5b97
style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
Krish Dholakia
b96f033c90
fix: prisma migrate deploy failures on pre-existing instances (#23655)
* fix: prisma migrate deploy failures on pre-existing instances

Fixes failed migrations due to idempotent schema changes on pre-existing litellm instances.

Problems:
1. P3018 recovery handler never returned True on successful resolution, causing "Database setup failed after multiple retries" even when the final recovery succeeded
2. _roll_back_migration exceptions escaped the P3018 handler, preventing _resolve_specific_migration from running
3. Migration SQL used ADD COLUMN/DROP COLUMN without IF [NOT] EXISTS, failing if schema was already modified

Changes:
- Add return True after successful P3018 idempotent error recovery
- Wrap _roll_back_migration in try/except to allow recovery continuation even if rollback fails
- Make migration.sql idempotent with IF NOT EXISTS / IF EXISTS clauses

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* test: add migration SQL idempotency safety tests

Adds TestMigrationSQLIdempotency test class that statically validates all
migration SQL files created after 2026-03-11 use idempotent DDL:
- ADD COLUMN must use IF NOT EXISTS
- DROP COLUMN must use IF EXISTS
- DROP INDEX must use IF EXISTS
- CREATE INDEX must use IF NOT EXISTS

This prevents the class of errors where prisma migrate deploy fails on
pre-existing instances because the schema was already modified.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: also catch TimeoutExpired in P3018 rollback handler

_roll_back_migration uses subprocess.run with timeout=60, so it can raise
subprocess.TimeoutExpired in addition to CalledProcessError. Without
catching this, a slow database during rollback would escape the handler
and bypass _resolve_specific_migration — the same class of bug.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: make all 85 migration SQL files idempotent, remove test cutoff

Fixed all existing migration files to use IF [NOT] EXISTS for DDL
statements (ADD COLUMN, DROP COLUMN, DROP INDEX, CREATE INDEX).
Removed the date cutoff from the idempotency tests so they now
validate all migrations, not just recent ones.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: make migration failure non-fatal by default, add --require_db_migration flag

By default the proxy now warns and continues when database migration
fails. Pass --require_db_migration (or set REQUIRE_DB_MIGRATION=true)
to restore the previous behavior of exiting with an error.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: wrap _resolve_specific_migration in try/except, guard RENAME COLUMN and ADD CONSTRAINT

Three fixes:

1. _resolve_specific_migration in the P3018 handler was not wrapped in
   try/except, so failures there would bypass the return True and
   propagate unexpectedly — partially defeating the rollback fix.

2. Bare RENAME COLUMN in 20260303000000_update_tool_table_policies was
   non-idempotent. Wrapped in DO $$ IF EXISTS block. Also wrapped all
   28 bare ADD CONSTRAINT statements across 9 migration files in
   DO $$ IF NOT EXISTS (pg_constraint) blocks.

3. Added test_rename_column_is_guarded and test_add_constraint_is_guarded
   to TestMigrationSQLIdempotency for full DDL coverage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: retry after resolving idempotent migration, guard DROP CONSTRAINT

Three fixes:

1. Both P3009 and P3018 idempotent handlers returned True after
   resolving a single migration, exiting before remaining pending
   migrations were applied. Now they continue the retry loop so
   prisma migrate deploy runs again for any remaining migrations.

2. Two migration files had bare DROP CONSTRAINT without a DO $$ IF
   EXISTS guard, which fails if the constraint was already dropped.
   Wrapped both in idempotent DO $$ blocks.

3. Added test_drop_constraint_is_guarded to catch unguarded DROP
   CONSTRAINT in future migrations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: P3009 try/except, CREATE TABLE IF NOT EXISTS, restore fail-fast default

Four fixes:

1. P3009 idempotent handler now has the same try/except around
   _roll_back_migration and _resolve_specific_migration as the P3018
   handler. Previously a rollback or resolve failure in the P3009 path
   would propagate and leave the migration unresolved.

2. Added IF NOT EXISTS to all 57 bare CREATE TABLE statements across
   34 migration files. Added test_create_table_uses_if_not_exists to
   catch this pattern.

3. Reverted the backwards-incompatible default behavior change: the
   proxy now fails fast on migration failure (original behavior).
   Added --skip_db_migration_check / SKIP_DB_MIGRATION_CHECK to
   opt into warn-and-continue instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
2026-03-14 16:54:21 -07:00