Keeps the base's rule that a non-admin id lookup matching no spend-log row answers 403, so the detail route never consults cold storage without an owner row
Adds object_permission.skills to keys and teams, enforces it on
/claude-code/marketplace.json?key=, /claude-code/plugins and
/claude-code/plugins/{name}, and exposes an Allowed Skills selector in
the key and team create/edit forms of the Admin UI
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The proxy probed for the Prisma CLI by spawning the bare `prisma` console
script, so a launcher whose PATH lacked the interpreter's bin directory
printed "prisma package not found", skipped every migration and served
traffic against an empty schema. Every Prisma command now falls back to
`python -m prisma` when the console script is not on PATH, and the boot
probe checks for the console script or the importable package instead of
spawning anything.
LiteLLM_JWTKeyMapping_token_fkey was created ON DELETE RESTRICT, so deleting
a virtual key that a JWT mapping pointed at failed with a foreign key
violation on every deletion path (/key/delete, Admin UI, alias delete,
team and user cascades). Declaring onDelete: Cascade on the relation lets
the database clean the mapping up uniformly, so the next JWT call from that
identity re-registers against the newly created key.
Rebase of #33703 onto current staging.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
A shadow eval job could only be scoped by identity, so "this user's traffic on model X
across every key they own" was not expressible and a models field on the start body was
silently dropped. The job now carries a models list that every target is narrowed to,
matched on the requested model group with model_group_alias resolved on both sides. An
unresolvable name is a 400 at start. Empty means every model, which is what every existing
row reads as. The dashboard start form gains an "Only on models" picker and the job
headline shows the scope.
A row that received both priced and unpriced increments used to collapse
to cost NULL, throwing away the priced subtotal and making every unit on
it read as untracked. The rollup now carries a second column,
untracked_units, that the aggregator increments for units with no known
price while cost keeps accruing for the rest, so cost covers exactly
units - untracked_units. Rows written before the migration keep cost
NULL and still read as untracked in full
The endpoints read untracked units off the column (or the whole row for
a legacy NULL) rather than from a NULL filter, and the policies overview
now fills totalUntrackedUsageUnits, which the previous commit missed
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
_setup_database_v2 decided the next attempt budget inline in eight
branches, each rebinding budget before continuing. The branches now live
in _budget_after_deploy_failure, which returns the budget the next pass
runs under, and the two identical idempotent-recovery blocks share
_mark_migration_applied. The loop backs off whenever a pass spent an
attempt, which is the same set of paths that slept before.
The v2 migration resolver gave `prisma migrate deploy` four attempts, and
every recovery path ended in a bare `continue`, so each one burned an attempt.
A database first brought up with `--use_prisma_db_push` has a full schema and
no migrations ledger, so the baseline spent attempt one and the first three
migrations whose objects already existed spent the rest. The proxy then exited
before binding its port, and that database could never be moved onto the
resolver.
The retry budget now counts only attempts that got nowhere. Creating the
baseline, and each migration newly marked applied, leaves the budget alone, so
a push-created database works through its pre-existing objects one pass at a
time. Timeouts, deadlock rollbacks, advisory-lock waits, and a repeat of a
recovery that already ran still spend an attempt, so a run that stops making
progress gives up exactly as before.
Every Prisma CLI call now goes through one runner that starts the command in
its own session and SIGKILLs the process group on timeout, so the Node process
and the Rust schema engine die together with the Python wrapper instead of
being reparented to pid 1, where they kept applying migrations after the proxy
had given up and held the Prisma advisory lock against every retry and every
later boot. Tests that faked subprocess.run now fake the runner, and the fake
Prisma CLI in the migration tests forks a grandchild that must not outlive a
timed-out migrate deploy.
Two replicas racing prisma migrate deploy can deadlock, and the loser dies
mid CREATE INDEX CONCURRENTLY, leaving the index INVALID. The retried
migration's IF NOT EXISTS then skips it, so the planner never uses it.
After migrations succeed, look for INVALID indexes on LiteLLM tables and
have one replica (advisory try-lock) REINDEX INDEX CONCURRENTLY each of
them, dropping _ccnew/_ccold leftovers of an interrupted rebuild instead.
The repair never blocks startup: a failed rebuild is logged and retried
on the next boot. Also encode DATABASE_URL query values with quote instead
of quote_plus so options=-c%20... reaches psycopg intact.
The daily guardrail usage rollup stored billable units per counter but no
cost, so the usage endpoints could only report units. The Bedrock hook now
stamps guardrail_cost_by_unit next to guardrail_usage, the spend-log
aggregator sums it into a new nullable cost column on
LiteLLM_DailyGuardrailUsageUnits, and /guardrails/usage/overview and
/guardrails/usage/detail/{id} return cost, totalCost and cost_by_unit /
cost_by_team / cost_by_key alongside the existing unit breakdowns.
Cost is nullable on purpose. Rows written before this migration, and rows
whose hook had no pricing entry, read as null rather than $0, and a single
unpriced increment keeps that row's cost unknown instead of partial.
guardrail_cost and the spend/budget path are untouched.
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
Two instances racing prisma migrate deploy on one database deadlock on
CREATE INDEX CONCURRENTLY: the victim gets P3018 with 40P01 and the
survivor then sees the failed ledger row as P3009. Both were treated as
unrecoverable, so neither instance came up.
Roll the deadlocked migration's ledger row back and retry the deploy on
P3018, consult the failed row's logs in _prisma_migrations to do the
same on P3009, and retry a deadlock reported without a Prisma error
code. Genuinely broken migrations still fail fast.
This reverts merge commit 2b1bd20834 (#31125)
Two CircleCI jobs on the staging-to-main promotion went red the moment
that PR landed. proxy_multi_instance_tests boots two proxies against one
database, and both now race the same migration:
Error: P3018 A migration failed to apply
Database error code: 40P01, deadlock detected
Process 73 waits for ShareLock on virtual transaction 4/11;
blocked by process 75. Process 75 waits for ExclusiveLock on
advisory lock [16384,0,72707369,1]; blocked by process 73
Neither proxy comes up, so the job times out after 300s waiting on
localhost:4000. The same wait took 36.5s on the last green run
Timeline: #31125 merged at 18:46:14Z and the failing run started at
18:49:59Z. The merge commit is not an ancestor of the last green
revision (194a3cc) and is an ancestor of the first failing one
(01de2837)
The v2 resolver was meant to avoid exactly this class of contention, so
the deadlock looks like a bug in it rather than a reason to abandon it.
Putting the default back to v1 buys time to fix it without holding up
the release
