From 306d699eee781390fea1d187f498450574e1cd18 Mon Sep 17 00:00:00 2001 From: Tin Chi Lo Date: Sat, 3 Oct 2026 12:42:59 -0700 Subject: [PATCH] docs(runbook): size the daily spend identity cutover window Adds the row-count query, the per-million-rows estimate measured on a 3M-row table, the one-transaction write-freeze note, and the migrate deploy timeout variable to the model-group identity cutover section. Co-Authored-By: Claude Fable 5.1 --- litellm-proxy-extras/migration_runbook.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/litellm-proxy-extras/migration_runbook.md b/litellm-proxy-extras/migration_runbook.md index 164952d3edf..e0b6305a4a2 100644 --- a/litellm-proxy-extras/migration_runbook.md +++ b/litellm-proxy-extras/migration_runbook.md @@ -98,6 +98,14 @@ When the guard fires: `20260922000000_daily_spend_model_group_identity` separates daily usage by the requested public model group. Schedule a maintenance window: the six replacement unique indexes scan existing daily spend tables and block writes while building +Size the window first. Count the rows the six builds will scan: + +```sql +SELECT relname, n_live_tup FROM pg_stat_user_tables WHERE relname LIKE 'LiteLLM_Daily%Spend' ORDER BY n_live_tup DESC; +``` + +Expect roughly 10 to 30 seconds per million rows per table on managed Postgres; a laptop built the 8-column index over 3 million rows in 17 seconds. The migration is one transaction, so each table's write lock is held until the last index commits: the write freeze is the sum of all six builds, while reads keep working throughout. The proxy runs `prisma migrate deploy` at boot, and the standalone job in `migrations/run.py` takes the same path, both under `LITELLM_PRISMA_MIGRATE_DEPLOY_TIMEOUT` (default 600 seconds). A timeout rolls the whole migration back and the retry starts from zero, so raise it above the summed estimate before upgrading a large installation + Drain incoming requests and all in-memory and shared Redis spend buffers, then stop every old proxy and spend collector before applying this migration. Start only upgraded writers after it completes. Old writers require the removed conflict target and cannot run alongside the expanded identity. The migration changes indexes only; previously merged attribution and legacy NULL groups remain as recorded PTU flat charges retain their deployment/day identity across public-model renames. New inference rows include the requested group in their identity. Rolling back requires stopping writers and reconciling group-separated rows before restoring the old unique indexes; simply starting an older image is unsupported