Every proxy process registers the same scheduler, and it registers it once per
uvicorn worker process rather than once per pod, so a job with a shared side
effect runs replicas times processes. The jobs that did elect an owner each
hand-rolled it, and the dialects disagree on the one case that matters:
acquire_lock reports contention and an unreachable Redis identically, so the
PTU rollup ran unguarded while key rotation, spend log cleanup and UI session
cleanup skipped, meaning a Redis outage silently stopped those three
everywhere at once.
Adds run_as_single_owner and claim_once_per_window, which name that decision
as WhenLockUnavailable and carry the two lease shapes the proxy actually uses:
a mutex held for one run, and a done-marker sized to a reporting window. The
lease is now renewed while the body runs, so the TTL is a failover deadline
rather than a run budget. On a live proxy that is the difference between a
spend log cleanup losing its 60s lock partway through a multi-minute sweep,
which a challenger could take at 75s, 120s and 180s, and holding it to
completion.
LITELLM_JOB_ROLE=serving registers none of these jobs, so an operator can run
them on a dedicated worker deployment instead of on every replica taking
traffic. A serving pod keeps the jobs that drain its own in-memory queues and
refresh its own model registry, so it still writes spend. The default is
unchanged, and an unrecognised value falls back to it with a warning.
Both Helm charts gain an opt-in worker Deployment, off by default and pinned
to one process. Live-Redis integration tests cover election, failover,
renewal, rolling restart and once-per-window claims.
Also fixes the ownership gauge, which only ever fired on a reentrant
re-acquisition and so was driven to 0 by every release and never back up.
2d0823abf7