mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-20 00:11:50 +00:00
2d0823abf7
Every proxy process registers the same scheduler, and it registers it once per uvicorn worker process rather than once per pod, so a job with a shared side effect runs replicas times processes. The jobs that did elect an owner each hand-rolled it, and the dialects disagree on the one case that matters: acquire_lock reports contention and an unreachable Redis identically, so the PTU rollup ran unguarded while key rotation, spend log cleanup and UI session cleanup skipped, meaning a Redis outage silently stopped those three everywhere at once. Adds run_as_single_owner and claim_once_per_window, which name that decision as WhenLockUnavailable and carry the two lease shapes the proxy actually uses: a mutex held for one run, and a done-marker sized to a reporting window. The lease is now renewed while the body runs, so the TTL is a failover deadline rather than a run budget. On a live proxy that is the difference between a spend log cleanup losing its 60s lock partway through a multi-minute sweep, which a challenger could take at 75s, 120s and 180s, and holding it to completion. LITELLM_JOB_ROLE=serving registers none of these jobs, so an operator can run them on a dedicated worker deployment instead of on every replica taking traffic. A serving pod keeps the jobs that drain its own in-memory queues and refresh its own model registry, so it still writes spend. The default is unchanged, and an unrecognised value falls back to it with a warning. Both Helm charts gain an opt-in worker Deployment, off by default and pinned to one process. Live-Redis integration tests cover election, failover, renewal, rolling restart and once-per-window claims. Also fixes the ownership gauge, which only ever fired on a reentrant re-acquisition and so was driven to 0 by every release and never back up. |
||
|---|---|---|
| .. | ||
| test_base_update_queue.py | ||
| test_daily_spend_update_queue.py | ||
| test_pod_lock_manager.py | ||
| test_redis_update_buffer.py | ||
| test_spend_logs_partition_manager.py | ||
| test_spend_update_queue.py | ||
| test_tool_discovery_queue.py | ||