mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
2d0823abf7
Every proxy process registers the same scheduler, and it registers it once per uvicorn worker process rather than once per pod, so a job with a shared side effect runs replicas times processes. The jobs that did elect an owner each hand-rolled it, and the dialects disagree on the one case that matters: acquire_lock reports contention and an unreachable Redis identically, so the PTU rollup ran unguarded while key rotation, spend log cleanup and UI session cleanup skipped, meaning a Redis outage silently stopped those three everywhere at once. Adds run_as_single_owner and claim_once_per_window, which name that decision as WhenLockUnavailable and carry the two lease shapes the proxy actually uses: a mutex held for one run, and a done-marker sized to a reporting window. The lease is now renewed while the body runs, so the TTL is a failover deadline rather than a run budget. On a live proxy that is the difference between a spend log cleanup losing its 60s lock partway through a multi-minute sweep, which a challenger could take at 75s, 120s and 180s, and holding it to completion. LITELLM_JOB_ROLE=serving registers none of these jobs, so an operator can run them on a dedicated worker deployment instead of on every replica taking traffic. A serving pod keeps the jobs that drain its own in-memory queues and refresh its own model registry, so it still writes spend. The default is unchanged, and an unrecognised value falls back to it with a warning. Both Helm charts gain an opt-in worker Deployment, off by default and pinned to one process. Live-Redis integration tests cover election, failover, renewal, rolling restart and once-per-window claims. Also fixes the ownership gauge, which only ever fired on a reentrant re-acquisition and so was driven to 0 by every release and never back up. |
||
|---|---|---|
| .. | ||
| actions | ||
| codeql | ||
| ISSUE_TEMPLATE | ||
| observatory | ||
| screenshots | ||
| scripts | ||
| workflows | ||
| ci-coverage-allowlist.yml | ||
| CODEOWNERS | ||
| dependabot.yaml | ||
| deploy-on-aws.png | ||
| deploy-on-gcp.png | ||
| deploy-to-aws.png | ||
| FUNDING.yml | ||
| pull_request_template.md | ||
| template.yaml | ||
