mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
2d0823abf7
Every proxy process registers the same scheduler, and it registers it once per uvicorn worker process rather than once per pod, so a job with a shared side effect runs replicas times processes. The jobs that did elect an owner each hand-rolled it, and the dialects disagree on the one case that matters: acquire_lock reports contention and an unreachable Redis identically, so the PTU rollup ran unguarded while key rotation, spend log cleanup and UI session cleanup skipped, meaning a Redis outage silently stopped those three everywhere at once. Adds run_as_single_owner and claim_once_per_window, which name that decision as WhenLockUnavailable and carry the two lease shapes the proxy actually uses: a mutex held for one run, and a done-marker sized to a reporting window. The lease is now renewed while the body runs, so the TTL is a failover deadline rather than a run budget. On a live proxy that is the difference between a spend log cleanup losing its 60s lock partway through a multi-minute sweep, which a challenger could take at 75s, 120s and 180s, and holding it to completion. LITELLM_JOB_ROLE=serving registers none of these jobs, so an operator can run them on a dedicated worker deployment instead of on every replica taking traffic. A serving pod keeps the jobs that drain its own in-memory queues and refresh its own model registry, so it still writes spend. The default is unchanged, and an unrecognised value falls back to it with a warning. Both Helm charts gain an opt-in worker Deployment, off by default and pinned to one process. Live-Redis integration tests cover election, failover, renewal, rolling restart and once-per-window claims. Also fixes the ownership gauge, which only ever fired on a reentrant re-acquisition and so was driven to 0 by every release and never back up. |
||
|---|---|---|
| .. | ||
| .coverage_baseline | ||
| __init__.py | ||
| _coverage_check.py | ||
| _pin_check.py | ||
| conftest.py | ||
| test_background_health.py | ||
| test_exception_handlers.py | ||
| test_harness_smoke.py | ||
| test_lifecycle.py | ||
| test_openapi_customization.py | ||
| test_proxy_config.py | ||
| test_routes_anthropic_beta.py | ||
| test_routes_assistants.py | ||
| test_routes_audio.py | ||
| test_routes_chat_completions.py | ||
| test_routes_completions.py | ||
| test_routes_config.py | ||
| test_routes_embeddings.py | ||
| test_routes_invitation.py | ||
| test_routes_login_sso.py | ||
| test_routes_misc.py | ||
| test_routes_model_cost_map.py | ||
| test_routes_model_info.py | ||
| test_routes_model_metrics.py | ||
| test_routes_models.py | ||
| test_routes_moderations.py | ||
| test_routes_onboarding.py | ||
| test_routes_queue.py | ||
| test_routes_threads.py | ||
| test_routes_utils.py | ||
| test_spend_counters.py | ||
| test_streaming_helpers.py | ||
| test_team_model_name_translation.py | ||