mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
c40828509b
* fix(reset_budget_job): advance budget_reset_at atomically with the spend cascade A postgres timeout mid-cascade previously left LiteLLM_BudgetTable rows stamped for the next window while team member, enduser, org and tag spend stayed at cap, so every later tick skipped them until the window rolled over. All cascade writes and the budget_reset_at advance now share one prisma batch transaction; a failed run persists nothing and the rows stay due for the next ~10 minute tick. Cache and counter invalidation runs only after commit, and the catch-all enduser log line now names the cascade. * fix(reset_budget_job): elect one runner per tick and chunk the reset scans Every pod and worker previously ran the reset job every ~10 minutes, each fetching every expired row with no limit and writing one giant transaction at the same calendar-aligned boundary; that concurrency is what piled up postgres lock contention and timeouts. The job now takes the shared PodLockManager redis lock (no redis keeps the old behavior), and each phase walks its due rows in 500-row chunks, one transaction per chunk, stopping when a chunk is short, makes no forward progress, or hits the per-run cap; leftovers wait for the next tick. * chore(lint): ratchet budget ceilings down for fixed violations * fix(reset_budget_job): harden chunk loop, fail open on redis errors, heartbeat the lock Review fixes on the two prior commits. Reset scans now skip rows with no budget_duration, so permanently due rows can neither starve a phase nor have a lifetime cap zeroed every tick. Chunk progress counts rows whose new budget_reset_at actually cleared the cutoff, so a zero-length duration cannot burn the per-run chunk cap. A failed lock acquire only skips the run when another pod verifiably holds the lock; a broken redis runs unguarded instead of silently disabling resets fleet-wide. Partial row failures report real progress and fire the failure hook without killing the phase. The leader re-asserts the lock between phases and stops if another pod took over, and the budget window advance uses update_many so a tier deleted mid-chunk cannot abort the transaction. Lint budget ceilings re-ratcheted for the net-fixed violations. * fix(reset_budget_job): renew the leader lease and reject non-positive budget durations Bot review follow-ups. PodLockManager now extends the lock TTL when the holding pod re-acquires, via an atomic compare-and-expire script with a plain SET fallback, so a run longer than the TTL keeps its lease instead of silently sharing the job with another pod. The positive-duration validation that team member endpoints already had is hoisted to management common_utils and applied to key, internal user, budget, customer and team intake, so a tenant can no longer create zero-duration budgets whose permanently due rows starve other tenants' resets. Such durations now return 400 at intake; existing rows are untouched. * refactor(reset_budget_job): defer leader election to a follow-up PR * fix(reset_budget_job): satisfy strict lint gates String defaults for the two getenv calls (PLW1508) and the chunk outcome returns moved to try/else (TRY300). |
||
|---|---|---|
| .. | ||
| management_v1 | ||
| policy_endpoints | ||
| scim | ||
| search_endpoints | ||
| usage_endpoints | ||
| test_access_group_endpoints.py | ||
| test_access_group_management.py | ||
| test_activity_tenant_scoping.py | ||
| test_auto_router_endpoints.py | ||
| test_budget_endpoints.py | ||
| test_cache_settings_endpoints.py | ||
| test_callback_management_endpoints.py | ||
| test_common_daily_activity.py | ||
| test_common_utils.py | ||
| test_compliance_endpoints.py | ||
| test_config_override_endpoints.py | ||
| test_coordination_redis_endpoints.py | ||
| test_cost_tracking_settings.py | ||
| test_credential_migration.py | ||
| test_customer_budget.py | ||
| test_customer_endpoints.py | ||
| test_delete_callbacks_endpoint.py | ||
| test_delete_verification_tokens_failed.py | ||
| test_encryption_endpoints.py | ||
| test_entraid_app_roles.py | ||
| test_gateway_request_endpoints.py | ||
| test_internal_user_endpoints.py | ||
| test_key_management_endpoints.py | ||
| test_mcp_management_endpoints.py | ||
| test_model_management_endpoints.py | ||
| test_org_admin_team_access.py | ||
| test_organization_endpoints.py | ||
| test_policy_endpoints.py | ||
| test_project_org_authz.py | ||
| test_ptu_model_settings.py | ||
| test_router_settings_endpoints.py | ||
| test_saml_sso.py | ||
| test_tag_management_endpoints.py | ||
| test_team_callback_endpoints.py | ||
| test_team_default_params.py | ||
| test_team_endpoints.py | ||
| test_team_model_alias_merge.py | ||
| test_tool_management_endpoints.py | ||
| test_ui_sso.py | ||
| test_workflow_management_endpoints.py | ||