mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-05 02:41:56 +00:00
6553bc8957
A model write is a read-modify-write of the shared `llm_router` global: read the db into a snapshot, then make the router match that snapshot. Nothing serialized it, so two of them interleaving was not a lost update but an eviction -- _delete_deployment removes every live deployment absent from the snapshot it was handed, so the request holding the older snapshot reconciles the newer request's model straight back out of the router. The row survives in the db, which is what makes it easy to miss: the pod simply stops serving a model it was told to serve until some later reload happens to put it back. clear_cache compounds it. It deletes every db model from the router before reloading them, so for the width of that reload the pod serves none of them -- and any concurrent write sampling the router in that window sees the hole. Fix is one lock (MODEL_RECONCILE_LOCK) held across both, so each reconcile reads the db and applies it atomically and no stale snapshot can evict a newer model. clear_cache holds it across wipe+reload and calls the already-locked _add_deployment_locked, since asyncio.Lock is not reentrant and routing back through the public add_deployment would deadlock the pod's whole model-write path. The verdict needed the same treatment. raise_if_reload_degraded_serving compared a desired-set read during the reload against a router snapshot taken after it, so a neighbouring reconcile's in-flight wipe was reported to the caller as collateral damage from its own reload -- a 500 on a create that had in fact succeeded. Reconciles now return a ReconcileOutcome carrying both the desired set and the post-reconcile serving state, captured before the lock is released, and the verdict judges against that. Omitting live_after keeps the old live re-read, which stays correct for the no-reconcile-ran case. Found by running the e2e suite with pytest-xdist at 8 workers: three unrelated tests failed together on "Previously served model id(s) [...] are also no longer being served by this pod", which is this. Serial runs concurrent enough to hit it are rare, which is why 78 minutes of sequential e2e never surfaced it -- but any customer provisioning models in parallel (terraform, CI) is in exactly this race. test_reconciles_serialize_so_no_stale_snapshot_can_evict fails with 5 == 1 without the lock. |
||
|---|---|---|
| .. | ||
| management_v1 | ||
| policy_endpoints | ||
| scim | ||
| search_endpoints | ||
| usage_endpoints | ||
| test_access_group_endpoints.py | ||
| test_access_group_management.py | ||
| test_activity_tenant_scoping.py | ||
| test_auto_router_endpoints.py | ||
| test_budget_endpoints.py | ||
| test_cache_settings_endpoints.py | ||
| test_callback_management_endpoints.py | ||
| test_common_daily_activity.py | ||
| test_common_utils.py | ||
| test_compliance_endpoints.py | ||
| test_config_override_endpoints.py | ||
| test_coordination_redis_endpoints.py | ||
| test_cost_tracking_settings.py | ||
| test_credential_migration.py | ||
| test_customer_budget.py | ||
| test_customer_endpoints.py | ||
| test_delete_callbacks_endpoint.py | ||
| test_delete_verification_tokens_failed.py | ||
| test_encryption_endpoints.py | ||
| test_entraid_app_roles.py | ||
| test_gateway_request_endpoints.py | ||
| test_internal_user_endpoints.py | ||
| test_key_management_endpoints.py | ||
| test_mcp_management_endpoints.py | ||
| test_model_management_endpoints.py | ||
| test_org_admin_team_access.py | ||
| test_organization_endpoints.py | ||
| test_policy_endpoints.py | ||
| test_project_org_authz.py | ||
| test_router_settings_endpoints.py | ||
| test_saml_sso.py | ||
| test_tag_management_endpoints.py | ||
| test_team_callback_endpoints.py | ||
| test_team_default_params.py | ||
| test_team_endpoints.py | ||
| test_team_model_alias_merge.py | ||
| test_tool_management_endpoints.py | ||
| test_ui_sso.py | ||
| test_workflow_management_endpoints.py | ||