mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-30 06:43:27 +00:00
7045139c3a
* fix(router): stop concurrent deployment deletes from crashing deployment lookups delete_deployment and upsert_deployment shrink model_list before they repair the index maps, so a lookup on another thread in that window read a stale index: IndexError, or the neighbouring deployment and its credentials. An add racing a delete could corrupt the index for good. One RLock now covers model_list and the three index maps. Delete, upsert, add and the set_model_list reset change them under it, and every lookup that turns an index into a deployment reads under it. Fixes #45242 * test(router): release the held delete when the read reaches the index lock The race tests released the held write after a fixed 0.2 s, so a reader the runner stalled past it read the repaired list and passed on unfixed code. The read now runs on its own thread and the write stays held until the read returns or reaches the router's index lock; the 10 s bound only fails a broken synchronization. The integration test gains the audit cells: chat, streaming, Responses and Anthropic messages calls (sync and async, by model name and team public name) during a held delete, the same for upsert, add and set_model_list, a request for the deployment being deleted, a provider error, a double delete, and thread and async bursts during upsert churn. * test(router): cover the early return of a delete and the team filter of get_model_ids delete_deployment now returns None from inside the index lock, and get_model_ids by name reads its deployments through _deployments_named; pin both: deleting an unknown id returns None and leaves the list alone, and exclude_team_models drops only the team deployment. * fix(router): read deployments by name under the index lock outside the Router The response polling provider check, the complexity router's adaptive pool and the savings baseline turned the name index into list positions themselves, so a concurrent delete could still hand them a stale slot. They now read through Router.deployments_named, which takes the deployment index lock. * fix(router): type the locked deployment lookups as DeploymentTypedDict deployments_named and _team_deployments_named returned tuple[Any, ...]. They now return DeploymentTypedDict rows, which is what model_list holds. The adaptive pool setup in the Router and the complexity router drop their branches for pydantic Deployment rows, which these lookups never return, and should_include_deployment accepts the typed row. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| adaptive_router | ||
| complexity_router | ||
| __init__.py | ||
| test_auto_router.py | ||
| test_base_routing_strategy.py | ||
| test_budget_limiter.py | ||
| test_budget_limiter_hotpath.py | ||
| test_complexity_router.py | ||
| test_complexity_tier_predictor.py | ||
| test_fuse_presets.py | ||
| test_lar1_routing.py | ||
| test_least_busy.py | ||
| test_litellm_encoder.py | ||
| test_llm_v2.py | ||
| test_lowest_cost.py | ||
| test_lowest_latency.py | ||
| test_lowest_tpm_rpm.py | ||
| test_lowest_tpm_rpm_v2.py | ||
| test_quality_router.py | ||
| test_router_routing_groups.py | ||
| test_router_routing_plugins.py | ||
| test_router_tag_regex_routing.py | ||
| test_router_tag_routing.py | ||
| test_savings_baseline.py | ||
| test_simple_shuffle.py | ||
| test_stall_detector.py | ||