scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-13 23:11:40 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 101

fix(auto-router): eliminate quadratic scan in reachability aggregation

Browse source 
Changed reachable_by_key construction from nested-loop dict comprehension O(keys*rows)
to single-pass accumulation O(rows). The original code scanned router_key_rows once per
distinct api_key to collect models for that key; with 100,000 rows and many distinct
keys, this caused ~10B comparisons.

Now builds the mapping in one pass by iterating rows once and accumulating their models
by api_key, then converts to frozenset at the end.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Abhimanyu Kapur 2026-08-08 13:08:54 -07:00
parent 941fd47804
commit b7f53ccded
1 changed files with 6 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
litellm/proxy/management_endpoints/auto_router_endpoints.py
View file

@ -552,12 +552,12 @@ def _quality_signals_for(
baseline_rows: Final = tuple(row for row in turns if row.router_name is None and row.api_key in router_keys)
router_key_rows: Final = tuple(row for row in turns if row.api_key in router_keys)
reachable_by_key: Final = MappingProxyType(
{
key: frozenset(row.model for row in router_key_rows if row.api_key == key)
for key in frozenset(row.api_key for row in router_key_rows)
}
)
reachable_by_key_dict: dict[str, set[str]] = {}
for row in router_key_rows:
if row.api_key not in reachable_by_key_dict:
reachable_by_key_dict[row.api_key] = set()
reachable_by_key_dict[row.api_key].add(row.model)
reachable_by_key: Final = MappingProxyType({key: frozenset(models) for key, models in reachable_by_key_dict.items()})
reachable_models: Final = tuple(frozenset(row.model for row in router_key_rows))
ranks: Final = rank_models_by_cost(llm_router, reachable_models) if llm_router is not None else MappingProxyType({})