mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-07 02:59:05 +00:00
f93d9b6b67
* feat(complexity_router): escalate oversized prompts to a tier that fits before dispatch The classifier scores complexity and never prompt size, so a long agentic session whose newest ask is trivial classifies SIMPLE onto a small-window tier and the provider rejects it with a context-window 400 that nothing retries. The gate runs after classification on every decision path (classify tail and session-affinity pin), estimates prompt tokens including the out-of-band carriers (top-level system, tools, instructions), and when the decided tier provably cannot hold the prompt moves the request to the lowest configured tier with a model whose declared window fits, restricting the pick to fitting models when the decided tier can keep it. Models with no resolvable window are never escalated away from or onto, escalated decisions are never written as session pins, and the decision records context_escalated plus the original tier in spend logs. Resolves LIT-6503 * fix(complexity_router): judge groups by smallest window, bound skips by bytes, filter adaptive picks Review-round rework, one mechanism per finding. A group is judged by its smallest resolvable deployment window, since the core router picks within a group with no fit check. The counting skip is gated on UTF-8 byte length, which BPE token counts can never exceed, so token-dense scripts cannot slip past it; only a real tokenizer count ever moves a request and a failed count leaves the placement alone. The fit facts now filter every adaptive phase including cold start and the tier fallbacks. Window questions adopt the declared provider and never resolve authenticating providers, and a router instance without get_model_list degrades the gate to a no-op. Tests rebuilt on real Router instances resolving deployment model_info end to end, plus a full-path test through async_get_available_deployment |
||
|---|---|---|
| .. | ||
| adaptive_router | ||
| test_auto_router.py | ||
| test_base_routing_strategy.py | ||
| test_budget_limiter_hotpath.py | ||
| test_complexity_router.py | ||
| test_lar1_routing.py | ||
| test_litellm_encoder.py | ||
| test_lowest_latency.py | ||
| test_quality_router.py | ||
| test_router_routing_groups.py | ||
| test_router_routing_plugins.py | ||
| test_router_tag_regex_routing.py | ||
| test_router_tag_routing.py | ||
| test_savings_baseline.py | ||