mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-13 08:52:02 +00:00
9a7f5eeec4
* fix(complexity_router): user_turn continuations no longer replay an earlier ask's pin A user_turn pin was stored per session, so when a new ask could not be pinned (classifier timeout, housekeeping excluded) its tool continuations replayed the previous ask's decision and could silently drop a hard task to the cheap tier Each pin now records a fingerprint of the ask that wrote it, and a continuation ignores a pin from a different ask. A new ask that cannot be pinned also overwrites the session pin with an empty one, which covers continuations whose history lives server side (previous_response_id) and carry no ask to fingerprint. Pins without a fingerprint keep replaying so rolling deploys stay consistent Fixes #44250 * fix(complexity_router): a new user_turn ask clears the previous ask's pin Replace the per-ask fingerprint with a simpler rule. In user_turn mode every new ask owns the session pin: a pinnable decision writes it as before, and one that cannot be pinned clears it, so the ask's continuations classify instead of replaying an earlier ask's model. This also covers continuations whose history lives server side (previous_response_id) Housekeeping calls never clear the pin. The check reads the classified decision before the health, modality and cache gates, which rewrite the cause of a failed-over title call Fixes #44250 * fix(complexity_router): hold the default model for a user_turn ask whose classifier failed Replace the clear-on-new-ask rule with a smaller change. In user_turn mode a classifier failure now pins the default model for that ask, so its tool continuations replay the default model instead of the previous ask's pin. The next new ask classifies again, so the hold ends with the ask, and session_affinity mode keeps never pinning a failed classification Fixes #44250 * fix(complexity_router): a plan-mode request never replays a pin without a tier A held default-model fallback has no tier when the default model sits outside the tier pools, so replaying it skipped plan_mode_min_tier. A plan-mode request now classifies instead of replaying a tier-less pin, which is what it did before the fallback was held --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| adaptive_router | ||
| complexity_router | ||
| switchyard_router | ||
| __init__.py | ||
| test_auto_router.py | ||
| test_auto_router_model_naming.py | ||
| test_auto_router_tuning_baseline.py | ||
| test_base_routing_strategy.py | ||
| test_baseline_request.py | ||
| test_budget_limiter.py | ||
| test_budget_limiter_hotpath.py | ||
| test_complexity_router.py | ||
| test_complexity_tier_predictor.py | ||
| test_fuse_presets.py | ||
| test_lar1_routing.py | ||
| test_least_busy.py | ||
| test_litellm_encoder.py | ||
| test_llm_v2.py | ||
| test_lowest_cost.py | ||
| test_lowest_latency.py | ||
| test_lowest_tpm_rpm.py | ||
| test_lowest_tpm_rpm_v2.py | ||
| test_quality_router.py | ||
| test_router_routing_groups.py | ||
| test_router_routing_plugins.py | ||
| test_router_tag_regex_routing.py | ||
| test_router_tag_routing.py | ||
| test_savings_baseline.py | ||
| test_simple_shuffle.py | ||
| test_stall_detector.py | ||