mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-04 02:31:27 +00:00
397c84678b
* feat(router): opt-in session affinity for complexity router Complexity router reclassified every turn, which could flip the routed model group mid-session and break provider-side prompt caching. Add a session_affinity config flag: when a session_id is resolvable, pin the model chosen on the first turn and reuse it for the rest of the session, skipping reclassification. Pinned turns still stamp the adaptive bandit's chosen-model metadata so reward feedback keeps working when adaptive=True. * fix(router): refresh session-affinity TTL on hit, scope pin by API key Two issues from review: the TTL was only set on the first classification, so an active session outliving session_affinity_ttl_seconds silently lost its pin instead of refreshing as documented. And the cache key was scoped only by session_id, which is client-supplied and unauthenticated, so two different callers reusing the same session_id could poison each other's routing pin. Refresh the TTL on every cache hit, and namespace the cache key by the proxy-derived API key hash when available. |
||
|---|---|---|
| .. | ||
| adaptive_router | ||
| test_auto_router.py | ||
| test_base_routing_strategy.py | ||
| test_budget_limiter_hotpath.py | ||
| test_complexity_router.py | ||
| test_lar1_routing.py | ||
| test_quality_router.py | ||
| test_router_routing_groups.py | ||
| test_router_routing_plugins.py | ||
| test_router_tag_regex_routing.py | ||
| test_router_tag_routing.py | ||