mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
c8bec20443
The ComplexityRouter's LLM classifier saw only the last user message, so on a multi-turn conversation it classified whatever happened to be last rather than what the human actually asked, and a near-constant classifier input pinned a whole session to one tier. The blindness turned out to be narrower than first diagnosed, and the fix is correspondingly smaller. Tool output was never the problem: on the Messages surface it rides a user turn as tool_result content blocks, which are not text parts, so flattening to `type == "text"` already dropped those turns; on chat completions it arrives on a `tool` role the extractor never read. Both surfaces were already handled before this change. What actually leaked through was the harness `<system-reminder>` block, which arrives as ordinary text, survives flattening, and became the current ask on any turn that carried one. So reminders are stripped rather than used to reject the turn, because a harness injects them alongside the live ask and not as a turn of their own; rejecting the turn would lose the ask, and keeping the block would feed the classifier the near-constant boilerplate that flattens tier selection in the first place. An earlier revision of this change also pattern-matched serialized tool_result payloads. That check only ever fired on a hand-serialized string neither request surface produces, it was where every review finding in this PR lived, and it is deleted here; the tests now pin the real shapes instead of the synthetic one they were built on. The classifier call is split into a system role carrying the rubric plus the caller's own system prompt, which stays byte-stable across a session so a provider can prompt-cache it, and a user role carrying the variable context: a bounded window of prior user turns, a conversation-depth signal, and the current ask. The caller's system prompt rides every turn, so task constraints are never dropped. The depth signal measures content-parts messages too, since counting only string content reported ~0 tokens for exactly the deep Messages-surface conversations that most need an expensive tier, and it is omitted entirely on the prompt-only path rather than asserting a false zero. Prior turns are excluded by matching the current ask rather than by dropping the newest turn positionally, because `aclassify` takes `prompt` and `messages` separately and a caller may classify something other than the newest turn. Truncated turns carry a marker so the classifier can tell a turn was clipped. Only the LLM classifier's input changes. The heuristic scorer, keyword overrides, escalation matching and semantic embedding still read the extracted current ask, which is why that extraction has to yield one clean human-authored string: those are substring and vector matchers, and an escalation keyword sitting inside a reminder blob would otherwise trip a tier jump on its own. Defaults keep single-turn classification equivalent to before. The prior-turn window is on by default so existing LLM-classifier deployments actually get the fix; the config field documents that those turns reach the classifier model, which may be a different provider than the routed completion model, and that the call already carries the current ask and the caller's system prompt in full. Scoped to the ComplexityRouter; the semantic AutoRouter is not touched.
29 lines
329 B
JSON
29 lines
329 B
JSON
{
|
|
"LIT001": {
|
|
"limit": 23253
|
|
},
|
|
"LIT002": {
|
|
"limit": 27427
|
|
},
|
|
"LIT003": {
|
|
"limit": 292
|
|
},
|
|
"LIT004": {
|
|
"limit": 44
|
|
},
|
|
"LIT005": {
|
|
"limit": 0
|
|
},
|
|
"LIT006": {
|
|
"limit": 1108
|
|
},
|
|
"LIT007": {
|
|
"limit": 0
|
|
},
|
|
"LIT008": {
|
|
"limit": 1004
|
|
},
|
|
"LIT009": {
|
|
"limit": 2474
|
|
}
|
|
}
|