mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
b8df48cd7f
* feat(auto-router): let operators replace the LLM classifier's system prompt The complexity router's LLM classifier has always sent one built-in rubric, so the router could only ever grade difficulty. Operators can now supply their own system prompt, which replaces the rubric outright and repurposes the same tier machinery for whatever taxonomy the prompt defines, data sensitivity being the obvious case. Replacement is total: neither the rubric nor its closing line is appended, since both describe grading difficulty over a "current message" and a prompt grading something else is entitled to contradict them. That closing paragraph is also the classifier's prompt-injection defense, so the config field and the dashboard editor both warn that a replacement omitting it lets a caller ask for a tier and get it. The heuristic fallback still scores complexity, which is meaningless for a repurposed taxonomy, so classifier_fallback now chooses between the heuristic scorer and routing straight to default_model. The default_model path bypasses tier pools, the adaptive bandit, and escalation, because no tier was decided and the point of that fallback is a known destination. It reports itself as default_model_fallback in the spend logs. The dashboard's prompt editor prefills from a new /auto_router/classifier/default_prompt endpoint rather than a copy of the rubric in the frontend, and stores no override when the draft matches the default, so later rubric improvements still reach every router that never customized it. Tier names stay SIMPLE/MEDIUM/COMPLEX/REASONING; a custom prompt redefines what they mean, not what they are called. * fix(complexity-router): don't let the default_model classifier fallback bypass routing plugins * fix(complexity-router): don't pin a session to the default model after a classifier failure * fix(complexity-router): omit the tier from a default-model-fallback routing decision The classifier never answered, so no tier was decided. The record reported the tier whose pool happens to hold default_model, which reads in the spend log and the UI as if the request was classified. Matches how default_fallback already records a route that no tier produced. * fix(proxy): allowlist /auto_router/ on the UI backend component The new GET /auto_router/classifier/default_prompt is a UI-consumed management route, so it belongs on the control plane. Without the prefix it was exposed by neither component and test_gateway_plus_backend_covers_full_app failed. * docs(ui): reword the classifier prompt disclaimer Frames the closing paragraph as a strong recommendation rather than a description of what gets dropped, names prompt injection explicitly, and notes the tier names stay fixed regardless of their display names. * fix(complexity-router): stop logging a fabricated tier on the plugin fallback path The classifier-failed fallback resolves a tier so the routing-plugin pipeline has a pool to filter, but nothing about the request produced that tier. The non-plugin short-circuit already dropped it from the logged decision; the plugin path still reported it, so a spend log claimed a classification the request never received. Record the pool as a plugin-filtered-pool signal instead. Also name the real problem when the resolved tier has no models at all: that raised "No candidate models left after routing-plugin filtering" and sent operators hunting for a policy plugin that never narrowed anything. |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||