mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
c76882b51b
* fix(auto-router): stop the embedding model's context window from failing long requests The auto-router embeds the last user message to pick a model and sent it to the embedding model unbounded. Embedding models carry 512 to 8k token windows while the chat models they route to carry 200k+, so any prompt over the encoder's window failed at the routing step with a 400 the destination model would never have raised. Cut every doc to a character cap inside LiteLLMRouterEncoder, which is the one choke point the auto-router, complexity-router, semantic guard and MCP tool filter all share. Default 2000 chars, roughly 500 tokens, which fits even a 512-token self-hosted encoder, overridable per deployment with auto_router_max_input_chars and globally with DEFAULT_MAX_EMBEDDING_INPUT_CHARS. Truncation alone cannot cover provider-side batch and byte limits, so any failure of the route call now falls back to the auto-router's default model instead of propagating. That path also fixes two latent bugs: a no-match left the auto-router alias in place as the model name, which fails downstream with "Unmapped LLM provider" rather than reaching default_model, and an empty route list raised IndexError. Fixes #17869 Fixes #20277 * fix(auto-router): make the embedding input cap opt-in so guards still see whole prompts Defaulting the cap inside the shared encoder truncated every consumer, not just the auto-router. The semantic guard builds the same encoder, so its pre-call check would have classified only the first 2000 characters while the full message still reached the model, which a benign opener in front of an injection payload walks straight past. The MCP tool filter and complexity router were silently narrowed the same way. The encoder now defaults to sending docs whole and cuts only when a caller passes max_input_chars. The auto-router is the only caller that does, so guard, MCP filter and complexity-router behaviour is unchanged from before this branch. DEFAULT_MAX_EMBEDDING_INPUT_CHARS becomes DEFAULT_AUTO_ROUTER_MAX_INPUT_CHARS, since it is now specific to the auto-router, and drops its env override: the per-deployment auto_router_max_input_chars already covers it, and every env var in constants.py has to be documented, which is what broke the documentation and code-quality checks. Also drops the added comments and the redundant type: ignore that review flagged. * test(auto-router): cover the max_input_chars wiring from litellm_params Nothing asserted that auto_router_max_input_chars on the deployment reaches the AutoRouter that embeds prompts. Dropping the wiring left every test green while the cap silently reverted to the default, so an operator with a 512-token embedding model could not lower it and every long prompt would fall back to the default model instead of being routed. * test(auto-router): cover the populated route-choice list branch The route layer can hand back a list, and picking its first element is where the IndexError lived: the empty case was covered but the populated one was not, so the branch that reads route_choice[0].name could be deleted with every test still green. |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||