mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-09 22:31:41 +00:00
0175c7da1c
A client behind an auto-router sends one max_tokens for every tier, so a value sized for the smallest tier starves a bigger tier's thinking budget and a value sized for the biggest is rejected by the smallest. After the complexity router picks a tier, its per-tier litellm_params now carry max_tokens set to the smallest max_output_tokens across that tier model's deployments (model_info, then the cost map), applied the same way a per-tier reasoning_effort already is, on every routing exit including plan mode, the empty-ask default and the classifier fallback. The router seam collapses whichever ceiling alias a tier carries onto the surface's own name, so one tier max_tokens reaches chat, /v1/messages and /v1/responses alike, drops the caller's other carriers of the same setting before the merge, and stamps the caller's original once so a fallback into a group no tier owns gets it back instead of a ceiling sized for the tier that failed. Proxy-level reservations were sized from the caller's cap before routing, so a raised cap left them short. Both owners now re-validate at the deployment hook: the v3 limiter tops up its combined-TPM and project-OTPM reservations to the final cap or writes the admitted cap back, and the budget limiter re-estimates on the chosen deployment and grows the reservation or writes the admitted cap back. An auto-router alias also reserves budget at its priciest tier model now instead of pricing to zero. An explicit per-tier max_tokens, max_completion_tokens or max_output_tokens still wins, and max_tokens_from_tier_model: false forwards the caller's value unchanged. |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||