mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
db15ef3742
Staying on one model writes the prompt cache once and reads it thereafter. Switching leaves the new model cold, so it pays to write the whole prompt again, and that charge exists only because the router switched. Both arms were priced as if each model wrote the cache, which credited the baseline a cache-creation charge it would never have paid again. On a sonnet to haiku switch mid-conversation that phantom write was larger than the entire real cost of the request: the route lost $0.0104 and was reported as having saved $0.0179, with the sign inverted. The baseline is now priced as the warm cache a single-model deployment would have had, so the cold-cache write counts against the saving. A request that read nothing from cache is a genuine first turn the baseline would have paid to write too, so both arms still write there and cold-start savings stay honest. The result is signed rather than floored at zero. A cache-thrashing route is a real cost and the dashboard has to be able to report it; flooring per request would leave a number that can only ever go up and would hide exactly the routing behaviour an operator needs to see. The donut plots only drivers that saved, since a negative slice has no meaning, while the card and the range total keep the signed truth. usd() now sizes and signs off the magnitude so a small loss reads as -$0.0004 rather than $-0.00. |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||