mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
4897f6ec88
Three defects, one cause: the driver was wired by hand at each stage of the savings pipeline instead of going through the owner of each stage. Pricing re-derived per-token arithmetic instead of calling the cost engine. `prompt_tokens` already includes cache-read and cache-creation tokens, so charging the whole total at the flat input rate and then subtracting a separate cache-write penalty priced those tokens twice. Both arms now price the identical usage through `generic_cost_per_token`, so each token is charged once in its own dimension and tiered rates, ephemeral cache-write tiers and regional uplifts stay consistent with what the request was actually billed. On a cache-heavy opus-to-haiku switch the old formula reported $0.0458 against a true $0.0717. The savings baseline was recorded by a second per-attempt writer sitting beside the routing decision, and the exit path that runs when no pre-routing strategy applies only cleared the decision. A fallback to a plain model group therefore kept the previous attempt's baseline, letting a caller who forces a router failure inflate the recorded savings. Both facts now travel from one response through one recorder, so no exit can clear one and leave the other. Aggregation summed every daily metric except this one, so two requests sharing a rollup key kept only the first value, and since the cross-pod Redis drain runs the same merge the field was dropped on every flush. The driver was also absent from the entire read path: no column in the rollup query, no accumulation, no field on the response model. The dashboard read a key the API never sent, so the card, donut segment and graph series would have rendered $0.00 forever however much routing saved. Wiring the write path without the read path is the failure this had already shipped, so the drivers are now enumerated from the response model itself and each is asserted to be summed, accumulated, carried and totalled. Also collapses the twelve hand-written per-field blocks in the daily upsert into one enumeration feeding both the create and the increment. |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||