mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-15 04:19:08 +00:00
2d4472efc1
* fix(vllm): write spend log rows for /vllm passthrough calls to router models The vLLM passthrough config never built a logged response for relayed calls, so stream and non-stream calls through /vllm with a router model wrote no spend row and never charged the key. vLLM now shares the OpenAI-shaped relay logging that Azure already had, moved into an openai_like helper both configs call. * ci: run the vllm passthrough unit tests in the llms-providers shard * fix(proxy): log /vllm text completions and ask router models for stream usage A chat stream relayed to a router model without stream_options now gets include_usage added on the way out, and the usage chunk vLLM appends is removed before the client sees it, so the spend row carries vLLM's own token counts instead of a local estimate. /vllm/completions calls now write a row too, stream and non-stream. * fix(vllm): keep usage, fingerprints, and disconnect rows correct on /vllm streams Inject and strip stream usage on /vllm/completions too, hold every finish frame for the fingerprint when n > 1, pass a forced-usage server's stream through untouched, stop turning continuous_usage_stats alone into per-chunk usage, close the upstream when the client hangs up so the row is written, and assemble text choices with one sort instead of a scan per choice * fix(vllm): keep n>1 stream order when holding finish chunks for the fingerprint A held finish chunk now goes out as soon as any other chunk arrives, so a choice that finishes while another still streams keeps its place. Only the finish chunks right before the usage chunk get vLLM's system_fingerprint. * fix(vllm): keep caller stream_options and log both success handlers Forward a stream_options the proxy cannot read, or a non-boolean include_usage, to vLLM as the caller sent it. Only a false always_include_stream_usage changes the /vllm relay, so clients get the same stream bytes under every setting. Record the passthrough origin when logging starts so the sync and async success handlers both parse the relayed body, hand callbacks a parsed text completion, and read relayed messages and prompt separately so a stray field no longer drops the local count. Add the audit's integration tests for the relay, the settings, and the Azure spend row. * test(vllm): expect the caller's stream under always_include_stream_usage true * fix(vllm): relay long SSE frames in linear time * test(vllm): time the relay's usage and finish_reason prefilter on one large frame --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| actions | ||
| assets/scaledown | ||
| codeql | ||
| e2e-stack | ||
| ISSUE_TEMPLATE | ||
| observatory | ||
| prompts | ||
| PULL_REQUEST_TEMPLATE | ||
| screenshots | ||
| scripts | ||
| workflows | ||
| ci-coverage-allowlist.yml | ||
| CODEOWNERS | ||
| dependabot.yaml | ||
| deploy-on-aws.png | ||
| deploy-on-gcp.png | ||
| deploy-to-aws.png | ||
| FUNDING.yml | ||
| issue-labels.json | ||
| mutmut-coverage.rc | ||
| template.yaml | ||
