mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-12 08:42:18 +00:00
1c68ec429a
* fix(proxy): bill an OpenAI pass-through background response once across repeated reads A finished background response now bills only on the first pass-through call that sees it finished with usage. The claim is an insert on the new LiteLLM_BackgroundResponseSettlement table, keyed by response id, so it holds across workers and replicas. Every later retrieve, cancel, streamed create, or resumed stream logs like an unbilled read. With no database or a failed insert the call still bills. * fix(proxy): keep pass-through background billing fail-open without prisma and skip foreground stream re-parse * fix(proxy): claim background responses on failed streams and bound the claim A resumed stream that fails after response.completed billed its usage through the partial-usage path without claiming the response id, so a later read billed it again. That path now claims the id and logs $0 when an earlier read paid. The claim uses create_many(skip_duplicates=True) so a repeat read no longer raises a unique violation in Postgres, and it waits at most 2 s before billing, since the claim now runs before the client gets its response. * fix(pass_through): keep the background response claim and its spend write together under cancellation and managed ids * test(pass_through): cover a failed stream whose failure logging cannot be scheduled * fix(proxy): start the background response settlement migration with a lock timeout * fix(passthrough): leave a failed background stream's charge to its next read A stream that failed after response.completed claimed the response id while its failure row billed $0, so no later read paid. The failure path is back to main's behavior, so the next finished read claims and bills once --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| dist | ||
| litellm_proxy_extras | ||
| tests | ||
| build_and_publish.md | ||
| LICENSE | ||
| migration_runbook.md | ||
| pyproject.toml | ||
| README.md | ||
Additional files for the proxy. Reduces the size of the main litellm package.
Currently, only stores the migration.sql files for litellm-proxy.
To install, run:
uv add litellm-proxy-extras
OR
uv tool install 'litellm[proxy]' # installs litellm-proxy-extras and other proxy dependencies
To use the migrations, run:
litellm --use_prisma_migrate