mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-06 07:48:06 +00:00
8ab0321816
* fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed
A custom video provider can report a video's cost on the status poll that
sees it completed. Every poll of a video shared the video id as its spend
log request_id, so the paid poll's row was dropped as a duplicate and every
later poll of the finished video charged the cost again.
Video status rows now carry their own `_video_cost` request_id, and a poll
that reports a cost goes through the same settle-once claim batches use, so
the first poll that sees the video completed charges it and writes the row,
and every later poll charges nothing.
* fix(proxy): charge a video's provider-reported cost once on model, router and session budgets too
A video status poll gets the _video_cost row suffix only when it reports a cost, so a $0 poll keeps the bare video id and adds no row of its own, and the spend log writer claims a row by that suffix.
The batch charge-once marker now covers videos, keyed on the video id under {spend_key}:video:{id}, with the batch marker key unchanged. Model and router budgets use it, and per-session budgets now charge a repolled batch or video once and ignore a cost that is not a finite number
* fix(proxy): charge a finished video's own cost once, keyed by the id the client polled
Guardrail cost on a running poll no longer takes the charge-once claim, only a completed poll charges, the session marker TTL refreshes like the others, and two deployments whose provider ids match each keep their own charge.
* fix(proxy): charge repeat poll guardrails, key video charges by canonical id and survive redaction
* fix(proxy): keep video status polls in-process for the collector and give guardrail-only job polls an unclaimable row id
* fix(proxy): keep repeat batch polls free and group guardrail-only batch rows under their batch's trace
* fix(proxy): settle a finished job whose own cost is below its poll's guardrail cost under the job's charge row
* test(proxy): count charged_poll_cost as covered through the finished job guardrail poll tests
* fix(proxy): settle a finished batch that cost nothing itself on its first priced retrieve
* fix(proxy): charge no budget for a batch retrieve before its cost is final, and read the retrieved batch from before redaction
* fix(proxy): charge sessions a poll's guardrail-inclusive cost, count a job cost row's guardrail run, and keep batch_cost_is_final importable from litellm.batches
* fix(proxy): claim a job and charge the session budget in one Redis call
The session limiter claimed the job marker and then raised spend in a second Redis call. A failure between the two left the job claimed while its charge sat only in local memory, so later polls skipped it. One script now does both, and a refused script claims and charges locally
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| litellm_skills | ||
| __init__.py | ||
| test_async_post_call_streaming_iterator_hook.py | ||
| test_autorouter_baseline_cache.py | ||
| test_banned_keyword_list.py | ||
| test_batch_enqueued_tokens.py | ||
| test_batch_file_validation.py | ||
| test_batch_rate_limiter.py | ||
| test_dynamic_rate_limiter.py | ||
| test_dynamic_rate_limiter_v3.py | ||
| test_image_generation_guardrails.py | ||
| test_key_management_event_hooks.py | ||
| test_max_budget_per_session_limiter.py | ||
| test_max_iterations_limiter.py | ||
| test_model_budget_shared_windows.py | ||
| test_model_max_budget_limiter.py | ||
| test_parallel_request_limiter.py | ||
| test_parallel_request_limiter_v3.py | ||
| test_post_call_failure_hook_integration.py | ||
| test_post_call_response_headers_hook.py | ||
| test_post_call_streaming_hook_integration.py | ||
| test_post_call_success_hook_integration.py | ||
| test_prompt_cache_observer.py | ||
| test_prompt_injection_detection.py | ||
| test_proxy_hooks_init.py | ||
| test_proxy_rate_limit_provider_field.py | ||
| test_proxy_track_cost_callback.py | ||
| test_rate_limiter_toctou.py | ||
| test_search_free_tier.py | ||
| test_send_invite_email.py | ||
| test_sensitive_data_routing.py | ||
| test_tpm_concurrent.py | ||
| test_unit_test_max_model_budget_limiter.py | ||
| test_user_management_event_hooks.py | ||