mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
ac1cc5111c
* fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions * fix(proxy): check the key's max_budget on /queue/chat/completions * fix(proxy): run budget, tag, and request policy checks on queued chat calls * fix(proxy): bill a dropped queued stream and run the trace ID and failure logging checks on queued calls A queued stream now hands its logging object to the stream generator, so a client that leaves mid-stream is billed for the reply it already got, and a stream left before the first model chunk releases its budget reservation. A team's require_trace_id and the failure logging for rejected calls now use the same route check as the other inference checks. * fix(proxy): settle a dropped queued stream's budget hold through the first-chunk step chat uses * fix(proxy): keep the trace ID, call ID, unpriced fallback filter, and JWT admin team header on queued chat calls * fix(proxy): send the call ID on a queued call the provider rejects before streaming * fix(proxy): release a dropped stream's budget hold only when nothing bills it A guardrail that buffers the stream can pull model chunks the client never saw. The early-drop release settled the hold at the input cost first, then the partial bill for those chunks added its full cost on top. The release now runs after partial billing and only when no bill fired * fix(proxy): fold the caller's litellm_metadata into a queued call's metadata A queued call that sent a litellm_metadata bucket, even an empty one, made tag routing and logging read that bucket, so the key and team tags the handler merged into metadata no longer narrowed deployments. The queue handler now folds the bucket into metadata and drops it, as chat does, after stripping the proxy-owned user_api_key_* and internal control fields from both buckets, so a caller can no longer set the queued spend row's organization ID either * fix(proxy): release a queued call's budget hold when the call is cancelled before it answers * fix(proxy): refund a cancelled queued call's hold in full until the model call claims it * fix(proxy): keep a cancelled queued call's hold release running when the call is cancelled again --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| .coverage_baseline | ||
| __init__.py | ||
| _coverage_check.py | ||
| _pin_check.py | ||
| conftest.py | ||
| test_background_health.py | ||
| test_exception_handlers.py | ||
| test_harness_smoke.py | ||
| test_lifecycle.py | ||
| test_openapi_customization.py | ||
| test_proxy_config.py | ||
| test_routes_anthropic_beta.py | ||
| test_routes_assistants.py | ||
| test_routes_audio.py | ||
| test_routes_chat_completions.py | ||
| test_routes_completions.py | ||
| test_routes_config.py | ||
| test_routes_embeddings.py | ||
| test_routes_invitation.py | ||
| test_routes_login_sso.py | ||
| test_routes_misc.py | ||
| test_routes_model_cost_map.py | ||
| test_routes_model_info.py | ||
| test_routes_model_metrics.py | ||
| test_routes_models.py | ||
| test_routes_moderations.py | ||
| test_routes_onboarding.py | ||
| test_routes_queue.py | ||
| test_routes_threads.py | ||
| test_routes_utils.py | ||
| test_spend_counters.py | ||
| test_streaming_helpers.py | ||
| test_team_model_name_translation.py | ||