mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-25 01:02:15 +00:00
e7cd97c6b6
* fix(proxy): release unclaimed budget reservations at request end * fix(proxy): release unclaimed budget reservations of websocket sessions too * test(proxy): drop the structural middleware inheritance check * fix(proxy): claim the budget reservation on streaming pass-through before its cost callback The SSE chunk processor hands its success handler to the logging worker after the response, so the request-end release freed the reservation first and left the key unguarded until the worker drained. Claim it at both end-of-stream hand-offs, the immediate enqueue and the coroutine parked for deferred dispatch. Give the xai realtime test double the litellm_params attribute every real Logging object carries, since the wrapper now reads it. * test(pass-through): give the vertex streaming test doubles a litellm_params dict The spec'd Logging mocks in test_vertex_ai_anthropic_streaming_cost_injection.py lacked the instance attribute the chunk processor now reads to claim the budget reservation. Also restores main's _lazy_openapi_snapshot.json: the branch's copy had been regenerated under Python 3.14, which dedents one docstring description that the CI regeneration on Python 3.12 keeps indented, and the PR adds no lazily loaded route, so main's file is the correct one. * fix(pass-through): claim the budget reservation only after its cost callback is enqueued Every pass-through success hand-off stamped callback_bound before handing the coroutine to the logging worker. When that enqueue raised, the reservation stayed claimed with no callback left to reconcile it, so the request-end release skipped it and the reserved cost stayed pinned on the key's counter. Enqueue first, then claim, so a failed hand-off leaves the reservation for the request-end release. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| test_admission_control_middleware.py | ||
| test_billable_request_metrics_middleware.py | ||
| test_budget_reservation_release_middleware.py | ||
| test_in_flight_requests_middleware.py | ||
| test_per_request_root_path_middleware.py | ||
| test_prometheus_auth_middleware.py | ||
| test_prometheus_auth_middleware_asgi.py | ||
| test_security_headers_middleware.py | ||