* fix(proxy): bill partial Responses API streams when the client disconnects
Streamed /v1/responses requests and the default-route OpenAI and Azure /v1/messages requests behind them logged a 499 with zero spend when the client went away mid-stream, while /v1/chat/completions already billed the chunks it had sent. The Responses API iterator now keeps the response.created payload and can hand the disconnect cleanup a response.incomplete event with counted usage and the served service_tier, and the messages-route byte stream and router wrappers delegate to it.
* fix(proxy): satisfy lint gates for partial Responses API disconnect billing
Make the disconnect helper public so the proxy, router and messages adapter can import it without reportPrivateUsage, type the SSE byte stream as an AsyncGenerator so aclose is known, read litellm_call_id off the logging object, and keep the new tests on Final locals with the recorder injected through dynamic_async_success_callbacks instead of litellm.callbacks.
* fix(router): bill the primary stream's tokens after a mid-stream hop and chat-bridge Responses streams on disconnect
The router's fallback wrapper now folds the primary Responses stream's relayed tokens into the fallback's disconnect event the way _combine_responses_fallback_usage does for a natural end of stream, and LiteLLMCompletionStreamingIterator (chat-only providers served through /v1/responses) builds its disconnect event from the chat completion chunks it collected. stamp_responses_usage_cost is now public so the bridge module can price that event.
* Potential fix for pull request finding 'CodeQL / Log Injection'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Backport of #47611 to stable/1.106.x
Cherry-picked from f0ae445cc4 (main)
* refactor(proxy): expose public names for private proxy helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep original class names behind public aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve internal callback filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep _PROXY_ class names for managed files hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names in package exports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve recursive auth helper name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align MCP limiter tests with server enforcement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): match main's MCP limiter tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names bound in importing modules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve compatibility imports through strict lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use exact pyright suppression in password helper test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): add reasons to compatibility import noqa comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update IN-list baseline for renamed helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): run end-of-stream post_call scan when the client disconnects mid-stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): close the guardrail stream chain in async_data_generator on client disconnect
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): leave the raw upstream response to the shielded finalizer on client disconnect
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep disconnect cleanup going when a streaming callback cleanup raises
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert the refund through a recorder instead of the mock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): inspect tool calls released before a disconnect under incremental_diff and record a failed scan marker
The incremental_diff transform stream now scans tool calls it already released when the client disconnects, and a disconnect scan whose translation raises after the guardrail recorded success also records guardrail_failed_to_respond
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): pin that a text-only disconnect scan is not handed a tool_calls finish
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover disconnect scans on every streaming endpoint and client, plus outage, worker-kill and cache-hit cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): prove the cache-hit twin is served from the cache
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): type the disconnect-close streams so basedpyright stops reporting unknown arguments
* fix(guardrails): give the guardrail metadata cast a reason so the type discipline gate accepts it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): scan released Messages and Responses tool calls on disconnect
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(guardrails): format the disconnect scan unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(guardrails): drop mutable-ok markers that no longer suppress a rule
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): scan released Responses output after a finished item and end only in-flight Chat choices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): type the disconnect scan test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): type request_data in the disconnect scan helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): type request_data in the disconnect scan test doubles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): pin that chat streams with no tool call in flight end as released
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): scan only released chunks on disconnect and skip it once a block owns the verdict
The disconnect scan now uses the chunks actually yielded to the client, copies them before scanning,
skips when a mid-stream block or HTTP error already settled the verdict, and the iterator wrapper
only closes hooks that are async generators so plain async iterator hooks keep working
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): pin that a delivered guardrail error or final chunk settles the disconnect verdict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): close any hook iterator that exposes aclose when the stream ends early
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): accept a synchronous aclose on custom streaming hook iterators
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): swallow callback aclose errors at end of stream
A custom callback whose async_post_call_streaming_iterator_hook returns a
non-generator async iterator with a raising aclose() failed the finished
stream: content plus usage reached the client and then the stream surfaced
an error SSE with no [DONE], or aborted a post_call pipeline's buffering
loop into a 500 with an empty body. Wrap the aclose invocation in
_wrap_streaming_iterator_with_enrichment in try/except and log a warning
naming the callback and the cleanup error, matching close_guarded_stream
and _close_guarded_layers. Iteration-time hook exceptions still propagate.
* fix(proxy): log only the error type when a callback aclose raises
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
