mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-27 06:13:07 +00:00
f768fb32ef
* fix: Send the include_usage chunk with an empty choices array Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Skip chunks with no choices when reading streams Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Strip usage from plain-iterator async stream chunks Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Expect empty choices on the usage chunk in the gemma tracing test Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage The usage chunk now has choices=[] only when the caller set stream_options.include_usage itself. When the OpenAI handler, the proxy's always_include_stream_usage setting, or the harness added include_usage on the caller's behalf, the chunk keeps its legacy single empty-delta choice, so clients that read choices[0] on every chunk keep working. The proxy and harness mark their injection with the private _litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy. Plain-iterator streams keep usage on their chunk when usage was not requested, and the LLM Shield trailing flush ignores chunks with no choices. * fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them. * fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route * refactor(router): move the stream usage injection check into router_utils * fix(router): read stream_options from pydantic deployment params when marking injected usage Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices. * fix(proxy): keep the stream usage injection marker out of stored request bodies * test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths * test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| faked_responses_case.py | ||
| test_assistants_evals_stream.py | ||
| test_azure_streaming_function_calls.py | ||
| test_chat_stream_guardrail_error_events.py | ||
| test_decision_model_buffering.py | ||
| test_decision_model_pipeline_buffering.py | ||
| test_fake_stream_reasoning_wire.py | ||
| test_faked_responses_stream_bridges.py | ||
| test_faked_responses_stream_cache_replay.py | ||
| test_faked_responses_stream_chaos.py | ||
| test_faked_responses_stream_guardrail_blocks.py | ||
| test_faked_responses_stream_items.py | ||
| test_file_content_streaming.py | ||
| test_google_genai_stream_usage_settings_wire.py | ||
| test_interception_converted_stream_options.py | ||
| test_mid_stream_fallback_continuation.py | ||
| test_noma_v2_streaming_settings.py | ||
| test_openai_like_stream_usage_wire.py | ||
| test_openai_sdk_plain_chunk_fields.py | ||
| test_openai_stream_passthrough_fields.py | ||
| test_responses_bridge_stream_chaos.py | ||
| test_responses_bridge_stream_errors.py | ||
| test_responses_top_level_error_events.py | ||
| test_stream_contracts.py | ||
| test_stream_open_error_provider_headers_proxy.py | ||
| test_stream_parallel_slot_release.py | ||
| test_text_completion_stream_guardrail_rebuild.py | ||
| test_ttft_keepalive.py | ||