mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
bda4e60f9b
* fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions stream_chunk_builder_text_completion read finish_reason from chunks[-1] and recounted usage from messages. With include_usage the last chunk is a usage-only trailer, so finish_reason came back None (or raised IndexError when the trailer had no choices), and prompt_tokens was 0 because a text-completion prompt is not in messages. Scan for the last finish_reason and prefer the provider-reported usage, as the chat path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): type the text completion chunk builder so its scans add no basedpyright debt Read chunk fields as typed attributes instead of untyped subscripts, which clears the reportUnknownArgumentType breach the lint gate flagged * fix(streaming): keep the provider's full usage, including token details, when rebuilding text completions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): rebuild dict-form text completion chunks again stream_chunk_builder routes dict chunks holding TextChoices to the text completion builder, which now reads attributes. Convert them to TextCompletionResponse at the routing point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): accept dict chunks in the text rebuild and derive its usage like the chat path Moves the dict to TextCompletionResponse conversion into stream_chunk_builder_text_completion so direct callers keep working, and builds usage through ChunkProcessor.calculate_usage over the chunks that report counts, so a trailer without total_tokens keeps its prompt and completion counts and the result is a fresh Usage * fix(streaming): read text completion usage through the shared usage extractor so usage dicts on typed chunks rebuild * fix(streaming): merge text completion logprobs across chunks and rebuild dict text chunks as text completions * fix(streaming): rebuild one text completion choice per index and keep null logprobs fields null * fix(streaming): keep dumped chat chunks with a choice-level text on the chat rebuild path Route a dict chunk to the text completion builder only when its choice has `text` and no `delta`. The Responses bridge snapshots chat chunks with model_dump(), and Together's chat choices carry `text` next to `delta`, so streamed /v1/responses calls to Together ended without response.completed. Group text completion choices by index in one sorted pass, so rebuilding an n>1 stream costs about the same as rebuilding an n=1 stream of the same size. * fix(streaming): keep text completion rebuilds working on mixed chunks and unreadable logprobs A chat chunk inside a text completion stream now converts to a text chunk that keeps its finish_reason and adds no text. A dict chunk whose logprobs fail validation is read again with logprobs dropped, and logprobs that are not a Logprobs object keep the last value sent. Each of these inputs raised or lost data at the previous commit, and each now rebuilds without raising Also adds a Purview test showing a multi-choice text stream is blocked like the non-streamed response, and Final annotations on the new test locals * fix(streaming): rebuild text completion choices without an index at index 0 Text completion choices whose index is None now rebuild under index 0, as on the merge base, where the sort by index raised at the previous commit. A chat chunk inside a text completion stream now keeps the usage it reports, and a choice whose logprobs mix a Logprobs object with another shape keeps the last value sent, as on the merge base * fix(streaming): rebuild text completion choices with a string index under that int A text completion choice whose index is a string now rebuilds under the int it names, and one whose index cannot be read as an int rebuilds at index 0, as on the merge base. A string index mixed with an int index raised TypeError at the previous commit, and a string-only index came back as a string * test(streaming): drop a docstring that repeats its test * fix(streaming): fall back to choice 0 for an infinite text completion index A float infinity index made int() raise OverflowError, which the index coercion did not catch, so the rebuild failed where the merge base returned choice 0. --------- Co-authored-by: Lai Wei <8022184+roywei@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| conftest.py | ||
| test_aiohttp_session_rebuild_wire.py | ||
| test_anthropic_messages_bridge_mid_stream_failure_sdk.py | ||
| test_azure_prompt_shield_tuple_messages.py | ||
| test_bedrock_converse_redacted_thinking_router_sdk.py | ||
| test_bedrock_converse_stream_sync_decoder_wire.py | ||
| test_bedrock_model_discovery_sdk.py | ||
| test_bedrock_native_output_fallback_sdk.py | ||
| test_bedrock_native_output_register_model_sdk.py | ||
| test_cache_control_census_pydantic_messages_sdk.py | ||
| test_caching_redis_sdk.py | ||
| test_callback_logging_cache.py | ||
| test_databricks_json_schema_refs_sdk.py | ||
| test_device_code_login_guard_sdk.py | ||
| test_dual_cache_redis.py | ||
| test_exa_search_envelope_sdk.py | ||
| test_gemini_long_decode_sync_sdk.py | ||
| test_gemini_tts_speech_logging_sdk.py | ||
| test_generic_api_callback.py | ||
| test_google_genai_adapter_stream_usage_sdk.py | ||
| test_http2_wire.py | ||
| test_huggingface_tokenizer_downloads_sdk.py | ||
| test_idle_upstream_reconnect_sdk.py | ||
| test_interception_converted_stream_options_sdk.py | ||
| test_langfuse_redaction_sdk.py | ||
| test_ocr_matrix_sdk.py | ||
| test_openai_sdk_client_sessions_wire.py | ||
| test_owned_redis_ports.py | ||
| test_provider_budget_redis.py | ||
| test_provider_error_and_extra_body_families_wire.py | ||
| test_redis_breaker_metrics_multiprocess_sdk.py | ||
| test_redis_cluster_iam_auth.py | ||
| test_redis_service_metrics.py | ||
| test_response_cache_sync_sdk.py | ||
| test_responses_bridge_stream_errors_sdk.py | ||
| test_responses_bridge_sync_reasoning_sdk.py | ||
| test_responses_bridge_tool_search_sdk.py | ||
| test_router_behavior_wire.py | ||
| test_router_budget_limiter_wire.py | ||
| test_router_cooldown_fallback_sdk.py | ||
| test_router_include_fallback_errors_wire.py | ||
| test_router_lowest_latency_wire.py | ||
| test_router_redis_tls_url.py | ||
| test_router_sync_stream_fallback_wire.py | ||
| test_router_tool_call_stream_fallback_budget_wire.py | ||
| test_router_unsupported_params_fallback_wire.py | ||
| test_sagemaker_embedding_sdk.py | ||
| test_slack_daily_report_redis.py | ||
| test_stream_error_api_base_sdk.py | ||
| test_stream_open_error_provider_headers_sdk.py | ||
| test_sync_stream_connection_release_wire.py | ||
| test_text_completion_stream_rebuild_sdk.py | ||
| test_tool_loop_server_tool_replay_sdk.py | ||
| test_topaz_image_variation_trailing_slash_sdk.py | ||
| test_usage_based_routing_sdk_redis_reads.py | ||
| test_usage_routing_counter_ttl.py | ||
| test_vertex_batch_row_line_errors_sdk.py | ||
| test_vertex_gemini_image_batch_cost_sdk.py | ||
| test_vertex_gemini_stream_timeout_wire.py | ||
| test_vertex_partner_count_tokens_sdk.py | ||