mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-06 02:48:13 +00:00
47bba14336
* fix(passthrough): parse Bedrock stream spend incrementally instead of buffering the whole response Bedrock pass-through streaming kept every relayed chunk in memory until EOF and then decoded, parsed and translated the whole stream again for spend logging. Large or concurrent streams could exhaust proxy worker memory. Sync and async passthrough wrappers now hand each chunk to a provider stream collector as it is relayed. Bedrock decodes event-stream frames incrementally, folds consecutive text deltas, and keeps only what stream_chunk_builder needs for usage, tool calls and metadata. Text deltas are no longer retained in the Bedrock and Anthropic stream decoders either. Providers without a collector keep the previous raw-bytes behavior. Collector failures are isolated so spend tracking can never interrupt the customer stream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(passthrough): assert the spend payload the collector builds instead of mock internals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(passthrough): type the Bedrock collector helpers by the collector protocol instead of asserting the class Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| batches | ||
| chat | ||
| count_tokens | ||
| embed | ||
| files | ||
| image | ||
| image_edit | ||
| invoke_agent | ||
| messages/invoke_transformations | ||
| passthrough | ||
| realtime | ||
| rerank | ||
| search | ||
| vector_stores | ||
| __init__.py | ||
| event_loop_probe.py | ||
| test_anthropic_beta_support.py | ||
| test_base_aws_llm.py | ||
| test_bedrock_common_utils.py | ||
| test_bedrock_ssl_verify.py | ||
| test_claude_platform_provider.py | ||
| test_converse_context_management.py | ||
| test_cross_region_inference_profile_mapping.py | ||
| test_mantle.py | ||
| test_nova_imported_models.py | ||
| test_request_metadata.py | ||
| test_web_identity_session_policy.py | ||