mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
b59f7993f9
* fix duplicate cost callbacks for anthropic streaming pass-through
Two bugs caused _PROXY_track_cost_callback to see stream=True +
complete_streaming_response=None on every streaming pass-through request,
making the dedup guard in dispatch_success_handlers permanently inactive:
1. pass_through_endpoints.py created the Logging object with stream=False
for all requests. _is_assembled_stream_success short-circuits on
self.stream is not True, so has_dispatched_final_stream_success was
never set and any second dispatch went through unchecked.
Fix: set logging_obj.stream = True after stream detection.
2. _create_anthropic_response_logging_payload set complete_streaming_response
inside the try block after litellm.completion_cost(), so a pricing error
caused an early return without setting it on model_call_details.
Fix: set complete_streaming_response before the try block.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix stream
* add stream to logging obj
* test(pass_through): give mock logging object a real model_call_details dict
The anthropic passthrough logging payload now records the assembled
response on model_call_details before cost calculation, which requires
model_call_details to support item assignment. In production it is always
a dict; the existing unit test stubbed the logging object with a bare Mock
whose attribute is not subscriptable, so the new assignment raised
TypeError. Use a real dict to match the production logging object.
* test(pass_through): cover streaming logging-obj stream flag
The streaming branch of pass_through_request that marks the logging object
as streaming (logging_obj.stream and model_call_details["stream"]) had no
unit coverage, so the patch coverage gate flagged it. Add a regression test
that drives a streaming pass-through request through pass_through_request and
asserts the logging object is flagged as a stream before dispatch.
* test(pass_through): cover SSE-response stream flag fallback branch
The auto-detected streaming branch of pass_through_request (when a request
that was not flagged as streaming returns a text/event-stream response) sets
logging_obj.stream and model_call_details["stream"] but had no unit coverage,
so the codecov patch gate failed at 60%. Drive a non-streaming pass-through
request whose upstream response is SSE through pass_through_request and assert
the logging object is flagged as a stream before dispatch.
* fix(pass_through): gate complete_streaming_response on stream flag
perform_redaction only scrubs complete_streaming_response when
model_call_details["stream"] is True. Setting it unconditionally for
non-streaming Anthropic pass-through responses left the assembled
response unredacted in model_call_details, which is handed to logging
callbacks as kwargs when message logging is disabled. Only record it for
actual streaming responses so redaction always applies.
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| llm_provider_handlers | ||
| test_llm_pass_through_endpoints.py | ||
| test_method_specific_routing.py | ||
| test_pass_through_endpoints.py | ||
| test_passthrough_auth_default.py | ||
| test_passthrough_endpoints_common_utils.py | ||
| test_passthrough_guardrails.py | ||
| test_passthrough_guardrails_field_targeting.py | ||
| test_passthrough_post_call_guardrails.py | ||
| test_streaming_handler_interrupt.py | ||
| test_vertex_ai_batch_passthrough.py | ||
| test_vertex_passthrough_load_balancing.py | ||