mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-04 02:31:27 +00:00
5cf269088c
* fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException The Bedrock-specific GuardrailInterventionNormalStringError predates the unified guardrails refactor and no proxy code path handles it, so a block with the flag set surfaced as an uncaught Exception -> HTTP 500 in pre_call mode and was silently discarded in during_call mode (model call proceeded in the parallel asyncio.gather; the block hook's data["mock_response"] mutation happened after route_request had already unpacked kwargs). Convert the block to ModifyResponseException at the raise site inside make_bedrock_api_request. That exception is the industry-standard proxy contract already caught in proxy_server, anthropic_endpoints, response_api _endpoints, and pass_through_endpoints; it turns into a 200 response with finish_reason=content_filter and the block message as content, which is exactly what the flag was documented to yield. Post-call blocks attach the LLM response to original_response so the synthetic reply reports the upstream call's real token usage instead of zero. Deletes the now-orphaned GuardrailInterventionNormalStringError class and the dead create_guardrail_blocked_response / mock_response plumbing in the Bedrock hooks; updates the existing tests that had locked in the buggy contract. Resolves LIT-4186 * chore(guardrails/bedrock): drop dead str branch in _update_messages_with_updated_bedrock_guardrail_response Follow-up to the disable_exception_on_block fix. That method used to receive either a BedrockGuardrailResponse or a plain string (the block message, when the flag was set). Now that a block always raises ModifyResponseException before this method runs, the string branch is unreachable; tighten the type to BedrockGuardrailResponse and delete the guard. * fix(guardrails/bedrock): streaming post_call block yields synthetic stream instead of surfacing as SSE 500 Regression from the LIT-4186 refactor: pre-refactor, the streaming post_call iterator caught GuardrailInterventionNormalStringError locally and replaced the assembled response with a synthetic content-filter message, then re-emitted it as chunks via MockResponseIterator. After the refactor the exception was re-raised as ModifyResponseException, which async_streaming_data_generator serializes as a proxy 500 error frame because the SSE response headers are already flushed by the time the block fires. Non-streaming paths still let ModifyResponseException propagate to the endpoint handler (which converts it into a 200). Streaming can't do that, so keep the local synthesis: on the exception, rebind the assembled response to a ModelResponse whose single choice carries the block message as content and finish_reason=content_filter, and let the downstream MockResponseIterator emit it as chunks. Same shape a non-streaming block produces. Adds a mapped-file regression test that mutation-kills the raise behavior and locks in the synthetic-stream contract. * fix(guardrails/bedrock): preserve upstream usage on streaming post_call block Non-streaming post_call blocks report the upstream LLM call's real token usage via ModifyResponseException.original_response, which the endpoint handler unwraps through _blocked_response_usage. Streaming post_call synthesizes its own ModelResponse locally (the exception can't escape the SSE generator), and previously left .usage unset, so the client saw accurate billing on non-streaming blocks and zero on streaming blocks -- silent revenue leak. Copy the assembled response's .usage onto the synthetic block response before yielding. Pre-refactor code had the same gap (create_guardrail_blocked_response never set usage); this is a net improvement, not a regression fix. * fix(proxy): capture logging_obj before post_call_failure_hook pops it in ModifyResponseException streaming path post_call_failure_hook removes litellm_logging_obj from request_data before iterating callbacks (it's not serialisable). The streaming branch of the ModifyResponseException handler read it from _data after that call, so it always received None and CustomStreamWrapper.__init__ crashed with AttributeError: NoneType has no attribute model_call_details. Capture it before the hook runs so the streaming path gets a valid object. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy): add regression for streaming ModifyResponseException logging_obj capture Covers the bug where logging_obj was read from request_data after post_call_failure_hook had already popped it, causing CustomStreamWrapper to crash with AttributeError. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy): drive real chat_completion in ModifyResponseException streaming logging_obj regression The original test inlined the fix pattern (capture before pop) in its own body rather than calling the actual chat_completion handler in proxy_server.py, so a revert of the fix left the test passing. Confirmed via mutation check: reverting the two-line source fix and re-running left the test green. Rewrite the test to drive chat_completion directly: - patch _read_request_body so chat_completion sees the seeded dict - patch ProxyBaseLLMRequestProcessing.base_process_llm_request to raise ModifyResponseException with the same request_data - patch proxy_logging_obj so post_call_failure_hook mutates the dict the way production does (pops litellm_logging_obj) - intercept CustomStreamWrapper.__init__ and assert logging_obj is the non-None object seeded in request_data Mutation-verified: reverting the source fix now surfaces the exact production crash inside CustomStreamWrapper's __init__ (AttributeError: NoneType has no attribute model_call_details) rather than a silently-passing test. Addresses Greptile P1 on PR #32665. --------- Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| guardrail_hooks | ||
| test_content_filter_path_traversal.py | ||
| test_content_utils.py | ||
| test_custom_code_security.py | ||
| test_deferred_guardrail_logging.py | ||
| test_guardrail_coverage.py | ||
| test_guardrail_endpoints.py | ||
| test_guardrail_registry.py | ||
| test_init_guardrails.py | ||
| test_llm_as_a_judge.py | ||
| test_mcp_jwt_signer.py | ||
| test_pillar_guardrails.py | ||
| test_prompt_security_guardrails.py | ||
| test_qostodian_nexus_guardrail.py | ||