Move the blocked-usage mapping for /v1/responses next to
blocked_response_usage in guardrail_translation utils, map bridged chat
prompt/completion tokens to Responses API input/output tokens, and let
raise_passthrough_exception attach the blocked response so post-call
guardrail blocks report real usage
* fix(prometheus): bound per-request budget metric emission with a timeout (#31632)
* fix(prometheus): bound per-request budget metric emission with a timeout
Wrap the per-request budget-metric gather in asyncio.wait_for so a slow Redis or DB lookup cannot consume the whole LoggingWorker watchdog and get the success-logging event cancelled. On timeout the emission is skipped in isolation; budget gauges are still refreshed by the periodic cron. The timeout is configurable via PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT and defaults to 5.0 seconds, falling back to the default on an invalid value instead of raising
* fix(prometheus): reject non-finite and non-positive budget-metrics timeout env
float() accepts 0, negatives, nan and inf, which bypass the fallback: a value <= 0 makes asyncio.wait_for time out immediately and skip every per-request emission, and inf reintroduces the unbounded wait the timeout was meant to bound. Validate the parsed value is finite and greater than zero before using it, otherwise fall back to the default
* fix: report the blocked LLM response's real token usage (#31217)
When a guardrail blocks a post-call response, the synthetic violation response
reported hard-coded zero usage, discarding the token usage the upstream call
had already consumed.
Fix the root cause rather than re-counting tokens:
- Add an optional `original_response` field to ModifyResponseException.
- The unified guardrail's post-call success hook attaches the blocked LLM
response to the exception.
- The /v1/messages and OpenAI-format (/v1/chat/completions, /v1/completions)
block handlers report `original_response.usage` directly. Pre-call blocks
never invoked the LLM, so usage is zero.
Mock-based tests cover the helper (returns original usage / zero), the success
hook attaching original_response, and the endpoint reporting it end-to-end.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guardrails): buffer + cleanly terminate streamed responses on block (#31389)
Streaming moderation improvements for the unified guardrail post-call
streaming iterator hook:
- streaming_buffer_until_moderated: withhold all chunks until end-of-stream
moderation passes, then release the original response (clean) or only the
block message (blocked) -- the original content is never delivered on a
block. Snapshot chunks with a shallow list() copy (end-of-stream builds a
separate assembled response; chunks aren't mutated in place).
- Clean Anthropic SSE on block: synthesize a well-formed termination sequence
instead of a bare data: {"error": ...} blob that truncates the stream.
Provider-specific synthesis lives in AnthropicMessagesHandler via
build_block_sse_chunks (format-agnostic routing stays in the hook).
- Mid-stream blocks continue the in-progress message (close open content
block, append block message, terminate) rather than emitting a second
message_start, which clients reject. Standalone envelope only when no chunks
were sent (buffered path).
- ModifyResponseException imported under TYPE_CHECKING + locally at runtime to
avoid a module-level cyclic import.
Adds regression tests for buffering (content withheld on block) and mid-stream
continuation (single message_start).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: report real usage on streaming blocks, disable buffered mode for content-rewriting guardrails
- _standalone_block_chunks and _block_continuation_chunks now read real
token usage from ModifyResponseException.original_response instead of
hardcoding zero, matching the non-streaming _blocked_response_usage path.
Shared helper moved to guardrail_translation/utils.py.
- streaming_buffer_until_moderated is now forced off when the guardrail has
mask_response_content=True, since buffered replay releases the withheld
original chunks verbatim -- unsafe for a guardrail that rewrites content
(e.g. PII masking).
- Fix inverted streaming-flag precedence comment.
* style: ruff format after greploop fixes
* fix: handle Anthropic streaming guardrail blocks
* fix(responses): check terminal event type for streaming guardrail end-of-stream detection
_check_streaming_has_ended assumed responses_so_far held ModelResponse
objects with .choices, but for the Responses API the accumulated chunks
are raw SSE event dicts, causing an AttributeError on every call
* fix: preserve Anthropic blocked stream usage
---------
Co-authored-by: FERNANDO IZAR <fizar@me.com>
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
