mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
347 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yucheng-berri
|
d4dc2c39e7
|
fix(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing (#36119)
* feat(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing
AWS's ApplyGuardrail API rejects requests whose content exceeds the
account's per-request "maximum input size in text units" quota with a
400 ValidationException. That cap is account/region/policy-dependent
and cannot be predicted from config, so it can only be reacted to.
_make_apply_guardrail_request now tries the whole-content call first
(no behavior change for requests that already fit). On a too-large
ValidationException it bisects the flat content list and retries each
half sequentially, recursing until every piece fits or cannot be split
further, then merges the per-chunk responses (action, assessments,
outputs, usage) into one so callers cannot tell chunking happened. A
real guardrail block on any (sub-)chunk still raises immediately.
Contextual-grounding requests are never chunked: grounding scores the
response holistically against the whole reference source, so
fragmenting it would produce misleading scores.
Each chunk call also gets a small exponential backoff retry on AWS
ThrottlingException (429), since chunking increases the number of
per-second API calls and can trade a 400 for a 429.
All new state is local to a single request's call stack (no shared
cache, no cross-process coordination), so this is safe for
single-pod, multi-pod, and cache-less LiteLLM proxy deployments alike.
* fix(guardrails): address Bedrock ApplyGuardrail chunking review feedback
Fixes three issues flagged in review of the chunking fallback: a single
oversized content item couldn't be split (only list-length bisection was
supported), a chunked request that got recovered still logged a stray
failure telemetry entry alongside the real outcome, and flattening chunk
outputs without positional bookkeeping could misalign masked text onto
the wrong original message once a chunk had nothing to mask.
* test(guardrails): add regression test for multi-level Bedrock guardrail chunking
Confirms the too-large bisection recursion isn't capped at a single split:
a payload that is still oversized after the first halving keeps splitting
until every piece fits, converging on however many chunks it takes rather
than only ever producing two.
* fix(guardrails): hybrid bin-pack+bisection chunking, whitespace-safe splits
Rework Bedrock ApplyGuardrail chunking from pure reactive bisection to a
hybrid strategy: bin-pack content into fixed-budget batches up front as
the fast path, falling back to the existing recursive bisection only for
a batch AWS still rejects as too large. Avoids paying O(log n) round
trips on every oversized request when a single pass would do.
Also switch single-item text splitting from a raw character midpoint to
the nearest whitespace boundary, so a fragment never starts or ends
mid-word. Closes the accidental-severing case from review; the residual
gap (a multi-word denied phrase deliberately straddling the boundary) is
documented as an accepted limitation, since fixing it would require an
overlap window reconciled against masked output with no documented
length-preservation guarantee from AWS.
* chore(ui): regenerate dashboard API types
* fix(guardrails): don't retry an oversized Bedrock guardrail call as a throttle
AWS reports an ApplyGuardrail request that exceeds the per-request
text-unit cap as a ThrottlingException (429), not only as the documented
ValidationException (400). Verified against a live guardrail with an
active content-filter policy: a 3273-text-unit request comes back as
"Input text size (3273 text units) exceeds the maximum allowed (1000 text
units) for the content filter policy (Classic tier)".
The throttle retry keyed off status 429 alone, so every oversized chunk
burned the full backoff-retry budget - each attempt a billed AWS call
preceded by a sleep - before the bisection fallback got a chance, at every
level of the recursion. A size error is not transient; re-posting the same
content can never succeed. It now short-circuits straight to bisection.
Also rename _is_input_too_large_validation_error to
_is_input_too_large_error (it never keyed off the status code, and the
error is not always a ValidationException), correct the docstrings that
asserted a 400, and log at warning level when a split happens so the
recovery is visible without --detailed_debug.
* Revert "chore(ui): regenerate dashboard API types"
This reverts commit ebf8ba2fd57f13bccf7aa6c5dfcac41c74db1ed9.
* fix(guardrails): group all fragments of one item and stop double-logging
Two defects found in review, both invisible to the existing tests.
Fragment grouping assumed a split content item always produces exactly two
adjacent fragments. That holds for one bisection level but not two: an item
split twice yields four fragments, which were regrouped in fixed pairs into
two output entries for a single message. Since masking walks the merged
outputs by a running index across the original, unchunked message list, that
message was written back truncated to its first half and every later message
shifted. Fragments now carry the size of the group they belong to, so any
number of them collapse back into exactly one output entry.
Telemetry was also double-counted. AsyncHTTPHandler.post calls
raise_for_status(), so every non-200 from Bedrock reaches _sign_and_post's
error path, which logged guardrail_failed_to_respond before re-raising as an
HTTPException that the consolidating caller then logged again. A request
recovered by chunking reported one failure per rejected attempt plus a
success. The ApplyGuardrail path now opts out of that per-attempt logging,
since it owns consolidated per-request logging; the connection-level branch
still logs, as nothing else records it.
The existing tests missed both because their mocks return a non-200 response
object, while the real client raises. Added a helper that raises a genuine
httpx.HTTPStatusError so these paths are covered the way production hits
them, plus a case asserting an unrecoverable failure still logs exactly once
rather than zero times.
* refactor(guardrails): move Bedrock chunking rationale into docstrings
The chunking work explained itself with inline comment blocks, which this
repo's conventions do not want. Folded that reasoning into the docstrings of
the functions it describes and dropped the comments, including the
module-level constant blocks and the test-file banner.
No behavior change. The banner also claimed AWS rejects an oversized request
with a 400 ValidationException, which live testing disproved, so removing it
drops a stale claim as well as an internal ticket reference from a public repo.
* feat(guardrails): match AWS default chunk budget and make it configurable
ApplyGuardrail's default quota is 25 text units, roughly 25,000 characters,
per second. Chunking has to respect that throughput limit rather than just the
per-request size, otherwise splitting an oversized request trades a size error
for a throttle. The budget now defaults to 25,000 to match that default for
every user, up from an arbitrary 20,000.
Accounts with raised quotas can spend fewer calls by setting
chunk_budget_chars on the guardrail. A value AWS still rejects as too large is
bisected automatically, so an over-large setting costs an extra round trip
rather than failing the request.
* fix(guardrails): never split a Bedrock text into an empty fragment
_nearest_whitespace_split_index could return len(text) when the only space at
or after the midpoint was the final character, so the first fragment came back
identical to the text AWS had just rejected as too large and the second came
back empty. AWS rejects the unchanged fragment again, and each retry re-splits
it into the same fragment, so an oversized single item shaped like a long
unbroken token with one trailing space exhausted the stack with a
RecursionError instead of scanning or surfacing Bedrock's error.
Candidate boundaries that would leave either side empty are now discarded, and
the raw midpoint is used when none remain. The midpoint is always safe because
_split_bedrock_content only calls this for text of at least two characters.
* style(guardrails): move chunking rationale out of comments and into docstrings
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Restores the source changes intended for
|
||
|
Aayush Gid
|
d332accabc
|
fix(proxy): improve Headroom 404 compression error diagnostics (#35952) | ||
|
mateo-berri
|
14d4897e55 |
fix(guardrails): refuse scan_only_tool_results combos that scan nothing
Prompt Security drops tool and function rows unless check_tool_results is on, so it now reports scan-only support from that setting and the registry refuses the pairing at boot. Pairing scan_only_tool_results with skip_tool_message_in_guardrail excludes every message, so guardrail initialization now rejects that combination too. |
||
|
mateo-berri
|
7d745521bf | fix(guardrails): merge synthesized tools under scan_only_tool_results and reject role-filtered no-op combos at init | ||
|
mateo-berri
|
c2998dea75 | fix(guardrails): guard tools write-back under scan_only_tool_results and warn on role-filtered no-op scans | ||
|
Mateo Wang
|
60d9e6012c
|
Merge pull request #35999 from BerriAI/litellm_guardrails_v1_messages_tool_traffic
fix(guardrails): scan /v1/messages tool traffic |
||
|
Mateo Wang
|
b9b239b0fb
|
Merge pull request #35980 from BerriAI/litellm_content_filter_post_mcp_call
fix(guardrails): allow litellm_content_filter to run on post_mcp_call |
||
|
mateo-berri
|
f16f3e23cd |
fix(tool_permission): fail closed on unverifiable SSE streams and end the turn when every tool call is denied
An SSE stream that cannot be positively identified as Anthropic (no parseable message_start event) now blocks instead of passing through unscanned, closing the bypass where any raw-SSE backend skipped tool permission checks entirely. Buffered chunks are joined back into one stream before parsing, so events split across network chunk boundaries assemble correctly instead of being silently dropped. Rewrite mode now resets finish_reason to stop when no tool call survives, so the re-encoded Anthropic stream reports stop_reason end_turn and clients do not wait for a tool result that never comes |
||
|
mateo-berri
|
bee787b4b5 |
fix(guardrails): scan /v1/messages tool traffic
Guardrails silently skipped three surfaces on the Anthropic Messages path, so an agent loop driven by /v1/messages ran unguarded: - The Anthropic input translation never walked tool_result blocks, so content returned by a local tool (a curl, a file read, an MCP call) reached the model unscanned in both the string and list content shapes, images inside a tool_result included. - tool_permission only understood ModelResponse, so an Anthropic non-streaming response or a raw SSE stream carrying tool_use blocks passed through with no rule ever evaluated. - ContentFilterGuardrail scanned inputs["texts"] but never inputs["tool_calls"], so the arguments a model proposes for a tool call went unchecked. Tool call arguments are parsed as JSON before filtering so a MASK action rewrites the value and leaves the payload valid JSON; non-JSON arguments fall back to scanning the raw string. Denied tool_use blocks are dropped from the Anthropic content array and replaced with a text block, and stop_reason resets to end_turn when nothing tool-shaped survives. |
||
|
Mateo Wang
|
332ec6c17a
|
Merge pull request #35926 from BerriAI/litellm_remove_types_ruff_exclusion
chore(lint): remove litellm/types from the ruff lint exclusion |
||
|
mateo-berri
|
83aca91dde |
fix(guardrails): allow litellm_content_filter to run on post_mcp_call
ContentFilterGuardrail implements apply_guardrail, which is everything the generic post_mcp_call_hook machinery needs to scan an MCP tool result before it reaches the model, but post_mcp_call was missing from get_supported_event_hooks. _validate_event_hook rejects any mode outside that list, so a config with `mode: post_mcp_call` failed proxy startup with "Event hook GuardrailEventHooks.post_mcp_call is not in the supported event hooks" instead of scanning tool output. Declaring the hook makes the indirect-prompt-injection case enforceable: an MCP fetch tool returns a page whose body carries "IGNORE ALL PREVIOUS INSTRUCTIONS ...", and the gateway blocks the result rather than handing it to the model. |
||
|
ryan-crabbe-berri
|
2792887e47
|
fix(proxy): give proxy_admin_viewer read parity with proxy_admin (#35851)
* fix(proxy): give proxy_admin_viewer read parity with proxy_admin Route-level checks already default-allow management GETs for the viewer role, but ~15 handlers compared user_role to PROXY_ADMIN only, dropping viewers into regular-user scoping (/key/list, /user/info, /model/info, guardrails, prompts, agents, memory, workflows, MCP catalog, coordination redis settings, credential migration check, enterprise projects). Swap those read paths to user_api_key_has_admin_view; write gates unchanged. The dashboard now presents the viewer session as Admin for all gating (effectiveSessionRole) so every page fetches with admin visibility, with userRoleLabel/isViewOnly preserving the account-menu label and the playground cost guard. The server remains the write authority. * refactor(agents): remove side-effectful health_check param from GET /v1/agents Addresses a security review finding on the admin viewer read parity change: listing agents with health_check=true made the proxy issue a server-side GET to every agent URL, so a read-scoped caller could trigger request fan-out beyond their object permissions. The list endpoint is now a pure read for every role. Removes the query param, the URL probing helper and its timeouts, the AgentHealthCheck httpx provider tag, and the dashboard's Health Check toggle. Requests still passing health_check=true get the full list back with the param ignored. * fix(proxy): keep credential encryption check proxy_admin only The residual scan behind GET /credentials/migrate-encryption/check loads every model, credential, MCP, team, and verification-token row and runs a decryption attempt on each stored value. Extending it to proxy_admin_viewer let a read-only account repeatedly trigger deployment-wide scans, so the route keeps its original full-admin gate. * fix(agents): restore health_check, keep list fast path proxy_admin only Restores the agent health_check feature exactly as before this PR: the query param, the URL probing helper, the httpx provider tag, and the dashboard toggle all return, so existing callers keep the filtering contract. The viewer expansion is instead reverted at its source: the GET /v1/agents admin fast path stays PROXY_ADMIN only, so a proxy_admin_viewer goes through the object-permission scoped branch as before and cannot fan out health checks beyond their allowlist. The viewer read of a single agent stays viewer-inclusive since it has no side effects. |
||
|
mateo-berri
|
4e32a8bf6a |
chore(lint): remove litellm/types from the ruff lint exclusion
ruff.toml has excluded litellm/types/* since 2024, so no lint rule ever ran on the types tree. Remove the exclusion, apply ruff --fix and ruff format across litellm/types, and hand-fix what autofix cannot reach so the pyupgrade budgets stay at zero: implicit type aliases converted to PEP 604 unions, RootModel[Union[...]] bases, duplicate imports, and a stray print. Load-bearing import X as X re-exports deleted by preview-mode F401 are restored, and the six star-imported hub modules keep their re-export surface via per-file F401 ignores. Star-import consumers that silently relied on typing names leaking from those hubs are modernized to builtin generics and PEP 604 unions. Runtime annotation introspection that only recognized typing.Union is taught types.UnionType (guardrail UI field schemas, volcengine response fill), with regression tests for both. Strict budget limits for the rules the types tree now trips are raised to exact measured totals, so any net-new violation still fails the gate |
||
|
yucheng-berri
|
bcce83a17e
|
fix(guardrails): scan model output on the /openai/v1/responses alias (#35818)
The proxy serves POST /openai/v1/responses alongside /responses and /v1/responses, but only the latter two were in API_ROUTE_TO_CALL_TYPES. UnifiedLLMGuardrails.async_post_call_success_hook resolves the call type from request_route, so on the alias it resolved to None and returned the response unscanned; model output reached the client with post-call guardrails never running. The key and team tool allowlist was unenforced on the same alias for the same reason. Register the alias family in API_ROUTE_TO_CALL_TYPES and in LiteLLMRoutes.openai_routes, mirroring how the /openai/v1/realtime aliases are registered, and log a warning at the two points where the unified guardrail skips post-call scanning so a future unmapped route is visible instead of silent. The Responses block of API_ROUTE_TO_CALL_TYPES moves from list to tuple literals because the LIT002 budget rejects net-new mutable-collection construction; the map is read-only, so it is now typed as a Mapping of Sequence and the budgets ratchet down accordingly. |
||
|
Yassin Kortam
|
3a429f3098
|
fix(guardrails): run bedrock guardrail on MCP tool calls in during_mcp_call mode (#35149)
A bedrock guardrail configured mode: during_mcp_call never ran. ProxyLogging remapped the event to during_mcp_call and dispatched, but bedrock's own async_moderation_hook then hard-coded during_call and re-checked, so the second check rejected the very requests the guardrail was configured for and the tool call proceeded unscanned with no error. Remap call_mcp_tool the way model_armor already does, which matches the remap ProxyLogging.during_call_hook itself performs, and teach the shared get_guardrails_messages_for_call_type helper that an MCP tool call carries its payload in the same messages key, without which the hook passes the gate and then bails on an empty message list. |
||
|
Mateo Wang
|
030370012c
|
Merge pull request #35259 from BerriAI/litellm_config_guardrail_info_lookup
fix(guardrails): serve config guardrails from list and info endpoints without a DB and make their ids stable |
||
|
tin-berri
|
b408b1d6dc
|
fix(guardrails/headroom): stop compressing the turn the model must act on (#35294)
The Headroom guardrail sent every message to /v1/compress, including the system prompt and the user's current instruction. On an agentic /v1/messages request the live turn is the largest compressible blob, so it came back as a hash marker; the model then called headroom_retrieve and got its own instruction returned in a tool_result block, which reads as data it fetched rather than a request to act on, so it described the content instead of doing the work. litellm already owns the policy for what a compressor may never rewrite: get_protected_indices covers the system rows, the last user row and the last assistant row, and compress() expands it over whole tool exchanges. Headroom now consults it (promoted from a private name and given tests) and expands it the same way, so the trailing tool result cannot come back as a marker standing in for the result of the call the model just made. Protected rows are withheld from the payload rather than pinned afterwards, so their tokens are not reported as savings that are never applied; the write-back discards a compressed system prompt outright, so that saving never existed. The cost is that a query-aware service no longer sees the newest user message. A response whose row count differs from what was sent can no longer be interleaved with the withheld rows, so it goes through the configured fail policy instead of being adopted. Fail-open now returns the caller's own inputs object: translation handlers detect a rewrite by identity, so a rebuilt copy sent an unchanged request through the Anthropic write-back for nothing. That write-back rebuilt the request with one anthropic_messages_pt call, which merges every run of consecutive user/tool rows, so a tool_result turn and the user turn after it arrived fused. Converting a row at a time would separate them but breaks tool pairing: with modify_params on, an assistant row whose results are converted separately reads as an orphaned tool call and the sanitizer answers it with a synthetic "tool execution skipped" result while dropping the real one. Conversion is now grouped by tool_call_id ownership, which satisfies both, and the same grouping decides which rows headroom protects, so the two agree by construction. The CCR follow-up also dropped any text the model wrote alongside its tool call, and echoed tool calls it had no results for. Both are fixed by reusing compresr's extraction helper, now shared instead of duplicated. Resolves LIT-5018 |
||
|
mateo-berri
|
5ae1f1530c | fix(guardrails): serve config guardrails from list and info endpoints without a DB and make their ids stable | ||
|
Tin Chi Lo
|
33fadd70a3 |
fix(guardrails): compress content-parts messages in headroom guardrail
Anthropic-format requests translate to messages whose content is a list of part dicts, which the headroom compression service's transforms silently skip (they only rewrite string content), so compression never applied to Anthropic client traffic while the guardrail still reported itself as applied. Flatten all-text part lists to plain strings for /v1/compress and restore the original shapes from the response: untouched rows keep their exact original parts, a rewritten row collapses to one part carrying the last declared cache_control breakpoint (a breakpoint caches the prefix ending at its part, so the last one and its TTL still describe the merged row). Rows with any non-text part are never flattened, since merging text across a non-text part would move a later breakpoint to the other side of it; they pass through the service untouched, matching its own behavior for non-string content. Flattening and write-back use the shared content_text helpers that compresr's breakpoint fix also uses. Resolves LIT-4795 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
tin-berri
|
10cd4288b6
|
Merge pull request #34660 from BerriAI/litellm_lit4804_compresr_cache_control
fix(guardrails): preserve cache_control breakpoints in compresr write-back |
||
|
devin-ai-integration[bot]
|
24123269cc
|
fix(guardrails): resolve judge_model credentials via lazy Router lookup in llm_as_a_judge (#34509)
* fix(guardrails): resolve judge_model credentials via Router in llm_as_a_judge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): wire llm_router into DB-backed judge guardrail init paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): assert patch endpoint forwards llm_router to sync Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(guardrails): resolve judge Router lazily and fix wildcard/alias dispatch Resolve the proxy Router at judge-call time via an injected provider instead of capturing it at construction, so a DB-backed judge guardrail created before the Router exists no longer captures None permanently. Select the Router path with router.get_model_list(model_name=judge_model) so wildcard routes and model_group_alias keys resolve, not just literal deployment names. Isolate the judge call from user-traffic routing with num_retries=0 and fallbacks=[]. Revert the llm_router threading through the DB sync/reinit/create/approve/patch paths since the lazy provider makes it unnecessary. Replace mocked-Router tests with real Router coverage for plain deployments, model_group_alias, and wildcard routes, plus lazy per-call resolution. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): harden judge verdict parsing and guard proxy import Strip markdown fences and surrounding prose before json.loads so fencing-prone judge models evaluate instead of failing open, guard the proxy_server import in _default_router_provider so an unimportable proxy falls back to the SDK, and snapshot/restore global callback lists in the DB-path judge registry tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): reject non-object judge verdicts instead of failing open as success * fix(guardrails): route hidden model_group_alias judge models through the Router --------- Co-authored-by: milan <milan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> |
||
|
Tin Chi Lo
|
c63e24bacf |
fix(guardrails): preserve cache_control breakpoints in compresr write-back
Anthropic cache_control breakpoints are positional: each one caches the prefix ending at the part that carries it. Compresr flattened every text part of a message into one string and wrote the compressed result back into the first text part only, which dropped every later breakpoint and, when a non-text part sat between text parts, moved the trailing text to the other side of it. The positional invariant now has one owner. guardrail_hooks/content_text.py holds content_to_text alongside is_all_text_parts and merge_rewritten_text_parts, so a compressed string is only ever written back over a contiguous run of text parts, and the merged part carries the last declared breakpoint and its TTL. Compresr consumes that owner at both ends: _select_targets no longer selects a row holding a non-text part, and _replace_text_in_content returns such a row unchanged rather than merging across it. Rows whose content is a plain string are unaffected. Mixed rows therefore stop being compressed, which is a deliberate trade; no single-string write-back can preserve a breakpoint across a non-text part, so the alternative is silently caching a different prefix than the caller configured. |
||
|
tin-berri
|
2d6b57407d
|
Merge pull request #34578 from BerriAI/litellm_headroom_tokens_saved
fix(guardrails): derive tokens_saved when Headroom compression service omits it |
||
|
yucheng-berri
|
76b0b10908
|
fix(guardrails): add /v1/messages support for Straiker plugin (#34548)
* fix(guardrails): add /v1/messages support for Straiker plugin - Pass prepared response data to Anthropic Messages streaming post-call hooks (litellm/llms/anthropic/chat/guardrail_translation/handler.py) - Normalize Straiker request, tool, finish-reason, and mode fields across Chat Completions, Messages, and Responses APIs * fix(guardrails): gate cross-surface message resolution and cover streaming request data Resolve request messages only for surfaces that have a mapped translation handler. The unguarded fallback tried every registered handler in turn, which raised AttributeError out of the guardrail's error handling on list-shaped `input` bodies, and synthesized a chat message that was never sent for bodies it happened to parse. Prepare request data on the mid-stream Anthropic branch as well, matching the terminal branch and the OpenAI handler, so guardrails that scan before end-of-stream still receive identity metadata. Read usage from Anthropic dict responses so non-streaming /v1/messages reports token counts instead of null. Add regression coverage for the streaming request data on both the terminal and mid-stream branches; reverting either now fails. --------- Co-authored-by: cs-mehta <chandra@straiker.ai> |
||
|
tin-berri
|
842f32dbaa
|
Merge pull request #34458 from BerriAI/litellm_lit4759_guardrail_metadata_bucket
fix(guardrails): keep guardrail information in spend logs when the caller sends its own metadata |
||
|
Tin Chi Lo
|
9bd89290cb |
fix(guardrails): derive tokens_saved when Headroom compression service omits it
The savings readers (extract_compression_saved_tokens, feeding compression_saved_tokens on the daily spend tables) key exclusively on tokens_saved in the guardrail_response stats, but the Headroom guardrail builds those stats as a filtered pass-through of the compression service response and the live service omits tokens_saved. Every compressed request recorded 0 saved tokens on the Cost Optimization dashboard. Derive tokens_saved = tokens_before - tokens_after when the key is absent and both operands are numeric; a service-sent value still wins. The two sibling writers (compresr, native compression interception) already derive it the same way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Tin Chi Lo
|
9777e9524a |
fix(guardrails): stop reporting a no-op guardrail as applied on passthrough
On passthrough requests the shared guardrail plumbing still dispatches headroom's pre_call apply_guardrail, but the passthrough translation hands it only `texts` and no `structured_messages`, so it early-returns a no-op. The @log_guardrail_information decorator then synthesized an "allow"/"success" StandardLoggingGuardrailInformation entry, and the unified hook added the guardrail to applied_guardrails, so spend logs reported the compression guardrail as succeeded even though nothing ran. Add a records_own_guardrail_information flag for guardrails that log their own execution (headroom). The decorator skips the synthetic success entry for them, and the unified hook lists such a guardrail in applied_guardrails only when it actually recorded a run. A guardrail that owns its logging must record every outcome it runs, so headroom now records a guardrail_failed_to_respond entry on the fail_open path (compression attempted, service unreachable, request forwarded uncompressed) instead of leaving it unlogged; fail_closed is still recorded by the decorator's error path, and a genuine no-op stays not_run. |
||
|
Tin Chi Lo
|
770f41b5fa |
fix(guardrails): keep guardrail information in spend logs when the caller sends its own metadata
The guardrail-information writer picked its metadata bucket with a hand-rolled precedence that preferred a caller-supplied `metadata` field, while every reader resolves the bucket through `get_metadata_variable_name_from_kwargs`, which prefers `litellm_metadata`. The two rules agree only when the caller sends no `metadata` of its own. Routes in `LITELLM_METADATA_ROUTES` seed `litellm_metadata`, so on /v1/messages and /v1/responses a caller that sends `metadata` sent the entry to a dict nothing reads; the spend log then reported `guardrail_status: not_run` with no `guardrail_information` even though the guardrail ran and the `x-litellm-applied-guardrails` header was present. Give the resolver one owner. `get_or_create_metadata_bucket` moves from the proxy layer into core_helpers next to the resolver it calls, so `litellm/integrations` can reach it without a proxy dependency, and the byte-identical duplicate of `get_metadata_variable_name_from_kwargs` in callback_utils is deleted. The writer now shares that owner with `add_guardrail_to_applied_guardrails_header`, so the response header and the spend log can no longer disagree. Two readers had to move with it or the fix would be a no-op on the affected routes. `_sync_guardrail_info_to_logging_obj`, which bridges request_data into the spend-log payload for passthrough routes, picked the first truthy bucket, so a non-empty caller `metadata` short-circuited it. The otel failure-path span reader `_emit_guardrail_spans_from_request_data` read a hard-coded `metadata` key, which also dropped the span whenever the entry lived in `litellm_metadata`. Model Armor already resolved the bucket for its file-scan results but wrote its text-scan and post-call results, and read them back in `_process_response`, through a hard-coded `metadata` key; on a seeded route that split the record so a file scan's evidence never reached the logger. All four Model Armor sites now use the shared resolver. The unified guardrail hook seeds `litellm_metadata` on every route, so the OpenAI moderation entry lands there too; spend-log output is unchanged because `merge_litellm_metadata` reads both buckets. |
||
|
Noah Nistler
|
8177230a29
|
feat(guardrails): add run_in_parallel opt-in for concurrent pre_call and post_call guardrails (#33770)
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails Pre-call guardrails run sequentially because each may mutate the request payload and later guardrails depend on earlier mutations. Deployments with several slow block-only pre_call guardrails (external moderation, Bedrock, LLM-judge) therefore pay the sum of their latencies. during_call guardrails run concurrently but alongside the LLM call, so a violating payload has already been sent, which is unacceptable when the request must never reach the model. This adds a per-guardrail run_in_parallel flag (default off). Guardrails that opt in are pulled out of the sequential loop and run concurrently via asyncio.gather after every sequential (payload-mutating) guardrail has run, so they observe the mutated payload and still form a hard barrier before the LLM call; the first to raise blocks the request. Their returned data is discarded since they are declared block-only. The flag is wired from LitellmParams onto the guardrail instance at the same generic choke point in initialize_guardrail that already sets skip_system_message_in_guardrail, so no per-provider initializer needs to change. * feat(guardrails): extend run_in_parallel opt-in to post_call guardrails post_call_success_hook ran guardrails sequentially for the same reason pre_call did: response-modifying guardrails thread the response forward. But block-only output scanners (which read the response and reject on violation without changing it) serialize for no benefit and add latency. This reuses the existing run_in_parallel flag for the post_call hook. Opted-in post_call guardrails are pulled out of the sequential loop and run concurrently via asyncio.gather after the sequential (response-modifying) guardrails and before the non-guardrail CustomLogger callbacks, so they inspect the final response and still block it from reaching the client if any raises. Their returned response is discarded since they are block-only. The apply_guardrail path sets data["guardrail_to_apply"] immediately before awaiting, and unified_guardrail pops it before its first suspension point, so concurrent guardrails never race on that key under asyncio's cooperative scheduling. * fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes Addresses review feedback on the run_in_parallel opt-in. asyncio.gather propagated the first exception without cancelling or awaiting the siblings, so a block at t=0 left the other guardrails running as unobserved background tasks (wasted external calls plus event-loop warnings), and a fast SensitiveDataRouteException/ModifyResponseException could return a reroute or passthrough before a slower block finished, letting crafted input bypass the block. Both the pre_call and post_call parallel batches now gather with return_exceptions=True so every guardrail runs to completion, then raise any blocking exception ahead of a flow-changing one. The registry choke point wrote bool(None)==False onto every instance when the config omitted run_in_parallel, silently disabling a constructor-set default; it now only writes when the config provides an explicit value. * fix(guardrails): record lifecycle logs for every concurrently-run guardrail The log_guardrail_information decorator skipped its auto-record when it saw that the count of standard_logging_guardrail_information entries in the shared request_data had grown during the wrapped call, taking that as proof the wrapped function had recorded its own richer entry. That heuristic breaks the moment guardrails run concurrently (parallel pre_call/post_call, during_call): a sibling guardrail's append inflates the shared count, so a guardrail that did not self-record wrongly concludes it already did and drops its own entry. The result is that enabling run_in_parallel silently loses per-guardrail lifecycle logs, so the Admin UI Request Lifecycle timeline and downstream loggers (Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent guardrails. Replace the shared-count heuristic with a ContextVar flag set when a guardrail records its own entry. asyncio copies the context into each gathered task, so the flag is isolated per concurrent guardrail while still catching the self-record-then-skip-auto-record case within a single invocation. * test(guardrails): declare run_in_parallel on post_call guardrail mocks The post_call partition reads run_in_parallel on every CustomGuardrail callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is set in __init__, not on the class) so the attribute access raised, and even a class-level default would return a truthy child mock that wrongly routes the double into the parallel batch. Declare the flag False on the shared mock factories so these pre-existing hook tests exercise the sequential path they assert on. * fix(guardrails): harden run_in_parallel reads and address review feedback Read run_in_parallel via getattr(..., False) in the pre_call and post_call partitions so a third-party CustomGuardrail subclass that overrides __init__ without chaining super().__init__() no longer raises AttributeError on a path that previously worked. Drop the redundant in-function GuardrailEventHooks import in _run_parallel_post_call_guardrails (already imported module-level). Remove the flaky wall-clock upper-bound assertions from the two concurrency tests; the all-start-before-any-end overlap assertion is the timing-independent signal that actually proves concurrency. |
||
|
devin-ai-integration[bot]
|
7257d0fc89
|
fix(guardrails/model_armor): handle None metadata in post_call _process_response (#34390) (#34405)
* fix(guardrails/model_armor): handle None metadata in post_call _process_response
On batch routes data["metadata"] is normalized to None (present key, None
value), so request_data.get("metadata", {}) returned None and _process_response
raised 'NoneType' object has no attribute 'get', 500ing every /v1/batches create
with a post_call Model Armor guardrail (regression from v1.93.0 activating the
post_call hook). Coalesce a falsy metadata to {}
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Clean up test case documentation
Remove regression comment from test_process_response_with_none_metadata_does_not_crash.
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
fa6b209165
|
feat(guardrails): add only_scan_new_messages for per-session incremental scanning (#33278)
* feat(guardrails): add only_scan_new_messages for per-session incremental scanning Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(guardrails): use fixed TTL constant and revert unrelated test formatting Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(guardrails): run only_scan_new_messages in the unified apply_guardrail path The initial wiring lived in BedrockGuardrail.async_pre_call_hook, but the proxy routes Bedrock through the unified apply_guardrail interface, so the flag had no effect live. Move incremental selection into apply_guardrail: filter the flat texts list against per-session scanned hashes, skip the Bedrock call when nothing is new, and mark hashes only after a successful (non-blocked) scan. Full-context fallback is preserved when there is no session id, the cache is unavailable, or a masking guardrail is configured. Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(guardrails): cover session-id fallbacks and mark_texts_scanned guards Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(guardrails): fall back to full scan when incremental guardrail masks content, use shared cache Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(guardrails): cover generic agent multi-turn incremental scan Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(guardrails): cover incremental scan cache resolver fallbacks Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(guardrails): cover flag interactions and /v1/messages incremental scan semantics * feat(guardrails): make GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS env configurable * test(guardrails): prove skip_system/skip_tool are enforced upstream of incremental scan --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: Yucheng Zhu <yucheng@berri.ai> |
||
|
yucheng-berri
|
049c6836d2
|
fix(model_armor): sanitize error details by default (#33908)
* fix(model_armor): sanitize error details by default Generated with AI Co-Authored-By: Claude Code * fix(model_armor): sanitize handler-raised HTTP errors and redact scanned content in guardrail logging The async HTTP handler raises MaskedHTTPStatusError on any non-2xx via raise_for_status, so the non-200 branch in make_model_armor_request never ran against a live API and the raw upstream body reached callers and logs. Catch the raised error and build the sanitized detail from the response status Replace the empty-dict guardrail logging payload with field-level redaction of the keys that echo scanned content (text, sanitizedText, findings) so guardrail traces keep filter states and block reasons while scanned content stays out Restore the upstream status code in the sanitized error detail, read guardrail metadata from the same key the hooks write, and keep guardrail_status within its typed literal values * fix(model_armor): bound redactor recursion depth and allowlist it in the recursion detector _redact_scanned_content walks provider JSON bounded by _REDACT_MAX_DEPTH=20 and fails closed by returning the redaction sentinel at the cap * fix(model_armor): honor fail_on_error for upstream API failures API failures now raise a dedicated ModelArmorAPIError so hooks can tell them apart from content-block HTTPExceptions; fail_on_error=False lets the request proceed on a Model Armor outage again while fail-closed configs get the same sanitized 400 as before Also addresses review notes: sanitize_error_detail constructor annotation matches the nullable config field, redaction is owned by the metadata write sites so _process_response no longer re-applies it, and the request and response debug log branches move into helpers * test(model_armor): cover fail_on_error routing on during-call, post-call, streaming, and file-scan paths * chore: remove accidentally committed pytest cache files * fix(model_armor): keep sanitize_error_detail coerced across in-memory config reloads update_in_memory_litellm_params assigns raw LitellmParams fields, so a hot reloaded config carrying an explicit null would silently disable sanitization; re-apply the only-explicit-False-opts-out coercion after the update * fix(model_armor): redact matched malicious URIs and reuse the shared recursion depth constant maliciousUriMatchedItems echoes the caller-supplied URL including path and query, so it joins the scanned-content key set; the redactor depth cap now comes from DEFAULT_MAX_RECURSE_DEPTH in litellm constants instead of a local literal * fix(model_armor): keep API failures out of the intervention trace status Fail-closed upstream failures re-raise ModelArmorAPIError instead of converting to HTTPException(400), so the shared guardrail logging keeps recording them as guardrail_failed_to_respond while content blocks stay guardrail_intervened. Callers see the same 500 shape as before this PR, with the sanitized message * chore(model_armor): drop explanatory comment per repository comment policy --------- Co-authored-by: eugene-yao-zocdoc <eugene.yao@zocdoc.com> |
||
|
yucheng-berri
|
9ad8698aab
|
feat: add deepkeep as custom guardrail (#33844)
* adding deepkeep as custom guardrail * adding deepkeep as a custom guardrail * adding deepkeep as a custom guardrail (hooks) * adding litellm/proxy/_experimental/out/ to .gitignore * adding deepkeep as custom guardrail in litellm * removing sentinel_fortress * comparing schema.prisma files * fix(deepkeep): address greptile review comments - extra_headers: fix type annotation (list -> Dict[str, str]) and actually merge them into _build_request_headers() so user-configured headers reach the DeepKeep API - user_api_key_hash: only fall back to user_api_key_token when no explicit hash is already set, avoiding silent overwrite - apply_guardrail: preserve tool_calls and structured_messages in the return value so downstream callers don't lose that content Adds tests for all four fixes. * fix(deepkeep): address greptile review comments - extra_headers: fix type annotation (list -> Dict[str, str]) and actually merge them into _build_request_headers() so user-configured headers reach the DeepKeep API - user_api_key_hash: only fall back to user_api_key_token when no explicit hash is already set, avoiding silent overwrite - apply_guardrail: preserve tool_calls and structured_messages in the return value so downstream callers don't lose that content Adds tests for all four fixes. * fix: add missing __init__.py and allowlist entries for upstream merge - tests/test_litellm/proxy/client/__init__.py: fixes pytest collection collision with tests/test_litellm/models/test_models.py (same basename) - tests/test_litellm/models/__init__.py: same fix - backend/routes/allowlist.py: add /config_overrides/ and /v1/unified_access_group prefixes for new routes added by upstream * fix(ui/tests): resolve frontend-lint failures in new test files - useLogDetails.test.ts: add Wrapper.displayName, replace 'null as any' with null, type resolveCall promise resolver properly - usePaginatedDailyActivity.test.ts: remove unused waitFor import, add Wrapper.displayName, change Record<string,any> to Record<string,unknown> - UsageViewSelect.adminFiltering.test.tsx: replace all props:any with explicit SelectProps/BadgeProps/SelectOption types, replace (X as any).displayName with direct X.displayName assignment no-explicit-any count: 2034 (budget: 2040). Prettier check: clean. * fix(ui): sync proxy/_experimental/out/ exactly to upstream 245 stale JS chunk files from earlier merges were left in the out/ directory but had been deleted in upstream. The Docker image in CI is built by copying this directory verbatim, so the stale artifacts caused the SERVER_ROOT_PATH redirect E2E to fail. Synced by: git checkout upstream/litellm_internal_staging -- out/ (adds new files) + git rm on every file present in HEAD but absent from upstream. * Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix(makefile): fall back to upstream/litellm_internal_staging for strict-budget gate origin/litellm_internal_staging exists on BerriAI's CI but not on forks that use a different remote name (e.g. Azure DevOps as origin). Fall back to upstream/litellm_internal_staging when the origin ref is absent. * linter reformat * fix(deepkeep): apply guardrail tool/tool_call redactions from API response When DeepKeep returns GUARDRAIL_INTERVENED with redacted tools or tool_calls, the previous code ignored those redactions and forwarded the original (potentially sensitive) values to the model — a guardrail bypass for content embedded in tool schemas or function arguments. Fix: prefer response_json["tools"] / response_json["tool_calls"] when present, falling back to the originals only when the guardrail did not return replacements — consistent with the existing pattern for texts and images. Refactor _build_return_inputs() into a private static helper to keep apply_guardrail() under the PLR0915 statement limit (50). Adds test_apply_guardrail_applies_tool_redactions_from_response to assert that redacted tool payloads from the API response are used. * Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix(lint): move base-ref fallback into ruff_strict_gate.py; revert Makefile The previous Makefile fix had a shell bug: 'git rev-parse --verify' writes the resolved SHA to stdout, so the $$(...) substitution captured both the SHA and the echo output, handing '--base <sha>\norigin/...' as two tokens to the Python script, causing exit code 1 in CI. Fix: revert Makefile to its original single-line invocation and add _resolve_base() to ruff_strict_gate.py. The function checks whether the requested ref resolves; if not, it tries the 'upstream/' equivalent before falling back to the original ref (letting git emit a clear error). Behaviour in BerriAI CI: origin/litellm_internal_staging resolves → used as before, no change. Behaviour on forks with a different 'origin': falls back to upstream/litellm_internal_staging transparently. * fix(lint): fix UP006/UP045/F401 in changed files; add depth guard to check_any_discipline - Replace Dict/List/Optional/Tuple typing imports with built-in equivalents (UP006, UP045) across files touched in this PR diff, then clean up the now-unused typing imports (F401). - Add _MAX_CONTAINS_ANY_DEPTH guard to check_any_discipline.contains_any() to prevent RecursionError on deeply-nested mypy types. * fix(lint): resolve all three CI lint job failures 1. lint (ruff_strict_gate) — UP006/UP045/F401 violations introduced on changed lines. Fixed Dict/List/Optional/Tuple → built-in equivalents across every file in the PR diff; cleaned up now-unused typing imports. 2. any-discipline — RecursionError in check_any_discipline.contains_any() on deeply-nested mypy types. Upstream fixed this by converting to an iterative stack-based algorithm (merged). Also added deepkeep.py to any-discipline-budget.json via 'make lint-any-budget-update' so the new file's Any count is baselined instead of failing against the zero-baseline default. 3. basedpyright reportMissingParameterType — **kwargs in DeepKeepGuardrail __init__ lacked a type annotation. Added **kwargs: Any. * Update litellm/deepkeep_tilt_config.yaml Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix(lint): black reformat after merge * fix(deepkeep): honour empty-list replacements in _build_return_inputs When DeepKeep returns GUARDRAIL_INTERVENED with an intentional empty replacement (e.g. texts:[], tool_calls:[]) the previous truthiness check treated [] as absent and forwarded the original content downstream — a guardrail bypass for any case where the firewall wants to fully clear a field. Fix: replace all response_json.get(field) truthiness checks with 'is not None' comparisons so that an empty list is respected as a deliberate replacement. Applies to texts, images, tools, tool_calls, and the original-input fallback guards. Adds test_apply_guardrail_honours_empty_list_replacements. * fix(test): replace live httpbin.org call with mocked transport in test_pass_through_with_httpbin_redirect Root cause of OOM: the test made a real HTTP request to https://httpbin.org inside a pytest-xdist worker. Under memory pressure the worker's httpx client and redirect-following logic allocated enough virtual memory to trip the OOM killer (confirmed by ulimit -v 16GB reproducing the crash with 'node down: Not properly terminated' on this exact test). Fix: replace the real network call with a custom httpx.AsyncBaseTransport that returns a pre-built 302 -> 200 response sequence in-memory. The test now runs hermetically with no network dependency and no excess memory allocation. ulimit -v 16GB: 24,284 passed (0 crashes) after this fix. * fix: merge upstream/litellm_internal_staging (197 commits), resolve conflicts 7 conflicts resolved: - 6 Python files: upstream added new code with old-style typing (Optional, Dict, List) on lines where we had ruff-fixed modern syntax (str | None, dict, list). Took upstream's version then re-ran ruff UP006/UP045/F401 --fix to keep both the new content and ruff compliance. - test_openapi_compliance.py: upstream replaced 'role' with 'steps' in output_fields and updated the spec comment. Took upstream's version. Also: added _resolve_base() fallback to type_check_gate.py and removed the hard 'git fetch origin litellm_internal_staging' from the Makefile's lint-basedpyright target (same pattern as ruff_strict_gate.py fix). * fix: merge upstream (41 commits), resolve .gitignore conflict, fix BLE001 - .gitignore: upstream removed package.json/out/ ignore entries; took theirs - deepkeep.py: added '# noqa: BLE001' on catch-all Exception handler (BLE001 rule newly enforced in ruff-strict-budget) - type_check_gate.py: added _resolve_base() fallback for basedpyright gate - Makefile: removed hard 'git fetch origin' from lint-basedpyright target * fix: merge upstream (57 commits), resolve conflicts - Makefile: upstream added lint-fetch-base target; made it tolerant of missing origin/litellm_internal_staging (git fetch || true) - test_websearch_chat_completion.py: took upstream's new assertions and skipif marker - anthropic_cache_control_hook.py: upstream added new code using List/Dict/Tuple which were undefined after our earlier UP006 cleanup; replaced with built-in list/dict/tuple * fix(coverage): revert ruff UP006/UP045 changes on upstream files The previous ruff fixes (Dict→dict, Optional→X|None) on 7 upstream files added ~500 changed lines of pure type-annotation no-ops to our PR diff. codecov/patch penalised these uncovered lines, dropping patch coverage to 51.35% (target 61.83%). Fix: revert these files to exactly match upstream/litellm_internal_staging. The ruff_strict_gate still passes because the violations exist equally in both the base and HEAD (total == base_count → no breach). * fix: merge upstream (130 commits), resolve Makefile + base_email conflicts - Makefile: upstream changed lint deps to $(LINT_DEP_INSTALL)/$(LINT_DEP_BASE); kept our --base removal (handled by _resolve_base in Python scripts) - base_email.py: took upstream's dedup cache addition - deepkeep.py: ruff format after merge * chore: remove lint/format-only changes and non-feature files Revert all lint-infra and black/ruff-reformat-only changes back to upstream/litellm_internal_staging so the PR diff shows only the DeepKeep guardrail feature: - Makefile, scripts/ruff_strict_gate.py, scripts/type_check_gate.py (lint-gate infra) - credential_migration.py + enterprise/* + assorted test files (black-reformat / xdist test-isolation drift) - backend/routes/allowlist.py (merge glue) Remove non-feature local artifacts: build-and-push.sh, deepkeep_tilt_config.yaml, stray __init__.py collision shims, and unrelated UI test files. * fix(lint): add reason to BLE001 noqa to satisfy type-discipline gate (LIT003) The type-discipline budget ratcheted LIT003's ceiling to 292 as upstream fixed reasonless suppressions, so our '# noqa: BLE001' (code but no reason) tipped the total to 293 and failed CI. Add a reason per the required '# noqa: CODE # <reason>' shape. * fix(deepkeep): apply structured_messages redactions returned by the guardrail API _build_return_inputs dropped any structured_messages the DeepKeep API returned and always forwarded the original input, so redactions on that field never took effect. Check the response first, same as texts/images/tools/tool_calls * chore(ui): drop redundant preserve prop from the guardrail form preserve defaults to true in rc-field-form (isMergedPreserve falls back to true when unset), so the explicit prop changed nothing and only widened this PR's blast radius to every guardrail provider in the shared form * fix(deepkeep): stop extra_headers list from crashing the guardrail call and name the real firewall id config key litellm_params.extra_headers is a list of header names to forward, so passing it straight into dict.update raised ValueError and, under fail_closed, took the request down with it. Only merge mapping values and warn otherwise The docstring example and the missing-secret error both said firewall_id, but initialize_guardrail only reads deepkeep_firewall_id, so anyone following them had their value silently ignored * refactor(proxy): drop normalize_callback change; split to its own PR (#33905) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Yaniv Israel <yaniv@deepkeep.ai> Co-authored-by: DK-yaniv <164404355+DK-yaniv@users.noreply.github.com> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
f759c75466
|
feat: add Straiker guardrail integration (#33781)
* feat: add Straiker guardrail integration Implements LLM security guardrails via Straiker with prompt and response inspection, multi-mode execution (pre_call, post_call), and configurable blocking or redaction of flagged content across providers, streaming, images, and tool calls. * fix(guardrails): harden straiker source attribution and error-path consistency Use the operator-configured source for Straiker application attribution instead of a caller-supplied agent_id metadata value, so a caller cannot spoof which application a detection is attributed to. Make _fail reuse _block so a post_call error raises ModifyResponseException like a deliberate post_call block rather than GuardrailRaisedException, and type the blocking helper as NoReturn so the type checker enforces that execution never falls through the BLOCKED branch. Serialize the webhook payload once and send it as raw content to avoid re-serializing on the size check and on every retry. * fix(guardrails): read straiker config and metadata from all supported shapes Handle a dict optional_params in _get_config_value so nested guardrail settings loaded from YAML or the DB (timeout, unreachable_fallback, and the rest) are applied instead of silently falling back to defaults; previously only attribute-style access was supported. Build the webhook metadata bag from the merged metadata so client tags stored under litellm_metadata on routes like /v1/messages reach Straiker the same way identity and application fields already do, and widen the internal-key skip prefix to user_api so proxy-injected budget values are not forwarded. * fix(guardrails): fail safe on straiker interventions without redactions Block instead of passing content through when Straiker returns GUARDRAIL_INTERVENED without replacement texts, so a positive intervention verdict can never silently forward the original flagged content. Fix the streamed-request detection to read the request body from proxy_server_request.body, where the proxy stores it, instead of a top-level body key that is never populated; the previous fallback was dead, so a streamed response whose stream flag was not lifted to the top level would have been redacted rather than blocked while buffering replayed the original chunks. * revert(guardrails): restore straiker caller agent_id application attribution Restore the original behavior where a request-scoped agent_id in metadata sets the Straiker application source, falling back to the configured source. This is the integration's intended per-application attribution; litellm already resolves a key-owned agent_id ahead of any caller-supplied value, so a configured key cannot be spoofed. * revert(guardrails): restore straiker webhook metadata scoping Restore the original behavior where the Straiker webhook metadata bag is built from request-scoped metadata only. Forwarding litellm_metadata was a scope change to what the integration sends to Straiker; keep the author's intended scoping. * fix(guardrails): keep proxy key material out of straiker webhook metadata Widen the internal-key skip prefix from user_api_key_ to user_api so the proxy-injected user_api_key hash and user_api_end_user_max_budget are not copied into the Straiker webhook metadata bag. The narrower prefix missed the bare user_api_key name, leaking the hashed key to the vendor. Keeps the request-scoped metadata source unchanged. --------- Co-authored-by: cs-mehta <chandra@straiker.ai> |
||
|
yuneng-jiang
|
04a5ebb94d
|
chore(ci): merge oss branch (#33784)
* fix(embeddings): accept encoding_format='float' for vertex_ai/gemini embeddings (#33617)
OpenAI SDKs (and litellm's own client since ~1.84) send
encoding_format='float' by default, but the vertex embedding config only
supports ['dimensions'], so get_optional_params_embeddings raised
UnsupportedParamsError at the provider default value. Any
OpenAI-compatible client talking to a litellm proxy with vertex
embedding models got a 400 unless the operator set proxy-wide
drop_params: true.
Float lists are exactly what the vertex API returns, so the param is a
no-op: pop it before validation. Other values (e.g. 'base64') keep the
existing unsupported-param behavior (dropped with drop_params, raise
otherwise).
Fixes #33173
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add Singulr guardrail integration for LiteLLM gateway (#31302)
* singulr guardrail support for litellm gateway
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix comments
* improvement
* fix: resolve review comments and implement requested improvements
* fix:Guardrail bypass through uninspected messages
* fix:tool text scanning
* fix: Legacy function definitions bypass scanning by adding indirect message scaning
* chore: remove unintended basedpyright budget file
* fix:Response schema bypasses guardrail scanning (response_format.json_schema)
* chore: restore basedpyright-code-budget.json and update lint baselines
Restores the file deleted in
|
||
|
devin-ai-integration[bot]
|
0d7b0f708b
|
fix(model_armor): restore reference attachments via skip_unscannable_attachments and remove the attachment count cap (#33554)
* fix(model_armor): add skip_unscannable_attachments to allow reference-only attachments through * fix(model_armor): wire skip_unscannable_attachments through guardrail config * fix(model_armor): make max_file_attachments configurable and scan overflow instead of dropping * fix(model_armor): remove the per-request attachment count cap and scan all attachments --------- Co-authored-by: yucheng <yucheng@berri.ai> |
||
|
yuneng-jiang
|
ff06119aa9
|
Merge pull request #32853 from BerriAI/litellm_/guardrail-monitor-details-fix-8845d6
fix(guardrails): show YAML-defined guardrails in the Guardrail Monitor |
||
|
devin-ai-integration[bot]
|
b907378f02
|
feat(guardrails): forward optional metadata on POST /guardrails/apply_guardrail (#33067)
Clients calling the standalone apply_guardrail endpoint had no way to pass per-request configuration to custom guardrail implementations. This adds an optional metadata field to ApplyGuardrailRequest and forwards it to CustomGuardrail.apply_guardrail via request_data, only when the client sends it. The messages guard is aligned to the same is-not-None semantics so an explicitly-sent empty list is forwarded instead of silently dropped. The Admin UI's Guardrail Test Playground gains an optional Metadata JSON input (validated client-side) wired through applyGuardrail in networking.tsx, so parameterized guardrails can be exercised from the dashboard. Tests cover metadata alone, metadata with messages, explicit empty values, the omitted-field passthrough, and the UI panel's parse/error behavior Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
yucheng-berri
|
587b8aca9b
|
feat(guardrails): add Compresr guardrail for query-aware context compression (#33295)
* feat(guardrails): add Compresr guardrail for query-aware context compression Adds a first-class guardrail that compresses bulky message content (tool outputs, RAG chunks, search results) through the Compresr API before the request reaches the LLM, via the apply_guardrail / structured_messages hook so it covers /chat/completions, /v1/messages, and /v1/responses (the latter through the texts channel, mirrored only when the replacement is unambiguous; anything ambiguous is left uncompressed). Distinct from whole-conversation compressors: - Query-aware: each message is compressed against the intent that produced it (a tool output against its originating tool call's name + arguments, resolved via tool_call_id; otherwise the last user message). - Recoverable: each compressed message carries a hash marker and the request gains a compresr_retrieve tool, so the model can pull the original content back through the agentic loop when the compressed version is not enough. Originals are cached in-process, scoped to the caller's virtual-key hash plus the request's litellm_call_id, with a TTL and a per-call byte cap; recovery is skipped when no caller scope is available so one caller can never read another's originals. The store is per-process, so multi-worker deployments need sticky routing (or enable_retrieval=false). Fail-closed by default (fail_open configurable), SSRF-validated api_base (alternate IP-literal encodings included), cross-tenant-isolated recovery store, and upstream errors redacted from client-facing responses. The outbound client follows redirects and re-resolves DNS per request, so the api_base host/IP checks are defense-in-depth, not a full SSRF guarantee; this is documented as a known limitation. Requests where nothing was actually compressed are returned untouched (same object identity) so handlers skip the write-back. Auto-discovered via the guardrail_hooks registry. * fix(guardrails): cap Compresr recovery store total memory The recovery store bounded bytes per call and entry count, but had no aggregate cap: 256 tracked call ids at the 10 MiB per-call default could retain ~2.5 GiB per worker. A flood of requests with distinct x-litellm-call-id values and large compressible tool outputs could exhaust a shared proxy worker. Add a global byte budget (_MAX_TOTAL_STORE_BYTES, 256 MiB) across all entries. A running total is maintained on every insert/eviction so the cap is enforced without re-encoding the whole store on the request path; oldest entries are evicted once the budget is exceeded, always keeping the most-recent entry so recovery still works for the request populating the store. +2 regression tests. * fix(guardrails): gate and bound Compresr recovery loop Two hardening fixes to the compresr_retrieve agentic loop: 1. Only run the loop when a retrieve call resolves to recovery state this guardrail actually created for the request. Previously the gate checked only that the caller-supplied tool list contained a compresr_retrieve function and that the model emitted a call, so a caller could define their own same-named tool and force an extra provider round-trip with nothing to recover. The plan now returns run_agentic_loop=False when no requested hash resolves. 2. Bound the follow-up against retrieval amplification: each distinct hash is expanded at most once (repeats get a short marker) and at most _MAX_RETRIEVALS_PER_LOOP calls are honored, so prompting the model to call compresr_retrieve many times with the same marker cannot balloon the follow-up. _retrieve_original now returns None on miss. +3 regression tests; two existing security tests updated to assert the stronger veto behavior (forged/cross-tenant hashes now stop the loop entirely instead of returning a not-found follow-up). * fix(guardrails): warn when Compresr recovery is skipped without auth scope When enable_retrieval is on (the default) but the proxy has no per-key auth, the request has no caller scope, so recovery is silently disabled: content is compressed but the compresr_retrieve tool is never injected and the originals are dropped, with no runtime indication. Emit a one-shot call-time warning so operators can see recovery is being suppressed and configure virtual-key auth. +1 regression test. * style(guardrails): tighten Compresr guardrail comments Condense the verbose multi-line inline comments and the api_base docstring to concise form. No behavior change. * fix(guardrails): keep injected tool on Responses API + bound recovery markers by byte cap Two fixes for reviewer-flagged defects in the Compresr guardrail: - Responses API: _merge_tools_after_guardrail iterated only over the request's original tools, dropping any tool a guardrail appended (the compresr_retrieve recovery tool) whenever the request already had tools. Keep the appended tools so recovery works on /v1/responses. - Recovery markers: markers + originals were built for every compressed target before the per-call byte cap trimmed the store, so an evicted original left a marker the model could never retrieve. Attach recovery only while the store (existing entries under the same key + this call's originals) stays within the cap, so a shipped marker is always retrievable -- including on a later turn that reuses the store key. Adds regression tests for both paths. * refactor(guardrails): extract _existing_originals to keep apply_guardrail under the complexity gate The byte-cap fix added a branch to apply_guardrail, tipping it past the C901 complexity ceiling. Move the store lookup into a small helper; no behavior change. * fix(guardrails): harden Compresr SSRF blocklist, re-arm no-scope warning, tolerate odd tool shapes * fix(guardrails): rerun input guardrails on Compresr retrieval follow-up * chore: remove unrelated deepkeep files committed by mistake --------- Co-authored-by: charafkamel <charafkamel@live.com> |
||
|
yucheng-berri
|
e3546c20af
|
feat(bedrock guardrails): add resource-less InvokeGuardrailChecks (detect-only) mode (#33299)
* feat(bedrock guardrails): add resource-less InvokeGuardrailChecks (detect-only) mode Adopted from #30830 by OS-joaocastilho; the original PR was merged into litellm_oss_staging_230626, which never landed, so this re-lands it on litellm_internal_staging Beyond the original diff, this fold includes the review fixups that were made on the staging branch (warn on unrecognized check keys, keep empty known checks as enable-with-defaults, fail fast when the checks block has no usable keys, tz-aware datetimes, stricter typing) and adapts the block path to the ModifyResponseException contract from LIT-4186, which replaced GuardrailInterventionNormalStringError after the original PR was written * fix(bedrock guardrails): only evaluate configured checks in violation collection An unsolicited score in the InvokeGuardrailChecks response (e.g. a future API revision returning checks the user never requested) previously fell through to the default 0.5 threshold and could block a request the user only asked to scan with other checks. Violation collection now skips any check absent from the configured checks block * fix(bedrock guardrails): fail closed on truncated PII results and tighten checks-path typing Truncated sensitiveInformation results now count as a violation when the PII check is configured: Bedrock omitted detections that were never scored, so sub-threshold visible entries no longer let the request pass. Also blocks on score == threshold per the documented contract (regression test added), rejects checks combined with guardrailVersion, turns a malformed 200 body into a logged guardrail_failed_to_respond 500 instead of a raw ValidationError, types the checks parameter and violations (BedrockChecksConfigModel, BedrockChecksViolation) instead of dict/object, types _sign_and_post against AWSPreparedRequest, hoists stdlib imports, and builds checks messages without intermediate mutation * fix(bedrock guardrails): tag all InvokeGuardrailChecks INPUT content as user Bedrock excludes system content from prompt-attack evaluation (per the AWS guardrails docs), so mapping a caller-supplied system/developer message onto the system role let a caller hide a prompt injection from the promptAttack check by self-labeling its role. At the proxy every INPUT message is caller-controlled, so all of it is now tagged as untrusted user input, which also matches AWS guidance to tag untrusted content as user input. OUTPUT stays assistant. Removes the now-unused role map; the input-message test asserts the new tagging as a regression * fix(bedrock guardrails): pass prepared request headers to httpx without dict coercion httpx accepts botocore's HTTPHeaders mapping directly, and wrapping it in dict() broke the existing test_bedrock_guardrail_make_api_request_passes_api_key which supplies a bare Mock as the prepared request (dict(Mock) calls Mock.keys()) --------- Co-authored-by: OS-joaocastilho <144790013+OS-joaocastilho@users.noreply.github.com> |
||
|
yucheng-berri
|
b2202cb1aa
|
feat(guardrails): streaming text transformation in generic_guardrail_api (#33110)
* feat(guardrails): support streaming text transformation in generic_guardrail_api * chore(guardrails): address PR review feedback * fix(guardrails): fail closed on tool-call and prefix-rewrite leaks in streaming transform * fix(guardrails): address Bugbot review on streaming transform correctness * fix(guardrails): coerce holdback in handler for in-process guardrails * fix(guardrails): harden streaming transform (holdback coercion, tool-call passthrough, n>1 finish_reason) * test(guardrails): targeted _mode_matches coverage for all guardrail_mode shapes * fix(guardrails): inspect streamed tool calls and harden incremental_diff edge cases * test: move ComplianceChecker mode tests to the compliance PR * fix(guardrails): strip content from tool-call passthrough so streamed text can't bypass the transform * fix(guardrails): four correctness fixes for incremental_diff streaming path Four bug fixes on top of the OSS PR's incremental_diff streaming text transformation, all inside the incremental_diff code paths only. No existing block_only, non-streaming, or pre_call behavior is touched. Fix #1 — Mixed content+tool_call finish_reason ordering _tool_call_passthrough_chunk now takes an optional finish_reason_per_choice map. For a choice carrying both delta.content and delta.tool_calls, finish_reason is stripped from the passthrough and recorded on the map so the final synthetic text chunk delivers it. Without this, SSE-compliant clients stopping at finish_reason drop the guardrailed text — defeating the redaction the whole feature exists for. (Greptile P1 twice, Veria.) Fix #2 — Choice index sort in _process_streaming_transform indices/texts_to_check were derived from dict insertion order. For n>1 streams where choice 1 emits before choice 0, guardrail-returned texts aligned to the input order mapped back to the wrong choice indices on write-back — wrong text goes to wrong choice. Sort raw_by_index.keys() up front so realignment is deterministic. (Bugbot Medium.) Fix #3 — Cross-chunk pre-tool-call text flush With default streaming_sampling_rate=5, text chunks followed by a pure tool-call chunk carrying finish_reason='tool_calls' would emit the passthrough with finish_reason before any transformed text delta had fired. Same failure mode as fix #1 but cross-chunk. Now we flush any accumulated text via _round(is_final=False) BEFORE yielding the tool-call passthrough. (Greptile P1.) Fix #4 — Terminator chunk for deferred finish_reason on empty mutated_text _build_transform_chunk returned None early when mutated_text_per_choice was empty. If a mixed content+tool_call chunk had deferred its finish_reason (via fix #1) and the guardrail then suppressed the text (empty return), the deferred finish_reason was never delivered. Now on is_final=True with empty mutated_text_per_choice, we emit a terminator carrying finish_reason per choice from finish_reason_per_choice. (Bugbot High.) Also normalized Optional[X] → X | None across the OSS PR's added surface via ruff UP045 autofix to keep the strict-rule gate within budget. Pure mechanical typing style change, no semantic effect. Regression tests for all four fixes: - test_mixed_chunk_finish_reason_arrives_after_transformed_text (#1) - test_text_flush_precedes_tool_call_passthrough (#3) - test_final_finish_reason_flushed_when_guardrail_suppresses_text (#4) - test_transform_sends_texts_sorted_by_choice_index (#2) All fixes reachable only when streaming_transform_mode == 'incremental_diff' is configured (via _run_incremental_transform_stream) or when a StreamTransformSink is present (via _process_streaming_transform). Verified scope-clean: no changes to block_only, non-streaming, pre_call, moderation, or sibling guardrails. --------- Co-authored-by: Marton Schneider <marton@schneider.co.nl> |
||
|
yucheng-berri
|
78e5c43301
|
feat(lasso): send source.type=litellm for Used By attribution (#33090)
Co-authored-by: Or Gershoni <org@lasso.security> |
||
|
yucheng-berri
|
ff2b690dd4
|
fix(guardrails): walk custom_tool_call_output items in _content_utils (#32969)
* fix(guardrails): walk custom_tool_call_output items in _content_utils * Change _OUTPUT_ITEM_TYPES to Frozenset type * fix(guardrails): use builtin frozenset generic for _OUTPUT_ITEM_TYPES annotation Frozenset is not a defined name (typing exports FrozenSet, the builtin is frozenset), so module import raised NameError and broke every proxy test suite. The builtin generic is valid on the supported python floor (3.10) and keeps the UP006 ruff-strict budget at its ceiling, which the typing alias would exceed |
||
|
yucheng-berri
|
f61fd2fb6d
|
fix(xecguard): sanitize scan result before recording it for logging (#32935) | ||
|
yucheng-berri
|
f947ef14a2
|
fix(xecguard): use StandardLoggingGuardrailInformation in logging hook (#32911)
XecGuard's async_logging_hook wrote a bare dict to standard_logging_object["guardrail_information"] while the typed contract is Optional[List[StandardLoggingGuardrailInformation]]. Readers that iterated the field walked dict keys, raised on info.get, or silently dropped the entry from guardrail usage tracking and spend-log writes Construct the typed entry and append it to the existing list or create a new one, matching the shared helper pattern. Record the configured guardrail name instead of a hardcoded "xecguard" and pass the GuardrailEventHooks enum for guardrail_mode |
||
|
yucheng-berri
|
e7f41442d0
|
feat(guardrails): add pre_mcp_call support to Content Filter (#32936)
* feat(guardrails): add pre_mcp_call support to Content Filter * test(guardrails): cover canonical MCP key gate under pre_mcp_call mode * fix(guardrails): scan MCP arguments per value and gate mixed-mode scans by call type * fix(guardrails): cap MCP argument scan depth and register the walker with the recursion detector * test(guardrails): update LIT-4226 UI settings tests for content filter pre_mcp_call support * fix(guardrails): use builtin generics in MCP scan annotations to satisfy strict-rule budget Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
yucheng-berri
|
69c5839cc0
|
fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712)
* fix(guardrails): filter Add-Guardrail mode dropdown per provider The GET /guardrails/ui/add_guardrail_settings endpoint returned every GuardrailEventHooks value in one flat supported_modes list, so the Admin UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving Content Filter or Tool Permission with pre_mcp_call then failed with a 400 because those guardrails' server-side supported_event_hooks list excludes it. Expose each guardrail's supported hooks as a get_supported_event_hooks classmethod on CustomGuardrail (mirrors the existing get_config_model pattern) and have the endpoint iterate guardrail_class_registry to build a supported_modes_by_provider map. The UI Mode dropdown filters by that map when the selected provider is known and falls back to the global list otherwise. __init__ now sources its own supported_event_hooks list from the classmethod so the two sides can't drift. Also register BedrockGuardrail, ToolPermissionGuardrail, lakera, lakera_v2, and presidio in guardrail_class_registry so they participate in the map (they were previously only in guardrail_initializer_registry and had no class-registry entry). Behavior change: guardrails that previously had no supported_event_hooks declared (aim, javelin, azure/text_moderation, cato_networks, crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx, prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai, lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now validate the configured mode at instantiation. Existing configs where the mode was silently a no-op will fail at proxy startup with a clear validation error rather than running as a broken guardrail. Resolves LIT-4226 * fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form Address Greptile P1 (startup break) and P2 (edit form UX): LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported event_hook, unchanged behavior for the guardrails validated pre-PR). Setting it to false logs a warning and continues, giving deployments an opt-out while they fix configs that now surface as errors instead of silently no-op'ing. Regression test covers both modes. Edit form now surfaces the currently-saved mode even when it is not in the filtered per-provider list, so a legacy row (e.g. content_filter saved with pre_mcp_call before this fix) no longer disappears from the dropdown; the option renders with a 'not supported by <provider>' note so the user knows to pick another. * fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint Audited every get_supported_event_hooks classmethod against the hooks each guardrail's own tests exercise and its handler methods. Five were too narrow and their tests caught it in CI: rubrik gains pre_call, presidio gains during_call and pre_mcp_call, prompt_security, onyx and qualifire gain during_call. The remaining classes match either their original __init__ declarations or their exercised modes exactly. Cursor review fixes: the Add form now drops selected modes the new provider does not support when the user switches providers, so a pre_mcp_call selection cannot ride along into a provider that rejects it at save; the edit form handles list-shaped stored modes instead of treating mode as always a string. Extracted shared toModeArray and getSupportedModesForProvider helpers into guardrail_info_helpers so both forms use one implementation, typed the remaining any usages in both forms, removed nested ternaries, and committed the ratcheted-down eslint metrics and pruned suppressions |
||
|
Yuneng Jiang
|
799a559871
|
fix(guardrails): show YAML-defined guardrails in the Guardrail Monitor
The /guardrails/usage/{overview,detail,logs} endpoints resolved guardrails only
from the litellm_guardrailstable Prisma table, so guardrails defined in
config.yaml (which live only in IN_MEMORY_GUARDRAIL_HANDLER) were invisible:
detail 404'd, overview omitted them or rendered them as Custom/Guardrail
orphans, and logs missed their logical-name alias.
Add config-owned accessors (list_config_guardrails, get_config_guardrail_by_id)
to the in-memory handler and use them in the usage endpoints, mirroring the
union/fallback already used by list_guardrails_v2 and get_guardrail_info. Also
preserve guardrail_info when storing a config guardrail (type/description were
dropped at initialize time) and read the Prisma-row / dict / LitellmParams
shapes uniformly.
Resolves LIT-2529
|
||
|
yucheng-berri
|
5cf269088c
|
fix(proxy): capture logging_obj before post_call_failure_hook pops it in ModifyResponseException streaming path (#32665)
* fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException The Bedrock-specific GuardrailInterventionNormalStringError predates the unified guardrails refactor and no proxy code path handles it, so a block with the flag set surfaced as an uncaught Exception -> HTTP 500 in pre_call mode and was silently discarded in during_call mode (model call proceeded in the parallel asyncio.gather; the block hook's data["mock_response"] mutation happened after route_request had already unpacked kwargs). Convert the block to ModifyResponseException at the raise site inside make_bedrock_api_request. That exception is the industry-standard proxy contract already caught in proxy_server, anthropic_endpoints, response_api _endpoints, and pass_through_endpoints; it turns into a 200 response with finish_reason=content_filter and the block message as content, which is exactly what the flag was documented to yield. Post-call blocks attach the LLM response to original_response so the synthetic reply reports the upstream call's real token usage instead of zero. Deletes the now-orphaned GuardrailInterventionNormalStringError class and the dead create_guardrail_blocked_response / mock_response plumbing in the Bedrock hooks; updates the existing tests that had locked in the buggy contract. Resolves LIT-4186 * chore(guardrails/bedrock): drop dead str branch in _update_messages_with_updated_bedrock_guardrail_response Follow-up to the disable_exception_on_block fix. That method used to receive either a BedrockGuardrailResponse or a plain string (the block message, when the flag was set). Now that a block always raises ModifyResponseException before this method runs, the string branch is unreachable; tighten the type to BedrockGuardrailResponse and delete the guard. * fix(guardrails/bedrock): streaming post_call block yields synthetic stream instead of surfacing as SSE 500 Regression from the LIT-4186 refactor: pre-refactor, the streaming post_call iterator caught GuardrailInterventionNormalStringError locally and replaced the assembled response with a synthetic content-filter message, then re-emitted it as chunks via MockResponseIterator. After the refactor the exception was re-raised as ModifyResponseException, which async_streaming_data_generator serializes as a proxy 500 error frame because the SSE response headers are already flushed by the time the block fires. Non-streaming paths still let ModifyResponseException propagate to the endpoint handler (which converts it into a 200). Streaming can't do that, so keep the local synthesis: on the exception, rebind the assembled response to a ModelResponse whose single choice carries the block message as content and finish_reason=content_filter, and let the downstream MockResponseIterator emit it as chunks. Same shape a non-streaming block produces. Adds a mapped-file regression test that mutation-kills the raise behavior and locks in the synthetic-stream contract. * fix(guardrails/bedrock): preserve upstream usage on streaming post_call block Non-streaming post_call blocks report the upstream LLM call's real token usage via ModifyResponseException.original_response, which the endpoint handler unwraps through _blocked_response_usage. Streaming post_call synthesizes its own ModelResponse locally (the exception can't escape the SSE generator), and previously left .usage unset, so the client saw accurate billing on non-streaming blocks and zero on streaming blocks -- silent revenue leak. Copy the assembled response's .usage onto the synthetic block response before yielding. Pre-refactor code had the same gap (create_guardrail_blocked_response never set usage); this is a net improvement, not a regression fix. * fix(proxy): capture logging_obj before post_call_failure_hook pops it in ModifyResponseException streaming path post_call_failure_hook removes litellm_logging_obj from request_data before iterating callbacks (it's not serialisable). The streaming branch of the ModifyResponseException handler read it from _data after that call, so it always received None and CustomStreamWrapper.__init__ crashed with AttributeError: NoneType has no attribute model_call_details. Capture it before the hook runs so the streaming path gets a valid object. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy): add regression for streaming ModifyResponseException logging_obj capture Covers the bug where logging_obj was read from request_data after post_call_failure_hook had already popped it, causing CustomStreamWrapper to crash with AttributeError. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy): drive real chat_completion in ModifyResponseException streaming logging_obj regression The original test inlined the fix pattern (capture before pop) in its own body rather than calling the actual chat_completion handler in proxy_server.py, so a revert of the fix left the test passing. Confirmed via mutation check: reverting the two-line source fix and re-running left the test green. Rewrite the test to drive chat_completion directly: - patch _read_request_body so chat_completion sees the seeded dict - patch ProxyBaseLLMRequestProcessing.base_process_llm_request to raise ModifyResponseException with the same request_data - patch proxy_logging_obj so post_call_failure_hook mutates the dict the way production does (pops litellm_logging_obj) - intercept CustomStreamWrapper.__init__ and assert logging_obj is the non-None object seeded in request_data Mutation-verified: reverting the source fix now surfaces the exact production crash inside CustomStreamWrapper's __init__ (AttributeError: NoneType has no attribute model_call_details) rather than a silently-passing test. Addresses Greptile P1 on PR #32665. --------- Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
6eed38bcfb
|
fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException (#32289)
* fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException The Bedrock-specific GuardrailInterventionNormalStringError predates the unified guardrails refactor and no proxy code path handles it, so a block with the flag set surfaced as an uncaught Exception -> HTTP 500 in pre_call mode and was silently discarded in during_call mode (model call proceeded in the parallel asyncio.gather; the block hook's data["mock_response"] mutation happened after route_request had already unpacked kwargs). Convert the block to ModifyResponseException at the raise site inside make_bedrock_api_request. That exception is the industry-standard proxy contract already caught in proxy_server, anthropic_endpoints, response_api _endpoints, and pass_through_endpoints; it turns into a 200 response with finish_reason=content_filter and the block message as content, which is exactly what the flag was documented to yield. Post-call blocks attach the LLM response to original_response so the synthetic reply reports the upstream call's real token usage instead of zero. Deletes the now-orphaned GuardrailInterventionNormalStringError class and the dead create_guardrail_blocked_response / mock_response plumbing in the Bedrock hooks; updates the existing tests that had locked in the buggy contract. Resolves LIT-4186 * chore(guardrails/bedrock): drop dead str branch in _update_messages_with_updated_bedrock_guardrail_response Follow-up to the disable_exception_on_block fix. That method used to receive either a BedrockGuardrailResponse or a plain string (the block message, when the flag was set). Now that a block always raises ModifyResponseException before this method runs, the string branch is unreachable; tighten the type to BedrockGuardrailResponse and delete the guard. * fix(guardrails/bedrock): streaming post_call block yields synthetic stream instead of surfacing as SSE 500 Regression from the LIT-4186 refactor: pre-refactor, the streaming post_call iterator caught GuardrailInterventionNormalStringError locally and replaced the assembled response with a synthetic content-filter message, then re-emitted it as chunks via MockResponseIterator. After the refactor the exception was re-raised as ModifyResponseException, which async_streaming_data_generator serializes as a proxy 500 error frame because the SSE response headers are already flushed by the time the block fires. Non-streaming paths still let ModifyResponseException propagate to the endpoint handler (which converts it into a 200). Streaming can't do that, so keep the local synthesis: on the exception, rebind the assembled response to a ModelResponse whose single choice carries the block message as content and finish_reason=content_filter, and let the downstream MockResponseIterator emit it as chunks. Same shape a non-streaming block produces. Adds a mapped-file regression test that mutation-kills the raise behavior and locks in the synthetic-stream contract. * fix(guardrails/bedrock): preserve upstream usage on streaming post_call block Non-streaming post_call blocks report the upstream LLM call's real token usage via ModifyResponseException.original_response, which the endpoint handler unwraps through _blocked_response_usage. Streaming post_call synthesizes its own ModelResponse locally (the exception can't escape the SSE generator), and previously left .usage unset, so the client saw accurate billing on non-streaming blocks and zero on streaming blocks -- silent revenue leak. Copy the assembled response's .usage onto the synthetic block response before yielding. Pre-refactor code had the same gap (create_guardrail_blocked_response never set usage); this is a net improvement, not a regression fix. |