mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-11 22:51:28 +00:00
7 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Deepanshu Lulla
|
72f1b3e969
|
feat(guardrails): add Lakera v2 skip-message honoring and advisory (inject_system_message) mode (#34940)
* feat(guardrails): honor Lakera v2 skip-message flags and add advisory (inject_system_message) mode Squashed rebase of bugfix/lakera-v2-skip-system-tool-messages onto latest litellm_internal_staging (900+ commits ahead; a commit-by-commit rebase hit repeated conflicts against the same files across earlier review-round commits, so the branch's cumulative diff was reapplied in one pass instead). Adds skip_system_message_in_guardrail/skip_tool_message_in_guardrail support to Lakera v2, a third on_flagged: "inject_system_message" advisory mode, and the associated masking-safety-guard hardening (multimodal content, non- maskable message fields, combined messages+input, and structured Responses- API input in advisory delivery) found across this PR's review rounds. * fix(guardrails): don't let one invalid guardrail config crash proxy boot init_guardrails_v2 had no try/except around initialize_guardrail, so a guardrail whose litellm_params fail validation at construction time (for example Lakera's on_flagged=inject_system_message combined with mode=during_call, or a malformed advisory_system_message template) raised uncaught and crashed the entire proxy at startup, taking down every other, correctly-configured guardrail in the list. Catch ValueError/TypeError per guardrail, log a warning, and skip it, matching the same pattern already used for the DB-driven guardrail-creation path in guardrail_endpoints.py. * fix(guardrails): preserve message fields and mask PII before advising in Lakera v2 Mask-in-place degraded to a hard block for any message carrying a field beyond role/content (tool_call_id, tool_calls, name, cache_control), for a message excluded by skip_system_message_in_guardrail/skip_tool_message_in_guardrail, or for a message with no inspectable text, since it rewrote data["messages"] wholesale from a synthetic role/content-only list built for the Lakera API call. That made masking effectively unusable for any real tool-calling conversation and made the skip flags flip every PII-only violation to a hard block instead of masking just the in-scope text. Replace the wholesale rewrite with a scope-index merge, reusing the same merge_guardrailed_scoped_messages helper the OpenAI/Anthropic guardrail translation handlers already use for this: patch content in place on a copy of each original message actually sent to Lakera, and leave every skipped/no-text/out-of-scope message untouched at its original position. This also fixes on_flagged="inject_system_message" (advisory mode) shipping raw unmasked PII to the model: a PII-only violation is now masked the same way regardless of on_flagged, and the advisory note is reserved for flags masking can't resolve on its own. Addresses maintainer-reported regressions on BerriAI/litellm#34940. * fix(guardrails): satisfy new lint gates for the masking/advisory fix Parameterize the write-back helper's dict param and suppress the two new lint rules that landed on the base while this branch was in flight: TQ008 (patching an internal collaborator) for two pre-existing tests unrelated to this change, and LIT001 for a param that genuinely needs to mutate the caller's request dict in place. * fix(guardrails): normalize role casing in Lakera v2 masking scope, log skipped guardrails louder Greptile finding: the masking scope helper compared roles case-sensitively while filter_messages_by_skip_flags (used to build what's actually sent to Lakera) normalizes casing, so an uppercase-cased "System"/"TOOL" role survived the scope filter but was excluded from the inspected list. The resulting length mismatch raised inside the strict positional zip, turning a maskable PII-only violation into an unhandled request failure. Lowercase the role comparison to match. Also, per veria-ai's finding that a skipped invalid guardrail now fails open: log it at error level with an explicit note that the proxy is starting without that guardrail, so it's not mistaken for routine info. * fix(guardrails): mask maskable PII in mixed violations before advising in Lakera v2 on_flagged="inject_system_message" only masked when a violation was PII-only; a mixed violation (PII plus a non-PII flag like prompt injection) fell straight through to the advisory branch with the raw PII still in place, in both async_pre_call_hook and async_moderation_hook. Mask whatever Lakera returned location data for before appending or logging the advisory, so a mixed violation never ships raw PII just because something else was also flagged. Also degrade to blocking, same as block mode already does, when nothing can be safely masked at all (multimodal content, or messages combined with a Responses API input field) instead of showing an advisory note next to raw, unredacted content. Widened call_v2_guard/_mask_pii_in_messages/the write-back helper's message parameters from list to Sequence to match what's actually passed through from _filter_skipped_messages, instead of duplicating list(...) casts at every call site. * fix(guardrails): don't hard-block advisory mode for non-PII flags on non-maskable input Bugbot finding: gating the entire inject_system_message branch on is_multimodal_input hard-blocked every flagged request on Responses instructions, combined messages+input, or multimodal content, including a prompt-injection-only violation with no PII at all. Masking safety only matters when there's actual PII to mask; a violation with no PII needs no masking, so the advisory should still be delivered normally. Only degrade to blocking when the breakdown actually contains a PII detection and masking isn't safely possible. Otherwise, mask whatever's maskable (if any) and deliver the advisory as before. * fix(guardrails): require payload and breakdown for Lakera v2 advisory mode Advisory mode's mixed-violation masking safety net can only redact detected PII when Lakera's response carries both the breakdown (to detect a PII hit at all) and payload (the location data to mask by). payload=False or breakdown=False alongside on_flagged='inject_system_message' silently forwarded raw PII next to the advisory note. Reject that combination at construction and hot-reload time instead. * fix(guardrails): skip_system_message_in_guardrail must not force-block Lakera masking _has_responses_instructions treated any non-empty data["instructions"] as unsafe to mask regardless of skip_system_message_in_guardrail, even though that flag excludes the instructions-derived synthetic system message from what Lakera ever inspects. PII detected purely in the maskable non-system content was force-blocked instead of masked. Also fixes pre-existing LIT010 (missing Final) violations in _has_responses_instructions, _breakdown_has_pii_violation, and async_post_call_success_hook that the rebase's lowered budget ceiling now flags. * chore: retrigger CI (GitHub Actions runner-acquisition failure on prior push) * fix(guardrails): address maintainer review findings on Lakera v2 advisory mode - Gate advisory_system_message template validation on on_flagged= 'inject_system_message', since block/monitor mode never reads it. - Allow on_flagged='inject_system_message' with mode='during_call' at construction/hot-reload instead of rejecting it; async_moderation_hook already degrades gracefully (masks if possible, else logs a warning). - reinitialize_guardrail now restores the previous live instance when the new config fails to initialize, instead of leaving the guardrail deleted entirely with nothing enforcing it. - PATCH /guardrails/{id} rolls back the DB write and returns 422 when the in-memory sync rejects the new config, instead of persisting a config that never actually took effect and returning 200. - Qualifire now rejects on_flagged values it doesn't implement (only Lakera should accept 'inject_system_message'; LitellmParams flattens the field across every guardrail config mixin). * fix(tests): satisfy lint gates and update collateral test for advisory-mode fixes - Add match= to a too-broad pytest.raises(ValueError), and suppress the new TQ008 mocker.patch findings (same pattern already used by sibling scenarios in this test). - test_init_guardrails_v2_skips_invalid_guardrail_instead_of_crashing_boot used mode='during_call' + on_flagged='inject_system_message' as its invalid-config example; that combination is now accepted, so swap in the payload/breakdown-missing case and add a test confirming during_call advisory mode constructs successfully. * docs(CLAUDE.md): auto-capture review learnings without being asked This session found three real bugs a human maintainer caught after eight rounds of bot review and live-proxy verification all missed them. Add a standing instruction to write learnings.md entries the moment a root cause is understood, in both the repo-wide file and any relevant skill's own file, instead of relying on being asked. * feat(guardrails): add scan_raw_request flag so YAML order can't change enforcement Maintainer finding on BerriAI/litellm#34940: guardrails for the same hook run sequentially over one shared, progressively-mutated request dict, so declaring a masking guardrail before a blocking one hides the violation from it (200 vs 400 depending purely on YAML order). scan_raw_request opts a guardrail into always evaluating a snapshot taken before any guardrail in the hook ran, regardless of its declared position. Same contract as run_in_parallel: block-only, its own mutations discarded. Verified live: real proxy, real Gemini call, two custom guardrails (a redactor then a blocker). Same request, same declared order -- without the flag the blocker never sees the raw secret (200); with it, the blocker correctly rejects before any provider call (400). * fix(guardrails): harden scan_raw_request against review findings - Use safe_deep_copy instead of a bare deepcopy for the raw-request snapshot; request payloads commonly carry unpicklable objects (e.g. an otel span in metadata), which previously raised on every guarded request when tracing was enabled (Bugbot, High). - Only compute the snapshot when a guardrail actually opted in, and take it before _maybe_execute_pipelines runs, so a pipeline-mutated payload can't hide a violation from a scan_raw_request guardrail outside the pipeline (veria-ai). - Log a warning when a scan_raw_request guardrail returns a modified payload, since that mutation is discarded and the combination is otherwise silently exploitable for a masking-capable integration misconfigured this way (veria-ai). * chore(openapi): regenerate lazy snapshot and dashboard schema types The lazy OpenAPI snapshot (litellm/proxy/_lazy_openapi_snapshot.json) and the derived dashboard schema.d.ts had drifted stale relative to the guardrail config model changes across this PR's rounds (advisory mode, scan_raw_request, and upstream additions picked up by rebasing). Regenerated via the CI's own documented fix: uv run python -m litellm.proxy._lazy_openapi_snapshot npm run gen:api (via make check) * chore(openapi): pick up cache_hit_filter field after rebase * fix(guardrails): stop scan_raw_request warning from firing on every call _process_guardrail_callback always returns a dict once a guardrail runs (mark_pre_call_hook_ran unconditionally stamps bookkeeping metadata), so comparing the result to non-None warned on every request even when the guardrail never touched the payload. Compare against a bookkeeping-only baseline instead, so only an actual content mutation triggers the warning. * fix(guardrails): make scan_raw_request snapshots independent of safe_memory_mode safe_deep_copy can return the original object under litellm.safe_memory_mode, or alias a per-key reference on copy failure. Under that mode, the scan_raw_request comparison baseline aliased raw_request_snapshot (and therefore the live request), letting mark_pre_call_hook_ran write a premature execution marker that a deployment-level guardrail sharing the same name would read as "already ran" and skip. Also affected the feature's core isolation guarantee: input_data itself could alias the live request under the same mode. Replace every scan_raw_request snapshot with _independent_snapshot, which never returns an alias, only a genuine copy or None. * fix(guardrails): gate during_call mixed-violation masking behind an actual PII check The during_call branch for a mixed violation under on_flagged=inject_system_message unconditionally masked and reassigned data["messages"], even for a pure prompt-injection violation with zero PII, unlike async_pre_call_hook which already gates the same call behind _breakdown_has_pii_violation. The unconditional reassignment touched shared request state during a hook documented as racing with the concurrent LLM dispatch, for no reason when there was nothing to mask. * fix(guardrails): stop scan_raw_request from silently no-op'ing on real requests _independent_snapshot did one whole-dict copy.deepcopy and returned None on any failure. Every real proxy request carries data["litellm_logging_obj"] (a Logging instance nesting a live OTel span with a real lock) by the time pre_call_hook runs, which can never be deep-copied, so the snapshot failed on every real request and silently fell back to the live, unisolated data with no warning -- defeating the entire feature in production while every existing test (none of which set litellm_logging_obj) kept passing. Rework the helper to deep-copy each top-level key independently, falling back to the original reference only for the specific key that fails, same crash tolerance as safe_deep_copy's own per-key fallback. It never returns None now; only the keys scan_raw_request actually depends on (messages/ input, metadata/litellm_metadata) need to be genuinely independent. * fix(guardrails): block during_call when PII can't be safely masked Greptile finding (P1, security): async_moderation_hook's inject_system_message branch had no equivalent to async_pre_call_hook's degrade-to-blocking case for a PII violation on input that can't be safely masked (e.g. combined messages+input). It fell through to the advisory no-op branch and let raw, unredacted PII reach the model with no protection at all. Raising still blocks the response from reaching the caller even though during_call races with the LLM dispatch, the same mechanism on_flagged="block" already relies on for this hook, so add the same block-instead-of-advisory branch pre_call already has. * chore(lint): fix LIT002 ceiling after rebase merge conflict resolution * fix(lint): suppress genuine LIT002 hits instead of padding the ceiling My earlier rebase conflict resolution for type-discipline-budget.json's LIT002 limit was too low, then overcorrected by padding it well above the actual measured count. Root-caused instead: _independent_snapshot and the PATCH-endpoint rollback path legitimately construct plain, mutable request-payload/config dicts (matching this file's existing precedent for the same shape), so suppress those four sites with `# mutable-ok:` rather than reshaping code that must stay a plain dict by contract. Set the limit to the exact current measured total; the small remaining gap vs upstream's own committed ceiling is pre-existing drift in litellm_internal_staging itself (its own tree already measures over its committed limit), not attributable to this PR. * fix(guardrails): stamp live request when a scan_raw_request guardrail runs _run_sequential_guardrail_callback and _run_parallel_pre_call_guardrails only called mark_pre_call_hook_ran on throwaway snapshot copies for a scan_raw_request guardrail, never on the live request returned to the caller. A later async_pre_call_deployment_hook (router-level guardrail re-check) reads that marker on live kwargs to decide whether to skip re-running the same guardrail; since it was never stamped there, the guardrail ran a second time on live data, doubling the external call and re-applying whatever scan_raw_request's contract says should be discarded. * fix(guardrails): revalidate Qualifire's on_flagged on live config reload on_flagged was validated only in __init__. The base CustomGuardrail.update_in_memory_litellm_params is a generic setattr loop with no revalidation, so a live config update (PUT /guardrails/{id}, no restart) could setattr on_flagged="inject_system_message" onto a running instance, bypassing the constructor's rejection -- silently blocking every flagged request under an "advisory" label. Mirrors LakeraAIGuardrail's own update_in_memory_litellm_params override added earlier in this PR. * fix(guardrails): honor scan_raw_request for pipeline-managed guardrails A scan_raw_request=True guardrail that is itself a pipeline step never saw raw_request_snapshot: PipelineExecutor.execute_steps had no way to receive it, and pipeline-managed guardrails are fully excluded from the normal sequential/parallel loops that implement the flag. Such a guardrail silently evaluated whatever an earlier pass_data step in the same pipeline had already rewritten, defeating the flag for pipeline-managed guardrails. Moves the snapshot helper (renamed independent_snapshot) from proxy/utils.py to litellm_core_utils/core_helpers.py so pipeline_executor.py can use the same independent-copy logic without a circular import, threads raw_request_snapshot through _maybe_execute_pipelines and PipelineExecutor.execute_steps/_run_step, and discards a scan_raw_request step's returned data the same way the sequential/parallel loops already do. * chore(openapi): pick up upstream drift after rebase onto litellm_internal_staging * fix(guardrails): stop attempting PII masking during during_call in Lakera v2 Greptile finding (P1, security): during_call runs concurrently with the LLM dispatch. In the common path, the provider call already binds its messages kwarg before this guardrail's coroutine gets a chance to run, let alone before its own network round trip to Lakera completes -- masking here can never reliably reach the outgoing request, and _apply_redacted_messages_back_ preserving_fields reassigns to a new list object rather than mutating in place, so even winning the race wouldn't help. This affected both the PII-only and mixed-violation masking branches, all added in this same PR. Remove masking from async_moderation_hook entirely and let PII violations fall through to the normal on_flagged branching: block under "block" or "inject_system_message" (extending the existing multimodal-only block to cover every PII case, since masking is proven non-functional regardless of input shape), log-and-allow under "monitor" -- consistent with how every other violation type in this hook is already handled. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> |
||
|
yucheng-berri
|
239ec955dc
|
fix(presidio): chunk oversized text before /analyze so large content blocks do not fail (#38483)
* fix(presidio): chunk oversized text before /analyze so large content blocks do not fail The Presidio PII guardrail sent each content block to the analyzer as a single /analyze call with no size check. Analyzer deployments commonly cap the request body (the reporting deployment rejects bodies over 1,000,000 bytes with HTTP 413), so large blocks failed closed, and analyzer latency grew linearly with payload size. analyze_text now splits texts larger than presidio_analyze_chunk_size_bytes (default 500,000 UTF-8 bytes, configurable per guardrail) into overlapping chunks, analyzes them concurrently, remaps each detection's start/end onto the original text, and deduplicates detections from the overlap regions. Anonymization, blocked-entity checks, score filtering, numbered-token unmasking, telemetry, and the dashboard entity positions all consume the remapped global offsets unchanged. Resolves LIT-4785 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(presidio): review-round hardening for chunked analyze - measure the chunk budget on the JSON-serialized text (non-ASCII escapes expand beyond raw UTF-8, so a raw-byte budget could still exceed the analyzer body limit) - share the chunk fan-out semaphore per event loop and instance instead of per call, so many oversized blocks cannot multiply concurrent analyzer calls - apply configured score thresholds and deny list per chunk BEFORE overlap resolution, so a below-threshold span cannot displace a detection the thresholds keep Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
6a0d03914c
|
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
|
||
|
yucheng-berri
|
d4dc2c39e7
|
fix(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing (#36119)
* feat(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing
AWS's ApplyGuardrail API rejects requests whose content exceeds the
account's per-request "maximum input size in text units" quota with a
400 ValidationException. That cap is account/region/policy-dependent
and cannot be predicted from config, so it can only be reacted to.
_make_apply_guardrail_request now tries the whole-content call first
(no behavior change for requests that already fit). On a too-large
ValidationException it bisects the flat content list and retries each
half sequentially, recursing until every piece fits or cannot be split
further, then merges the per-chunk responses (action, assessments,
outputs, usage) into one so callers cannot tell chunking happened. A
real guardrail block on any (sub-)chunk still raises immediately.
Contextual-grounding requests are never chunked: grounding scores the
response holistically against the whole reference source, so
fragmenting it would produce misleading scores.
Each chunk call also gets a small exponential backoff retry on AWS
ThrottlingException (429), since chunking increases the number of
per-second API calls and can trade a 400 for a 429.
All new state is local to a single request's call stack (no shared
cache, no cross-process coordination), so this is safe for
single-pod, multi-pod, and cache-less LiteLLM proxy deployments alike.
* fix(guardrails): address Bedrock ApplyGuardrail chunking review feedback
Fixes three issues flagged in review of the chunking fallback: a single
oversized content item couldn't be split (only list-length bisection was
supported), a chunked request that got recovered still logged a stray
failure telemetry entry alongside the real outcome, and flattening chunk
outputs without positional bookkeeping could misalign masked text onto
the wrong original message once a chunk had nothing to mask.
* test(guardrails): add regression test for multi-level Bedrock guardrail chunking
Confirms the too-large bisection recursion isn't capped at a single split:
a payload that is still oversized after the first halving keeps splitting
until every piece fits, converging on however many chunks it takes rather
than only ever producing two.
* fix(guardrails): hybrid bin-pack+bisection chunking, whitespace-safe splits
Rework Bedrock ApplyGuardrail chunking from pure reactive bisection to a
hybrid strategy: bin-pack content into fixed-budget batches up front as
the fast path, falling back to the existing recursive bisection only for
a batch AWS still rejects as too large. Avoids paying O(log n) round
trips on every oversized request when a single pass would do.
Also switch single-item text splitting from a raw character midpoint to
the nearest whitespace boundary, so a fragment never starts or ends
mid-word. Closes the accidental-severing case from review; the residual
gap (a multi-word denied phrase deliberately straddling the boundary) is
documented as an accepted limitation, since fixing it would require an
overlap window reconciled against masked output with no documented
length-preservation guarantee from AWS.
* chore(ui): regenerate dashboard API types
* fix(guardrails): don't retry an oversized Bedrock guardrail call as a throttle
AWS reports an ApplyGuardrail request that exceeds the per-request
text-unit cap as a ThrottlingException (429), not only as the documented
ValidationException (400). Verified against a live guardrail with an
active content-filter policy: a 3273-text-unit request comes back as
"Input text size (3273 text units) exceeds the maximum allowed (1000 text
units) for the content filter policy (Classic tier)".
The throttle retry keyed off status 429 alone, so every oversized chunk
burned the full backoff-retry budget - each attempt a billed AWS call
preceded by a sleep - before the bisection fallback got a chance, at every
level of the recursion. A size error is not transient; re-posting the same
content can never succeed. It now short-circuits straight to bisection.
Also rename _is_input_too_large_validation_error to
_is_input_too_large_error (it never keyed off the status code, and the
error is not always a ValidationException), correct the docstrings that
asserted a 400, and log at warning level when a split happens so the
recovery is visible without --detailed_debug.
* Revert "chore(ui): regenerate dashboard API types"
This reverts commit ebf8ba2fd57f13bccf7aa6c5dfcac41c74db1ed9.
* fix(guardrails): group all fragments of one item and stop double-logging
Two defects found in review, both invisible to the existing tests.
Fragment grouping assumed a split content item always produces exactly two
adjacent fragments. That holds for one bisection level but not two: an item
split twice yields four fragments, which were regrouped in fixed pairs into
two output entries for a single message. Since masking walks the merged
outputs by a running index across the original, unchunked message list, that
message was written back truncated to its first half and every later message
shifted. Fragments now carry the size of the group they belong to, so any
number of them collapse back into exactly one output entry.
Telemetry was also double-counted. AsyncHTTPHandler.post calls
raise_for_status(), so every non-200 from Bedrock reaches _sign_and_post's
error path, which logged guardrail_failed_to_respond before re-raising as an
HTTPException that the consolidating caller then logged again. A request
recovered by chunking reported one failure per rejected attempt plus a
success. The ApplyGuardrail path now opts out of that per-attempt logging,
since it owns consolidated per-request logging; the connection-level branch
still logs, as nothing else records it.
The existing tests missed both because their mocks return a non-200 response
object, while the real client raises. Added a helper that raises a genuine
httpx.HTTPStatusError so these paths are covered the way production hits
them, plus a case asserting an unrecoverable failure still logs exactly once
rather than zero times.
* refactor(guardrails): move Bedrock chunking rationale into docstrings
The chunking work explained itself with inline comment blocks, which this
repo's conventions do not want. Folded that reasoning into the docstrings of
the functions it describes and dropped the comments, including the
module-level constant blocks and the test-file banner.
No behavior change. The banner also claimed AWS rejects an oversized request
with a 400 ValidationException, which live testing disproved, so removing it
drops a stale claim as well as an internal ticket reference from a public repo.
* feat(guardrails): match AWS default chunk budget and make it configurable
ApplyGuardrail's default quota is 25 text units, roughly 25,000 characters,
per second. Chunking has to respect that throughput limit rather than just the
per-request size, otherwise splitting an oversized request trades a size error
for a throttle. The budget now defaults to 25,000 to match that default for
every user, up from an arbitrary 20,000.
Accounts with raised quotas can spend fewer calls by setting
chunk_budget_chars on the guardrail. A value AWS still rejects as too large is
bisected automatically, so an over-large setting costs an extra round trip
rather than failing the request.
* fix(guardrails): never split a Bedrock text into an empty fragment
_nearest_whitespace_split_index could return len(text) when the only space at
or after the midpoint was the final character, so the first fragment came back
identical to the text AWS had just rejected as too large and the second came
back empty. AWS rejects the unchanged fragment again, and each retry re-splits
it into the same fragment, so an oversized single item shaped like a long
unbroken token with one trailing space exhausted the stack with a
RecursionError instead of scanning or surfacing Bedrock's error.
Candidate boundaries that would leave either side empty are now discarded, and
the raw midpoint is used when none remain. The midpoint is always safe because
_split_bedrock_content only calls this for text of at least two characters.
* style(guardrails): move chunking rationale out of comments and into docstrings
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Restores the source changes intended for
|
||
|
Noah Nistler
|
8177230a29
|
feat(guardrails): add run_in_parallel opt-in for concurrent pre_call and post_call guardrails (#33770)
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails Pre-call guardrails run sequentially because each may mutate the request payload and later guardrails depend on earlier mutations. Deployments with several slow block-only pre_call guardrails (external moderation, Bedrock, LLM-judge) therefore pay the sum of their latencies. during_call guardrails run concurrently but alongside the LLM call, so a violating payload has already been sent, which is unacceptable when the request must never reach the model. This adds a per-guardrail run_in_parallel flag (default off). Guardrails that opt in are pulled out of the sequential loop and run concurrently via asyncio.gather after every sequential (payload-mutating) guardrail has run, so they observe the mutated payload and still form a hard barrier before the LLM call; the first to raise blocks the request. Their returned data is discarded since they are declared block-only. The flag is wired from LitellmParams onto the guardrail instance at the same generic choke point in initialize_guardrail that already sets skip_system_message_in_guardrail, so no per-provider initializer needs to change. * feat(guardrails): extend run_in_parallel opt-in to post_call guardrails post_call_success_hook ran guardrails sequentially for the same reason pre_call did: response-modifying guardrails thread the response forward. But block-only output scanners (which read the response and reject on violation without changing it) serialize for no benefit and add latency. This reuses the existing run_in_parallel flag for the post_call hook. Opted-in post_call guardrails are pulled out of the sequential loop and run concurrently via asyncio.gather after the sequential (response-modifying) guardrails and before the non-guardrail CustomLogger callbacks, so they inspect the final response and still block it from reaching the client if any raises. Their returned response is discarded since they are block-only. The apply_guardrail path sets data["guardrail_to_apply"] immediately before awaiting, and unified_guardrail pops it before its first suspension point, so concurrent guardrails never race on that key under asyncio's cooperative scheduling. * fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes Addresses review feedback on the run_in_parallel opt-in. asyncio.gather propagated the first exception without cancelling or awaiting the siblings, so a block at t=0 left the other guardrails running as unobserved background tasks (wasted external calls plus event-loop warnings), and a fast SensitiveDataRouteException/ModifyResponseException could return a reroute or passthrough before a slower block finished, letting crafted input bypass the block. Both the pre_call and post_call parallel batches now gather with return_exceptions=True so every guardrail runs to completion, then raise any blocking exception ahead of a flow-changing one. The registry choke point wrote bool(None)==False onto every instance when the config omitted run_in_parallel, silently disabling a constructor-set default; it now only writes when the config provides an explicit value. * fix(guardrails): record lifecycle logs for every concurrently-run guardrail The log_guardrail_information decorator skipped its auto-record when it saw that the count of standard_logging_guardrail_information entries in the shared request_data had grown during the wrapped call, taking that as proof the wrapped function had recorded its own richer entry. That heuristic breaks the moment guardrails run concurrently (parallel pre_call/post_call, during_call): a sibling guardrail's append inflates the shared count, so a guardrail that did not self-record wrongly concludes it already did and drops its own entry. The result is that enabling run_in_parallel silently loses per-guardrail lifecycle logs, so the Admin UI Request Lifecycle timeline and downstream loggers (Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent guardrails. Replace the shared-count heuristic with a ContextVar flag set when a guardrail records its own entry. asyncio copies the context into each gathered task, so the flag is isolated per concurrent guardrail while still catching the self-record-then-skip-auto-record case within a single invocation. * test(guardrails): declare run_in_parallel on post_call guardrail mocks The post_call partition reads run_in_parallel on every CustomGuardrail callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is set in __init__, not on the class) so the attribute access raised, and even a class-level default would return a truthy child mock that wrongly routes the double into the parallel batch. Declare the flag False on the shared mock factories so these pre-existing hook tests exercise the sequential path they assert on. * fix(guardrails): harden run_in_parallel reads and address review feedback Read run_in_parallel via getattr(..., False) in the pre_call and post_call partitions so a third-party CustomGuardrail subclass that overrides __init__ without chaining super().__init__() no longer raises AttributeError on a path that previously worked. Drop the redundant in-function GuardrailEventHooks import in _run_parallel_post_call_guardrails (already imported module-level). Remove the flaky wall-clock upper-bound assertions from the two concurrency tests; the all-start-before-any-end overlap assertion is the timing-independent signal that actually proves concurrency. |
||
|
Yuneng Jiang
|
799a559871
|
fix(guardrails): show YAML-defined guardrails in the Guardrail Monitor
The /guardrails/usage/{overview,detail,logs} endpoints resolved guardrails only
from the litellm_guardrailstable Prisma table, so guardrails defined in
config.yaml (which live only in IN_MEMORY_GUARDRAIL_HANDLER) were invisible:
detail 404'd, overview omitted them or rendered them as Custom/Guardrail
orphans, and logs missed their logical-name alias.
Add config-owned accessors (list_config_guardrails, get_config_guardrail_by_id)
to the in-memory handler and use them in the usage endpoints, mirroring the
union/fallback already used by list_guardrails_v2 and get_guardrail_info. Also
preserve guardrail_info when storing a config guardrail (type/description were
dropped at initialize time) and read the Prisma-row / dict / LitellmParams
shapes uniformly.
Resolves LIT-2529
|
||
|
Krish Dholakia
|
ef42461c1e
|
Litellm fix GitHub action testing (#11163)
* test: add __init__.py files * refactor: rename test folder to avoid naming conflict * test: update workflows * test: update tests * test: update imports * test: update tests * test: remove unused import * ci(test-litellm.yml): add pytest retry to github workflow * test: fix test |
Renamed from tests/litellm/proxy/guardrails/test_init_guardrails.py (Browse further)