mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-09 03:18:44 +00:00
8177230a29
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails Pre-call guardrails run sequentially because each may mutate the request payload and later guardrails depend on earlier mutations. Deployments with several slow block-only pre_call guardrails (external moderation, Bedrock, LLM-judge) therefore pay the sum of their latencies. during_call guardrails run concurrently but alongside the LLM call, so a violating payload has already been sent, which is unacceptable when the request must never reach the model. This adds a per-guardrail run_in_parallel flag (default off). Guardrails that opt in are pulled out of the sequential loop and run concurrently via asyncio.gather after every sequential (payload-mutating) guardrail has run, so they observe the mutated payload and still form a hard barrier before the LLM call; the first to raise blocks the request. Their returned data is discarded since they are declared block-only. The flag is wired from LitellmParams onto the guardrail instance at the same generic choke point in initialize_guardrail that already sets skip_system_message_in_guardrail, so no per-provider initializer needs to change. * feat(guardrails): extend run_in_parallel opt-in to post_call guardrails post_call_success_hook ran guardrails sequentially for the same reason pre_call did: response-modifying guardrails thread the response forward. But block-only output scanners (which read the response and reject on violation without changing it) serialize for no benefit and add latency. This reuses the existing run_in_parallel flag for the post_call hook. Opted-in post_call guardrails are pulled out of the sequential loop and run concurrently via asyncio.gather after the sequential (response-modifying) guardrails and before the non-guardrail CustomLogger callbacks, so they inspect the final response and still block it from reaching the client if any raises. Their returned response is discarded since they are block-only. The apply_guardrail path sets data["guardrail_to_apply"] immediately before awaiting, and unified_guardrail pops it before its first suspension point, so concurrent guardrails never race on that key under asyncio's cooperative scheduling. * fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes Addresses review feedback on the run_in_parallel opt-in. asyncio.gather propagated the first exception without cancelling or awaiting the siblings, so a block at t=0 left the other guardrails running as unobserved background tasks (wasted external calls plus event-loop warnings), and a fast SensitiveDataRouteException/ModifyResponseException could return a reroute or passthrough before a slower block finished, letting crafted input bypass the block. Both the pre_call and post_call parallel batches now gather with return_exceptions=True so every guardrail runs to completion, then raise any blocking exception ahead of a flow-changing one. The registry choke point wrote bool(None)==False onto every instance when the config omitted run_in_parallel, silently disabling a constructor-set default; it now only writes when the config provides an explicit value. * fix(guardrails): record lifecycle logs for every concurrently-run guardrail The log_guardrail_information decorator skipped its auto-record when it saw that the count of standard_logging_guardrail_information entries in the shared request_data had grown during the wrapped call, taking that as proof the wrapped function had recorded its own richer entry. That heuristic breaks the moment guardrails run concurrently (parallel pre_call/post_call, during_call): a sibling guardrail's append inflates the shared count, so a guardrail that did not self-record wrongly concludes it already did and drops its own entry. The result is that enabling run_in_parallel silently loses per-guardrail lifecycle logs, so the Admin UI Request Lifecycle timeline and downstream loggers (Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent guardrails. Replace the shared-count heuristic with a ContextVar flag set when a guardrail records its own entry. asyncio copies the context into each gathered task, so the flag is isolated per concurrent guardrail while still catching the self-record-then-skip-auto-record case within a single invocation. * test(guardrails): declare run_in_parallel on post_call guardrail mocks The post_call partition reads run_in_parallel on every CustomGuardrail callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is set in __init__, not on the class) so the attribute access raised, and even a class-level default would return a truthy child mock that wrongly routes the double into the parallel batch. Declare the flag False on the shared mock factories so these pre-existing hook tests exercise the sequential path they assert on. * fix(guardrails): harden run_in_parallel reads and address review feedback Read run_in_parallel via getattr(..., False) in the pre_call and post_call partitions so a third-party CustomGuardrail subclass that overrides __init__ without chaining super().__init__() no longer raises AttributeError on a path that previously worked. Drop the redundant in-function GuardrailEventHooks import in _run_parallel_post_call_guardrails (already imported module-level). Remove the flaky wall-clock upper-bound assertions from the two concurrency tests; the all-start-before-any-end overlap assertion is the timing-independent signal that actually proves concurrency.
88 lines
3.5 KiB
Python
88 lines
3.5 KiB
Python
import json
|
|
import os
|
|
import sys
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
import pytest
|
|
|
|
sys.path.insert(0, os.path.abspath("../../..")) # Adds the parent directory to the system path
|
|
|
|
from litellm.proxy.guardrails.guardrail_registry import InMemoryGuardrailHandler
|
|
from litellm.types.guardrails import SupportedGuardrailIntegrations
|
|
|
|
|
|
def test_initialize_presidio_guardrail():
|
|
"""
|
|
Test that initialize_guardrail correctly uses registered initializers
|
|
for presidio guardrail
|
|
"""
|
|
# Setup test data for a non-custom guardrail (using Presidio as an example)
|
|
test_guardrail = {
|
|
"guardrail_name": "test_presidio_guardrail",
|
|
"litellm_params": {
|
|
"guardrail": SupportedGuardrailIntegrations.PRESIDIO.value,
|
|
"mode": "pre_call",
|
|
"presidio_analyzer_api_base": "https://fakelink.com/v1/presidio/analyze",
|
|
"presidio_anonymizer_api_base": "https://fakelink.com/v1/presidio/anonymize",
|
|
},
|
|
}
|
|
|
|
# Call the initialize_guardrail method
|
|
guardrail_handler = InMemoryGuardrailHandler()
|
|
result = guardrail_handler.initialize_guardrail(
|
|
guardrail=test_guardrail,
|
|
)
|
|
|
|
assert result["guardrail_name"] == "test_presidio_guardrail"
|
|
assert result["litellm_params"].guardrail == SupportedGuardrailIntegrations.PRESIDIO.value
|
|
assert result["litellm_params"].mode == "pre_call"
|
|
|
|
|
|
def test_initialize_guardrail_preserves_guardrail_info():
|
|
"""
|
|
Regression (LIT-2529): initialize_guardrail must carry guardrail_info into the
|
|
stored in-memory Guardrail. Dropping it left the Guardrail Monitor's usage
|
|
endpoints unable to render type/description for YAML-defined guardrails.
|
|
"""
|
|
test_guardrail = {
|
|
"guardrail_name": "test_presidio_with_info",
|
|
"litellm_params": {
|
|
"guardrail": SupportedGuardrailIntegrations.PRESIDIO.value,
|
|
"mode": "pre_call",
|
|
"presidio_analyzer_api_base": "https://fakelink.com/v1/presidio/analyze",
|
|
"presidio_anonymizer_api_base": "https://fakelink.com/v1/presidio/anonymize",
|
|
},
|
|
"guardrail_info": {"type": "PII", "description": "masks PII"},
|
|
}
|
|
|
|
guardrail_handler = InMemoryGuardrailHandler()
|
|
result = guardrail_handler.initialize_guardrail(guardrail=test_guardrail)
|
|
|
|
assert result is not None
|
|
assert result["guardrail_info"] == {"type": "PII", "description": "masks PII"}
|
|
stored = guardrail_handler.IN_MEMORY_GUARDRAILS[result["guardrail_id"]]
|
|
assert stored["guardrail_info"] == {"type": "PII", "description": "masks PII"}
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"config_value, expected",
|
|
[(True, True), (False, False), (None, False)],
|
|
)
|
|
def test_initialize_guardrail_sets_run_in_parallel(config_value, expected):
|
|
"""run_in_parallel from litellm_params must reach the built guardrail instance."""
|
|
litellm_params = {
|
|
"guardrail": SupportedGuardrailIntegrations.PRESIDIO.value,
|
|
"mode": "pre_call",
|
|
"presidio_analyzer_api_base": "https://fakelink.com/v1/presidio/analyze",
|
|
"presidio_anonymizer_api_base": "https://fakelink.com/v1/presidio/anonymize",
|
|
}
|
|
if config_value is not None:
|
|
litellm_params["run_in_parallel"] = config_value
|
|
|
|
guardrail_handler = InMemoryGuardrailHandler()
|
|
result = guardrail_handler.initialize_guardrail(
|
|
guardrail={"guardrail_name": "test_parallel_flag", "litellm_params": litellm_params},
|
|
)
|
|
|
|
custom_guardrail = guardrail_handler.guardrail_id_to_custom_guardrail[result["guardrail_id"]]
|
|
assert custom_guardrail.run_in_parallel is expected
|