scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102
litellm/ruff-strict-budget.json
Noah Nistler 8177230a29
feat(guardrails): add run_in_parallel opt-in for concurrent pre_call and post_call guardrails (#33770) 
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails

Pre-call guardrails run sequentially because each may mutate the request
payload and later guardrails depend on earlier mutations. Deployments with
several slow block-only pre_call guardrails (external moderation, Bedrock,
LLM-judge) therefore pay the sum of their latencies. during_call guardrails
run concurrently but alongside the LLM call, so a violating payload has
already been sent, which is unacceptable when the request must never reach
the model.

This adds a per-guardrail run_in_parallel flag (default off). Guardrails that
opt in are pulled out of the sequential loop and run concurrently via
asyncio.gather after every sequential (payload-mutating) guardrail has run, so
they observe the mutated payload and still form a hard barrier before the LLM
call; the first to raise blocks the request. Their returned data is discarded
since they are declared block-only.

The flag is wired from LitellmParams onto the guardrail instance at the same
generic choke point in initialize_guardrail that already sets
skip_system_message_in_guardrail, so no per-provider initializer needs to
change.

* feat(guardrails): extend run_in_parallel opt-in to post_call guardrails

post_call_success_hook ran guardrails sequentially for the same reason
pre_call did: response-modifying guardrails thread the response forward. But
block-only output scanners (which read the response and reject on violation
without changing it) serialize for no benefit and add latency.

This reuses the existing run_in_parallel flag for the post_call hook. Opted-in
post_call guardrails are pulled out of the sequential loop and run concurrently
via asyncio.gather after the sequential (response-modifying) guardrails and
before the non-guardrail CustomLogger callbacks, so they inspect the final
response and still block it from reaching the client if any raises. Their
returned response is discarded since they are block-only.

The apply_guardrail path sets data["guardrail_to_apply"] immediately before
awaiting, and unified_guardrail pops it before its first suspension point, so
concurrent guardrails never race on that key under asyncio's cooperative
scheduling.

* fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes

Addresses review feedback on the run_in_parallel opt-in.

asyncio.gather propagated the first exception without cancelling or awaiting
the siblings, so a block at t=0 left the other guardrails running as
unobserved background tasks (wasted external calls plus event-loop warnings),
and a fast SensitiveDataRouteException/ModifyResponseException could return a
reroute or passthrough before a slower block finished, letting crafted input
bypass the block. Both the pre_call and post_call parallel batches now gather
with return_exceptions=True so every guardrail runs to completion, then raise
any blocking exception ahead of a flow-changing one.

The registry choke point wrote bool(None)==False onto every instance when the
config omitted run_in_parallel, silently disabling a constructor-set default;
it now only writes when the config provides an explicit value.

* fix(guardrails): record lifecycle logs for every concurrently-run guardrail

The log_guardrail_information decorator skipped its auto-record when it saw
that the count of standard_logging_guardrail_information entries in the shared
request_data had grown during the wrapped call, taking that as proof the
wrapped function had recorded its own richer entry. That heuristic breaks the
moment guardrails run concurrently (parallel pre_call/post_call, during_call):
a sibling guardrail's append inflates the shared count, so a guardrail that did
not self-record wrongly concludes it already did and drops its own entry. The
result is that enabling run_in_parallel silently loses per-guardrail lifecycle
logs, so the Admin UI Request Lifecycle timeline and downstream loggers
(Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent
guardrails.

Replace the shared-count heuristic with a ContextVar flag set when a guardrail
records its own entry. asyncio copies the context into each gathered task, so
the flag is isolated per concurrent guardrail while still catching the
self-record-then-skip-auto-record case within a single invocation.

* test(guardrails): declare run_in_parallel on post_call guardrail mocks

The post_call partition reads run_in_parallel on every CustomGuardrail
callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is
set in __init__, not on the class) so the attribute access raised, and even
a class-level default would return a truthy child mock that wrongly routes
the double into the parallel batch. Declare the flag False on the shared
mock factories so these pre-existing hook tests exercise the sequential
path they assert on.

* fix(guardrails): harden run_in_parallel reads and address review feedback

Read run_in_parallel via getattr(..., False) in the pre_call and post_call
partitions so a third-party CustomGuardrail subclass that overrides __init__
without chaining super().__init__() no longer raises AttributeError on a path
that previously worked. Drop the redundant in-function GuardrailEventHooks
import in _run_parallel_post_call_guardrails (already imported module-level).
Remove the flaky wall-clock upper-bound assertions from the two concurrency
tests; the all-start-before-any-end overlap assertion is the timing-independent
signal that actually proves concurrency.
2026-07-24 13:25:58 -07:00

368 lines
4.1 KiB
JSON
Raw Permalink Blame History

{
"ANN001": {
"limit": 3152
},
"ANN002": {
"limit": 69
},
"ANN003": {
"limit": 835
},
"ANN201": {
"limit": 2138
},
"ANN202": {
"limit": 944
},
"ANN204": {
"limit": 724
},
"ANN205": {
"limit": 127
},
"ANN206": {
"limit": 130
},
"ANN401": {
"limit": 2075
},
"ASYNC230": {
"limit": 14
},
"B004": {
"limit": 4
},
"B006": {
"limit": 190
},
"B008": {
"limit": 505
},
"B009": {
"limit": 84
},
"B010": {
"limit": 197
},
"B018": {
"limit": 5
},
"B019": {
"limit": 4
},
"B021": {
"limit": 4
},
"B026": {
"limit": 6
},
"B033": {
"limit": 4
},
"BLE001": {
"limit": 2902
},
"C401": {
"limit": 11
},
"C404": {
"limit": 4
},
"C405": {
"limit": 23
},
"C408": {
"limit": 14
},
"C414": {
"limit": 7
},
"C419": {
"limit": 4
},
"C901": {
"limit": 316
},
"D419": {
"limit": 9
},
"DTZ001": {
"limit": 5
},
"DTZ003": {
"limit": 33
},
"DTZ005": {
"limit": 241
},
"DTZ006": {
"limit": 13
},
"DTZ007": {
"limit": 23
},
"DTZ011": {
"limit": 6
},
"EXE001": {
"limit": 7
},
"EXE002": {
"limit": 6
},
"F401": {
"limit": 23
},
"FURB136": {
"limit": 4
},
"FURB168": {
"limit": 4
},
"FURB188": {
"limit": 52
},
"I001": {
"limit": 273
},
"LOG015": {
"limit": 8
},
"N999": {
"limit": 4
},
"PERF102": {
"limit": 30
},
"PERF401": {
"limit": 146
},
"PERF402": {
"limit": 9
},
"PERF403": {
"limit": 74
},
"PIE790": {
"limit": 278
},
"PIE800": {
"limit": 4
},
"PIE804": {
"limit": 24
},
"PIE810": {
"limit": 44
},
"PLC0206": {
"limit": 31
},
"PLC0208": {
"limit": 4
},
"PLC0414": {
"limit": 38
},
"PLR0124": {
"limit": 4
},
"PLR0206": {
"limit": 4
},
"PLR0402": {
"limit": 9
},
"PLR1704": {
"limit": 6
},
"PLR1711": {
"limit": 34
},
"PLR1714": {
"limit": 265
},
"PLR1730": {
"limit": 10
},
"PLR2044": {
"limit": 4
},
"PLW0127": {
"limit": 44
},
"PLW0133": {
"limit": 4
},
"PLW0602": {
"limit": 230
},
"PLW0603": {
"limit": 193
},
"PLW1508": {
"limit": 198
},
"PLW1510": {
"limit": 5
},
"PYI030": {
"limit": 5
},
"PYI036": {
"limit": 5
},
"PYI041": {
"limit": 12
},
"PYI064": {
"limit": 5
},
"RET501": {
"limit": 38
},
"RET504": {
"limit": 719
},
"RUF010": {
"limit": 874
},
"RUF012": {
"limit": 168
},
"RUF015": {
"limit": 11
},
"RUF019": {
"limit": 41
},
"RUF022": {
"limit": 85
},
"RUF023": {
"limit": 5
},
"RUF046": {
"limit": 8
},
"RUF051": {
"limit": 6
},
"RUF059": {
"limit": 73
},
"RUF100": {
"limit": 480
},
"S110": {
"limit": 236
},
"S112": {
"limit": 24
},
"SIM101": {
"limit": 63
},
"SIM102": {
"limit": 324
},
"SIM103": {
"limit": 129
},
"SIM113": {
"limit": 6
},
"SIM114": {
"limit": 113
},
"SIM115": {
"limit": 5
},
"SIM117": {
"limit": 10
},
"SIM118": {
"limit": 114
},
"SIM201": {
"limit": 4
},
"SIM210": {
"limit": 12
},
"SIM211": {
"limit": 4
},
"SIM222": {
"limit": 4
},
"SIM401": {
"limit": 12
},
"TC004": {
"limit": 8
},
"TC005": {
"limit": 9
},
"TID251": {
"limit": 2701
},
"TRY002": {
"limit": 548
},
"TRY004": {
"limit": 98
},
"TRY201": {
"limit": 424
},
"TRY203": {
"limit": 123
},
"TRY300": {
"limit": 883
},
"UP006": {
"limit": 12789
},
"UP007": {
"limit": 2570
},
"UP008": {
"limit": 5
},
"UP012": {
"limit": 7
},
"UP018": {
"limit": 21
},
"UP024": {
"limit": 15
},
"UP028": {
"limit": 5
},
"UP031": {
"limit": 5
},
"UP032": {
"limit": 629
},
"UP034": {
"limit": 4
},
"UP035": {
"limit": 2284
},
"UP036": {
"limit": 4
},
"UP037": {
"limit": 105
},
"UP045": {
"limit": 18461
}
}
Reference in a new issue View git blame Copy permalink