mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-15 04:19:08 +00:00
220bbfc0ce
* feat(guardrails): add Airia guardrail integration Adds `airia` as a built-in guardrail provider. The hook sends the prompt under `pre_call` and the model's response under `post_call` to the Airia AI Gateway, which answers allow, block, or redacted content; the only configuration is `api_base`, `api_key`, and an optional `timeout`. `during_call` is not offered: it runs concurrently with the model call, so a block could land after the prompt has already reached the provider. A BLOCKED verdict, or any action this version does not recognise, raises with `blocked_content=True`; a transport error, a non-2xx, or a response body that is not a JSON object raises with `blocked_content=False`, so callers can tell "could not evaluate" from "evaluated and blocked" while both still fail closed. On GUARDRAIL_INTERVENED a copy of the inputs is returned with every rewritten field substituted; an intervention carrying no applicable rewrite, a rewrite of the wrong shape, or a rewrite whose item count does not match what was sent (which would misalign the positional write-back downstream) blocks instead of letting the original through. Streamed responses are moderated whole and then emitted redacted: the hook opts into the unified hook's `incremental_diff` mode (the default `block_only` drops rewrites) with `streaming_end_of_stream_only`, so a redaction can never span transform rounds and underflow. Covered by a test that drives the real streaming iterator hook end to end and checks the assembled client output, not just the guardrail's own config attributes. The HTTP client is dependency-injected (an `async_handler` constructor parameter, defaulting to a real client) rather than mutated post-construction, so tests inject a fake client instead of monkeypatching an instance method. Known limitation, not fixable from this guardrail alone: LiteLLM's unified guardrail framework yields a tool-call chunk to the client immediately during `incremental_diff` streaming, inspecting the assembled tool calls against the guardrail's verdict only once the stream ends (`_run_incremental_transform_stream` in `unified_guardrail.py`). A blocked tool call can therefore reach an executing client before the terminal error arrives. Switching to `block_only` with `streaming_buffer_until_moderated` does not close this without a regression: that flag has no effect under `incremental_diff` (the mode branches away before it is read), and `block_only` itself replays the original, unredacted chunks on release, silently dropping any REDACT verdict on streamed text. Relatedly, LiteLLM's OpenAI chat completions guardrail translation skips `post_call` guardrails entirely for a response with no text content, so a tool-call-only response bypasses this guardrail (and every other post-call guardrail) even outside streaming. Both are shared-framework behavior, not specific to Airia; fixing either is out of scope for an isolated single-guardrail contribution. * fix(guardrails): fail closed on an Airia rewrite for a field that was not sent Bugbot finding on #40784. An INTERVENED verdict carrying `structured_messages: []` with `texts` omitted passed `_rewritten` unchanged: an empty list has the same length as an omitted field, so it counted as an applied rewrite while the `texts` that actually carried the content went through untouched. post_call sends only `texts`, so this was reachable. A rewrite for a field the request did not send is now a mismatch, same as a wrong length, and the verdict blocks. Regression test covers both the empty and the non-empty case. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(guardrails): fail closed on streamed non-chat routes the Airia rewrite cannot reach Veria finding on #40784. incremental_diff, the mode that carries a rewrite into the stream, exists only for OpenAI chat completions; UnifiedLLMGuardrails sends every other streamed route (/v1/responses, /v1/messages) to block_only, which streams the original text live and drops rewrites. So with post_call on, a streamed Responses or Anthropic call could receive text that Airia then blocked or redacted. Two changes, both confined to this guardrail: - streaming_buffer_until_moderated is set, so on the block_only fallback the framework holds every chunk until the end-of-stream verdict instead of releasing it live. The flag is never read on the incremental_diff path, so streamed chat completions are unchanged. - A REDACT verdict on a streamed response whose route cannot carry a rewrite is enforced as a block, since block_only would otherwise replay the original. The route check mirrors _resolve_transform_call_type so the two cannot disagree; an unresolvable route fails closed. Request-side rewrites and non-streamed calls are untouched. The end-to-end test drives the real UnifiedLLMGuardrails iterator over an Anthropic /v1/messages SSE stream and asserts not one original chunk is released for either verdict. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(guardrails): keep the Airia stream test's pytest.raises block to one statement CI's test-tree ruff run flags PT012 on the non-chat streaming test: the `async for` that collected released chunks sat inside `pytest.raises`, so a raise from the loop body would be indistinguishable from the one the test is about. The loop now lives in a `_drain_into` helper and the `with` block awaits it alone. Same assertion, same partial-leak capture. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(guardrails): keep the Airia guardrail within the per-rule basedpyright ceilings scripts/type_check_gate.py now gates every rule on its merge-base count (#40193), and the guardrail added sixteen strict-rule errors it previously absorbed under the global budget. - The parsed gateway response goes through `_as_object` once, so `action`, `blocked_reason` and the rewrite fields are read from `dict[str, object]` rather than `dict[Unknown, Unknown]`; `blocked_reason` is only used as the message when it is a non-empty string. - `get_config_model` returns the concrete `type[AiriaGuardrailConfigModel]`, and the config model carries the same pyright ignore as its siblings for the unparameterised base. - The remaining untyped upstream members (the decorator, the client factory, `super().__init__`'s kwargs, the handler's `post`) are ignored per line with the reason, as the Wingback guardrail does. The registry reads `guardrail["guardrail_name"]`, a required TypedDict key. - Tests reach the injected mock and the handler timeout through typed helpers instead of the real handler's partially typed attributes. No behaviour change: 46/46 tests, ruff default and strict, and both lint gates pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(guardrails): clear the Airia guardrail's type-discipline gate violations Drop the mutable-ok comments that suppress nothing (LIT013), give the two deferred-import noqa lines a reason (LIT003), and annotate the parsed JSON object as Mapping instead of dict (LIT001). --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _support | ||
| base_sdk_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| e2e_harness | ||
| fixtures | ||
| harness_e2e | ||
| integration | ||
| rust-python-harness | ||
| test_litellm_rust | ||
| typing | ||
| unit | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _master_key.py | ||
| _openai_record_replay_proxy.py | ||
| _process_helpers.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| AGENTS.md | ||
| capturing_transport.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_logging.conf | ||
| test_rust_python_harness.py | ||
| VCR_TESTING.md | ||
| white_100x100.png | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/unit
This folder can only run mock tests.