19 commits

Author	SHA1	Message	Date
mateo-berri	92b46538e1	fix(policy_engine): restore request guardrails list after pipeline allow	2026-08-31 16:38:44 -07:00
Deepanshu Lulla	72f1b3e969	feat(guardrails): add Lakera v2 skip-message honoring and advisory (inject_system_message) mode (#34940) ... * feat(guardrails): honor Lakera v2 skip-message flags and add advisory (inject_system_message) mode Squashed rebase of bugfix/lakera-v2-skip-system-tool-messages onto latest litellm_internal_staging (900+ commits ahead; a commit-by-commit rebase hit repeated conflicts against the same files across earlier review-round commits, so the branch's cumulative diff was reapplied in one pass instead). Adds skip_system_message_in_guardrail/skip_tool_message_in_guardrail support to Lakera v2, a third on_flagged: "inject_system_message" advisory mode, and the associated masking-safety-guard hardening (multimodal content, non- maskable message fields, combined messages+input, and structured Responses- API input in advisory delivery) found across this PR's review rounds. * fix(guardrails): don't let one invalid guardrail config crash proxy boot init_guardrails_v2 had no try/except around initialize_guardrail, so a guardrail whose litellm_params fail validation at construction time (for example Lakera's on_flagged=inject_system_message combined with mode=during_call, or a malformed advisory_system_message template) raised uncaught and crashed the entire proxy at startup, taking down every other, correctly-configured guardrail in the list. Catch ValueError/TypeError per guardrail, log a warning, and skip it, matching the same pattern already used for the DB-driven guardrail-creation path in guardrail_endpoints.py. * fix(guardrails): preserve message fields and mask PII before advising in Lakera v2 Mask-in-place degraded to a hard block for any message carrying a field beyond role/content (tool_call_id, tool_calls, name, cache_control), for a message excluded by skip_system_message_in_guardrail/skip_tool_message_in_guardrail, or for a message with no inspectable text, since it rewrote data["messages"] wholesale from a synthetic role/content-only list built for the Lakera API call. That made masking effectively unusable for any real tool-calling conversation and made the skip flags flip every PII-only violation to a hard block instead of masking just the in-scope text. Replace the wholesale rewrite with a scope-index merge, reusing the same merge_guardrailed_scoped_messages helper the OpenAI/Anthropic guardrail translation handlers already use for this: patch content in place on a copy of each original message actually sent to Lakera, and leave every skipped/no-text/out-of-scope message untouched at its original position. This also fixes on_flagged="inject_system_message" (advisory mode) shipping raw unmasked PII to the model: a PII-only violation is now masked the same way regardless of on_flagged, and the advisory note is reserved for flags masking can't resolve on its own. Addresses maintainer-reported regressions on BerriAI/litellm#34940. * fix(guardrails): satisfy new lint gates for the masking/advisory fix Parameterize the write-back helper's dict param and suppress the two new lint rules that landed on the base while this branch was in flight: TQ008 (patching an internal collaborator) for two pre-existing tests unrelated to this change, and LIT001 for a param that genuinely needs to mutate the caller's request dict in place. * fix(guardrails): normalize role casing in Lakera v2 masking scope, log skipped guardrails louder Greptile finding: the masking scope helper compared roles case-sensitively while filter_messages_by_skip_flags (used to build what's actually sent to Lakera) normalizes casing, so an uppercase-cased "System"/"TOOL" role survived the scope filter but was excluded from the inspected list. The resulting length mismatch raised inside the strict positional zip, turning a maskable PII-only violation into an unhandled request failure. Lowercase the role comparison to match. Also, per veria-ai's finding that a skipped invalid guardrail now fails open: log it at error level with an explicit note that the proxy is starting without that guardrail, so it's not mistaken for routine info. * fix(guardrails): mask maskable PII in mixed violations before advising in Lakera v2 on_flagged="inject_system_message" only masked when a violation was PII-only; a mixed violation (PII plus a non-PII flag like prompt injection) fell straight through to the advisory branch with the raw PII still in place, in both async_pre_call_hook and async_moderation_hook. Mask whatever Lakera returned location data for before appending or logging the advisory, so a mixed violation never ships raw PII just because something else was also flagged. Also degrade to blocking, same as block mode already does, when nothing can be safely masked at all (multimodal content, or messages combined with a Responses API input field) instead of showing an advisory note next to raw, unredacted content. Widened call_v2_guard/_mask_pii_in_messages/the write-back helper's message parameters from list to Sequence to match what's actually passed through from _filter_skipped_messages, instead of duplicating list(...) casts at every call site. * fix(guardrails): don't hard-block advisory mode for non-PII flags on non-maskable input Bugbot finding: gating the entire inject_system_message branch on is_multimodal_input hard-blocked every flagged request on Responses instructions, combined messages+input, or multimodal content, including a prompt-injection-only violation with no PII at all. Masking safety only matters when there's actual PII to mask; a violation with no PII needs no masking, so the advisory should still be delivered normally. Only degrade to blocking when the breakdown actually contains a PII detection and masking isn't safely possible. Otherwise, mask whatever's maskable (if any) and deliver the advisory as before. * fix(guardrails): require payload and breakdown for Lakera v2 advisory mode Advisory mode's mixed-violation masking safety net can only redact detected PII when Lakera's response carries both the breakdown (to detect a PII hit at all) and payload (the location data to mask by). payload=False or breakdown=False alongside on_flagged='inject_system_message' silently forwarded raw PII next to the advisory note. Reject that combination at construction and hot-reload time instead. * fix(guardrails): skip_system_message_in_guardrail must not force-block Lakera masking _has_responses_instructions treated any non-empty data["instructions"] as unsafe to mask regardless of skip_system_message_in_guardrail, even though that flag excludes the instructions-derived synthetic system message from what Lakera ever inspects. PII detected purely in the maskable non-system content was force-blocked instead of masked. Also fixes pre-existing LIT010 (missing Final) violations in _has_responses_instructions, _breakdown_has_pii_violation, and async_post_call_success_hook that the rebase's lowered budget ceiling now flags. * chore: retrigger CI (GitHub Actions runner-acquisition failure on prior push) * fix(guardrails): address maintainer review findings on Lakera v2 advisory mode - Gate advisory_system_message template validation on on_flagged= 'inject_system_message', since block/monitor mode never reads it. - Allow on_flagged='inject_system_message' with mode='during_call' at construction/hot-reload instead of rejecting it; async_moderation_hook already degrades gracefully (masks if possible, else logs a warning). - reinitialize_guardrail now restores the previous live instance when the new config fails to initialize, instead of leaving the guardrail deleted entirely with nothing enforcing it. - PATCH /guardrails/{id} rolls back the DB write and returns 422 when the in-memory sync rejects the new config, instead of persisting a config that never actually took effect and returning 200. - Qualifire now rejects on_flagged values it doesn't implement (only Lakera should accept 'inject_system_message'; LitellmParams flattens the field across every guardrail config mixin). * fix(tests): satisfy lint gates and update collateral test for advisory-mode fixes - Add match= to a too-broad pytest.raises(ValueError), and suppress the new TQ008 mocker.patch findings (same pattern already used by sibling scenarios in this test). - test_init_guardrails_v2_skips_invalid_guardrail_instead_of_crashing_boot used mode='during_call' + on_flagged='inject_system_message' as its invalid-config example; that combination is now accepted, so swap in the payload/breakdown-missing case and add a test confirming during_call advisory mode constructs successfully. * docs(CLAUDE.md): auto-capture review learnings without being asked This session found three real bugs a human maintainer caught after eight rounds of bot review and live-proxy verification all missed them. Add a standing instruction to write learnings.md entries the moment a root cause is understood, in both the repo-wide file and any relevant skill's own file, instead of relying on being asked. * feat(guardrails): add scan_raw_request flag so YAML order can't change enforcement Maintainer finding on BerriAI/litellm#34940: guardrails for the same hook run sequentially over one shared, progressively-mutated request dict, so declaring a masking guardrail before a blocking one hides the violation from it (200 vs 400 depending purely on YAML order). scan_raw_request opts a guardrail into always evaluating a snapshot taken before any guardrail in the hook ran, regardless of its declared position. Same contract as run_in_parallel: block-only, its own mutations discarded. Verified live: real proxy, real Gemini call, two custom guardrails (a redactor then a blocker). Same request, same declared order -- without the flag the blocker never sees the raw secret (200); with it, the blocker correctly rejects before any provider call (400). * fix(guardrails): harden scan_raw_request against review findings - Use safe_deep_copy instead of a bare deepcopy for the raw-request snapshot; request payloads commonly carry unpicklable objects (e.g. an otel span in metadata), which previously raised on every guarded request when tracing was enabled (Bugbot, High). - Only compute the snapshot when a guardrail actually opted in, and take it before _maybe_execute_pipelines runs, so a pipeline-mutated payload can't hide a violation from a scan_raw_request guardrail outside the pipeline (veria-ai). - Log a warning when a scan_raw_request guardrail returns a modified payload, since that mutation is discarded and the combination is otherwise silently exploitable for a masking-capable integration misconfigured this way (veria-ai). * chore(openapi): regenerate lazy snapshot and dashboard schema types The lazy OpenAPI snapshot (litellm/proxy/_lazy_openapi_snapshot.json) and the derived dashboard schema.d.ts had drifted stale relative to the guardrail config model changes across this PR's rounds (advisory mode, scan_raw_request, and upstream additions picked up by rebasing). Regenerated via the CI's own documented fix: uv run python -m litellm.proxy._lazy_openapi_snapshot npm run gen:api (via make check) * chore(openapi): pick up cache_hit_filter field after rebase * fix(guardrails): stop scan_raw_request warning from firing on every call _process_guardrail_callback always returns a dict once a guardrail runs (mark_pre_call_hook_ran unconditionally stamps bookkeeping metadata), so comparing the result to non-None warned on every request even when the guardrail never touched the payload. Compare against a bookkeeping-only baseline instead, so only an actual content mutation triggers the warning. * fix(guardrails): make scan_raw_request snapshots independent of safe_memory_mode safe_deep_copy can return the original object under litellm.safe_memory_mode, or alias a per-key reference on copy failure. Under that mode, the scan_raw_request comparison baseline aliased raw_request_snapshot (and therefore the live request), letting mark_pre_call_hook_ran write a premature execution marker that a deployment-level guardrail sharing the same name would read as "already ran" and skip. Also affected the feature's core isolation guarantee: input_data itself could alias the live request under the same mode. Replace every scan_raw_request snapshot with _independent_snapshot, which never returns an alias, only a genuine copy or None. * fix(guardrails): gate during_call mixed-violation masking behind an actual PII check The during_call branch for a mixed violation under on_flagged=inject_system_message unconditionally masked and reassigned data["messages"], even for a pure prompt-injection violation with zero PII, unlike async_pre_call_hook which already gates the same call behind _breakdown_has_pii_violation. The unconditional reassignment touched shared request state during a hook documented as racing with the concurrent LLM dispatch, for no reason when there was nothing to mask. * fix(guardrails): stop scan_raw_request from silently no-op'ing on real requests _independent_snapshot did one whole-dict copy.deepcopy and returned None on any failure. Every real proxy request carries data["litellm_logging_obj"] (a Logging instance nesting a live OTel span with a real lock) by the time pre_call_hook runs, which can never be deep-copied, so the snapshot failed on every real request and silently fell back to the live, unisolated data with no warning -- defeating the entire feature in production while every existing test (none of which set litellm_logging_obj) kept passing. Rework the helper to deep-copy each top-level key independently, falling back to the original reference only for the specific key that fails, same crash tolerance as safe_deep_copy's own per-key fallback. It never returns None now; only the keys scan_raw_request actually depends on (messages/ input, metadata/litellm_metadata) need to be genuinely independent. * fix(guardrails): block during_call when PII can't be safely masked Greptile finding (P1, security): async_moderation_hook's inject_system_message branch had no equivalent to async_pre_call_hook's degrade-to-blocking case for a PII violation on input that can't be safely masked (e.g. combined messages+input). It fell through to the advisory no-op branch and let raw, unredacted PII reach the model with no protection at all. Raising still blocks the response from reaching the caller even though during_call races with the LLM dispatch, the same mechanism on_flagged="block" already relies on for this hook, so add the same block-instead-of-advisory branch pre_call already has. * chore(lint): fix LIT002 ceiling after rebase merge conflict resolution * fix(lint): suppress genuine LIT002 hits instead of padding the ceiling My earlier rebase conflict resolution for type-discipline-budget.json's LIT002 limit was too low, then overcorrected by padding it well above the actual measured count. Root-caused instead: _independent_snapshot and the PATCH-endpoint rollback path legitimately construct plain, mutable request-payload/config dicts (matching this file's existing precedent for the same shape), so suppress those four sites with `# mutable-ok:` rather than reshaping code that must stay a plain dict by contract. Set the limit to the exact current measured total; the small remaining gap vs upstream's own committed ceiling is pre-existing drift in litellm_internal_staging itself (its own tree already measures over its committed limit), not attributable to this PR. * fix(guardrails): stamp live request when a scan_raw_request guardrail runs _run_sequential_guardrail_callback and _run_parallel_pre_call_guardrails only called mark_pre_call_hook_ran on throwaway snapshot copies for a scan_raw_request guardrail, never on the live request returned to the caller. A later async_pre_call_deployment_hook (router-level guardrail re-check) reads that marker on live kwargs to decide whether to skip re-running the same guardrail; since it was never stamped there, the guardrail ran a second time on live data, doubling the external call and re-applying whatever scan_raw_request's contract says should be discarded. * fix(guardrails): revalidate Qualifire's on_flagged on live config reload on_flagged was validated only in __init__. The base CustomGuardrail.update_in_memory_litellm_params is a generic setattr loop with no revalidation, so a live config update (PUT /guardrails/{id}, no restart) could setattr on_flagged="inject_system_message" onto a running instance, bypassing the constructor's rejection -- silently blocking every flagged request under an "advisory" label. Mirrors LakeraAIGuardrail's own update_in_memory_litellm_params override added earlier in this PR. * fix(guardrails): honor scan_raw_request for pipeline-managed guardrails A scan_raw_request=True guardrail that is itself a pipeline step never saw raw_request_snapshot: PipelineExecutor.execute_steps had no way to receive it, and pipeline-managed guardrails are fully excluded from the normal sequential/parallel loops that implement the flag. Such a guardrail silently evaluated whatever an earlier pass_data step in the same pipeline had already rewritten, defeating the flag for pipeline-managed guardrails. Moves the snapshot helper (renamed independent_snapshot) from proxy/utils.py to litellm_core_utils/core_helpers.py so pipeline_executor.py can use the same independent-copy logic without a circular import, threads raw_request_snapshot through _maybe_execute_pipelines and PipelineExecutor.execute_steps/_run_step, and discards a scan_raw_request step's returned data the same way the sequential/parallel loops already do. * chore(openapi): pick up upstream drift after rebase onto litellm_internal_staging * fix(guardrails): stop attempting PII masking during during_call in Lakera v2 Greptile finding (P1, security): during_call runs concurrently with the LLM dispatch. In the common path, the provider call already binds its messages kwarg before this guardrail's coroutine gets a chance to run, let alone before its own network round trip to Lakera completes -- masking here can never reliably reach the outgoing request, and _apply_redacted_messages_back_ preserving_fields reassigns to a new list object rather than mutating in place, so even winning the race wouldn't help. This affected both the PII-only and mixed-violation masking branches, all added in this same PR. Remove masking from async_moderation_hook entirely and let PII violations fall through to the normal on_flagged branching: block under "block" or "inject_system_message" (extending the existing multimodal-only block to cover every PII case, since masking is proven non-functional regardless of input shape), log-and-allow under "monitor" -- consistent with how every other violation type in this hook is already handled. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>	2026-08-28 14:13:49 -07:00
yuneng-jiang	6bce3dce0d	test(callbacks): unwind the callbacks global the policy engine and realtime tests scaffold around (#37826) ... * test(policy-engine): unwind the callback global the pipeline tests scaffold around Every one of the 16 tests in this file set litellm.callbacks by hand, each wrapping its body in a try/finally to put the old value back, and each capturing that old value with a .copy() first. That is 32 TQ005 violations and about 70 lines of scaffolding to say what monkeypatch.setattr says in one. The write also sat outside the try, so the block that restores it did not cover the statement that changed it. 16 tests pass either way, and litellm.callbacks reads restored on both sides, because the conftest snapshot already lists it. The point is that these tests stop depending on that snapshot to clean up after them. * test(realtime): unwind the same callback global in the realtime streaming tests Same global, same shape as the previous commit. 25 writes to litellm.callbacks, 2 of them wrapped in a try/finally that resets to [] rather than to the old value, and 12 tests that write it with no protection at all. monkeypatch.setattr replaces all of them, and the sys.path.insert with its now-unused os and sys imports goes too. Both sides read restored here as well, for the same reason as the previous commit: litellm.callbacks is in the conftest snapshot. What changes is that these tests no longer lean on it. 101 tests pass in this file, 16 in the policy engine one. * style(realtime): wrap the one signature the monkeypatch param pushed past 120	2026-08-21 21:19:35 -07:00
ryan-crabbe-berri	b76def0e5d	test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) ... `pytest.raises(Exception)` with no `match=` passes on any error that broad. A TypeError from a refactor, a botched fixture, an import that moved: all of them read as the rejection the test claims to police, so the test goes green for the wrong reason and stays green after the behaviour it guards is gone. PT011 closes that gap for the 317 sites B017 could not reach, because B017 only fires on a single-statement body with no `as e` binding. Each pattern here is the message the code actually raised, recorded by running the sites under a plugin that logged the concrete type and text per call site, so the assertions describe observed behaviour rather than a guess. Where a site raises more than one message across its parametrize cases, the pattern is an alternation of what was seen; where the exception carries an empty `str()` and puts the text on `.message`, the site keeps a narrow `noqa` with the reason. PT014 removes four parametrize cases that were listed twice. The duplicate re-runs an assertion that already passed, and it usually marks a case someone meant to vary and forgot to edit.	2026-08-20 20:24:49 -07:00
yucheng-berri	1139012b45	fix(guardrails): scan text on /guardrails/apply_guardrail for Azure Content Safety (#36894) ... * fix(guardrails): scan text on /guardrails/apply_guardrail for Azure Content Safety The two Azure Content Safety guardrails never implemented apply_guardrail, so the endpoint fell through to the base no-op and answered 200 with the caller's text echoed back, having scanned nothing. Implementing that method also flips the proxy's unified-vs-native dispatch, which would move request traffic off these guardrails' own hooks. Add an opt-out that keeps every lifecycle event on the native hooks, so only the endpoint changes. * test(guardrails): cover the remaining native-hook opt-out dispatch sites Adds regression tests for the parallel post-call path, the MCP post-call hook, and the policy engine step, so every read of the opt-out flag fails when removed.	2026-08-17 11:18:40 -07:00
mateo-berri	b42ef469cf	fix(policy_engine): warn that the config-defined policy reactivates when all DB versions are deleted	2026-07-30 19:48:35 -07:00
mateo-berri	35f770f43e	fix(policy_engine): restore config policy immediately when its DB override is removed and keep same-named DB drafts reachable in the UI	2026-07-30 19:23:50 -07:00
mateo-berri	ec016d1bd8	fix(policy_engine): decide config policy suppression from fresh db query only	2026-07-30 16:01:57 -07:00
mateo-berri	91290c6020	fix(policy_engine): only hide config policies behind production DB versions in policies list	2026-07-30 15:33:52 -07:00
mateo-berri	4e9df55392	fix(policy_engine): preserve config-defined policies across DB sync and expose them via list APIs	2026-07-30 12:21:03 -07:00
ryan-crabbe-berri	23873f8447	fix(policies): reject non-existent team/key/model scope entries on attachment create (#32131) ... * fix(policies): reject non-existent team/key/model scope entries on attachment create Creating a policy attachment accepted arbitrary team, key, and model values with no validation, so a typo'd or non-existent team was silently persisted (LIT-4199). The create endpoint now rejects a concrete (non-wildcard) team, key, or model that does not resolve to a real entity, wiring the previously-dead PolicyValidator existence checks and reusing RouteChecks._is_wildcard_pattern so validation agrees with request-time matching, where only a trailing "*" is a wildcard. Wildcard patterns are still allowed through since they may match zero entities today and more later, and tags stay free-form. The Admin UI's Teams field validates the same rule for immediate feedback when its team list has loaded, deferring to the backend otherwise. * style(policies): use builtin list generics and | None in scope validator Keeps the new find_invalid_scope_entries signature off the UP006/UP045 strict ruff budgets instead of copying the surrounding legacy typing.List/Optional idiom. * fix(policies): separate multiple attachment scope errors with ' | ' Addresses Greptile review: joining per-entry validation messages with a bare space read as one run-on sentence; ' | ' makes the multi-error 400 detail easier to parse for users and programmatically.	2026-07-04 11:58:29 -07:00
Yassin Kortam	c14329128b	fix(guardrails): match policy-pipeline block response to direct guardrail attachment (#31421) ... When a guardrail blocked a request through a flow-builder policy pipeline, the proxy discarded the guardrail's own exception and synthesized a generic guardrail_pipeline_error response, so the same guardrail produced a different HTTP response and trace span depending on whether it was attached directly or via a policy. The pipeline now carries the guardrail's original exception and re-raises it verbatim on block, enriching it with the blocking guardrail's name and mode exactly as the direct path does, so the two attachment methods are indistinguishable to clients and tracing. The generic pipeline error remains only as a fallback for blocks with no underlying exception (e.g. a guardrail that could not be found). Resolves LIT-4041	2026-06-26 14:25:10 -07:00
Shivam Rawat	4c3efe9c7c	fix(guardrails): return HTTP 400 for litellm content filter blocks (#28418) ... * fix(guardrails): return HTTP 400 for litellm content filter blocks Align litellm_content_filter hard rejects with the standard guardrail block status code so clients receive 400 instead of 403. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): return HTTP 400 for custom code guardrail blocks Pre-call custom code guardrail blocks now raise HTTPException(400) instead of using the passthrough ModifyResponseException path that returned a synthetic 200 response. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): preserve custom code passthrough blocks Keep standalone custom code guardrail blocks on the passthrough contract while covering policy pipeline block handling for passthrough-style guardrail interventions. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-30 14:28:46 -07:00
Ishaan Jaffer	e8461b5b97	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
ishaan-berri	51876292a0	Litellm ishaan april4 2 (#25150) ... * feat(router): integrate allowed_fails_policy into health check failures (#24988) * feat(router): integrate allowed_fails_policy into health check failures Health check failures now increment the same per-deployment failure counters used by allowed_fails_policy, so users can control how many health check failures of each error type are required before a deployment enters cooldown. - ahealth_check() preserves the original exception in its return dict - run_with_timeout() returns a litellm.Timeout on health check timeout - _perform_health_check() propagates exceptions to unhealthy endpoints - _write_health_state_to_router_cache() calls _set_cooldown_deployments for each unhealthy endpoint that has an exception - When allowed_fails_policy is set, the binary health check filter is bypassed so cooldown is the sole routing exclusion mechanism - Safety net: if all deployments are in cooldown with enable_health_check_routing=True, the cooldown filter is bypassed Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(router): add health_check_ignore_transient_errors flag When enabled, health check failures with 429 (rate limit) or 408 (timeout) status codes are skipped from the cooldown pipeline. These are transient load issues, not broken deployments. Auth errors (401), 404, and 5xx errors still increment counters and trigger cooldown as before. Config (general_settings): health_check_ignore_transient_errors: true Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(router): also exclude 429/408 from health state cache when ignore_transient_errors set The previous fix only skipped cooldown counter increments. The health state cache was still marking 429/408 endpoints as is_healthy=False, causing the binary health check filter to exclude them from routing. Now, when health_check_ignore_transient_errors=True, 429/408 endpoints are also excluded from the unhealthy list passed to build_deployment_health_states(), so the binary filter treats them as unaffected (not unhealthy). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(router): add health check driven routing guide New standalone page covering the full health check routing feature: allowed_fails_policy integration, health_check_ignore_transient_errors, architecture SVG, step-by-step setup, and gotchas (TTL, AllowedFails semantics). Replaces the inline section in health.md with a link to the new page. Added to the Routing & Load Balancing sidebar. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(health-check-routing): fix three CI failures - Add "exception" to ILLEGAL_DISPLAY_PARAMS in health_check.py so the exception object is stripped before the health endpoint serializes results to JSON (fixes TypeError: 'URL' object is not iterable) - Add allowed_fails_policy = None to FakeRouter stubs in test_router_health_check_routing.py (fixes AttributeError) - Add health_check_ignore_transient_errors to config_settings.md router settings reference table (fixes documentation test) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix litellm/tests/proxy_unit_tests/test_proxy_server.py * fix(router): address greptile review comments - Narrow cooldown safety-net bypass: only fires when allowed_fails_policy is set (cooldown is health-check driven). Without a policy, cooldowns are from real request failures and must not be bypassed. - Restore cooldown deployments DEBUG log that was accidentally removed. - Fix test_health TypeError: move exception extraction to a separate exceptions_by_model_id dict returned alongside endpoints, so exception objects never appear in the endpoint dicts that get JSON-serialized by the /health response. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(health-check-routing): properly isolate exceptions from health response Return exceptions_by_model_id as a separate third value from _perform_health_check / perform_health_check so exception objects (which contain non-JSON-serializable httpx URL types) never appear in the endpoint dicts that get serialized by the /health response. Callers updated: _health_endpoints.py, shared_health_check_manager.py, proxy_server.py background loop. All use the exceptions dict only for cooldown integration, not for display. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(shared-health-check): fix remaining 2-value return sites and update type annotation * fix(health-check-routing): fix P0 cooldown integration never firing The cooldown loop was reading endpoint.get("exception") which is always None because exceptions are now returned via exceptions_by_model_id, not stored in endpoint dicts. Fixed to use _exceptions.get(model_id). Also fixes the transient-error filter to use _exceptions instead of endpoint.get("exception"), and fixes all remaining 2-value return sites in shared_health_check_manager.py. Tests updated to pass exceptions via exceptions_by_model_id parameter instead of endpoint dicts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(health-check-routing): fix P1 transient-error filter broken on cache hits When SharedHealthCheckManager returns cached results, exceptions_by_model_id is always {} so the transient-error filter defaulted to status 500 for all endpoints, incorrectly marking 429/408 endpoints as unhealthy. Fix: store integer exception_status on each unhealthy endpoint dict in _perform_health_check. _get_endpoint_exception_status() uses the live exception object when available (direct path) and falls back to the stored integer (cache-hit path). The integer is JSON-serializable and survives the shared cache round-trip. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(health-check-routing): gate cooldown loop behind allowed_fails_policy Without the policy, cooldown is not the routing exclusion mechanism. Firing _set_cooldown_deployments for all enable_health_check_routing users was a backwards-incompatible change — 401s would immediately cooldown deployments that the binary filter would have recovered on the next cycle. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * revert: undo allowed_fails_policy gate on cooldown loop Cooldown integration via health checks is intentional for all enable_health_check_routing users, not just those with allowed_fails_policy. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(docs+tests): fix health_check_ignore_transient_errors doc section and test coverage - Move health_check_ignore_transient_errors from router_settings to general_settings in config_settings.md (code reads it from general_settings) - Remove duplicate enable_health_check_routing / health_check_staleness_threshold entries that were incorrectly listed under router_settings - Replace TestHealthCheckEndpointExceptionPropagation tests with ones that exercise the real _perform_health_check code path via mocked ahealth_check, verifying exceptions appear in exceptions_by_model_id and NOT in endpoint dicts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tests+docs): fix tuple unpacking and docs test failures - Update test mocks that return (healthy, unhealthy) to return (healthy, unhealthy, {}) to match the new 3-value signature - Update test unpackings of perform_shared_health_check to use healthy, unhealthy, _ = ... - Add health_check_ignore_transient_errors to router_settings section in config_settings.md (it is a Router constructor param, so the doc test requires it there; it also lives in general_settings for proxy use) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix CodeQL errors * fix(tests): fix 2-value unpackings of _perform_health_check in test_health_check.py * fix(tests): fix mock _perform_health_check returning 2-tuple instead of 3 * fix team routing --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add distributed lock for key rotation job (#23364) * fix: add distributed lock for key rotation job * fix: address Greptile review feedback on key rotation lock (#23834) * fix: address Greptile review feedback on key rotation lock * fix req changes greptile * feat(proxy): Optional on_error for guardrail pipeline (API / technical failures) (#24831) * guardrails fallback * docs * docs: add LITELLM_KEY_ROTATION_LOCK_TTL_SECONDS to environment variables reference * fix(mypy): accept Union[Dict, Any] in _get_deployment_order and use typed list to fix min() type error * fix(mypy): use Optional[str] for api_base in PydanticAI provider to match superclass signature --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: Shivam Rawat <shivam@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai>	2026-04-04 23:09:42 +00:00
Krish Dholakia	76ccc9e844	Guardrail Policy Versioning (#21862) ... * feat: initial commit, adding support for policy versioning on litellm * fix(policy_registry): support policy versioning * fix: multiple QA fixes for policy flow builder with guardrail versioning on litellm * feat: ui improvements * feat: add prisma migration * fix: address greptile fixes	2026-02-21 20:14:31 -08:00
Ishaan Jaff	08b61f49a4	[Guardrails] Add guardrail pipeline support for conditional sequential execution (#21177) ... * Add pipeline type definitions for guardrail pipelines PipelineStep, GuardrailPipeline, PipelineStepResult, PipelineExecutionResult with validation for actions (allow/block/next/modify_response) and modes. * Export pipeline types from policy_engine types package * Add optional pipeline field to Policy model * Add pipeline executor for sequential guardrail execution * Parse pipeline config in policy registry * Add pipeline validation in policy validator * Add pipeline resolution and managed guardrail tracking * Resolve pipelines and exclude managed guardrails in pre-call * Integrate pipeline execution into proxy pre_call_hook * Add test guardrails for pipeline E2E testing * Add example pipeline config YAML * Add unit tests for pipeline type definitions * Add unit tests for pipeline executor * Update litellm/proxy/policy_engine/pipeline_executor.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update litellm/proxy/utils.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>	2026-02-13 19:49:12 -08:00
Ishaan Jaff	f83620157e	[Feat] Policies - Allow connecting Policies to Tags, Simulating Policies, Viewing how many keys, teams it applies on (#20904) ... * init schema with TAGS * ui: add policy test * resolvePoliciesCall * add_policy_sources_to_metadata + headers * types Policy * preview Impact * def _describe_match_reason( * match based on TAGs * TestTagBasedAttachments * test fixes * add policy_resolve_router * add_guardrails_from_policy_engine * TestMatchAttribution * refactor * fix * fix: address Greptile review feedback on policy resolve endpoints - Track unnamed keys/teams as separate counts instead of inflating affected_keys_count with duplicate "(unnamed key)" placeholders. Added unnamed_keys_count and unnamed_teams_count to response. - Push alias pattern matching to DB via _build_alias_where() which converts exact patterns to Prisma "in" and suffix wildcards to "startsWith" filters. - Gate sync_policies_from_db/sync_attachments_from_db behind force_sync query param (default false) to avoid 2 DB round-trips on every /policies/resolve request. - Remove worktree-only conftest.py that cleared sys.modules at import time — no longer needed since code moved to main repo. - Rename MAX_ESTIMATE_IMPACT_ROWS → MAX_POLICY_ESTIMATE_IMPACT_ROWS. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: eliminate duplicate DB queries and fix header delimiter ambiguity - Fetch teams table once in estimate_attachment_impact and reuse for both tag-based and alias-based lookups (was querying teams twice when both tag_patterns and team_patterns were provided). - Convert tag/team filter functions from async DB queries to sync filters that operate on pre-fetched data (_filter_keys_by_tags, _filter_teams_by_tags). - Fix comma ambiguity in x-litellm-policy-sources header: use '; ' as entry delimiter since matched_via values can contain commas. - Use '+' as the within-value separator in matched_via reason strings (e.g. "tag:healthcare+team:health-team") to avoid conflict with header delimiters. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update litellm/proxy/policy_engine/policy_resolve_endpoints.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>	2026-02-10 17:50:37 -08:00
Ishaan Jaff	c23e4b87dc	[Feat] New LiteLLM Policy engine - create policies to manage guardrails, conditions - permissions per Key, Team (#19612) ... * init PolicyMatcher * TestPolicyMatcherGetMatchingPolicies * TestPolicyMatcherGetMatchingPolicies * feat: init PolicyResolver * init resolver types * init policy from config * inint PolicyValidator * validate policy * init Architecture Diagram * test_add_guardrails_from_policy_engine * init _init_policy_engine * test updates * test fixws * new attachment config * simplify types * TestPolicyResolverInheritance * fix policy resolver * fix policies * fix applied policy * docs fix * docs fix * fix linting + QA checks * fix linting + QA fixes * test fixes	2026-01-22 19:49:53 -08:00