diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md index ea2c1700eea..90d71fb95c8 100644 --- a/docs/my-website/docs/proxy/config_settings.md +++ b/docs/my-website/docs/proxy/config_settings.md @@ -944,7 +944,7 @@ router_settings: | QDRANT_URL | Connection URL for Qdrant database | QDRANT_VECTOR_SIZE | Vector size for Qdrant operations. Default is 1536 | REDIS_CONNECTION_POOL_TIMEOUT | Timeout in seconds for Redis connection pool. Default is 5 -| REDIS_CLUSTER_NODES | JSON-formatted list of Redis cluster startup nodes for Redis Cluster mode. Example: '[{"host": "node1", "port": 6379}]' +| REDIS_CLUSTER_NODES | JSON-formatted list of Redis cluster startup nodes for Redis Cluster mode. Example: `[{"host": "node1", "port": 6379}]` | REDIS_HOST | Hostname for Redis server | REDIS_PASSWORD | Password for Redis service | REDIS_PORT | Port number for Redis server diff --git a/docs/my-website/docs/proxy/guardrails/guardrail_policies.md b/docs/my-website/docs/proxy/guardrails/guardrail_policies.md index e2cb839203e..f4411553c69 100644 --- a/docs/my-website/docs/proxy/guardrails/guardrail_policies.md +++ b/docs/my-website/docs/proxy/guardrails/guardrail_policies.md @@ -309,6 +309,10 @@ Response: +## Policy Flow Builder + +For conditional execution (e.g., run a second guardrail only if the first fails), use the [Policy Flow Builder](./policy_flow_builder) to define pipelines with per-step pass/fail actions. + ## Config Reference ### `policies` @@ -323,6 +327,7 @@ policies: remove: [...] condition: model: ... + pipeline: ... # optional; see Policy Flow Builder ``` | Field | Type | Description | @@ -332,6 +337,7 @@ policies: | `guardrails.add` | `list[string]` | Guardrails to enable. | | `guardrails.remove` | `list[string]` | Guardrails to disable (useful with inheritance). | | `condition.model` | `string` or `list[string]` | Optional. Only apply when model matches. Supports regex. | +| `pipeline` | `object` | Optional. Ordered guardrail execution with per-step actions. See [Policy Flow Builder](./policy_flow_builder). | ### `policy_attachments` diff --git a/docs/my-website/docs/proxy/guardrails/policy_flow_builder.md b/docs/my-website/docs/proxy/guardrails/policy_flow_builder.md new file mode 100644 index 00000000000..2a83f3768ab --- /dev/null +++ b/docs/my-website/docs/proxy/guardrails/policy_flow_builder.md @@ -0,0 +1,219 @@ +# Policy Flow Builder + +The Policy Flow Builder lets you design guardrail pipelines with **conditional execution**. Instead of running guardrails independently, you chain them into ordered steps and control what happens when each guardrail passes or fails. + +Two powerful patterns it enables: **guardrail fallbacks** (try a different guardrail when one fails) and **retrying the same guardrail** (run the same guardrail again if it fails, e.g. to handle transient errors). + +## When to use the Flow Builder + +| Approach | Use case | +|----------|----------| +| **Simple policy** (`guardrails.add`) | All guardrails run in parallel; any failure blocks the request. | +| **Flow Builder** (pipeline) | Guardrails run in sequence; you choose actions per step (next, block, allow, custom response). | + +Use the Flow Builder when you need: + +- **Guardrail fallbacks** — use `on_fail: next` to try a different guardrail when one fails (e.g., fast filter → stricter filter) +- **Retrying the same guardrail** — add the same guardrail as multiple steps; if it fails, `on_fail: next` moves to the next step, which can be the same guardrail again (useful for transient API errors or rate limits) +- **Conditional routing** — e.g., if a fast guardrail fails, run a more advanced one instead of blocking immediately +- **Custom responses** — return a specific message when a guardrail fails instead of a generic block +- **Data chaining** — pass modified data (e.g., PII-masked content) from one step to the next +- **Fine-grained control** — different actions on pass vs. fail per step + +## Concepts + +### Pipeline + +A pipeline has: + +- **Mode**: `pre_call` (before the LLM) or `post_call` (after the LLM) +- **Steps**: Ordered list of guardrail steps + +### Step actions + +Each step defines what happens when the guardrail **passes** and when it **fails**: + +| Action | Description | +|--------|-------------| +| **Next Step** | Continue to the next guardrail in the pipeline | +| **Allow** | Stop the pipeline and allow the request to proceed | +| **Block** | Stop the pipeline and block the request | +| **Custom Response** | Return a custom message instead of the default block | + +### Step options + +| Field | Type | Description | +|-------|------|--------------| +| `guardrail` | `string` | Name of the guardrail to run | +| `on_pass` | `string` | Action when guardrail passes: `next`, `allow`, `block`, `modify_response` | +| `on_fail` | `string` | Action when guardrail fails: `next`, `allow`, `block`, `modify_response` | +| `pass_data` | `boolean` | Forward modified request data (e.g., PII-masked) to the next step | +| `modify_response_message` | `string` | Custom message when using `modify_response` action | + +## Using the Flow Builder (UI) + +1. Go to **Policies** in the LiteLLM Admin UI +2. Click **+ Create New Policy** or **Edit** on an existing policy +3. Select **Flow Builder** (instead of the simple form) +4. Design your flow: + - **Trigger** — Incoming LLM request (runs when the policy matches) + - **Steps** — Add guardrails, set ON PASS and ON FAIL actions per step + - **End** — Request proceeds to the LLM +5. Use the **+** between steps to insert new steps +6. Use the **Test** panel to run sample messages through the pipeline before saving +7. Click **Save** to create or update the policy + +## Config (YAML) + +Define a pipeline in your policy config: + +```yaml showLineNumbers title="config.yaml" +guardrails: + - guardrail_name: pii_masking + litellm_params: + guardrail: presidio + mode: pre_call + + - guardrail_name: prompt_injection + litellm_params: + guardrail: lakera + mode: pre_call + +policies: + my-pipeline-policy: + description: "PII mask first, then check for prompt injection" + guardrails: + add: + - pii_masking + - prompt_injection + pipeline: + mode: pre_call + steps: + - guardrail: pii_masking + on_pass: next + on_fail: block + pass_data: true + - guardrail: prompt_injection + on_pass: allow + on_fail: block + +policy_attachments: + - policy: my-pipeline-policy + scope: "*" +``` + +## Fallbacks and retries + +### Guardrail fallbacks + +Use `on_fail: next` to fall back to another guardrail when one fails. Run a lightweight guardrail first; if it fails, escalate to a stricter or different provider: + +```yaml +policies: + fallback-policy: + guardrails: + add: + - fast_content_filter + - strict_content_filter + pipeline: + mode: pre_call + steps: + - guardrail: fast_content_filter + on_pass: allow + on_fail: next + - guardrail: strict_content_filter + on_pass: allow + on_fail: block +``` + +If `fast_content_filter` passes → allow. If it fails → run `strict_content_filter`; pass → allow, fail → block. + +### Retrying the same guardrail + +Add the same guardrail as multiple steps to retry on failure. Useful for transient errors (API timeouts, rate limits): + +```yaml +policies: + retry-policy: + guardrails: + add: + - lakera_prompt_injection + pipeline: + mode: pre_call + steps: + - guardrail: lakera_prompt_injection + on_pass: allow + on_fail: next + - guardrail: lakera_prompt_injection + on_pass: allow + on_fail: block +``` + +First attempt passes → allow. First attempt fails → retry the same guardrail; second pass → allow, second fail → block. + +## Example: Custom response on fail + +Return a branded message instead of a generic block: + +```yaml +policies: + branded-block-policy: + guardrails: + add: + - pii_detector + pipeline: + mode: pre_call + steps: + - guardrail: pii_detector + on_pass: allow + on_fail: modify_response + modify_response_message: "Your message contains sensitive information. Please remove PII and try again." +``` + +## Test a pipeline (API) + +Test a pipeline with sample messages before attaching it: + +```bash +curl -X POST "http://localhost:4000/policies/test-pipeline" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "pipeline": { + "mode": "pre_call", + "steps": [ + { + "guardrail": "pii_masking", + "on_pass": "next", + "on_fail": "block", + "pass_data": true + }, + { + "guardrail": "prompt_injection", + "on_pass": "allow", + "on_fail": "block" + } + ] + }, + "test_messages": [ + {"role": "user", "content": "What is 2+2?"}, + {"role": "user", "content": "My SSN is 123-45-6789"} + ] + }' +``` + +Response includes per-step outcomes (pass/fail/error), actions taken, and timing. + +## Pipeline vs simple policy + +When a policy has a `pipeline`, the pipeline defines execution order and actions. The `guardrails.add` list must include all guardrails used in the pipeline steps. + +| Policy type | Execution | +|-------------|-----------| +| Simple (`guardrails.add` only) | All guardrails run; any failure blocks | +| Pipeline (`pipeline` present) | Steps run in order; actions control flow | + +## Related docs + +- [Guardrail Policies](./guardrail_policies) — Policy basics, attachments, inheritance +- [Policy Templates](./policy_templates) — Pre-built policy templates diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index d032b8ca393..8383b826f0d 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -100,6 +100,7 @@ const sidebars = { label: "Policies", items: [ "proxy/guardrails/guardrail_policies", + "proxy/guardrails/policy_flow_builder", "proxy/guardrails/policy_templates", "proxy/guardrails/policy_tags", ],