* feat(guardrails): optional skip system message in unified guardrail inputs
Made-with: Cursor
* feat(dashboard): skip_system_message_in_guardrail in guardrail UI
Add a tri-state control (inherit / yes / no) when creating or editing
guardrails so admins can set litellm_params.skip_system_message_in_guardrail
without YAML. Table edit merges existing litellm_params before PUT to avoid
wiping content-filter and other provider fields.
Document the dashboard flow in the guardrails quick start with a screenshot.
Made-with: Cursor
* fix(guardrails): type structured_messages as AllMessageValues for mypy
Use AllMessageValues in openai_messages_without_system and cast adapter
request messages so GenericGuardrailAPIInputs matches TypedDict.
Made-with: Cursor
Bug 1: internal users hit route-level 403 on /guardrails/submissions.
The route wasn't in self_managed_routes, so the route allowlist rejected
non-admin callers before our endpoint's team-scoping ran. Added
/guardrails/submissions and /guardrails/submissions/{guardrail_id} to
self_managed_routes.
Bug 2: register_guardrail 403'd non-admins registering for teams in
their user.teams list. It used get_team_membership() which reads the
litellm_teammembership join table, but that row is only created when
the team has a budget (management_helpers/utils.py:225). Switched to
the _get_user_team_ids helper (reads user_obj.teams), making it
consistent with list_guardrail_submissions.
UI: moved the Test Playground tab inside the isAdmin conditional in
guardrails.tsx. Internal users now see only the Submitted Guardrails
tab; admins still see all four.
Tests: added coverage for non-admin register paths (cross-team allowed
and cross-team forbidden).
Backend:
- list_guardrail_submissions no longer 403s non-admins; it returns only
submissions whose team_id matches one of the caller's teams (via
get_user_object.teams). Admins still see all.
- Filtering by a team the caller is not in returns 403.
- Users with no team memberships get an empty list (no DB query).
- get_guardrail_submission applies the same scoping to single-item GETs.
Frontend:
- Remove admin-only bail-out in TeamGuardrailsTab.fetchSubmissions so
internal users actually load their team's submissions.
- Finish antd migration in guardrails.tsx: drop the last Tremor Button.
- Remove guardrailsList.length === 0 gate on the Test Playground tab;
the playground already renders a "No guardrails available" inline
empty state, which is more discoverable than a disabled tab.
Tests:
- Cover non-admin scoped access, empty teams, cross-team filter 403,
and per-submission GET scoping.
* Litellm ishaan april1 (#25103)
* fix(proxy): enforce upperbound key params on key/update and add custom_key_update hook
The /key/update endpoint did not enforce upperbound_key_generate_params,
allowing users to bypass configured limits (tpm_limit, rpm_limit,
max_budget, duration, budget_duration) by updating an existing key
instead of generating a new one.
Extract the upperbound enforcement logic from _common_key_generation_helper()
into a standalone _enforce_upperbound_key_params() function and call it from
both the generate and update paths. For updates, None values are skipped
(not filled with defaults) since they mean "don't change this field".
Also adds a custom_key_update config option and user_custom_key_update global,
mirroring the existing custom_key_generate pattern, so custom key validation
logic can fire during key updates as well.
* fix(proxy): invoke custom_key_update hook in bulk update path
The user_custom_key_update hook was only called in update_key_fn
(single key update) but not in _process_single_key_update (bulk
update path), allowing custom validation to be bypassed via the
/key/update/bulk endpoint. Mirror the hook invocation in both paths.
* fix(proxy): pass UpdateKeyRequest to hook in bulk path, not BulkUpdateKeyRequestItem
Move the custom_key_update hook invocation to after UpdateKeyRequest
is constructed so the hook receives the same type in both single and
bulk update paths. Previously the bulk path passed
BulkUpdateKeyRequestItem (5 fields only), which would cause
AttributeError for hooks accessing fields like tpm_limit or models.
* fix(bedrock): promote cache usage to message_delta for Claude Code (#24850)
Ensure Bedrock/Anthropic-compatible streaming exposes cache usage where Claude Code reads it by promoting message_stop usage onto message_delta and preserving usage fields in fake-streamed message_delta events.
Made-with: Cursor
* fix(search): Support self-hosted Firecrawl response format in search transform (#24866)
The `transform_search_response` method only handled Firecrawl Cloud (v2)
response format where `data` is a dict with `web`/`news` keys. Self-hosted
Firecrawl (v1) returns `data` as a flat list of result objects, causing an
`AttributeError: 'list' object has no attribute 'get'`.
Detect the response format by checking if `data` is a list (self-hosted)
or dict (cloud) and handle both cases.
Cloud format: {"data": {"web": [...], "news": [...]}}
Self-hosted: {"success": true, "data": [{"url": "...", "title": "...", ...}]}
Co-authored-by: Synergy <synergyoclaw@gmail.com>
* feat: add environment and user tracking to prompt management (#24855)
* feat: add environment and user tracking to prompt management
- Add environment (development/staging/production) and created_by columns to LiteLLM_PromptTable
- Update unique constraint to [prompt_id, version, environment]
- All CRUD endpoints support environment filtering and user tracking
- Redesigned prompt detail page with environment tabs and version history
- UI: environment filter on list page, environment selector in editor
- 8 new tests for environment and user tracking
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Black formatting and add environments to PromptInfoResponse TypeScript type
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address Greptile review findings
- P1: delete_prompt scopes in-memory cleanup to environment when provided
- P2: dotprompt_content parsed directly regardless of environment flag
- P2: use distinct for environments query
- P2: fix double-fetch on initial mount in prompt_info.tsx
- fix: remove unsupported select kwarg from find_many
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address remaining Greptile review comments
- Remove unused useCallback import (index.tsx)
- Remove unused ENV_COLORS variable (prompt_info.tsx)
- P1: in-memory fallback in get_prompt_versions now respects environment filter
- P1: reset selectedEnv when promptId changes to avoid stale state
- Cyclic imports are pre-existing pattern, not introduced by this PR
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: scope patch_prompt to environment using primary key
- Add environment query param to patch_prompt endpoint
- Look up target row by composite key (prompt_id + version + environment)
- Update by primary key (id) to target exactly one row
- Fixes Greptile finding: patch with multiple environments no longer ambiguous
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use actual start_time for failed request spend logs (#24906)
async_post_call_failure_hook set both start_time and end_time to
datetime.now(), making all failed requests show duration=0. Use the
actual start_time from litellm_logging_obj instead, so spend logs
reflect the real request duration on timeout and other failures.
Fixes#24888
* feat(bedrock): add nova canvas image edit support (#24869)
* feat(bedrock): add nova canvas image edit support
* fix(bedrock): support PathLike inputs for nova image edit
* chore: sync schema.prisma copies from root
* fix(mypy): correct type-ignore code for delta_usage arg-type
* fix(mypy): cast status_code to str, suppress intentional str yield
* fix(lint): extract _create_content_block_chunks to fix PLR0915
* fix(lint): extract helpers to fix PLR0915 in prompt endpoints
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: redhelix <amin.lalji@gmail.com>
Co-authored-by: Synergy <synergyoclaw@gmail.com>
Co-authored-by: Talha Anwar <37379131+talhaanwarch@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: madhu19991 <madhu@thunkai.com>
Co-authored-by: Srikanth @adobe <devarakondasrikanth@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(test): update model armor streaming test to handle string or int error code
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: redhelix <amin.lalji@gmail.com>
Co-authored-by: Synergy <synergyoclaw@gmail.com>
Co-authored-by: Talha Anwar <37379131+talhaanwarch@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: madhu19991 <madhu@thunkai.com>
Co-authored-by: Srikanth @adobe <devarakondasrikanth@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
When Model Armor blocks a streaming response, it correctly raises
HTTPException(status_code=400) but create_response() catches it with a
bare except Exception and hardcodes a 500 response, discarding the
original status code.
Fix create_response() to preserve status_code from HTTPException instead
of hardcoding 500. Also update Model Armor's streaming hook to yield an
SSE error event instead of raising (matching the Prisma Airs pattern),
and fix make_model_armor_request() to return 400 for upstream API
failures instead of passing through the upstream status code.
asyncio.create_task in CSW.__anext__ scheduled the deferred logging
callback as an independent task that raced with unified_guardrail's
end-of-stream block. For short-stream providers (Vertex AI, Azure,
Anthropic), the logging fired before guardrail_information was written,
causing post_call guardrail entries to be missing from
StandardLoggingPayload.
Move the deferred callback trigger from CSW.__anext__ to
ProxyLogging.async_post_call_streaming_iterator_hook (after the full
streaming pipeline completes). CSW now stores the assembled response
args; the outer consumer fires the callback after all guardrail
end-of-stream blocks finish. Also skip apply_guardrail guardrails in
_run_deferred_stream_guardrails to eliminate duplicate API calls.
Two independent bugs prevented post-call OpenAI Moderation guardrail
results from reaching downstream logging callbacks (Langfuse, Datadog).
Bug 1: process_output_response() created a throwaway request_data dict,
so guardrail info written by @log_guardrail_information was discarded.
Fixed by threading the real request_data from the unified guardrail
dispatcher through all 13 BaseTranslation handlers, with litellm_metadata
injection preserved for third-party guardrails (Zscaler, Prompt Security).
Also extended to process_output_streaming_response for consistency.
Bug 2: The @log_guardrail_information decorator collapsed the full
moderation API response (categories, scores, flagged status) to "allow".
Fixed by overriding _process_response/_process_error on
OpenAIModerationGuardrail to stash and log the full response, following
the established Model Armor pattern.
Wrap _run_deferred_stream_guardrails initialization (UnifiedLLMGuardrails
constructor and _check_and_merge_model_level_guardrails) in try/finally
so logging always fires even if init throws. Prevents silent logging loss
on transient errors.
Move fastapi.HTTPException import from module-level to local test-function
scope. Add test_logging_fires_even_if_guardrail_init_raises to verify the
try/finally guard.
Use the merged guardrail_data dict (from _check_and_merge_model_level_guardrails)
for hook invocations in _run_deferred_stream_guardrails, instead of the original
captured_data. This ensures model-level non-default guardrails are visible to
inner should_run_guardrail re-checks inside UnifiedLLMGuardrails.
Rewrite three hand-crafted closure tests to exercise the production
_run_deferred_stream_guardrails exception-handling path. Add three new tests
that use deep-copy mocks to prove hooks receive the merged dict.
guardrail_information is None in StandardLoggingPayload because logging
fires before post-call guardrails write to metadata.
Non-streaming: wrapper_async stores a closure instead of calling
create_task immediately. The proxy fires it in a try/finally after
post_call_success_hook so the SLP is built with guardrail info.
Streaming: a closure on logging_obj is called by CSW.__anext__ at
stream end. The closure runs only guardrail hooks (not all callbacks)
on the assembled response, then fires both logging handlers. This
avoids behavioral changes for non-guardrail callbacks on streaming.
* feat(xai): add grok-4.20 beta 2 models with pricing (#23900)
Add three grok-4.20 beta 2 model variants from xAI:
- grok-4.20-multi-agent-beta-0309 (reasoning + multi-agent)
- grok-4.20-beta-0309-reasoning (reasoning)
- grok-4.20-beta-0309-non-reasoning
Pricing (from https://docs.x.ai/docs/models):
- Input: $2.00/1M tokens ($0.20/1M cached)
- Output: $6.00/1M tokens
- Context: 2M tokens
All variants support vision, function calling, tool choice, and web search.
Closes LIT-2171
* docs: add Quick Install section for litellm --setup wizard (#23905)
* docs: add Quick Install section for litellm --setup wizard
* docs: clarify setup wizard is for local/beginner use
* feat(setup): interactive setup wizard + install.sh (#23644)
* feat(setup): add interactive setup wizard + install.sh
Adds `litellm --setup` — a Claude Code-style TUI onboarding wizard that
guides users through provider selection, API key entry, and proxy config
generation, then optionally starts the proxy immediately.
- litellm/setup_wizard.py: wizard with ASCII art, numbered provider menu
(OpenAI, Anthropic, Azure, Gemini, Bedrock, Ollama), API key prompts,
port/master-key config, and litellm_config.yaml generation
- litellm/proxy/proxy_cli.py: adds --setup flag that invokes the wizard
- scripts/install.sh: curl-installable script (detect OS/Python, pip
install litellm[proxy], launch wizard)
Usage:
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh
litellm --setup
* fix(install.sh): remove orange color, add LITELLM_BRANCH env var for branch installs
* fix(install.sh): install from git branch so --setup is available for QA
* fix(install.sh): remove stale LITELLM_BRANCH reference that caused unbound variable error
* fix(install.sh): force-reinstall from git to bypass cached PyPI version
* fix(install.sh): show pip progress bar during install
* fix(install.sh): always launch wizard via $PYTHON_BIN -m litellm, not PATH binary
* fix(install.sh): use litellm.proxy.proxy_cli module (no __main__.py exists)
* fix(install.sh): suppress RuntimeWarning from module invocation
* fix(install.sh): use Python bin-dir litellm binary to avoid CWD sys.path shadowing
* fix(install.sh): use sysconfig.get_path('scripts') to find pip-installed litellm binary
* fix(install.sh): redirect stdin from /dev/tty on exec so wizard gets terminal, not exhausted pipe
* fix(install.sh): warn about git clone duration, drop --no-cache-dir so re-runs are faster
* feat(setup_wizard): arrow-key selector, updated model names
* fix(setup_wizard): use sysconfig binary to start proxy, not python -m litellm
* feat(setup_wizard): credential validation after key entry + clear next-steps after proxy start
* style(install.sh): show git clone warning in blue
* refactor(setup_wizard): class with static methods, use check_valid_key from litellm.utils
* address greptile review: fix yaml escaping, port validation, display name collisions, tests
- setup_wizard.py: add _yaml_escape() for safe YAML embedding of API keys
- setup_wizard.py: add _styled_input() with readline ANSI ignore markers
- setup_wizard.py: change DIVIDER to _divider() fn to avoid import-time color capture
- setup_wizard.py: validate port range 1-65535, initialize before loop
- setup_wizard.py: qualify azure display names (azure-gpt-4o) to avoid collision with openai
- setup_wizard.py: work on env_copy in _build_config to avoid mutating caller's dict
- setup_wizard.py: skip model_list entries for providers with no credentials
- setup_wizard.py: prompt for azure deployment name
- setup_wizard.py: wrap os.execlp in try/except with friendly fallback
- setup_wizard.py: wrap config write in try/except OSError
- setup_wizard.py: fix _validate_and_report to use two print lines (no \r overwrite)
- setup_wizard.py: add .gitignore tip next to key storage notice
- setup_wizard.py: fix run_setup_wizard() return type annotation to None
- scripts/install.sh: drop pipefail (not supported by dash on Ubuntu when invoked as sh)
- scripts/install.sh: use litellm[proxy] from PyPI (not hardcoded dev branch)
- scripts/install.sh: guard /dev/tty read with -r check for Docker/CI compat
- scripts/install.sh: remove --force-reinstall to avoid downgrading dependencies
- tests/test_litellm/test_setup_wizard.py: 13 unit tests for _build_config and _yaml_escape
* style: black format setup_wizard.py
* fix: address remaining greptile issues - Windows compat, YAML quoting, credential flow
- guard termios/tty imports with try/except ImportError for Windows compat
- quote master_key as YAML double-quoted scalar (same as env vars)
- remove unused port param from _build_config signature
- _validate_and_report now returns the final key so re-entered creds are stored
- add test for master_key YAML quoting
* fix: add --port to suggested command, guard /dev/tty exec in install.sh
* fix: quote api_base in YAML, skip azure if no deployment, only redraw on state change
* fix: address greptile review comments
- _yaml_escape: add control character escaping (\n, \r, \t)
- test: fix tautological assertion in test_build_config_azure_no_deployment_skipped
- test: add tests for control character escaping in _yaml_escape
* feat(ui): remove Chat UI page link and banner from sidebar and playground (#23908)
* feat(guardrails): MCPJWTSigner - built-in guardrail for zero trust MCP auth (#23897)
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* feat(guardrails): add MCPJWTSigner built-in guardrail for zero trust MCP auth
Signs outbound MCP tool calls with a LiteLLM-issued RS256 JWT so MCP servers
can trust a single signing authority instead of every upstream IdP.
Enable in config.yaml:
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
JWT carries sub (user_id), act.sub (team_id, RFC 8693), tool-level scope, iss,
aud, iat/exp/nbf. RSA-2048 keypair auto-generated at startup unless
MCP_JWT_SIGNING_KEY env var is set.
Adds /.well-known/jwks.json endpoint and jwks_uri to /.well-known/openid-configuration
so MCP servers can verify LiteLLM-issued tokens via OIDC discovery.
* Update MCPServerManager to raise HTTPException with status code 400 for extra headers in OpenAPI-backed servers. Adjust tests to verify the correct status code and exception message.
* fix: address P1 issues in MCPJWTSigner
- OpenAPI servers: warn + skip header injection instead of 500
- JWKS Cache-Control: 5min for auto-generated keys, 1h for persistent
- sub claim: fallback to apikey:{token_hash} for anonymous callers
- ttl_seconds: validate > 0 at init time
* docs: add MCP zero trust auth guide with architecture diagram
* docs: add FastMCP JWT verification guide to zero trust doc
* fix: address remaining Greptile review issues (round 2)
- mcp_server_manager: warn when hook Authorization overwrites existing header
- __init__: remove _mcp_jwt_signer_instance from __all__ (private internal)
- discoverable_endpoints: copy dict instead of mutating in-place on OIDC augmentation
- test docstring: reflect warn-and-continue behavior for OpenAPI servers
- test: update scope assertions for least-privilege (no mcp:tools/list on tool-call JWTs)
* fix: address Greptile round 3 feedback
- initialize_guardrail: validate mode='pre_mcp_call' at init time — misconfigured
mode silently bypasses JWT injection, which is a zero-trust bypass
- _build_claims: remove duplicate inline 'import re' (module-level import already present)
- _types.py: add TODO comment explaining jwt_claims is forward-compat plumbing
for a follow-up PR that will forward upstream IdP claims into outbound MCP JWTs
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes
Addresses all missing pieces from the scoping doc review:
FR-5 (Verify + re-sign): MCPJWTSigner now accepts access_token_discovery_uri
and token_introspection_endpoint. When set, the incoming Bearer token is
extracted from raw_headers (threaded through pre_call_tool_check), verified
against the IdP's JWKS (JWT) or introspected (opaque), and only re-signed if
valid. Falls back to user_api_key_dict.jwt_claims for LiteLLM JWT-auth mode.
FR-12 (Configurable end-user identity mapping): end_user_claim_sources
ordered list drives sub resolution — sources: token:<claim>, litellm:user_id,
litellm:email, litellm:end_user_id, litellm:team_id.
FR-13 (Claim operations): add_claims (insert-if-absent), set_claims (always
override), remove_claims (delete) applied in that order.
FR-14 (Two-token model): channel_token_audience + channel_token_ttl issue a
second JWT injected as x-mcp-channel-token: Bearer <token>.
FR-15 (Incoming claim validation): required_claims raises HTTP 403 when any
listed claim is absent; optional_claims passes listed claims from verified
token into the outbound JWT.
FR-9 (Debug headers): debug_headers: true emits x-litellm-mcp-debug with kid,
sub, iss, exp, scope.
FR-10 (Configurable scopes): allowed_scopes replaces auto-generation. Also
fixed: tool-call JWTs no longer grant mcp:tools/list (overpermission).
P1 fixes:
- proxy/utils.py: _convert_mcp_hook_response_to_kwargs merges rather than
replaces extra_headers, preserving headers from prior guardrails.
- mcp_server_manager.py: warns when hook injects Authorization alongside a
server-configured authentication_token (previously silent).
- mcp_server_manager.py: pre_call_tool_check now accepts raw_headers and
extracts incoming_bearer_token so FR-5 verification has the raw token.
- proxy/utils.py: remove stray inline import inspect inside loop (pre-existing
lint error, now cleaned up).
Tests: 43 passing (28 new tests covering all FR flags + P1 fixes).
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes (core)
Remaining files from the FR implementation:
mcp_jwt_signer.py — full rewrite with all new params:
FR-5: access_token_discovery_uri, token_introspection_endpoint,
verify_issuer, verify_audience + _verify_incoming_jwt(),
_introspect_opaque_token()
FR-12: end_user_claim_sources ordered resolution chain
FR-13: add_claims, set_claims, remove_claims
FR-14: channel_token_audience, channel_token_ttl → x-mcp-channel-token
FR-15: required_claims (raises 403), optional_claims (passthrough)
FR-9: debug_headers → x-litellm-mcp-debug
FR-10: allowed_scopes; tool-call JWTs no longer over-grant tools/list
mcp_server_manager.py:
- pre_call_tool_check gains raw_headers param to extract incoming_bearer_token
- Silent Authorization override warning fixed: now fires when server has
authentication_token AND hook injects Authorization
tests/test_mcp_jwt_signer.py:
28 new tests covering all FR flags + P1 fixes (43 total, all passing)
* fix(mcp_jwt_signer): address pre-landing review issues
- Remove stale TODO comment on UserAPIKeyAuth.jwt_claims — the field is
already populated and consumed by MCPJWTSigner in the same PR
- Fix _get_oidc_discovery to only cache the OIDC discovery doc when
jwks_uri is present; a malformed/empty doc now retries on the next
request instead of being permanently cached until proxy restart
- Add FR-5 test coverage for _fetch_jwks (cache hit/miss),
_get_oidc_discovery (cache/no-cache on bad doc), _verify_incoming_jwt
(valid token, expired token), _introspect_opaque_token (active,
inactive, no endpoint), and the end-to-end 401 hook path — 53 tests
total, all passing
* docs(mcp_zero_trust): rewrite as use-case guide covering all new JWT signer features
Add scenario-driven sections for each new config area:
- Verify+re-sign with Okta/Azure AD (access_token_discovery_uri,
end_user_claim_sources, token_introspection_endpoint)
- Enforcing caller attributes with required_claims / optional_claims
- Adding metadata via add_claims / set_claims / remove_claims
- Two-token model for AWS Bedrock AgentCore Gateway
(channel_token_audience / channel_token_ttl)
- Controlling scopes with allowed_scopes
- Debugging JWT rejections with debug_headers
Update JWT claims table to reflect configurable sub (end_user_claim_sources)
* fix(mcp_jwt_signer): wire all config.yaml params through initialize_guardrail
The factory was only passing issuer/audience/ttl_seconds to MCPJWTSigner.
All FR-5/9/10/12/13/14/15 params (access_token_discovery_uri,
end_user_claim_sources, add/set/remove_claims, channel_token_audience,
required/optional_claims, debug_headers, allowed_scopes, etc.) were
silently dropped, making every advertised advanced feature non-functional
when loaded from config.yaml.
Add regression test that asserts every param is wired through correctly.
* docs(mcp_zero_trust): add hero image
* docs(mcp_zero_trust): apply Linear-style edits
- Lead with the problem (unsigned direct calls bypass access controls)
- Shorter statement section headers instead of question-form headers
- Move diagram/OIDC discovery block after the reader is bought in
- Add 'read further only if you need to' callout after basic setup
- Two-token section now opens from the user problem not product jargon
- Add concrete 403 error response example in required_claims section
- Debug section opens from the symptom (MCP server returning 401)
- Lowercase claims reference header for consistency
* fix(mcp_jwt_signer): fix algorithm confusion attack + add OIDC discovery 24h TTL
- Remove alg from unverified JWT header; use signing_jwk.algorithm_name from JWKS key instead.
Reading alg from attacker-controlled headers enables alg:none / HS256 confusion attacks.
- Add _oidc_discovery_fetched_at timestamp and _OIDC_DISCOVERY_TTL = 86400 (24h).
Without a TTL the cached discovery doc never refreshes, so IdP key rotation is invisible.
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* fix(ci): stabilize CI - formatting, type errors, test polling, security CVEs, router bug, batch resolution
Fix 1: Run Black formatter on 35 files
Fix 2: Fix MyPy type errors:
- setup_wizard.py: add type annotation for 'selected' set variable
- user_api_key_auth.py: remove redundant type annotation on jwt_claims reassignment
Fix 3: Fix spend accuracy test burst 2 polling to wait for expected total
spend instead of just 'any increase' from burst 2
Fix 4: Bump Next.js 16.1.6 -> 16.1.7 to fix CVE-2026-27978, CVE-2026-27979,
CVE-2026-27980, CVE-2026-29057
Fix 5: Fix router _pre_call_checks model variable being overwritten inside
loop, causing wrong model lookups on subsequent deployments. Use local
_deployment_model variable instead.
Fix 6: Add missing resolve_output_file_ids_to_unified call in batch retrieve
non-terminal-to-terminal path (matching the terminal path behavior)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* chore: regenerate poetry.lock to sync with pyproject.toml
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: format merged files from main and regenerate poetry.lock
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): annotate jwt_claims as Optional[dict] to fix type incompatibility
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): update router region test to use gpt-4.1-mini (fix flaky model lookup)
Replace deprecated gpt-3.5-turbo-1106 with gpt-4.1-mini + mock_response in
test_router_region_pre_call_check, following the same pattern used in commit
717d37cc5b for test_router_context_window_check_pre_call_check_out_group.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* ci: retry flaky logging_testing (async event loop race condition)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): aggregate all mock calls in langfuse e2e test to fix race condition
The _verify_langfuse_call helper only inspected the last mock call
(mock_post.call_args), but the Langfuse SDK may split trace-create and
generation-create events across separate HTTP flush cycles. This caused
an IndexError when the last call's batch contained only one event type.
Fix: iterate over mock_post.call_args_list to collect batch items from
ALL calls. Also add a safety assertion after filtering by trace_id and
mark all langfuse e2e tests with @pytest.mark.flaky(retries=3) as an
extra safety net for any residual timing issues.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): black formatting + update OpenAPI compliance tests for spec changes
- Apply Black 26.x formatting to litellm_logging.py (parenthesized style)
- Update test_input_types_match_spec to follow $ref to InteractionsInput schema
(Google updated their OpenAPI spec to use $ref instead of inline oneOf)
- Update test_content_schema_uses_discriminator to handle discriminator without
explicit mapping (Google removed the mapping key from Content discriminator)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* revert: undo incorrect Black 26.x formatting on litellm_logging.py
The file was correctly formatted for Black 23.12.1 (the version pinned
in pyproject.toml). The previous commit applied Black 26.x formatting
which was incompatible with the CI's Black version.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): deduplicate and sort langfuse batch events after aggregation
The Langfuse SDK may send the same event (e.g., trace-create) in
multiple flush cycles, causing duplicates when we aggregate from all
mock calls. After filtering by trace_id, deduplicate by keeping only
the first event of each type, then sort to ensure trace-create is at
index 0 and generation-create at index 1.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: Register DynamoAI guardrail initializer and enum entry
Fix the "Unsupported guardrail: dynamoai" error by:
1. Adding DYNAMOAI to SupportedGuardrailIntegrations enum
2. Implementing initialize_guardrail() and registries in dynamoai/__init__.py
The DynamoAI guardrail was added in PR #15920 but never properly registered
in the initialization system. The __init__.py was missing the
guardrail_initializer_registry and guardrail_class_registry dictionaries
that the dynamic discovery mechanism looks for at module load time.
Fixes#22773
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Update litellm/proxy/guardrails/guardrail_hooks/dynamoai/__init__.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update litellm/proxy/guardrails/guardrail_hooks/dynamoai/__init__.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* test: Add tests for DynamoAI guardrail registration
Verifies enum entry, initializer registry, class registry,
instance creation, and global registry discovery.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat: Add Canadian PII protection patterns and PIPEDA-compliant policy template
Adds 6 new Canadian PII regex pattern detectors to patterns.json:
- ca_sin: Social Insurance Number (PIPEDA Privacy Act, Income Tax Act)
- ca_ohip: Ontario Health Insurance Plan Number (PHIPA)
- ca_on_drivers_licence: Ontario driver's licence (HTA, PIPEDA)
- ca_immigration_doc: IRCC immigration docs (UCI, work/study permits, IMM refs)
- ca_bank_account: Canadian bank account routing (transit-institution-account)
- ca_postal_code: Canadian postal code (Canada Post spec)
Adds comprehensive policy template 'canadian-pii-protection' (id: canadian-pii-protection)
with 5 sub-guardrails grouping patterns by data type. All patterns include contextual
keyword matching (English + French keywords where applicable) to reduce false positives.
Complements existing passport_canada pattern.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat: Add Canadian PII compliance dataset and tests (57 tests)
Adds:
- test_ca_patterns.py: 30 unit tests for regex pattern matching (SIN, OHIP,
driver's licence, immigration docs, bank account, postal code)
- test_ca_policy_e2e.py: 27 end-to-end tests running the full
ContentFilterGuardrail pipeline with MASK action — validates detection
of real PII and pass-through of clean prompts
- canadianPiiCompliancePrompts.ts: 21-prompt compliance dataset for UI
evaluation, wired into the main compliancePrompts framework
Fixes keyword_pattern alternation ordering in patterns.json — longer
alternatives (e.g. "social insurance number") now precede shorter ones
("social insurance") to avoid excessive gap-word count when the regex
engine selects the shorter match first.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat: Add University of Toronto FIPPA identifier patterns and tests (36 tests)
Add 3 UofT institutional identifiers (student/employee number, UTORid, TCard)
covered under Ontario FIPPA. Includes pattern definitions, policy template
sub-guardrail, compliance prompts, unit tests, and e2e tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Correct test assertion bug and inaccurate docstring
Fix test_utorid_masked checking `result` (dict) instead of `output` (string).
Update test_ca_policy_e2e.py docstring to clarify scope vs UofT tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Address Greptile review feedback
- Tighten ca_postal_code keyword_pattern: replace broad "address" with
specific compound terms (mailing/street/shipping/home address)
- Add missing "PIPEDA" tag to policy_templates.json for discoverability
- Add us_phone pattern to test_ca_policy_e2e.py setup to match deployed template
- Add phone number e2e test for complete coverage
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Tighten patterns to reduce false positives and add missing test coverage
- ca_sin: reject leading-zero SINs ([1-9]\d{2}), set allow_word_numbers to false
- ca_immigration_doc: require separators in UCI pattern (prevent bare \d{10} match)
- uoft_utorid: qualify generic keywords (acorn -> acorn login, quercus -> quercus login)
- uoft_tcard: remove generic keywords (student card, id card, library card) that
overlap with credit card contexts; keep only UofT-specific terms (tcard, campus card)
- Add visa/mastercard/amex/iban patterns to test_ca_policy_e2e.py setup to match
deployed template; add Visa card masking test
- Add test verifying "student card" no longer triggers TCard redaction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
* azure content enhancement...
* rafactored to increase confidence score
* improvements based on additional feedback
* removed unused import
* Force-split any word longer than max length allowed
* preserve whitespace in text splitting
* moving common initialization to base class
* consolidate enforcement into async_make_request as single point, remove redundant caller-side checks, extract shared init/HTTP logic into base, and fix stale log messages
* clean up
* clean up tests
* style(ui/): distinguish agent calls from llm calls on ui
* feat: initial grouping working
* feat: set stable contextid for a2a calls - allows for easily passing to downstream llm/mcp calls
* feat(a2a_endpoints.py): fix tracing to avoid recreating logging objects for the same call
allows stable trace id usage
* fix(guardrail_endpoints): handle string ui_type values in _build_field_dict
_build_field_dict unconditionally called .value on ui_type, which crashes
for guardrail configs that use plain strings (e.g. BlockCodeExecutionGuardrailConfigModel
uses "multiselect" and "percentage"). Now checks with hasattr before calling .value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: propagate trace/session id from headers in MCP server calls
Cherry-picked mcp_server/server.py fixes from 6feb9bab: adds
get_chain_id_from_headers to extract x-litellm-trace-id /
x-litellm-session-id from raw headers, and uses it in call_tool
and list_tools to keep spend logs and tracing consistent with A2A.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(guardrails): team-based guardrail registration and approval workflow
Add team-based guardrail submission system where teams can register
Generic Guardrail API guardrails for admin review. Includes:
- POST /guardrails/register endpoint for team-scoped submissions
- Admin review endpoints (list/get/approve/reject submissions)
- Team Guardrails tab in the UI dashboard
- extra_headers support for forwarding client headers to guardrail APIs
- Prisma schema migration for status, submitted_at, reviewed_at fields
- Documentation for team-based guardrails and static/dynamic headers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(guardrails): address review feedback - SSRF, silent failure, redundant query
- Validate api_base URL scheme (http/https only) and hostname in
register_guardrail to prevent SSRF via team submissions
- Return warning field in approve response when in-memory initialization
fails so admins know the guardrail won't work until next sync cycle
- Eliminate redundant DB query in list_guardrail_submissions by fetching
all team guardrails once and deriving both filtered list and summary
counts from the single result set
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(guardrails): add pending_review status guard to reject endpoint
Prevent rejecting already-active or already-rejected guardrails, which
would create a DB/memory inconsistency (active in memory but rejected
in DB). Now mirrors the approve endpoint's status check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add CrowdStrike AIDR guardrail hook
* fixup! use apply_guardrail event hook
* fixup! update imports
* fix(guardrails): include AI response in CrowdStrike AIDR output events
Issue:
_build_guard_input_for_response() was:
- Sending only the original user input (messages).
- Not sending the AI provider response.
This fix will:
- Extract response.choices from the ModelResponse object and include them in guard_input payload.
- Thus, ensure AIDR output rules receive the AI-generated content for analysis.
- Fix and update tests.
* fix(guardrails): prevent duplicate input events in CrowdStrike AIDR guardrail
Issue:
The CrowdStrike AIDR guardrail was running on during_call hooks wihtout event_hook configured.
This fix will:
- Set event_hook to ["pre_call", "post_call"] (AIDR admins will control what policy is applied)
This change will:
- Require default_on parameter
- Prevent duplicate API calls to AIDR for the same input
- Avoid unchecked AI provider API calls on during_call hook
* docs: add CrowdStrike AIDR to the list of Guardrails under Integrations
* docs: update CrowdStrike AIDR documentation page
---------
Co-authored-by: Konstantin Lapine <konstantin.lapine@crowdstrike.com>
Enables guardrail processing for OCR requests and responses. Adds OCR handler under litellm/llms/mistral/ocr/guardrail_translation/ to process document URLs on input and extracted page markdown on output. Includes route-to-call-type mappings for /ocr and /v1/ocr endpoints. Adds 14 unit tests and 4 e2e tests verifying handler discovery, input/output processing, and integration with UnifiedLLMGuardrails.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Adds a new block_code_execution guardrail that detects markdown fenced code blocks
in request/response content and blocks or masks them by language. Includes full
UI integration, type definitions, compliance test dataset, and 26 unit tests.
Key guardrail capabilities:
- Regex-based fenced code block detection with configurable blocked languages
- Confidence scoring with tunable threshold
- Execution-intent heuristics (request-side only) with conflict resolution
- Block or mask actions for detected code
- Support for pre_call, post_call, and during_call event hooks
Security hardening:
- Response-side blocking skips intent heuristics (LLM output doesn't contain
user intent phrases, so checking would silently disable post_call blocking)
- No-execution short-circuit includes conflict resolution: if both no-execution
and execution phrases match, execution intent wins
- Tightened overly broad phrases to prevent trivial bypass
- _normalize_escaped_newlines only applies to pure-escaped payloads to avoid
corrupting content that discusses escape sequences
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add post-call hook for Lakera guardrail and mask PII in responses
* Add post-call hook for Lakera and mask PII in responses
* Fix post-call hook: pass event_type to call_v2_guard
* Address Greptile review: return ModelResponse, fix mutation, add header, test location, mask order
- PII masking path: return ModelResponse instead of dict so deployment hook accepts it
- Avoid mutating request data: deep copy original_messages and messages in _mask_pii_in_messages
- Add guardrail header in PII-only return path
- Add test in tests/test_litellm/ (test_lakera_ai_v2.py) per PR checklist
- Sort PII payload spans by (start,end) descending so multiple spans in one message mask correctly
Co-authored-by: Cursor <cursoragent@cursor.com>
* Updated ponteital for index mismatch when choices have null content and inconsistent on_flagged access pattern
* Update litellm/proxy/guardrails/guardrail_hooks/lakera_ai_v2.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update to explicitly state supported endpoints - chat completions
* Fix minor lint error on masked_entity_count
---------
Co-authored-by: Steve <steve.giguere@lakera.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(proxy): tool policies - auto-discover tools, manage policies, guardrail enforcement
- New LiteLLM_ToolTable in schema.prisma to store discovered tools
- Auto-discovery: tools seen in LLM responses get upserted via ToolDiscoveryQueue
(hooks into DBSpendUpdateWriter, same pipeline as spend tracking)
- Management endpoints: GET /v1/tool/list, GET /v1/tool/{name}, POST /v1/tool/policy
- ToolPolicyGuardrail: blocks tool_calls in responses based on policy setting
- UI: Tool Policies page under Guardrails section with policy selector,
filters by policy/team/key, live tail, sortable table
- Unit tests for queue, writer, endpoints, guardrail
* feat(tool-policies): track call_count + discover tools from request body and /messages API
- Add call_count column to LiteLLM_ToolTable; incremented on every flush
- Extract tools from request body too (not just response tool_calls):
- OpenAI /chat/completions: tools[].function.name
- Anthropic /messages pass-through: request_body.tools[].name
- Show call_count column in UI table (sortable)
- UI: drop dual_llm option, keep only trusted/blocked
* fix: address greptile review feedback
- Remove redundant @@index([tool_name]) from schema.prisma (tool_name has @unique which already creates an index)
- Replace gen_random_uuid()::text with str(uuid.uuid4()) for portability
- Rewrite test_tool_registry_writer.py to mock execute_raw/query_raw (actual implementation) instead of Prisma model methods
- Fix test patches in test_tool_management_endpoints.py to target source modules since imports are inside function bodies
- Add "Tool Policies" page title to ToolPolicies.tsx
* fix: address greptile review round 2
- Replace NOW() with Python datetime parameter in tool_registry_writer (SQLite portability)
- Fix cache key collision in tool_policy_guardrail: use null-byte separator instead of colon
- Remove type==function filter from request-side tool extraction to match response-side behavior
- Clear seen_tool_names on flush so call_count increments per batch cycle not per pod lifetime
* fix: address greptile review round 3
- Fix test_seen_names_persist_across_flushes to match actual per-flush-cycle behavior
- Update module docstring in tool_discovery_queue.py to accurately describe flush behavior
- Add created_at/updated_at to raw SQL INSERT in batch_upsert_tools and update_tool_policy
* fix: cache tool policies per tool name not per combination
Previously the cache key was built from the full set of tool names in a
request, so each unique combination of tools got its own cold cache entry
and triggered a separate DB query. With N distinct tools across requests
this was effectively a DB hit on every request.
Now each tool name is cached individually. Cache hits are checked per
tool, only missing tools are fetched from DB in a single batch query,
and each result is cached separately. Once a tool's policy is warm,
any subsequent request using that tool benefits from the cache regardless
of what other tools are in the request.
* Update ui/litellm-dashboard/src/components/ToolPolicies.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: feat: add litellm_system_prompt support
* feat: support new 'litellm_agent' model provider
* feat: ui/ - new agent builder ui
* fix(anthropic/chat/transformation.py): normalize max_tokens if decimal
* feat(agentbuilderview.tsx): run compliance datasets against litellm agent
* feat: new response rejection detector
* fix: multiple fixes
* feat: add mcp tools support to agent builder
create an agent with access to llm's + mcp servers
* feat: add competitor name blocker guardrail
* fix: fix batch test endpoint for compliance playground
* fix(airline.py): add list of all known airlines to airline competitor name detector
prevent competitor discussion on company chatbot
* feat: ui tweaks for prod
* Add keyword-based topic blocker implementation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add embedding-based topic blocker using MiniLM
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add topic blocker package init with exports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add synthetic engine eval set (34 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment questions eval set (207 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine eval synthetic policy config
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MiniLM eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MPNet eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment TF-IDF eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add unified eval runner with confusion matrix reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add benchmarks comparison table in markdown
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Clean up topic blocker: remove unused blockers, add phrase_patterns to content filter
- Remove embedding_blocker.py, api_embedding_blocker.py, nli_blocker.py,
tfidf_blocker.py, onnx_blocker.py (heavy deps not in Docker, inferior accuracy)
- Remove airline_off_topic_restriction policy template and its test
- Fix __init__.py to only export DeniedTopic and TopicBlocker (no eager import crash)
- Add phrase_patterns support to ContentFilterGuardrail for regex-based paraphrase detection
- Rewrite denied_financial_advice.yaml with conditional matching (identifier + block word),
always-block keywords, phrase patterns, and exception phrases
- Clean up test_eval.py: only keyword blocker + content filter tests remain (no network calls)
- All 207 eval cases pass at 100% F1, 0 FP, 0 FN, <0.1ms latency
Addresses all Greptile review comments:
- Eager import crash (embedding deps) → fixed
- Undeclared dependencies → fixed (files deleted)
- lru_cache memory leak → fixed (file deleted)
- Real network calls in tests → fixed (embedding tests removed)
- Unused Dict import → already fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add LLM-as-judge eval and update BENCHMARKS.md
- Add TestInvestmentLlmJudgeGpt4oMini and TestInvestmentLlmJudgeClaude
test classes that use litellm.completion() to classify messages
- System prompt instructs LLM to act as airline chatbot content moderator
- Tests skip gracefully when API keys aren't set
- Update BENCHMARKS.md with production results table, historical comparison,
and instructions for running LLM judge evals
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move evals and benchmarks to guardrail_benchmarks folder
Move eval runner, eval data (JSONL), and results from
tests/test_litellm/.../topic_blocker/ into the guardrail implementation
folder at litellm/.../litellm_content_filter/guardrail_benchmarks/.
This keeps benchmarks co-located with the guardrail code they test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove standalone topic_blocker package, consolidate into content_filter
The standalone keyword_blocker.py was redundant with content_filter.py +
denied_financial_advice.yaml. Removed the entire topic_blocker/ package,
engine eval files, and old keyword blocker results. Simplified test_eval.py
to only test ContentFilter + LLM judge baselines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix compliance playground batch scoring bug, add display_name support
The compliance playground was sending all texts in a single batch API call,
but the content filter raises HTTPException on the first blocked text. This
caused a single blocked/allowed result to be applied to all rows, producing
incorrect scores (e.g. 41% instead of 100%). Fix by sending each text
individually to get per-text results with progressive UI updates.
Also add display_name field support for category YAML files so
denied_financial_advice shows as "Denied Financial / Investment Advice"
in the UI dropdown.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add block_investment CSV eval set and update benchmark result JSON
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* address greptile review feedback (greploop iteration 1)
Fix stale test path in denied_financial_advice.yaml comment.
Other comments were on files already deleted in prior commits.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(schema.prisma): add object permissions for end users
allows controlling if end user can call specific mcp servers
* feat: cleanup for customer_endpoints support of object permission id
* fix: cleanup str
* feat(customers/): enforce end user can only call allowed mcps - if configured
* docs: document customer/end user object permission usage
* feat: enforce end user permissions on MCP tool calls
This commit implements end user permission enforcement for MCP servers:
1. Always add server prefixes to MCP tool names
- Removed conditional logic that only added prefixes when multiple servers existed
- Now always adds server prefix for consistent tool naming across all scenarios
- Updated 5 locations in server.py (list_tools, get_prompts, get_resources,
get_resource_templates, get_prompt)
2. Created MCP End User Permission Guardrail Hook
- New guardrail hook: litellm/proxy/guardrails/guardrail_hooks/mcp_end_user_permission.py
- Runs on post_call to validate tool calls in LLM responses
- Extracts MCP server name from tool names (splits on first '-')
- Checks if end_user_id has permissions for the MCP server
- Raises GuardrailRaisedException if end user lacks permission
- Supports both streaming and non-streaming responses
3. Added comprehensive tests
- Test file: tests/test_litellm/proxy/guardrails/guardrail_hooks/test_mcp_end_user_permission.py
- Tests cover: authorized/unauthorized tools, non-MCP tools, no end_user scenarios
- Tests permission checking logic and exception raising
The hook integrates with the existing MCPRequestHandler._get_allowed_mcp_servers_for_end_user
to fetch end user permissions and enforce access control at the response level.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* refactor: remove redundant add_prefix variable assignments
Simplified the code by removing intermediate `add_prefix` variable
assignments and passing `True` directly to function calls since
we now always add server prefixes.
Changes:
- Removed `add_prefix = True` variable assignments in 5 locations
- Changed `add_prefix=add_prefix` to `add_prefix=True` in function calls
- Added inline comments to clarify the behavior
This makes the code more concise and clearer in intent.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat(auth_utils.py): support safety_identifier as a valid way of passing the end user id for responses api
* feat(llms): ensure 'tools' is correctly updated for responses api
* fix: fix greptile feedback
* feat: transformation.py
proper responses api tool handling for guardrail translation layer
---------
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
