litellm/tests/e2e
yucheng-berri 69c5839cc0
fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712)
* fix(guardrails): filter Add-Guardrail mode dropdown per provider

The GET /guardrails/ui/add_guardrail_settings endpoint returned every
GuardrailEventHooks value in one flat supported_modes list, so the Admin
UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving
Content Filter or Tool Permission with pre_mcp_call then failed with a
400 because those guardrails' server-side supported_event_hooks list
excludes it.

Expose each guardrail's supported hooks as a get_supported_event_hooks
classmethod on CustomGuardrail (mirrors the existing get_config_model
pattern) and have the endpoint iterate guardrail_class_registry to build
a supported_modes_by_provider map. The UI Mode dropdown filters by that
map when the selected provider is known and falls back to the global
list otherwise. __init__ now sources its own supported_event_hooks list
from the classmethod so the two sides can't drift.

Also register BedrockGuardrail, ToolPermissionGuardrail, lakera,
lakera_v2, and presidio in guardrail_class_registry so they participate
in the map (they were previously only in guardrail_initializer_registry
and had no class-registry entry).

Behavior change: guardrails that previously had no supported_event_hooks
declared (aim, javelin, azure/text_moderation, cato_networks,
crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx,
prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai,
lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now
validate the configured mode at instantiation. Existing configs where
the mode was silently a no-op will fail at proxy startup with a clear
validation error rather than running as a broken guardrail.

Resolves LIT-4226

* fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form

Address Greptile P1 (startup break) and P2 (edit form UX):

LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported
event_hook, unchanged behavior for the guardrails validated pre-PR).
Setting it to false logs a warning and continues, giving deployments an
opt-out while they fix configs that now surface as errors instead of
silently no-op'ing. Regression test covers both modes.

Edit form now surfaces the currently-saved mode even when it is not in
the filtered per-provider list, so a legacy row (e.g. content_filter
saved with pre_mcp_call before this fix) no longer disappears from the
dropdown; the option renders with a 'not supported by <provider>' note
so the user knows to pick another.

* fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint

Audited every get_supported_event_hooks classmethod against the hooks
each guardrail's own tests exercise and its handler methods. Five were
too narrow and their tests caught it in CI: rubrik gains pre_call,
presidio gains during_call and pre_mcp_call, prompt_security, onyx and
qualifire gain during_call. The remaining classes match either their
original __init__ declarations or their exercised modes exactly.

Cursor review fixes: the Add form now drops selected modes the new
provider does not support when the user switches providers, so a
pre_mcp_call selection cannot ride along into a provider that rejects
it at save; the edit form handles list-shaped stored modes instead of
treating mode as always a string.

Extracted shared toModeArray and getSupportedModesForProvider helpers
into guardrail_info_helpers so both forms use one implementation, typed
the remaining any usages in both forms, removed nested ternaries, and
committed the ratcheted-down eslint metrics and pruned suppressions
2026-07-11 14:51:27 -07:00
..
access_control test(e2e): migrate access-control and inference-endpoint regression tests (#32016) 2026-07-05 01:39:10 +00:00
batches fix(e2e): batch credentials wiring and compose harness for live proxy suite (#32744) 2026-07-10 11:31:40 -07:00
budgets ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
coverage_registry ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
llm_translation Merge pull request #32918 from BerriAI/litellm_e2e_basedpyright_gate 2026-07-11 12:01:50 -07:00
logging fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712) 2026-07-11 14:51:27 -07:00
management ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
spend_tracking test(e2e): probe the full spend read surface including schema-hidden routes (#32267) 2026-07-06 14:02:05 -07:00
bob_the_builder.py test: litellm fix failing tests (#32577) 2026-07-09 13:54:45 -07:00
CLAUDE.md ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
conftest.py test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857) 2026-07-11 14:11:00 -04:00
CONTRIBUTING.md ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
docker-compose.yml fix(e2e): batch credentials wiring and compose harness for live proxy suite (#32744) 2026-07-10 11:31:40 -07:00
e2e_config.py fix(ui): scope key models dropdown options to the key's team (#32382) 2026-07-07 18:54:19 -07:00
e2e_gateway.py test: litellm fix failing tests (#32577) 2026-07-09 13:54:45 -07:00
e2e_http.py test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857) 2026-07-11 14:11:00 -04:00
lifecycle.py test(e2e): add live batches suite across providers and routing scenarios (#30958) 2026-07-02 08:05:23 -07:00
models.py Merge pull request #32918 from BerriAI/litellm_e2e_basedpyright_gate 2026-07-11 12:01:50 -07:00
pytest.ini test: add e2e tests for spend, budgets and llms (#30869) 2026-06-24 15:01:57 -07:00
test_e2e_gateway.py test(e2e): make dynamic model provisioning robust on split deployments (#32670) 2026-07-09 13:39:47 -07:00
test_lifecycle.py test: add e2e tests for spend, budgets and llms (#30869) 2026-06-24 15:01:57 -07:00
test_transport.py test(e2e): probe the full spend read surface including schema-hidden routes (#32267) 2026-07-06 14:02:05 -07:00
transport.py test(e2e): probe the full spend read surface including schema-hidden routes (#32267) 2026-07-06 14:02:05 -07:00