litellm/tests/e2e/logging
yucheng-berri 69c5839cc0
fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712)
* fix(guardrails): filter Add-Guardrail mode dropdown per provider

The GET /guardrails/ui/add_guardrail_settings endpoint returned every
GuardrailEventHooks value in one flat supported_modes list, so the Admin
UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving
Content Filter or Tool Permission with pre_mcp_call then failed with a
400 because those guardrails' server-side supported_event_hooks list
excludes it.

Expose each guardrail's supported hooks as a get_supported_event_hooks
classmethod on CustomGuardrail (mirrors the existing get_config_model
pattern) and have the endpoint iterate guardrail_class_registry to build
a supported_modes_by_provider map. The UI Mode dropdown filters by that
map when the selected provider is known and falls back to the global
list otherwise. __init__ now sources its own supported_event_hooks list
from the classmethod so the two sides can't drift.

Also register BedrockGuardrail, ToolPermissionGuardrail, lakera,
lakera_v2, and presidio in guardrail_class_registry so they participate
in the map (they were previously only in guardrail_initializer_registry
and had no class-registry entry).

Behavior change: guardrails that previously had no supported_event_hooks
declared (aim, javelin, azure/text_moderation, cato_networks,
crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx,
prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai,
lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now
validate the configured mode at instantiation. Existing configs where
the mode was silently a no-op will fail at proxy startup with a clear
validation error rather than running as a broken guardrail.

Resolves LIT-4226

* fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form

Address Greptile P1 (startup break) and P2 (edit form UX):

LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported
event_hook, unchanged behavior for the guardrails validated pre-PR).
Setting it to false logs a warning and continues, giving deployments an
opt-out while they fix configs that now surface as errors instead of
silently no-op'ing. Regression test covers both modes.

Edit form now surfaces the currently-saved mode even when it is not in
the filtered per-provider list, so a legacy row (e.g. content_filter
saved with pre_mcp_call before this fix) no longer disappears from the
dropdown; the option renders with a 'not supported by <provider>' note
so the user knows to pick another.

* fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint

Audited every get_supported_event_hooks classmethod against the hooks
each guardrail's own tests exercise and its handler methods. Five were
too narrow and their tests caught it in CI: rubrik gains pre_call,
presidio gains during_call and pre_mcp_call, prompt_security, onyx and
qualifire gain during_call. The remaining classes match either their
original __init__ declarations or their exercised modes exactly.

Cursor review fixes: the Add form now drops selected modes the new
provider does not support when the user switches providers, so a
pre_mcp_call selection cannot ride along into a provider that rejects
it at save; the edit form handles list-shaped stored modes instead of
treating mode as always a string.

Extracted shared toModeArray and getSupportedModesForProvider helpers
into guardrail_info_helpers so both forms use one implementation, typed
the remaining any usages in both forms, removed nested ternaries, and
committed the ratcheted-down eslint metrics and pruned suppressions
2026-07-11 14:51:27 -07:00
..
conftest.py test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857) 2026-07-11 14:11:00 -04:00
logging_client.py fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712) 2026-07-11 14:51:27 -07:00
test_langfuse_e2e.py test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857) 2026-07-11 14:11:00 -04:00
test_prometheus_cardinality_e2e.py test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction (#32165) 2026-07-04 18:56:52 -07:00