litellm

scriptoriumadmin/litellm

Fork 0

mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00

Commit graph

Author	SHA1	Message	Date
yucheng-berri	69c5839cc0	fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712 ) * fix(guardrails): filter Add-Guardrail mode dropdown per provider The GET /guardrails/ui/add_guardrail_settings endpoint returned every GuardrailEventHooks value in one flat supported_modes list, so the Admin UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving Content Filter or Tool Permission with pre_mcp_call then failed with a 400 because those guardrails' server-side supported_event_hooks list excludes it. Expose each guardrail's supported hooks as a get_supported_event_hooks classmethod on CustomGuardrail (mirrors the existing get_config_model pattern) and have the endpoint iterate guardrail_class_registry to build a supported_modes_by_provider map. The UI Mode dropdown filters by that map when the selected provider is known and falls back to the global list otherwise. __init__ now sources its own supported_event_hooks list from the classmethod so the two sides can't drift. Also register BedrockGuardrail, ToolPermissionGuardrail, lakera, lakera_v2, and presidio in guardrail_class_registry so they participate in the map (they were previously only in guardrail_initializer_registry and had no class-registry entry). Behavior change: guardrails that previously had no supported_event_hooks declared (aim, javelin, azure/text_moderation, cato_networks, crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx, prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai, lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now validate the configured mode at instantiation. Existing configs where the mode was silently a no-op will fail at proxy startup with a clear validation error rather than running as a broken guardrail. Resolves LIT-4226 * fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form Address Greptile P1 (startup break) and P2 (edit form UX): LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported event_hook, unchanged behavior for the guardrails validated pre-PR). Setting it to false logs a warning and continues, giving deployments an opt-out while they fix configs that now surface as errors instead of silently no-op'ing. Regression test covers both modes. Edit form now surfaces the currently-saved mode even when it is not in the filtered per-provider list, so a legacy row (e.g. content_filter saved with pre_mcp_call before this fix) no longer disappears from the dropdown; the option renders with a 'not supported by <provider>' note so the user knows to pick another. * fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint Audited every get_supported_event_hooks classmethod against the hooks each guardrail's own tests exercise and its handler methods. Five were too narrow and their tests caught it in CI: rubrik gains pre_call, presidio gains during_call and pre_mcp_call, prompt_security, onyx and qualifire gain during_call. The remaining classes match either their original __init__ declarations or their exercised modes exactly. Cursor review fixes: the Add form now drops selected modes the new provider does not support when the user switches providers, so a pre_mcp_call selection cannot ride along into a provider that rejects it at save; the edit form handles list-shaped stored modes instead of treating mode as always a string. Extracted shared toModeArray and getSupportedModesForProvider helpers into guardrail_info_helpers so both forms use one implementation, typed the remaining any usages in both forms, removed nested ternaries, and committed the ratcheted-down eslint metrics and pruned suppressions	2026-07-11 14:51:27 -07:00
mubashir1osmani	1bf98c0687	test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857 ) * test(e2e): cover Langfuse logging.yaml P0 logs_spend cells Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat traffic and assert calculatedTotalCost matches StandardLogging response_cost and proxy spend. Also assert tool calls and applied guardrails land on the trace. Missing env or proxy is a hard failure, never a skip * test(e2e): use langfuse_otel callback for Langfuse spend coverage Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to Langfuse) instead of the classic langfuse SDK. Match generations named litellm_request by prompt marker and user_api_key_alias * test(e2e): require Langfuse spend assert; drop AGENTS.md Guardrail path no longer soft-gates logs_spend. Non-stream responses must return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md * test(e2e): fail when Langfuse spend is missing on guardrail path Always run logs_spend assertions for tool_permission; require positive x-litellm-response-cost on non-stream and positive /spend/logs spend * test(e2e): do not fall back to unmatched spend log rows poll_proxy_spend_for_key returns None when response_id or positive-spend filters match nothing, instead of silently using rows[0]	2026-07-11 14:11:00 -04:00
mubashir1osmani	31c1ffc5a4	test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction (#32165 ) * fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects spend_tracking/spend_e2e_client.py imported SpendTagsResponse and TagSpend from models, but neither was ever defined, so importing the client raised ImportError and pytest aborted collection for the whole e2e session. The tag-spend tests had never run. Model /spend/tags as it actually answers: a bare array of per-tag aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field that also wouldn't match the array shape; it now reads .root, matching how spend_logs consumes its RootModel. * test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction Adds regression nets and gap-surfacing tests: A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document that reasoning_effort='none' and thinking type='disabled' are silently dropped (LIT-3686 / GH #27453) A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py): parametrized regression net asserting real completion content, not just a 200, across the configured providers for /chat/completions and /responses (GH #28991) A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is honored and prompt-cache read tokens grow on a repeated cacheable prefix A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call rate limiter runs, then asserts no unattributed spend row is left behind by the internal input-file retrieval (LIT-3266) A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct key_alias and asserts each alias gets its own labeled series on /metrics A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict) regression proving eviction must not close an httpx client still held by an in-flight caller (LIT-3221 / GH #13034) Extends tests/e2e/models.py with the typed request and response fields these tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage fields, spend-log api_key) Co-authored-by: Cursor <cursoragent@cursor.com> * test(e2e): drop unused litellm-regression-tests submodule The e2e suite migrated the regression cases into this repo; nothing imports the submodule at runtime (only a provenance comment references it), so the .gitmodules entry and gitlink pointing at a personal repo would just make upstream CI init a submodule it never uses. Remove both to keep the change test-only. * test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread cleanup vs shared client teardown) and belongs in its own PR, not this e2e coverage PR, so revert the file to its base state. --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-07-04 18:56:52 -07:00

Author

SHA1

Message

Date

yucheng-berri

69c5839cc0

fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712 )

* fix(guardrails): filter Add-Guardrail mode dropdown per provider

The GET /guardrails/ui/add_guardrail_settings endpoint returned every
GuardrailEventHooks value in one flat supported_modes list, so the Admin
UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving
Content Filter or Tool Permission with pre_mcp_call then failed with a
400 because those guardrails' server-side supported_event_hooks list
excludes it.

Expose each guardrail's supported hooks as a get_supported_event_hooks
classmethod on CustomGuardrail (mirrors the existing get_config_model
pattern) and have the endpoint iterate guardrail_class_registry to build
a supported_modes_by_provider map. The UI Mode dropdown filters by that
map when the selected provider is known and falls back to the global
list otherwise. __init__ now sources its own supported_event_hooks list
from the classmethod so the two sides can't drift.

Also register BedrockGuardrail, ToolPermissionGuardrail, lakera,
lakera_v2, and presidio in guardrail_class_registry so they participate
in the map (they were previously only in guardrail_initializer_registry
and had no class-registry entry).

Behavior change: guardrails that previously had no supported_event_hooks
declared (aim, javelin, azure/text_moderation, cato_networks,
crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx,
prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai,
lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now
validate the configured mode at instantiation. Existing configs where
the mode was silently a no-op will fail at proxy startup with a clear
validation error rather than running as a broken guardrail.

Resolves LIT-4226

* fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form

Address Greptile P1 (startup break) and P2 (edit form UX):

LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported
event_hook, unchanged behavior for the guardrails validated pre-PR).
Setting it to false logs a warning and continues, giving deployments an
opt-out while they fix configs that now surface as errors instead of
silently no-op'ing. Regression test covers both modes.

Edit form now surfaces the currently-saved mode even when it is not in
the filtered per-provider list, so a legacy row (e.g. content_filter
saved with pre_mcp_call before this fix) no longer disappears from the
dropdown; the option renders with a 'not supported by <provider>' note
so the user knows to pick another.

* fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint

Audited every get_supported_event_hooks classmethod against the hooks
each guardrail's own tests exercise and its handler methods. Five were
too narrow and their tests caught it in CI: rubrik gains pre_call,
presidio gains during_call and pre_mcp_call, prompt_security, onyx and
qualifire gain during_call. The remaining classes match either their
original __init__ declarations or their exercised modes exactly.

Cursor review fixes: the Add form now drops selected modes the new
provider does not support when the user switches providers, so a
pre_mcp_call selection cannot ride along into a provider that rejects
it at save; the edit form handles list-shaped stored modes instead of
treating mode as always a string.

Extracted shared toModeArray and getSupportedModesForProvider helpers
into guardrail_info_helpers so both forms use one implementation, typed
the remaining any usages in both forms, removed nested ternaries, and
committed the ratcheted-down eslint metrics and pruned suppressions

2026-07-11 14:51:27 -07:00

mubashir1osmani

1bf98c0687

test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857 )

* test(e2e): cover Langfuse logging.yaml P0 logs_spend cells

Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat
traffic and assert calculatedTotalCost matches StandardLogging response_cost
and proxy spend. Also assert tool calls and applied guardrails land on the
trace. Missing env or proxy is a hard failure, never a skip

* test(e2e): use langfuse_otel callback for Langfuse spend coverage

Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to
Langfuse) instead of the classic langfuse SDK. Match generations named
litellm_request by prompt marker and user_api_key_alias

* test(e2e): require Langfuse spend assert; drop AGENTS.md

Guardrail path no longer soft-gates logs_spend. Non-stream responses must
return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md

* test(e2e): fail when Langfuse spend is missing on guardrail path

Always run logs_spend assertions for tool_permission; require positive
x-litellm-response-cost on non-stream and positive /spend/logs spend

* test(e2e): do not fall back to unmatched spend log rows

poll_proxy_spend_for_key returns None when response_id or positive-spend
filters match nothing, instead of silently using rows[0]

2026-07-11 14:11:00 -04:00

mubashir1osmani

31c1ffc5a4

test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction (#32165 )

* fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects

spend_tracking/spend_e2e_client.py imported SpendTagsResponse and
TagSpend from models, but neither was ever defined, so importing the
client raised ImportError and pytest aborted collection for the whole
e2e session. The tag-spend tests had never run.

Model /spend/tags as it actually answers: a bare array of per-tag
aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the
existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field
that also wouldn't match the array shape; it now reads .root, matching
how spend_logs consumes its RootModel.

* test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction

Adds regression nets and gap-surfacing tests:

A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the
DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document
that reasoning_effort='none' and thinking type='disabled' are silently dropped
(LIT-3686 / GH #27453)

A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py):
parametrized regression net asserting real completion content, not just a 200,
across the configured providers for /chat/completions and /responses (GH #28991)

A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is
honored and prompt-cache read tokens grow on a repeated cacheable prefix

A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call
rate limiter runs, then asserts no unattributed spend row is left behind by the
internal input-file retrieval (LIT-3266)

A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct
key_alias and asserts each alias gets its own labeled series on /metrics

A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict)
regression proving eviction must not close an httpx client still held by an
in-flight caller (LIT-3221 / GH #13034)

Extends tests/e2e/models.py with the typed request and response fields these
tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage
fields, spend-log api_key)

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(e2e): drop unused litellm-regression-tests submodule

The e2e suite migrated the regression cases into this repo; nothing
imports the submodule at runtime (only a provenance comment references
it), so the .gitmodules entry and gitlink pointing at a personal repo
would just make upstream CI init a submodule it never uses. Remove both
to keep the change test-only.

* test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage

The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client
close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread
cleanup vs shared client teardown) and belongs in its own PR, not this e2e
coverage PR, so revert the file to its base state.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

2026-07-04 18:56:52 -07:00

3 commits