The sidebar toggle sits absolutely positioned over the drawer's flex row. With
the sidebar expanded it lands on the sidebar header, but once collapsed it lands
on the drawer header, which is sticky at z-chrome (10). The named-z-scale
refactor moved the toggle from z-20 to z-raised (1), so from then on the header
painted over it and swallowed the click: collapse the trace list and there was
no way to bring it back.
Moves the toggle to z-floating (30) and folds the two mirrored buttons into one,
since they only ever differed by icon, label and handler.
Covered by a Playwright spec, which is the tier that can see the layering: the
button stays visible and enabled either way, so the pre-fix failure is a click
interception that jsdom cannot reproduce.
* feat(mcp): support RS256 signing for MCP gateway session tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format session token modules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): enforce key strength on rotated public keys and unique kids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): honor allow_requests_on_db_unavailable in readiness probe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): bound readiness DB check and pass reconnect timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): bound whole readiness DB check with one deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): keep readiness deadline fallback within lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(health): suppress TQ008 for proxy-global readiness patches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): release reconnect lock when a waiting reconnect is cancelled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
* test: add logging e2e coverage (s3_v2, gcs_bucket, team langfuse callback, datadog failure)
Five new live e2e scenarios raising Logging & Guardrails registry coverage:
s3_v2 success and failure objects read back from the real S3 bucket,
gcs_bucket success record read back through the GCS JSON API (with
nextPageToken pagination and per-request bearer minting), team-scoped
Langfuse callback delivery with non-team isolation, and DataDog failure
event delivery queried by indexed model_group. datadog_reader gains
query-based variants of the marker search; the langfuse cell is a new
registry row. Bucket readers settle past a full flush interval so a
late duplicate cannot hide from the exactly-one assertions
* test: cover clock-skew day prefix in gcs read-back and retry team callback propagation
* test: key the s3 failure read-back on the provider error, not payload absence
* chore: rerun ci
* chore: rerun ci after config sync
* chore: rerun ci with pr lane env
* chore: rerun ci
* chore: rerun ci
* chore: rerun ci
* chore: rerun ci
* chore: rerun ci
* test: add guardrail e2e coverage (presidio masking, bedrock post and during call, moderation on messages) (#38553)
* test: add guardrail e2e coverage (presidio masking, bedrock post/during, moderation on messages)
* test: require the phone placeholder positively in the presidio masking predicate
* test: count only the 400 verdict body as a bedrock post_call block
* test(e2e): exempt the guardrail config echo from the post_call leak assertion
* test(e2e): pin the fail-closed contract for an unknown guardrail name (skipped, product gap)
* test(e2e): tolerate the readiness 503 from a transient db blip in the callback-config probes
PR #38593 stopped forwarding temperature to reasoning models, which left
test_extra_body_merges_with_request_data raising UnsupportedParamsError
and test_bad_request_bad_param_error no longer getting a rejection from
OpenAI because drop_params now eats the param. Both repairs are the same
hunks PR #38739 carries, so the branches merge clean in either order
Adds litellm/litellm_core_utils/aws_partition.py mapping a region to its
AWS partition (aws, aws-cn, aws-us-gov, and the iso partitions), its DNS
suffix, and its ARN prefix, and uses it at every AWS host and ARN build
site: bedrock (runtime, agent, agentcore, legacy client, batches, files,
realtime), sagemaker, polly, secrets manager, s3 log uploads, bedrock
passthrough routes, and rag ingestion. ARN detection now accepts
arn:aws-cn: and arn:aws-us-gov: prefixes.
STS region resolution now falls back to the configured aws_region_name
after the aws_sts_endpoint host and the AWS_REGION/AWS_DEFAULT_REGION env
vars, so cn and gov role assumption no longer silently signs against
us-west-2.
A partition sweep test walks every endpoint builder with cn regions and
asserts no amazonaws.com host or arn:aws: prefix comes out, plus an AST
guard that fails on any new f-string hardcoding either literal.
* feat(terraform): add litellm_jwt_key_mapping resource
Adds a Terraform resource for the proxy's JWT to virtual key mappings, so a
JWT client identified by a claim such as client_id, azp or sub maps to a
virtual key and inherits its models, budgets, rate limits and spend tracking.
Covers the four mapping endpoints: /jwt/key/mapping/new, /info, /update and
/delete. is_active is applied through a follow-up update because the create
endpoint always starts a mapping active, a dropped description is sent as an
empty string because the update endpoint ignores absent fields, changing the
mapped key rotates it in place, and changing the claim name or value forces
replacement since the update endpoint cannot change them.
* fix(terraform): revert key on failed jwt_key_mapping update
Classic SDKv2 persists a failed Update's diff-applied values to state
regardless of the error, so a rejected key rotation left the new key in
state while the proxy kept the old one and the next plan falsely converged.
Revert key via GetChange and resync description/is_active/computed fields
from a post-failure Read, since Read alone can't recover key (the proxy
never returns it).
Also drop the case-insensitive "mapping not found" body match: the proxy
raises 404 for all three not-found paths (info, update, delete), so
checking the status code alone is sufficient.
Clarify the docs: referencing a litellm_key resource's write-only key is
not a null-then-400 situation, it's a static "Missing required argument"
error at plan time, in every apply ordering.
* fix(terraform): stop leaving an active mapping behind on failed cleanup
Two issues flagged by review:
- Create has no way to ask the proxy for an inactive mapping, so an
is_active=false mapping is briefly active while the follow-up
deactivation runs. If that deactivation call itself fails, the mapping
used to stay active and untracked. It's now deleted instead, closing
the exposure rather than leaving it open indefinitely.
- On a failed update, only `key` was reverted before the recovery read.
If that read also failed, description/is_active kept the rejected
values, so a later plan could report false convergence. Now all three
are reverted before the read runs.
Both come with regression tests, mutation-verified against the pre-fix
code.
* fix(deps): bump restrictedpython to 8.5 for GHSA-ffg3-p8fm-mjx2
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: retrigger ci
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): stub anthropic judge credentials in funnel seeding test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(deps): keep uv.lock unchanged to keep the PR terraform-only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Fabrice Pont <fabrice.pont@doctolib.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The attempt row now prices the real arm (the payload's response_cost plus its own
routing classifier when it routed) beside the shadow arm (completion plus the
classifier cost the routing decision writes back), and flags turns litellm's
response cache served. A per-leg funnel table counts the eligible requests that
produced no row (lost the sampling dice, unjudgeable shape, concurrency shed),
so results can weigh judged rows against the traffic they stand for. Job results
gain per-slice and overall arm spends plus the coverage counts, the budget gates
charge the shadow arm's classifier spend against max_budget, and the dashboard
shows the measured cost comparison beside the win rate
Resolves LIT-6358
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(proxy): reset a key's budget-window counters and broadcast the reset cross-pod
/key/{id}/reset_spend already reset the key's lifetime spend counter in
Redis, but a key with its own budget_limits (an extra time-windowed cap,
e.g. a daily budget layered on top of the lifetime max_budget) kept its
window counter untouched, so the key stayed 429'd on
"ExceededBudget: Key over <duration> budget" even after the admin action
reported spend back to $0.
Force-expire each window on reset: zero its Redis counter and restart the
window from now (window_start is derived as reset_at - budget_duration,
so reset_at must float to now + duration, not the next calendar boundary
get_budget_reset_time gives key creation - that boundary can still be
in the past relative to the spend that triggered the block).
Also close a second, narrower race: _delete_cache_key_object evicted the
cached key object only on the handling pod, so another pod could keep
serving the stale pre-reset object (and re-derive the pre-reset spend
counter via its own floor-marker cache) until its own TTL expired. It now
broadcasts the eviction, matching the pattern already used for team,
team-member, customer, and tag caches.
* fix(proxy): evict the cached key object after every reset_spend DB write
Greptile P1: eviction ran before the window-reset DB write committed, so a
request racing the reset could re-fetch and re-cache the pre-write row,
pinning that pod to the stale budget_limits for the rest of its own cache
TTL even after the write went through. Move the eviction to run last.
* test: pin real cache state and satisfy the test-quality gate
test_delete_cache_key_object_broadcasts_invalidation now asserts a real
UserApiKeyCache no longer holds the evicted entry, rather than only
inspecting a mock's call args. Suppress test-quality-ok on the
hash_token/_check_proxy_or_team_admin_for_key/_delete_cache_key_object/
publish_auth_cache_invalidation patches: none has an HTTP boundary to
fake, matching the pattern the file already uses for these same targets.
* fix(proxy): narrow budget_limits by the str branch, not the list branch
isinstance(x, list) in the else branch still leaves Sequence[object] | str
(a tuple satisfies Sequence without being a list), so json.loads() saw a
possible non-str argument. Check isinstance(x, str) instead, which narrows
each branch to exactly the type it needs.
* fix(proxy): persist advanced budget-window boundaries before zeroing counters
Greptile P1: publishing a zeroed window counter before the new reset_at
committed let a request racing the write compute window_start from the
stale boundary, re-sum the historical spend log rows the reset was
clearing, and put the counter right back above budget. Compute every
window's new boundary, persist all of them in one DB write, then zero
each window's Redis counter only once that write has landed.
An auto-router tier pin injects reasoning_effort into request_kwargs, but
provider translations give a caller-supplied thinking, output_config.effort,
or reasoning carrier precedence over the reasoning_effort alias, so the pin
never reached the wire whenever the client expressed effort natively. Drop
the client's other encodings of the setting at the tier-param merge; a
client output_config keeps its non-effort fields
Every complexity router now derives and records its savings baseline from
its hardest configured tier, and the spend writer always prices against the
decision-recorded baseline model and deployment id. A leftover
litellm_settings.autorouter_savings_baseline_model key is inert
PR #38586 changed the fallback-stamp scrub in async_function_with_fallbacks to
rebind kwargs[sibling] to a scrubbed copy instead of popping in place. Every
other router bucket write mutates the caller's dict in place, and everything
below the router resolves the metadata bucket by key presence, so on a proxy
request that carries litellm_metadata the copy becomes a detached object: the
proxy's post_call guardrail write-backs land in request_data while the spend
row is built from the router's copy. Result: guardrail_information and the
guardrail cost silently drop from the spend row on any request that planted a
reserved key, and an SDK caller aliasing one dict as both buckets loses the
router stamps entirely.
Scrub in place again, and move the anti-spoof to the proxy boundary: strip
attempted_fallbacks and original_model_group from client-supplied metadata and
litellm_metadata in add_litellm_data_to_request, next to the pricing-field
strip, so proxy traffic never carries a reserved key and the in-place pop only
ever fires for an SDK caller that planted one. Keep #38586's hop-stamp ordering
fix (caller keys first, stamps appended) untouched.
* fix(router): drop a tier param the routed target cannot take
A complexity tier's litellm_params are an operator override applied to every request that tier
routes, and they were written into the request kwargs unconditionally. When the tier set a param
the target does not declare, get_optional_params raised UnsupportedParamsError before the request
left the proxy, so the whole tier answered 400. The bundled Lite preset sets reasoning_effort on
its complex tier, and four of the thirteen kimi-k3 map entries reject that param, so a router
built from a first-party template failed on every complex prompt
Filter the tier params at both store sites against what the group's deployments declare. The
candidate set is asked of the module that raises rather than derived from a second list, so
credentials, endpoint and transport controls are never at risk: base_url, timeout,
default_headers, organization and deployment_id are not chat completion params and never reach
that comparison. A param survives if any deployment could take it, since routing has not picked
one yet, and it survives an unresolvable provider or an empty group, since a best-effort filter
must not narrow what the request already did
The skip list _check_valid_arg applies before rejecting a param now has one owner both it and the
router read, so the two cannot drift
* fix(router): honor allowed_openai_params when gating tier params
* test(router): cover _declared_param_allowlist malformed declarations
* fix(router): never ask an authenticating provider whether it takes a tier param
Resolving github_copilot or chatgpt runs their OAuth device flow, so the
capability question _deployment_accepts_param asks would freeze the event
loop for minutes inside async_get_available_deployment. Promote
register_model's local skip set to
constants.PROVIDERS_THAT_AUTHENTICATE_ON_PROVIDER_INFO and fail open on
those providers before any lookup
* fix(utils): adopt a declared authenticating prefix instead of resolving it
The tier-param guard alone was not enough: the savings baseline and the
model-info funnels also resolve deployments during routing, and each
resolution of github_copilot or chatgpt runs their OAuth device flow.
declared_authenticating_provider gives every metadata funnel
(get_supported_openai_params, _get_potential_model_names,
_supports_factory, canonical_model) the resolver's answer by string, so
the whole routing path answers without authenticating. A through-test
drives async_get_available_deployment with a copilot deployment and
records that no copilot resolution happens
* feat(guardrails): honor Lakera v2 skip-message flags and add advisory (inject_system_message) mode
Squashed rebase of bugfix/lakera-v2-skip-system-tool-messages onto latest
litellm_internal_staging (900+ commits ahead; a commit-by-commit rebase hit
repeated conflicts against the same files across earlier review-round
commits, so the branch's cumulative diff was reapplied in one pass instead).
Adds skip_system_message_in_guardrail/skip_tool_message_in_guardrail support
to Lakera v2, a third on_flagged: "inject_system_message" advisory mode, and
the associated masking-safety-guard hardening (multimodal content, non-
maskable message fields, combined messages+input, and structured Responses-
API input in advisory delivery) found across this PR's review rounds.
* fix(guardrails): don't let one invalid guardrail config crash proxy boot
init_guardrails_v2 had no try/except around initialize_guardrail, so a
guardrail whose litellm_params fail validation at construction time (for
example Lakera's on_flagged=inject_system_message combined with
mode=during_call, or a malformed advisory_system_message template) raised
uncaught and crashed the entire proxy at startup, taking down every other,
correctly-configured guardrail in the list. Catch ValueError/TypeError per
guardrail, log a warning, and skip it, matching the same pattern already
used for the DB-driven guardrail-creation path in guardrail_endpoints.py.
* fix(guardrails): preserve message fields and mask PII before advising in Lakera v2
Mask-in-place degraded to a hard block for any message carrying a field
beyond role/content (tool_call_id, tool_calls, name, cache_control), for a
message excluded by skip_system_message_in_guardrail/skip_tool_message_in_guardrail,
or for a message with no inspectable text, since it rewrote data["messages"]
wholesale from a synthetic role/content-only list built for the Lakera API
call. That made masking effectively unusable for any real tool-calling
conversation and made the skip flags flip every PII-only violation to a hard
block instead of masking just the in-scope text.
Replace the wholesale rewrite with a scope-index merge, reusing the same
merge_guardrailed_scoped_messages helper the OpenAI/Anthropic guardrail
translation handlers already use for this: patch content in place on a copy
of each original message actually sent to Lakera, and leave every
skipped/no-text/out-of-scope message untouched at its original position.
This also fixes on_flagged="inject_system_message" (advisory mode) shipping
raw unmasked PII to the model: a PII-only violation is now masked the same
way regardless of on_flagged, and the advisory note is reserved for flags
masking can't resolve on its own.
Addresses maintainer-reported regressions on BerriAI/litellm#34940.
* fix(guardrails): satisfy new lint gates for the masking/advisory fix
Parameterize the write-back helper's dict param and suppress the two new
lint rules that landed on the base while this branch was in flight: TQ008
(patching an internal collaborator) for two pre-existing tests unrelated to
this change, and LIT001 for a param that genuinely needs to mutate the
caller's request dict in place.
* fix(guardrails): normalize role casing in Lakera v2 masking scope, log skipped guardrails louder
Greptile finding: the masking scope helper compared roles case-sensitively
while filter_messages_by_skip_flags (used to build what's actually sent to
Lakera) normalizes casing, so an uppercase-cased "System"/"TOOL" role
survived the scope filter but was excluded from the inspected list. The
resulting length mismatch raised inside the strict positional zip, turning
a maskable PII-only violation into an unhandled request failure. Lowercase
the role comparison to match.
Also, per veria-ai's finding that a skipped invalid guardrail now fails
open: log it at error level with an explicit note that the proxy is
starting without that guardrail, so it's not mistaken for routine info.
* fix(guardrails): mask maskable PII in mixed violations before advising in Lakera v2
on_flagged="inject_system_message" only masked when a violation was
PII-only; a mixed violation (PII plus a non-PII flag like prompt injection)
fell straight through to the advisory branch with the raw PII still in
place, in both async_pre_call_hook and async_moderation_hook. Mask whatever
Lakera returned location data for before appending or logging the advisory,
so a mixed violation never ships raw PII just because something else was
also flagged.
Also degrade to blocking, same as block mode already does, when nothing
can be safely masked at all (multimodal content, or messages combined with
a Responses API input field) instead of showing an advisory note next to
raw, unredacted content.
Widened call_v2_guard/_mask_pii_in_messages/the write-back helper's message
parameters from list to Sequence to match what's actually passed through
from _filter_skipped_messages, instead of duplicating list(...) casts at
every call site.
* fix(guardrails): don't hard-block advisory mode for non-PII flags on non-maskable input
Bugbot finding: gating the entire inject_system_message branch on
is_multimodal_input hard-blocked every flagged request on Responses
instructions, combined messages+input, or multimodal content, including
a prompt-injection-only violation with no PII at all. Masking safety only
matters when there's actual PII to mask; a violation with no PII needs no
masking, so the advisory should still be delivered normally.
Only degrade to blocking when the breakdown actually contains a PII
detection and masking isn't safely possible. Otherwise, mask whatever's
maskable (if any) and deliver the advisory as before.
* fix(guardrails): require payload and breakdown for Lakera v2 advisory mode
Advisory mode's mixed-violation masking safety net can only redact
detected PII when Lakera's response carries both the breakdown (to
detect a PII hit at all) and payload (the location data to mask by).
payload=False or breakdown=False alongside on_flagged='inject_system_message'
silently forwarded raw PII next to the advisory note. Reject that
combination at construction and hot-reload time instead.
* fix(guardrails): skip_system_message_in_guardrail must not force-block Lakera masking
_has_responses_instructions treated any non-empty data["instructions"]
as unsafe to mask regardless of skip_system_message_in_guardrail, even
though that flag excludes the instructions-derived synthetic system
message from what Lakera ever inspects. PII detected purely in the
maskable non-system content was force-blocked instead of masked.
Also fixes pre-existing LIT010 (missing Final) violations in
_has_responses_instructions, _breakdown_has_pii_violation, and
async_post_call_success_hook that the rebase's lowered budget ceiling
now flags.
* chore: retrigger CI (GitHub Actions runner-acquisition failure on prior push)
* fix(guardrails): address maintainer review findings on Lakera v2 advisory mode
- Gate advisory_system_message template validation on on_flagged=
'inject_system_message', since block/monitor mode never reads it.
- Allow on_flagged='inject_system_message' with mode='during_call' at
construction/hot-reload instead of rejecting it; async_moderation_hook
already degrades gracefully (masks if possible, else logs a warning).
- reinitialize_guardrail now restores the previous live instance when the
new config fails to initialize, instead of leaving the guardrail deleted
entirely with nothing enforcing it.
- PATCH /guardrails/{id} rolls back the DB write and returns 422 when the
in-memory sync rejects the new config, instead of persisting a config
that never actually took effect and returning 200.
- Qualifire now rejects on_flagged values it doesn't implement (only
Lakera should accept 'inject_system_message'; LitellmParams flattens
the field across every guardrail config mixin).
* fix(tests): satisfy lint gates and update collateral test for advisory-mode fixes
- Add match= to a too-broad pytest.raises(ValueError), and suppress the
new TQ008 mocker.patch findings (same pattern already used by sibling
scenarios in this test).
- test_init_guardrails_v2_skips_invalid_guardrail_instead_of_crashing_boot
used mode='during_call' + on_flagged='inject_system_message' as its
invalid-config example; that combination is now accepted, so swap in
the payload/breakdown-missing case and add a test confirming during_call
advisory mode constructs successfully.
* docs(CLAUDE.md): auto-capture review learnings without being asked
This session found three real bugs a human maintainer caught after eight
rounds of bot review and live-proxy verification all missed them. Add a
standing instruction to write learnings.md entries the moment a root
cause is understood, in both the repo-wide file and any relevant skill's
own file, instead of relying on being asked.
* feat(guardrails): add scan_raw_request flag so YAML order can't change enforcement
Maintainer finding on BerriAI/litellm#34940: guardrails for the same hook
run sequentially over one shared, progressively-mutated request dict, so
declaring a masking guardrail before a blocking one hides the violation
from it (200 vs 400 depending purely on YAML order).
scan_raw_request opts a guardrail into always evaluating a snapshot taken
before any guardrail in the hook ran, regardless of its declared position.
Same contract as run_in_parallel: block-only, its own mutations discarded.
Verified live: real proxy, real Gemini call, two custom guardrails (a
redactor then a blocker). Same request, same declared order -- without the
flag the blocker never sees the raw secret (200); with it, the blocker
correctly rejects before any provider call (400).
* fix(guardrails): harden scan_raw_request against review findings
- Use safe_deep_copy instead of a bare deepcopy for the raw-request
snapshot; request payloads commonly carry unpicklable objects (e.g. an
otel span in metadata), which previously raised on every guarded
request when tracing was enabled (Bugbot, High).
- Only compute the snapshot when a guardrail actually opted in, and take
it before _maybe_execute_pipelines runs, so a pipeline-mutated payload
can't hide a violation from a scan_raw_request guardrail outside the
pipeline (veria-ai).
- Log a warning when a scan_raw_request guardrail returns a modified
payload, since that mutation is discarded and the combination is
otherwise silently exploitable for a masking-capable integration
misconfigured this way (veria-ai).
* chore(openapi): regenerate lazy snapshot and dashboard schema types
The lazy OpenAPI snapshot (litellm/proxy/_lazy_openapi_snapshot.json) and
the derived dashboard schema.d.ts had drifted stale relative to the
guardrail config model changes across this PR's rounds (advisory mode,
scan_raw_request, and upstream additions picked up by rebasing).
Regenerated via the CI's own documented fix:
uv run python -m litellm.proxy._lazy_openapi_snapshot
npm run gen:api (via make check)
* chore(openapi): pick up cache_hit_filter field after rebase
* fix(guardrails): stop scan_raw_request warning from firing on every call
_process_guardrail_callback always returns a dict once a guardrail runs
(mark_pre_call_hook_ran unconditionally stamps bookkeeping metadata), so
comparing the result to non-None warned on every request even when the
guardrail never touched the payload. Compare against a bookkeeping-only
baseline instead, so only an actual content mutation triggers the warning.
* fix(guardrails): make scan_raw_request snapshots independent of safe_memory_mode
safe_deep_copy can return the original object under litellm.safe_memory_mode,
or alias a per-key reference on copy failure. Under that mode, the
scan_raw_request comparison baseline aliased raw_request_snapshot (and
therefore the live request), letting mark_pre_call_hook_ran write a
premature execution marker that a deployment-level guardrail sharing the
same name would read as "already ran" and skip. Also affected the feature's
core isolation guarantee: input_data itself could alias the live request
under the same mode. Replace every scan_raw_request snapshot with
_independent_snapshot, which never returns an alias, only a genuine copy
or None.
* fix(guardrails): gate during_call mixed-violation masking behind an actual PII check
The during_call branch for a mixed violation under on_flagged=inject_system_message
unconditionally masked and reassigned data["messages"], even for a pure
prompt-injection violation with zero PII, unlike async_pre_call_hook which
already gates the same call behind _breakdown_has_pii_violation. The
unconditional reassignment touched shared request state during a hook
documented as racing with the concurrent LLM dispatch, for no reason when
there was nothing to mask.
* fix(guardrails): stop scan_raw_request from silently no-op'ing on real requests
_independent_snapshot did one whole-dict copy.deepcopy and returned None on
any failure. Every real proxy request carries data["litellm_logging_obj"]
(a Logging instance nesting a live OTel span with a real lock) by the time
pre_call_hook runs, which can never be deep-copied, so the snapshot failed
on every real request and silently fell back to the live, unisolated data
with no warning -- defeating the entire feature in production while every
existing test (none of which set litellm_logging_obj) kept passing.
Rework the helper to deep-copy each top-level key independently, falling
back to the original reference only for the specific key that fails, same
crash tolerance as safe_deep_copy's own per-key fallback. It never returns
None now; only the keys scan_raw_request actually depends on (messages/
input, metadata/litellm_metadata) need to be genuinely independent.
* fix(guardrails): block during_call when PII can't be safely masked
Greptile finding (P1, security): async_moderation_hook's inject_system_message
branch had no equivalent to async_pre_call_hook's degrade-to-blocking case for
a PII violation on input that can't be safely masked (e.g. combined
messages+input). It fell through to the advisory no-op branch and let raw,
unredacted PII reach the model with no protection at all. Raising still
blocks the response from reaching the caller even though during_call races
with the LLM dispatch, the same mechanism on_flagged="block" already relies
on for this hook, so add the same block-instead-of-advisory branch pre_call
already has.
* chore(lint): fix LIT002 ceiling after rebase merge conflict resolution
* fix(lint): suppress genuine LIT002 hits instead of padding the ceiling
My earlier rebase conflict resolution for type-discipline-budget.json's
LIT002 limit was too low, then overcorrected by padding it well above the
actual measured count. Root-caused instead: _independent_snapshot and the
PATCH-endpoint rollback path legitimately construct plain, mutable
request-payload/config dicts (matching this file's existing precedent for
the same shape), so suppress those four sites with `# mutable-ok:` rather
than reshaping code that must stay a plain dict by contract. Set the limit
to the exact current measured total; the small remaining gap vs upstream's
own committed ceiling is pre-existing drift in litellm_internal_staging
itself (its own tree already measures over its committed limit), not
attributable to this PR.
* fix(guardrails): stamp live request when a scan_raw_request guardrail runs
_run_sequential_guardrail_callback and _run_parallel_pre_call_guardrails only
called mark_pre_call_hook_ran on throwaway snapshot copies for a
scan_raw_request guardrail, never on the live request returned to the
caller. A later async_pre_call_deployment_hook (router-level guardrail
re-check) reads that marker on live kwargs to decide whether to skip
re-running the same guardrail; since it was never stamped there, the
guardrail ran a second time on live data, doubling the external call and
re-applying whatever scan_raw_request's contract says should be discarded.
* fix(guardrails): revalidate Qualifire's on_flagged on live config reload
on_flagged was validated only in __init__. The base
CustomGuardrail.update_in_memory_litellm_params is a generic setattr loop
with no revalidation, so a live config update (PUT /guardrails/{id}, no
restart) could setattr on_flagged="inject_system_message" onto a running
instance, bypassing the constructor's rejection -- silently blocking every
flagged request under an "advisory" label. Mirrors LakeraAIGuardrail's own
update_in_memory_litellm_params override added earlier in this PR.
* fix(guardrails): honor scan_raw_request for pipeline-managed guardrails
A scan_raw_request=True guardrail that is itself a pipeline step never saw
raw_request_snapshot: PipelineExecutor.execute_steps had no way to receive
it, and pipeline-managed guardrails are fully excluded from the normal
sequential/parallel loops that implement the flag. Such a guardrail silently
evaluated whatever an earlier pass_data step in the same pipeline had
already rewritten, defeating the flag for pipeline-managed guardrails.
Moves the snapshot helper (renamed independent_snapshot) from proxy/utils.py
to litellm_core_utils/core_helpers.py so pipeline_executor.py can use the
same independent-copy logic without a circular import, threads
raw_request_snapshot through _maybe_execute_pipelines and
PipelineExecutor.execute_steps/_run_step, and discards a scan_raw_request
step's returned data the same way the sequential/parallel loops already do.
* chore(openapi): pick up upstream drift after rebase onto litellm_internal_staging
* fix(guardrails): stop attempting PII masking during during_call in Lakera v2
Greptile finding (P1, security): during_call runs concurrently with the LLM
dispatch. In the common path, the provider call already binds its messages
kwarg before this guardrail's coroutine gets a chance to run, let alone
before its own network round trip to Lakera completes -- masking here can
never reliably reach the outgoing request, and _apply_redacted_messages_back_
preserving_fields reassigns to a new list object rather than mutating in
place, so even winning the race wouldn't help. This affected both the
PII-only and mixed-violation masking branches, all added in this same PR.
Remove masking from async_moderation_hook entirely and let PII violations
fall through to the normal on_flagged branching: block under "block" or
"inject_system_message" (extending the existing multimodal-only block to
cover every PII case, since masking is proven non-functional regardless of
input shape), log-and-allow under "monitor" -- consistent with how every
other violation type in this hook is already handled.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
