A post_call pipeline guardrail that rewrites a streamed tool call (its
arguments or its name) now has that rewrite written back across the buffered
chunks on chat, Responses, and Messages streams, so the client receives the
rewritten tool call instead of the original. The chat handler rewrites the
first fragment of each tool-call index and blanks the rest, the Responses
handler syncs the function_call output items and their argument events, and
the Messages handler rewrites the tool_use content_block_start and
input_json_delta events in both dict and SSE-bytes chunks.
The delivers_ended_stream_text_rewrites flag becomes
delivers_ended_stream_rewrites, since the write-back now covers both text and
tool calls, and the executor only discards a tool-call rewrite on translations
without write-back or on a shape the translation refuses.
The streaming loop skipped every guardrail stepped by a post_call pipeline, even when the pipeline was dropped from the stream for lacking the unified apply_guardrail interface, so a default_on guardrail that only implements async_post_call_streaming_iterator_hook stopped governing streams it governed on the merge base. The skip set now comes from the pipelines that will gate the stream
A post_call pipeline now releases the original stream instead of refusing the
request on every shape it has no handler for: a background request, a pipeline
guardrail without the unified apply_guardrail interface, a route with no
endpoint translation, a buffered stream no translation resolves, and a rewrite
the translation cannot write back (tool-call edits, text edits on translations
without write-back, n>1 chat, an unended Anthropic stream, a Responses dump
with no event envelope). Each case logs a warning naming the policy and
guardrail. Real blocks and writable text masks are unchanged.
* feat(proxy): resolve root_path per request from SERVER_ROOT_PATHS
One deployment can encode exactly one client-visible URL path prefix
today: SERVER_ROOT_PATH is a scalar stamped onto the app at startup, so
a pod fronting several ingress prefixes 404s every prefix but one before
any handler runs, and MCP OAuth discovery can emit only one prefix's
URLs (RFC 9728 section 3 exact-match fails for the rest).
Add an opt-in outermost ASGI middleware that matches the request path
against a configured prefix list (SERVER_ROOT_PATHS, comma-separated) on
a segment boundary and sets scope["root_path"] for that request only.
Everything downstream is stock Starlette: route matching strips
root_path so routes stay registered root-relative, and request.base_url
re-includes it, so the discovery documents' resource and the 401
challenges' resource_metadata land under the prefix the client actually
called — with no discovery-builder changes.
LazyFeatureMiddleware now strips the scope root_path (falling back to
the cached SERVER_ROOT_PATH scalar) before feature prefix matching, so
lazily-registered routers — the MCP OAuth discovery router among them —
load under per-request prefixes.
Follow-up to the routing discussion on #35226; composes with, but does
not depend on, #35576.
* fix(proxy): import Sequence from collections.abc (ruff UP035 strict-budget gate)
* review(greptile): trim implementation commentary; fixture-own MCP registry state in tests
Addresses both P2s from the first Greptile pass:
- per_request_root_path_middleware.py (and the related _lazy_features /
proxy_server comments) cut down to the constraints the code cannot
express, per repo comment guidance
- the new discovery tests no longer clear/repopulate the shared MCP
registry inline; a fixture snapshots it, hands the test an empty
registry, and restores it afterwards so no state leaks between cases
* fix(lint): mutable-ok marker on the prefix accumulator (LIT002 type-discipline gate)
* fix(proxy): tie 401 challenges and get_custom_url to the per-request root_path
The per-request root_path middleware sets scope["root_path"] to the
prefix the client actually called, but the OAuth 401 challenges
(raise_user_oauth_challenge / raise_token_exchange_challenge) still
built their resource_metadata from SERVER_ROOT_PATH. On a pod fronting
several prefixes, the challenge advertised a discovery URL under a
different prefix than the discovery document served — the two
disagreed on where the resource metadata lives, and a strict RFC 9728
client refused the challenge. Route the challenges through a small
ContextVar the middleware populates so they read the same effective
root_path Starlette resolves the request under.
The same accessor fixes get_custom_url: when a request lives under a
SERVER_ROOT_PATHS-matched prefix, request.base_url already carries it,
so appending the SERVER_ROOT_PATH scalar on top produced e.g.
/tenant-a/legacy/sso/callback — a path that does not exist. Reading
the per-request prefix instead (and relying on join_paths's tail-dedup)
keeps SSO login/callback URLs under one prefix — the one the request
actually arrived on.
Fallback: outside a request (module-load-time UI URL builders,
background tasks) the ContextVar is unset and the accessor reads
SERVER_ROOT_PATH, matching get_server_root_path() so scalar-only
deployments are byte-identical.
* fix(mcp): challenge URL under per-request prefix must route, and mock parity
Two follow-ups to the review fix that made the 401 challenge use the
per-request root_path:
1. oauth_protected_resource_path must pick the URL structure that
actually routes for the mechanism in use:
- The scalar SERVER_ROOT_PATH deployment registers the well-known
routes with the prefix INSERTED (via well_known_root_suffix at
import time), matching RFC 8414 §3. The challenge URL must use the
same insertion or a client fetching it 404s.
- The per-request SERVER_ROOT_PATHS deployment can't register routes
per prefix; PerRequestRootPathMiddleware strips the prefix from
scope["path"] and the router matches the un-inserted route. The
URL must place the prefix BEFORE .well-known so the strip leaves a
matching path.
The previous fix used the insertion form for both, which 404'd the
discovery fetch on the per-request path — the discovery doc and the
challenge would then disagree on where the resource metadata lives,
the very failure the review flagged. End-to-end verified: the URL
the challenge advertises routes and the doc's `resource` field
equals the URL the client originally called (RFC 9728 §3).
2. get_request_root_path now delegates its fallback through
get_server_root_path() instead of reading the env directly, so every
existing `monkeypatch.setattr("litellm.proxy.utils.get_server_root_path"`
test override keeps working. This unstubbed the mock on the /v2/login
test that failed on the last CI run.
Plus the lint budget: annotate the local accumulator Final, tag the
scope["root_path"] rewrite as an intentional ASGI-contract mutation,
tag the reused `path`/`root_path` rebinds in LazyFeatureMiddleware, and
add reason strings to the two new PLC0415 lazy-import noqas.
* test(mcp): pin the reviewer's expected end-state — challenge URL routes, resource matches called URL
End-to-end regression test that mounts the discoverable router + the
per-request root_path middleware, hits an MCP endpoint that raises
raise_user_oauth_challenge, fetches the resource_metadata URL the
challenge advertises, and checks the returned document's `resource`
equals the URL the client originally called (RFC 9728 §3 exact match).
Covers /tenant-a, /tenant-b, and the unprefixed path on the same app so
a regression on any prefix — challenge URL 404s, or doc emits a
different prefix than the client called — fails at this test rather
than in a strict MCP client's discovery.
---------
Co-authored-by: gym-cmd <186399764+gym-cmd@users.noreply.github.com>
* fix(proxy): retry deadlocks and requeue spend logs on any DB write error
update_spend_logs dequeued the batch and only retried/requeued on transport
errors. A 40P01 deadlock surfaced as a plain prisma DataError and went through
poison-row isolation, which dropped every row it hit; every other DB error was
re-raised with the batch already gone from the queue.
Treat deadlocks as transient (retry, then requeue), keep them out of poison-row
isolation, and requeue the batch at the head of the queue on any other prisma
error so it lands once the DB is healthy.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop redundant docstrings and tighten test typing for spend-log requeue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert deadlock retries from mock call history instead of mutable lists
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): deliver budget alerts on webhook-only alerting and accept ALERTING_WEBHOOK_URL
ProxyLogging.budget_alerts forwarded to the alerting pipeline only when
'slack' was in general_settings.alerting, so alerting: ['webhook'] plus
WEBHOOK_URL silently never delivered a budget alert (the config
/health/services?service=webhook exists to test). Forward when 'webhook'
is present too; SlackAlerting.send_alert already fans out per channel.
Also accept a provider-neutral ALERTING_WEBHOOK_URL env fallback for the
Slack-format channel (any Slack-compatible receiver works), mark it as a
sensitive var, and de-brand the admin UI alerting copy.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): format settings.tsx with prettier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for updated alerting description
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger checks after ALERTING_WEBHOOK_URL docs merged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Buffered streams governed by post_call policy pipelines now deliver text
rewrites back into the stream per surface (chat SSE, responses SSE,
anthropic messages SSE) instead of rejecting the request with a 400
upfront. Rewrites chain across pipeline steps; tool-call rewrites and
translations without stream write-back still withhold the stream.
* fix(proxy): run SMTP send_email off the event loop with a connection timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): format utils.py and update _create_smtp_connection tests for timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep malformed SMTP_TIMEOUT inside the email error boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: retrigger ci
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: exclude misaligned circleci coverage flag from merged codecov report
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: retrigger ci for codecov and benchmarks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: disable carryforward for the circleci codecov flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: exclude carried-forward coverage from the codecov patch status
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: stop carrying forward the dead circleci codecov flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A litellm_content_filter step with a MASK action masks chat streams through
its own iterator hook under guardrails.add, which pipeline-managed guardrails
skip, so the pipeline path released the stream unmasked. CustomGuardrail now
declares rewrites_streamed_output (mask_response_content by default, any MASK
action for the content filter) and the upfront streaming check names such
steps in the same 400 it gives mask_response_content and incremental_diff
Adds regression tests for the modify_response block on the Anthropic route,
the gate with no iterator overrides, and the per-chunk hook skipping
pipeline-managed guardrails. Corrects the gate docstring: an allow releases
the chunks as the endpoint translation left them, not verbatim
* fix(health): honor allow_requests_on_db_unavailable in readiness probe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): bound readiness DB check and pass reconnect timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): bound whole readiness DB check with one deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): keep readiness deadline fallback within lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(health): suppress TQ008 for proxy-global readiness patches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): release reconnect lock when a waiting reconnect is cancelled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Same-day debt cleanup on code that landed in the last 24 hours. No behavior change.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Post_call pipelines run step hooks against a copied request dict, so guardrail
writes into the metadata bucket (applied_guardrails for the response header,
standard_logging_guardrail_information for spend logs) were dropped when the
guardrail was the first writer. Merge those writes back onto the request on the
post_call allow path, keeping the request payload and the executor's per-step
guardrails activation flag out of it.
Background /v1/responses requests dodge the streaming 400: pre_call sees stream
unset, then the polling task forces stream=true with pre-call logic skipped and
the streaming branch returns before post_call_success_hook, silently bypassing
post_call pipelines. Reject background=true at pre_call the same way as
stream=true.
Also pin the run_in_parallel pipeline-managed exclusion in both hook loops with
regression tests.
* feat(guardrails): honor Lakera v2 skip-message flags and add advisory (inject_system_message) mode
Squashed rebase of bugfix/lakera-v2-skip-system-tool-messages onto latest
litellm_internal_staging (900+ commits ahead; a commit-by-commit rebase hit
repeated conflicts against the same files across earlier review-round
commits, so the branch's cumulative diff was reapplied in one pass instead).
Adds skip_system_message_in_guardrail/skip_tool_message_in_guardrail support
to Lakera v2, a third on_flagged: "inject_system_message" advisory mode, and
the associated masking-safety-guard hardening (multimodal content, non-
maskable message fields, combined messages+input, and structured Responses-
API input in advisory delivery) found across this PR's review rounds.
* fix(guardrails): don't let one invalid guardrail config crash proxy boot
init_guardrails_v2 had no try/except around initialize_guardrail, so a
guardrail whose litellm_params fail validation at construction time (for
example Lakera's on_flagged=inject_system_message combined with
mode=during_call, or a malformed advisory_system_message template) raised
uncaught and crashed the entire proxy at startup, taking down every other,
correctly-configured guardrail in the list. Catch ValueError/TypeError per
guardrail, log a warning, and skip it, matching the same pattern already
used for the DB-driven guardrail-creation path in guardrail_endpoints.py.
* fix(guardrails): preserve message fields and mask PII before advising in Lakera v2
Mask-in-place degraded to a hard block for any message carrying a field
beyond role/content (tool_call_id, tool_calls, name, cache_control), for a
message excluded by skip_system_message_in_guardrail/skip_tool_message_in_guardrail,
or for a message with no inspectable text, since it rewrote data["messages"]
wholesale from a synthetic role/content-only list built for the Lakera API
call. That made masking effectively unusable for any real tool-calling
conversation and made the skip flags flip every PII-only violation to a hard
block instead of masking just the in-scope text.
Replace the wholesale rewrite with a scope-index merge, reusing the same
merge_guardrailed_scoped_messages helper the OpenAI/Anthropic guardrail
translation handlers already use for this: patch content in place on a copy
of each original message actually sent to Lakera, and leave every
skipped/no-text/out-of-scope message untouched at its original position.
This also fixes on_flagged="inject_system_message" (advisory mode) shipping
raw unmasked PII to the model: a PII-only violation is now masked the same
way regardless of on_flagged, and the advisory note is reserved for flags
masking can't resolve on its own.
Addresses maintainer-reported regressions on BerriAI/litellm#34940.
* fix(guardrails): satisfy new lint gates for the masking/advisory fix
Parameterize the write-back helper's dict param and suppress the two new
lint rules that landed on the base while this branch was in flight: TQ008
(patching an internal collaborator) for two pre-existing tests unrelated to
this change, and LIT001 for a param that genuinely needs to mutate the
caller's request dict in place.
* fix(guardrails): normalize role casing in Lakera v2 masking scope, log skipped guardrails louder
Greptile finding: the masking scope helper compared roles case-sensitively
while filter_messages_by_skip_flags (used to build what's actually sent to
Lakera) normalizes casing, so an uppercase-cased "System"/"TOOL" role
survived the scope filter but was excluded from the inspected list. The
resulting length mismatch raised inside the strict positional zip, turning
a maskable PII-only violation into an unhandled request failure. Lowercase
the role comparison to match.
Also, per veria-ai's finding that a skipped invalid guardrail now fails
open: log it at error level with an explicit note that the proxy is
starting without that guardrail, so it's not mistaken for routine info.
* fix(guardrails): mask maskable PII in mixed violations before advising in Lakera v2
on_flagged="inject_system_message" only masked when a violation was
PII-only; a mixed violation (PII plus a non-PII flag like prompt injection)
fell straight through to the advisory branch with the raw PII still in
place, in both async_pre_call_hook and async_moderation_hook. Mask whatever
Lakera returned location data for before appending or logging the advisory,
so a mixed violation never ships raw PII just because something else was
also flagged.
Also degrade to blocking, same as block mode already does, when nothing
can be safely masked at all (multimodal content, or messages combined with
a Responses API input field) instead of showing an advisory note next to
raw, unredacted content.
Widened call_v2_guard/_mask_pii_in_messages/the write-back helper's message
parameters from list to Sequence to match what's actually passed through
from _filter_skipped_messages, instead of duplicating list(...) casts at
every call site.
* fix(guardrails): don't hard-block advisory mode for non-PII flags on non-maskable input
Bugbot finding: gating the entire inject_system_message branch on
is_multimodal_input hard-blocked every flagged request on Responses
instructions, combined messages+input, or multimodal content, including
a prompt-injection-only violation with no PII at all. Masking safety only
matters when there's actual PII to mask; a violation with no PII needs no
masking, so the advisory should still be delivered normally.
Only degrade to blocking when the breakdown actually contains a PII
detection and masking isn't safely possible. Otherwise, mask whatever's
maskable (if any) and deliver the advisory as before.
* fix(guardrails): require payload and breakdown for Lakera v2 advisory mode
Advisory mode's mixed-violation masking safety net can only redact
detected PII when Lakera's response carries both the breakdown (to
detect a PII hit at all) and payload (the location data to mask by).
payload=False or breakdown=False alongside on_flagged='inject_system_message'
silently forwarded raw PII next to the advisory note. Reject that
combination at construction and hot-reload time instead.
* fix(guardrails): skip_system_message_in_guardrail must not force-block Lakera masking
_has_responses_instructions treated any non-empty data["instructions"]
as unsafe to mask regardless of skip_system_message_in_guardrail, even
though that flag excludes the instructions-derived synthetic system
message from what Lakera ever inspects. PII detected purely in the
maskable non-system content was force-blocked instead of masked.
Also fixes pre-existing LIT010 (missing Final) violations in
_has_responses_instructions, _breakdown_has_pii_violation, and
async_post_call_success_hook that the rebase's lowered budget ceiling
now flags.
* chore: retrigger CI (GitHub Actions runner-acquisition failure on prior push)
* fix(guardrails): address maintainer review findings on Lakera v2 advisory mode
- Gate advisory_system_message template validation on on_flagged=
'inject_system_message', since block/monitor mode never reads it.
- Allow on_flagged='inject_system_message' with mode='during_call' at
construction/hot-reload instead of rejecting it; async_moderation_hook
already degrades gracefully (masks if possible, else logs a warning).
- reinitialize_guardrail now restores the previous live instance when the
new config fails to initialize, instead of leaving the guardrail deleted
entirely with nothing enforcing it.
- PATCH /guardrails/{id} rolls back the DB write and returns 422 when the
in-memory sync rejects the new config, instead of persisting a config
that never actually took effect and returning 200.
- Qualifire now rejects on_flagged values it doesn't implement (only
Lakera should accept 'inject_system_message'; LitellmParams flattens
the field across every guardrail config mixin).
* fix(tests): satisfy lint gates and update collateral test for advisory-mode fixes
- Add match= to a too-broad pytest.raises(ValueError), and suppress the
new TQ008 mocker.patch findings (same pattern already used by sibling
scenarios in this test).
- test_init_guardrails_v2_skips_invalid_guardrail_instead_of_crashing_boot
used mode='during_call' + on_flagged='inject_system_message' as its
invalid-config example; that combination is now accepted, so swap in
the payload/breakdown-missing case and add a test confirming during_call
advisory mode constructs successfully.
* docs(CLAUDE.md): auto-capture review learnings without being asked
This session found three real bugs a human maintainer caught after eight
rounds of bot review and live-proxy verification all missed them. Add a
standing instruction to write learnings.md entries the moment a root
cause is understood, in both the repo-wide file and any relevant skill's
own file, instead of relying on being asked.
* feat(guardrails): add scan_raw_request flag so YAML order can't change enforcement
Maintainer finding on BerriAI/litellm#34940: guardrails for the same hook
run sequentially over one shared, progressively-mutated request dict, so
declaring a masking guardrail before a blocking one hides the violation
from it (200 vs 400 depending purely on YAML order).
scan_raw_request opts a guardrail into always evaluating a snapshot taken
before any guardrail in the hook ran, regardless of its declared position.
Same contract as run_in_parallel: block-only, its own mutations discarded.
Verified live: real proxy, real Gemini call, two custom guardrails (a
redactor then a blocker). Same request, same declared order -- without the
flag the blocker never sees the raw secret (200); with it, the blocker
correctly rejects before any provider call (400).
* fix(guardrails): harden scan_raw_request against review findings
- Use safe_deep_copy instead of a bare deepcopy for the raw-request
snapshot; request payloads commonly carry unpicklable objects (e.g. an
otel span in metadata), which previously raised on every guarded
request when tracing was enabled (Bugbot, High).
- Only compute the snapshot when a guardrail actually opted in, and take
it before _maybe_execute_pipelines runs, so a pipeline-mutated payload
can't hide a violation from a scan_raw_request guardrail outside the
pipeline (veria-ai).
- Log a warning when a scan_raw_request guardrail returns a modified
payload, since that mutation is discarded and the combination is
otherwise silently exploitable for a masking-capable integration
misconfigured this way (veria-ai).
* chore(openapi): regenerate lazy snapshot and dashboard schema types
The lazy OpenAPI snapshot (litellm/proxy/_lazy_openapi_snapshot.json) and
the derived dashboard schema.d.ts had drifted stale relative to the
guardrail config model changes across this PR's rounds (advisory mode,
scan_raw_request, and upstream additions picked up by rebasing).
Regenerated via the CI's own documented fix:
uv run python -m litellm.proxy._lazy_openapi_snapshot
npm run gen:api (via make check)
* chore(openapi): pick up cache_hit_filter field after rebase
* fix(guardrails): stop scan_raw_request warning from firing on every call
_process_guardrail_callback always returns a dict once a guardrail runs
(mark_pre_call_hook_ran unconditionally stamps bookkeeping metadata), so
comparing the result to non-None warned on every request even when the
guardrail never touched the payload. Compare against a bookkeeping-only
baseline instead, so only an actual content mutation triggers the warning.
* fix(guardrails): make scan_raw_request snapshots independent of safe_memory_mode
safe_deep_copy can return the original object under litellm.safe_memory_mode,
or alias a per-key reference on copy failure. Under that mode, the
scan_raw_request comparison baseline aliased raw_request_snapshot (and
therefore the live request), letting mark_pre_call_hook_ran write a
premature execution marker that a deployment-level guardrail sharing the
same name would read as "already ran" and skip. Also affected the feature's
core isolation guarantee: input_data itself could alias the live request
under the same mode. Replace every scan_raw_request snapshot with
_independent_snapshot, which never returns an alias, only a genuine copy
or None.
* fix(guardrails): gate during_call mixed-violation masking behind an actual PII check
The during_call branch for a mixed violation under on_flagged=inject_system_message
unconditionally masked and reassigned data["messages"], even for a pure
prompt-injection violation with zero PII, unlike async_pre_call_hook which
already gates the same call behind _breakdown_has_pii_violation. The
unconditional reassignment touched shared request state during a hook
documented as racing with the concurrent LLM dispatch, for no reason when
there was nothing to mask.
* fix(guardrails): stop scan_raw_request from silently no-op'ing on real requests
_independent_snapshot did one whole-dict copy.deepcopy and returned None on
any failure. Every real proxy request carries data["litellm_logging_obj"]
(a Logging instance nesting a live OTel span with a real lock) by the time
pre_call_hook runs, which can never be deep-copied, so the snapshot failed
on every real request and silently fell back to the live, unisolated data
with no warning -- defeating the entire feature in production while every
existing test (none of which set litellm_logging_obj) kept passing.
Rework the helper to deep-copy each top-level key independently, falling
back to the original reference only for the specific key that fails, same
crash tolerance as safe_deep_copy's own per-key fallback. It never returns
None now; only the keys scan_raw_request actually depends on (messages/
input, metadata/litellm_metadata) need to be genuinely independent.
* fix(guardrails): block during_call when PII can't be safely masked
Greptile finding (P1, security): async_moderation_hook's inject_system_message
branch had no equivalent to async_pre_call_hook's degrade-to-blocking case for
a PII violation on input that can't be safely masked (e.g. combined
messages+input). It fell through to the advisory no-op branch and let raw,
unredacted PII reach the model with no protection at all. Raising still
blocks the response from reaching the caller even though during_call races
with the LLM dispatch, the same mechanism on_flagged="block" already relies
on for this hook, so add the same block-instead-of-advisory branch pre_call
already has.
* chore(lint): fix LIT002 ceiling after rebase merge conflict resolution
* fix(lint): suppress genuine LIT002 hits instead of padding the ceiling
My earlier rebase conflict resolution for type-discipline-budget.json's
LIT002 limit was too low, then overcorrected by padding it well above the
actual measured count. Root-caused instead: _independent_snapshot and the
PATCH-endpoint rollback path legitimately construct plain, mutable
request-payload/config dicts (matching this file's existing precedent for
the same shape), so suppress those four sites with `# mutable-ok:` rather
than reshaping code that must stay a plain dict by contract. Set the limit
to the exact current measured total; the small remaining gap vs upstream's
own committed ceiling is pre-existing drift in litellm_internal_staging
itself (its own tree already measures over its committed limit), not
attributable to this PR.
* fix(guardrails): stamp live request when a scan_raw_request guardrail runs
_run_sequential_guardrail_callback and _run_parallel_pre_call_guardrails only
called mark_pre_call_hook_ran on throwaway snapshot copies for a
scan_raw_request guardrail, never on the live request returned to the
caller. A later async_pre_call_deployment_hook (router-level guardrail
re-check) reads that marker on live kwargs to decide whether to skip
re-running the same guardrail; since it was never stamped there, the
guardrail ran a second time on live data, doubling the external call and
re-applying whatever scan_raw_request's contract says should be discarded.
* fix(guardrails): revalidate Qualifire's on_flagged on live config reload
on_flagged was validated only in __init__. The base
CustomGuardrail.update_in_memory_litellm_params is a generic setattr loop
with no revalidation, so a live config update (PUT /guardrails/{id}, no
restart) could setattr on_flagged="inject_system_message" onto a running
instance, bypassing the constructor's rejection -- silently blocking every
flagged request under an "advisory" label. Mirrors LakeraAIGuardrail's own
update_in_memory_litellm_params override added earlier in this PR.
* fix(guardrails): honor scan_raw_request for pipeline-managed guardrails
A scan_raw_request=True guardrail that is itself a pipeline step never saw
raw_request_snapshot: PipelineExecutor.execute_steps had no way to receive
it, and pipeline-managed guardrails are fully excluded from the normal
sequential/parallel loops that implement the flag. Such a guardrail silently
evaluated whatever an earlier pass_data step in the same pipeline had
already rewritten, defeating the flag for pipeline-managed guardrails.
Moves the snapshot helper (renamed independent_snapshot) from proxy/utils.py
to litellm_core_utils/core_helpers.py so pipeline_executor.py can use the
same independent-copy logic without a circular import, threads
raw_request_snapshot through _maybe_execute_pipelines and
PipelineExecutor.execute_steps/_run_step, and discards a scan_raw_request
step's returned data the same way the sequential/parallel loops already do.
* chore(openapi): pick up upstream drift after rebase onto litellm_internal_staging
* fix(guardrails): stop attempting PII masking during during_call in Lakera v2
Greptile finding (P1, security): during_call runs concurrently with the LLM
dispatch. In the common path, the provider call already binds its messages
kwarg before this guardrail's coroutine gets a chance to run, let alone
before its own network round trip to Lakera completes -- masking here can
never reliably reach the outgoing request, and _apply_redacted_messages_back_
preserving_fields reassigns to a new list object rather than mutating in
place, so even winning the race wouldn't help. This affected both the
PII-only and mixed-violation masking branches, all added in this same PR.
Remove masking from async_moderation_hook entirely and let PII violations
fall through to the normal on_flagged branching: block under "block" or
"inject_system_message" (extending the existing multimodal-only block to
cover every PII case, since masking is proven non-functional regardless of
input shape), log-and-allow under "monitor" -- consistent with how every
other violation type in this hook is already handled.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* refactor(proxy): move the shared list framework to a surface-neutral package
The list framework and its RFC 9457 problem machinery sat under
management_endpoints/management_v1/, which was the right home while
/management/v1 was its only consumer. The public surface is about to build
on the same framework, and a control-plane package is the wrong thing for a
public route to import.
Moves list_framework.py in full, plus everything in common.py except
MANAGEMENT_V1_PREFIX, to litellm/proxy/list_api/. Every importer is updated
directly instead of leaving re-export shims, so each symbol keeps exactly
one import path. ManagementProblem keeps its name: renaming it would touch
the app-wide exception handler and every call site for no behavioural gain.
The framework's own tests move alongside the code they cover. The fastapi
removed-name guard in test_common.py now globs both packages, so budgets.py
and spend_logs.py stay covered after leaving the framework's directory.
Pure move, no behaviour change: the 179 tests across both packages pass
unchanged.
* feat(proxy): add paginated GET /public/v1/model_hub
The public Model Hub page loads every public model group in one call.
Measured on a live proxy with 300 published groups, /public/model_hub
answers with 328 KB in a single response and the page renders all 300 rows
into the DOM. At a few thousand models that is multiple megabytes and a
page that stops responding, which is what a customer reported.
Adds GET /public/v1/model_hub, the first resource on the unauthenticated
/public/v1 surface. It is built on the shared list framework, so it gets
the {data, meta, links} envelope, RFC 9457 problems, strict unknown and
duplicate query parameter rejection, and sort validation without
reimplementing any of it. Sorting covers model_group, mode, the token
limits and the per-token costs, `q` searches model_group, and the filters
are the ones the page actually offers: mode and providers. Default sort is
alphabetical, which is what a browse list wants and what these rows can
support: they carry no creation timestamp.
/public/model_hub is untouched. The shipped UI still calls it and its
migration is a separate change, so this is purely additive alongside it.
Model hub rows are computed off the running router rather than read from a
table, so this adds InMemoryListExecutor: the same QueryPlan applied in
Python instead of rendered to SQL. It matches the SQL executors where it
counts, NULLS LAST in both sort directions and NULL satisfying no
comparison, so a filter means the same thing on either. The other three
public hubs have the same shape and can reuse it as is.
The fix itself is ordering. The endpoint being superseded reads every
latest health check and joins it against the whole model list, so paging
the response alone would have changed nothing. Here the health lookup is
an injected dependency the executor calls on the page slice, after the
filter and the sort, so it resolves health for the rows being served and
no others. PrismaClient gains a bounded read for that, next to the
unbounded one it mirrors. The regression test pins the ordering by
asserting which model groups the lookup is asked about, and fails against
an enrich-then-slice implementation.
* fix(proxy): address self-review findings on the public model hub list
Five adversarial review passes over the branch. What they found:
`is_null` was the one predicate in the in-memory executor that read a
repeated field's container instead of its elements, so a field holding only
nulls was indistinguishable from a populated one. It now lifts over elements
like every other predicate does. Not reachable through this endpoint, whose
only repeated field grants `contains` alone, but the executor is written to
be reused by the other three hubs and the inconsistency was a trap for them.
The fastapi removed-name guard globbed the framework packages but not
`public_endpoints/public_v1`, which `proxy_server` also imports unguarded at
module level, so the new package had none of the protection the test claims
to give. It now covers all three.
Regenerates the dashboard's API types, which the OpenAPI sync check requires
whenever the proxy's route surface moves. The diff is the 65 generated lines
for the new operation and nothing else; no dashboard code changes here.
Also trims comments and docstrings that argued for a decision or restated a
signature rather than explaining code, and wraps a docstring line that ran
past 120 characters.
* ci: run the relocated list framework tests in the proxy-endpoints shard
The framework's tests moved from tests/test_litellm/proxy/management_endpoints,
which the proxy-endpoints shard claims, into a new tests/test_litellm/proxy/list_api
that no shard named. Both coverage guards caught it: the semantic shards have no
catch-all bucket, so the directory would have run nowhere.
Claims it alongside management_endpoints, where the same tests ran before.
* docs(proxy): stop restating the list spec in the model hub route docstring
The docstring listed every sortable field, the page-size cap and the filter
set, all of which already live in MODEL_HUB_LIST_SPEC and all of which the
endpoint hands back in the allowed array of a rejected request. Two copies of
one spec is a prose update owed on every change to the real one.
Keeps what a caller cannot derive from the endpoint itself: what the resource
is, that it needs no authentication, and a working example. Regenerates the
dashboard types, which carry the docstring as the operation description.
* fix(proxy): reject a repeated sort field instead of sorting by it twice
sort took any number of comma-separated keys, and the in-memory executor runs
one full sorted() pass per key before slicing. Naming one allowed field N times
therefore bought N passes over every published model group, synchronously on the
event loop, from a route that needs no credentials. Measured on 300 groups:
0.001s for one key, 0.034s for a thousand, 0.166s for five thousand, and it
grows with the catalogue this endpoint exists to make large.
A repeated field cannot change the ordering, so rejecting repeats costs a caller
nothing and bounds the passes at len(sortable), a number the spec author picks
rather than the caller. That beats an arbitrary cap: no magic number, and the
bound holds for every resource built on the framework.
The tiebreaker is appended after parsing, so sorting explicitly by it stays legal.
Budgets renders one ORDER BY in SQL and never had the amplification, but the
check belongs with the rest of the sort validation rather than in one executor.
* fix(proxy): make the search disjunction one level deep by type
Two CI gates, one cause. AnyOf declared its clauses as Predicate, so both
consumers had to recurse to evaluate one: the SQL renderer through
_render/_render_all, and the in-memory executor through _holds. The recursion
detector flags the latter, and its reason is the same one this PR already ran
into once, a caller-controlled cost that shows up as CPU.
Nothing actually builds a nested AnyOf. _search_predicate is its only producer
anywhere in the repo and it emits Compare leaves, in every call site and every
test. Declaring clauses as tuple[Compare, ...] makes that a fact the type
checker keeps rather than a comment, and _holds then evaluates a disjunction of
leaves with no recursion at all.
Also marks the new health read's broad except, which the strict gate counts,
and covers the ordering comparison operators. The endpoint exposes only
eq/in/contains, so gt/gte/lt/lte were live code no test evaluated.
* fix(proxy): keep the new health read inside the type-discipline ceiling
The bounded health query added ten LIT002 violations, which pushed the
codebase total past its budget. The gate counts across the tree and compares
to the merge base, so a file already carrying debt does not absorb new
violations.
Returns an empty tuple rather than an empty list on the two no-result paths:
the signature already promises a Sequence, so that is a free two-violation
reduction and a better type. Builds prisma's order argument from a tuple of
pairs, which turns four literals into one. The three that remain are prisma's
own API shape and each carries its reason.
Both budget gates now pass against the merge base.
* fix(proxy): clear the two basedpyright errors the new route added
The type-check budget is over its ceiling on the base already, so the gate
blames any increase: reportArgumentType 2574/2564 and reportPrivateUsage
1815/1808, one each, both from this file.
fastapi types a route's tags as list[str | Enum], so the tuple was an argument
error; budgets.py has the same one and it is part of what put the rule over.
Passing a list is what the signature asks for, marked because an inline list
is a construction the discipline gate counts.
_get_model_group_info is private by name but is the shared reader the endpoint
this supersedes imports the same way, so the import carries a rule-scoped
ignore with that reason rather than a copy of the function.
basedpyright now reports zero errors across both new modules, and all three
budget gates pass against the merge base.
LiteLLM_PromptTable is unique on (prompt_id, version, environment) and
version numbering restarts at 1 per environment, but the in-memory
registry keyed prompts as {prompt_id}.v{version} with no environment, so
environments sharing a prompt id shadowed each other and only one
environment's template ever served.
Registry entries are now keyed {versioned_id}::{environment}, and serve
time resolution goes through resolve_prompt_spec(base_id, version,
environment): production > staging > development when no environment is
requested, latest version within the chosen environment when no version
is requested. Chat requests can pin an environment with a new optional
prompt_environment body param, filtered from provider-bound params like
prompt_id and prompt_version. The newest-updated_at dedupe in
_init_prompts_in_db is dropped since registry keys can no longer
collide, and the key-parsing serve helpers plus dead registry getters
are removed
On /v1/responses the prompt template ran inside litellm.aresponses, after the
router had already resolved a deployment and injected its api_key/api_base, so a
prompt whose metadata.model pointed at another provider sent the old
deployment's credentials cross-provider (401). The proxy now runs the prompt
template for aresponses in the pre-call hook, before routing, so the router
picks the deployment that matches the swapped model. As a backstop, the SDK
refuses a cross-provider swap when explicit credentials are already present
instead of forwarding them.
ignore_prompt_manager_model and ignore_prompt_manager_optional_params saved on
a prompt were only read by the generic manager, so dotprompt prompts ignored
them on every endpoint. PromptManagementBase now merges the prompt spec's flags
with the per-request ones for every manager, and the generic manager no longer
drops caller flags when no spec is present.
The session lookup reads spend logs straight out of the database, so a
follow-up sent seconds after the turn it chains off found nothing while the
row was still queued in the worker that served it, and the conversation was
dropped without an error. Responses calls now ask the spend-log writer to
flush on its next pass instead of waiting out its poll interval, and the
lookup gives a just-finished turn a short second chance.
Replaying a session also accepted `input` only as a string or a single dict,
so the standard list shape dropped every user turn and left the model with
assistant messages alone.
* fix(guardrails): resolve guardrail pipelines from the canonical metadata bucket
Policy-resolved pipelines are stored in litellm_metadata on routes like /v1/messages, but the pre_call reader fell back to the caller-supplied metadata field first, so a request that sends its own top-level metadata (Claude Code sends metadata.user_id) skipped every pipeline-managed guardrail.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): drive the pipeline regression through a registered guardrail
Exercise the real executor with a guardrail in litellm.callbacks instead of patching PipelineExecutor.execute_steps at class scope.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): read pipeline state from the bucket the policy engine wrote
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): type the policy pipeline state accessors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): annotate policy pipeline state casts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): scan batch records with the content hooks that are not guardrails
Guardrails were made to run on batch uploads by scanning each record through the pre-call hook
with the walk limited to guardrails. That limit exists because the same branch carries the rate
limiters and budget accounting, which must count an upload once rather than once per line. It
also excluded every enforcement hook written as a plain CustomLogger, so prompt-injection
detection, Azure content safety, banned keywords and the blocked-user check never saw a batch
record at all. Content that is a hard 400 online reached the provider verbatim through batch.
A CustomLogger now declares whether its pre-call hook judges the payload or merely counts the
request. The four that judge it opt in, the walk admits them, and both short-circuits learn
about them, including the one that decides whether the file is streamed off disk in the first
place: a proxy configured only with one of these hooks was skipping the scan entirely. Nothing
that counts a request is marked, so an upload still costs one slot and one budget check.
* refactor(proxy): drop the per-hook comment the attribute contract already states
* test(proxy): make the classification a ledger, and pin the wiring with a real hook
The classification test listed the two non-enterprise hooks by hand, so unmarking either
enterprise one changed nothing and the mutation matrix passed with both surviving. It now walks
the hook registries and fails on any pre-call CustomLogger that is on neither side, which also
gives the flag the forcing function it lacked: an enforcement hook added later would otherwise
default to off and silently skip batch records, which is the bug being fixed here.
Nothing exercised the path the bug actually lived on either, since every test raised its own
exception rather than a real hook's. One test now drives the shipped prompt-injection hook
through the scan, which pins the part no synthetic exception reaches: a chained exception reads
as a failure to judge, so refactoring any of these hooks to `raise ... from` would turn every
per-record drop into an aborted upload.
Also records why a hook that rewrites the payload for routing stays unmarked, and that only the
leaf class is consulted.
* test(proxy): set the callback list through monkeypatch rather than writing the global
The Prisma query engine is a separate process whose resident memory grows with
what it is asked to hold and glibc never returns it, so a pod's memory floor
ratchets up to its worst statement and stays there for the life of the worker.
#34956 bounded a spend-log flush by payload bytes, which caps that floor when
prompts are stored and does nothing when they are not: rows carrying only
attribution metadata run about 1.2 KB, so a 1000-row statement is roughly
1.2 MB, the 2 MB byte budget never binds, and every statement stays at 1000
rows forever.
The engine charges per row as well as per byte. Measured on a container running
the same engine build (5.4.2) against real Postgres, with rows shaped like a
store_prompts_in_spend_logs=false deployment, writing the same 200,000 rows:
rows/statement engine RSS still resident after the flush
1000 179 MB
500 91 MB
250 41 MB
100 19 MB
None of those statements came near the byte budget, so the whole difference is
row count. The floor is a plateau rather than a leak: 1,000,000 rows written at
1000 per statement settles around 229 MB and stops climbing.
Adds SPEND_LOG_WRITE_BATCH_MAX_ROWS, default 100, applied alongside the
existing byte budget so whichever binds first splits the statement. Both are
needed, since bytes are what track a prompt-carrying row and rows are what
track the engine's per-row bookkeeping.
One consequence worth naming: a flush now issues more statements, and a
statement that fails under a poison flood costs one insert before any
isolation runs, so the irreducible floor rises by the statement count. The
isolation budget still caps the amplification on top of that, and the tests
assert the bound derived from the configured row cap rather than a constant.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* fix(proxy): run pre-call guardrails on batch input file uploads
POST /v1/files with purpose=batch was the only route in files_endpoints that
never reached pre_call_hook, so guardrails did not see batch content at all and
records reached the provider unscanned.
Stream the uploaded JSONL a record at a time and run each record's body through
the existing pre_call_hook dispatch under the call type its url maps to, so
guardrail resolution, key and team config, and the per-endpoint translations are
reused rather than reimplemented. The hook gains a guardrails_only mode for this,
since the same callback loop also drives rate limiters, budget hooks, prompt
templates and hanging-request alerting, none of which should fire once per record.
A guardrail that blocks raises its own exception, which propagates untouched so
its status code survives. A record a guardrail would rewrite, a record that
cannot be parsed, and a record whose url cannot be scanned all reject the upload,
since silently skipping any of them is the bypass this is meant to close.
Per-record redaction lands separately.
The scan only runs when a guardrail that actually runs pre_call, or a guardrail
pipeline, is configured, so deployments without one are byte for byte unchanged.
* fix(proxy): compare the dict a batch guardrail returns, not the one it was given
async_pre_call_hook may return a replacement dict instead of mutating its input, and
process_pre_call_hook_response then makes that replacement the request. The scan only
inspected the dict it passed in, so a guardrail that redacts by returning a copy was
treated as a no-op and its record uploaded unchanged.
* fix(proxy): treat a missing batch body key as different from a null one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(proxy): document the guardrails_only mode on pre_call_hook
* fix(proxy): resolve a batch record's scan type from its body when the url is unfamiliar
The scanner only accepted five exact urls, but callers write that field by hand and the
provider transformers are far more permissive: bedrock treats any non-empty url as chat
and vertex strips query strings and trailing slashes. Uploads that work today would have
started failing the moment a pre-call guardrail was configured.
Normalize the url before lookup and fall back to the body shape when it is unfamiliar, so
a record we can still read is a record we still scan. Only a body with no messages, prompt
or input is now refused, and the error says so instead of listing urls that were never the
whole set.
Also pins the default side of the guardrails_only gate: the hanging-request alert and
prompt templating are asserted to still fire when the flag is absent.
* refactor(proxy): drop batch guardrail checks the upload validation already makes
check_batch_file_upload now runs first and rejects a line that does not parse, a line that
is not an object, and a line missing custom_id, method, url or body, so the guardrail scan
can rely on all four. Its own parse handling was unreachable through the endpoint and is
gone, along with the tests for it. What is left is the case that validation does not cover,
a body whose value is not an object, since it only checks that the key is present.
* fix(proxy): resolve a batch record's call type from the url path, not the whole url
A record naming its route in full, which is how callers actually write batch files, matched
no known route, so it fell through to the body shape. A Responses record carries `input`,
and that reads as an embedding, so the record was scanned as the wrong call type and any
guardrail scoped to chat or Responses skipped it while the upload was accepted. Chat records
survived only because their body shape happens to map back to the same call type. The url is
now reduced to its path before matching.
Guardrails that pick their policy from a request header, such as noma choosing an application
id, saw no headers at all during the scan and fell back to a default, so a batch record could
be evaluated under a different policy than the same content sent online. The sanitized headers
the proxy already stores in request metadata now travel with the scan.
Also drops the bare `dict` annotation, the unreachable non-dict branch on the guardrail chain's
own return, and the type alias that was missing its `TypeAlias`, which together were failing
the lint gate.
* fix(proxy): give each batch record its own copy of the scan metadata
The narrowed metadata was handed to every record as a shallow copy, so `headers` and `tags`
stayed shared with the upload request and with the other records in the same window. A guardrail
that writes into one of those in place, which several do to record their own bookkeeping, would
have its write show up in every record scanned after it and in the request itself. The narrowing
already removed the values that cannot be copied, so each record now gets a deep copy.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Azure Database for PostgreSQL Flexible Server takes a Microsoft Entra ID access
token as the connection password, and those tokens last about an hour, so a
proxy pointed at one dies shortly after boot unless something keeps minting
fresh ones
Set AZURE_POSTGRESQL_AUTH=True (or pass --azure_postgresql_auth) alongside
DATABASE_HOST, DATABASE_USER, and DATABASE_NAME, and the proxy mints a token at
startup, assembles the connection URL around it, and refreshes it in the
background for as long as the process runs. That is the same shape
IAM_TOKEN_DB_AUTH already had for AWS RDS, so the two now share one code path:
a tagged union picks the minting strategy once, and the wrapper, the read
replica, and the refresh loop all read the choice off it instead of each
guessing from the environment. Setting both toggles is a startup error, in the
chart as well as in Python
The helm chart gets database.writer.useAzureEntraAuth and the matching reader
knob next to the existing useIAMAuth
Fixes#29661
Co-authored-by: David Balatoni <balcsida@gmail.com>
