* fix(guardrails): resolve guardrail pipelines from the canonical metadata bucket
Policy-resolved pipelines are stored in litellm_metadata on routes like /v1/messages, but the pre_call reader fell back to the caller-supplied metadata field first, so a request that sends its own top-level metadata (Claude Code sends metadata.user_id) skipped every pipeline-managed guardrail.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): drive the pipeline regression through a registered guardrail
Exercise the real executor with a guardrail in litellm.callbacks instead of patching PipelineExecutor.execute_steps at class scope.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): read pipeline state from the bucket the policy engine wrote
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): type the policy pipeline state accessors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): annotate policy pipeline state casts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): scan batch records with the content hooks that are not guardrails
Guardrails were made to run on batch uploads by scanning each record through the pre-call hook
with the walk limited to guardrails. That limit exists because the same branch carries the rate
limiters and budget accounting, which must count an upload once rather than once per line. It
also excluded every enforcement hook written as a plain CustomLogger, so prompt-injection
detection, Azure content safety, banned keywords and the blocked-user check never saw a batch
record at all. Content that is a hard 400 online reached the provider verbatim through batch.
A CustomLogger now declares whether its pre-call hook judges the payload or merely counts the
request. The four that judge it opt in, the walk admits them, and both short-circuits learn
about them, including the one that decides whether the file is streamed off disk in the first
place: a proxy configured only with one of these hooks was skipping the scan entirely. Nothing
that counts a request is marked, so an upload still costs one slot and one budget check.
* refactor(proxy): drop the per-hook comment the attribute contract already states
* test(proxy): make the classification a ledger, and pin the wiring with a real hook
The classification test listed the two non-enterprise hooks by hand, so unmarking either
enterprise one changed nothing and the mutation matrix passed with both surviving. It now walks
the hook registries and fails on any pre-call CustomLogger that is on neither side, which also
gives the flag the forcing function it lacked: an enforcement hook added later would otherwise
default to off and silently skip batch records, which is the bug being fixed here.
Nothing exercised the path the bug actually lived on either, since every test raised its own
exception rather than a real hook's. One test now drives the shipped prompt-injection hook
through the scan, which pins the part no synthetic exception reaches: a chained exception reads
as a failure to judge, so refactoring any of these hooks to `raise ... from` would turn every
per-record drop into an aborted upload.
Also records why a hook that rewrites the payload for routing stays unmarked, and that only the
leaf class is consulted.
* test(proxy): set the callback list through monkeypatch rather than writing the global
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* fix(proxy): run pre-call guardrails on batch input file uploads
POST /v1/files with purpose=batch was the only route in files_endpoints that
never reached pre_call_hook, so guardrails did not see batch content at all and
records reached the provider unscanned.
Stream the uploaded JSONL a record at a time and run each record's body through
the existing pre_call_hook dispatch under the call type its url maps to, so
guardrail resolution, key and team config, and the per-endpoint translations are
reused rather than reimplemented. The hook gains a guardrails_only mode for this,
since the same callback loop also drives rate limiters, budget hooks, prompt
templates and hanging-request alerting, none of which should fire once per record.
A guardrail that blocks raises its own exception, which propagates untouched so
its status code survives. A record a guardrail would rewrite, a record that
cannot be parsed, and a record whose url cannot be scanned all reject the upload,
since silently skipping any of them is the bypass this is meant to close.
Per-record redaction lands separately.
The scan only runs when a guardrail that actually runs pre_call, or a guardrail
pipeline, is configured, so deployments without one are byte for byte unchanged.
* fix(proxy): compare the dict a batch guardrail returns, not the one it was given
async_pre_call_hook may return a replacement dict instead of mutating its input, and
process_pre_call_hook_response then makes that replacement the request. The scan only
inspected the dict it passed in, so a guardrail that redacts by returning a copy was
treated as a no-op and its record uploaded unchanged.
* fix(proxy): treat a missing batch body key as different from a null one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(proxy): document the guardrails_only mode on pre_call_hook
* fix(proxy): resolve a batch record's scan type from its body when the url is unfamiliar
The scanner only accepted five exact urls, but callers write that field by hand and the
provider transformers are far more permissive: bedrock treats any non-empty url as chat
and vertex strips query strings and trailing slashes. Uploads that work today would have
started failing the moment a pre-call guardrail was configured.
Normalize the url before lookup and fall back to the body shape when it is unfamiliar, so
a record we can still read is a record we still scan. Only a body with no messages, prompt
or input is now refused, and the error says so instead of listing urls that were never the
whole set.
Also pins the default side of the guardrails_only gate: the hanging-request alert and
prompt templating are asserted to still fire when the flag is absent.
* refactor(proxy): drop batch guardrail checks the upload validation already makes
check_batch_file_upload now runs first and rejects a line that does not parse, a line that
is not an object, and a line missing custom_id, method, url or body, so the guardrail scan
can rely on all four. Its own parse handling was unreachable through the endpoint and is
gone, along with the tests for it. What is left is the case that validation does not cover,
a body whose value is not an object, since it only checks that the key is present.
* fix(proxy): resolve a batch record's call type from the url path, not the whole url
A record naming its route in full, which is how callers actually write batch files, matched
no known route, so it fell through to the body shape. A Responses record carries `input`,
and that reads as an embedding, so the record was scanned as the wrong call type and any
guardrail scoped to chat or Responses skipped it while the upload was accepted. Chat records
survived only because their body shape happens to map back to the same call type. The url is
now reduced to its path before matching.
Guardrails that pick their policy from a request header, such as noma choosing an application
id, saw no headers at all during the scan and fell back to a default, so a batch record could
be evaluated under a different policy than the same content sent online. The sanitized headers
the proxy already stores in request metadata now travel with the scan.
Also drops the bare `dict` annotation, the unreachable non-dict branch on the guardrail chain's
own return, and the type alias that was missing its `TypeAlias`, which together were failing
the lint gate.
* fix(proxy): give each batch record its own copy of the scan metadata
The narrowed metadata was handed to every record as a shallow copy, so `headers` and `tags`
stayed shared with the upload request and with the other records in the same window. A guardrail
that writes into one of those in place, which several do to record their own bookkeeping, would
have its write show up in every record scanned after it and in the request itself. The narrowing
already removed the values that cannot be copied, so each record now gets a deep copy.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): expose client HTTP headers to logging callbacks and hooks
MCP protocol tool calls built a synthetic Request with only content-type, so metadata.headers reaching logging callbacks and guardrails was empty while /mcp-rest/tools/call exposed the full set. Rebuild the synthetic request from the connection's raw headers (shared with the sampling path), and pass sanitized headers to the pre-call hook, the MCP to LLM guardrail bridge and the Responses API MCP bridge. Credential headers stay masked and proxy key headers stripped.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): strip custom proxy key and upstream MCP credential headers from logging copies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): make client side auth header name accessor public
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): strip custom proxy key and client redaction opt-out from mcp headers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): drop custom proxy key header in the synthetic request builder
Strips general_settings.litellm_key_header_name in build_synthetic_mcp_request so every caller, including sampling, is covered, and reverts passing general_settings into add_litellm_data_to_request on the tool call path since that also switches on enforced_params.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
* fix(alerting): dedupe scheduled Slack spend reports across pods
Every pod ran its own weekly/monthly spend report jobs, prometheus
fallback stats cron, and daily report loop, so deployments with
multiple replicas or uvicorn workers received one copy per pod.
Gate each scheduled send behind the shared PodLockManager redis lock.
The lock is never released: its TTL (the full reporting window for the
weekly interval job, whose per-pod anchors drift by boot time and
jitter) doubles as a sent-this-window marker. acquire_lock returning
None (no redis wired) proceeds, preserving single-pod behavior.
Also generalize the pod lock could-not-acquire log line, which claimed
to be about spend tracking for every consumer.
Fixes#14809
* fix(alerting): harden spend report locks after adversarial review
Weekly lock TTL gets an hour haircut: with ttl equal to the interval,
the winner re-fires just before its own key expires, reacquires without
a TTL refresh, and the key then lapses in time for a trailing pod to
re-send. Job/lock ids move to litellm/constants.py per convention, and
spend_report_frequency now rejects non-positive day counts, which
previously coerced to an every-second schedule and would now compute a
negative lock TTL that silently never sends.
Adds the missing test coverage the review flagged: startup_event's
pod_lock_manager wiring (identity-asserted), the prometheus closure's
positive path, and the ungated immediate prometheus send pinned to
exactly one await.
* test(alerting): consolidate spend_report_frequency validator coverage
Drops a duplicate non-positive-days test and parametrizes the survivor
over the suffix half of the validator too
* fix(alerting): route the startup prometheus fallback send through the pod lock
Greptile caught that the boot-time send still ran once per pod when
PROMETHEUS_URL is set, the same duplication class this PR removes
* fix(alerting): make report lock acquisition non-reentrant
Greptile caught that a pod booting within an hour of the fallback stats
cron sent twice: the startup send takes the lock, then the cron fire
hits acquire_lock's reacquire branch, which returns True for the
holder. Window-marker gates now pass allow_reentrant=False so a live
lock blocks everyone including its holder; leader-election consumers
keep the reentrant default
* test(proxy): give spec'd ProxyLogging mocks a db_spend_update_writer
_initialize_slack_alerting_jobs now reads it for the pod lock manager,
and spec=ProxyLogging blocks instance-only attributes
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails
Pre-call guardrails run sequentially because each may mutate the request
payload and later guardrails depend on earlier mutations. Deployments with
several slow block-only pre_call guardrails (external moderation, Bedrock,
LLM-judge) therefore pay the sum of their latencies. during_call guardrails
run concurrently but alongside the LLM call, so a violating payload has
already been sent, which is unacceptable when the request must never reach
the model.
This adds a per-guardrail run_in_parallel flag (default off). Guardrails that
opt in are pulled out of the sequential loop and run concurrently via
asyncio.gather after every sequential (payload-mutating) guardrail has run, so
they observe the mutated payload and still form a hard barrier before the LLM
call; the first to raise blocks the request. Their returned data is discarded
since they are declared block-only.
The flag is wired from LitellmParams onto the guardrail instance at the same
generic choke point in initialize_guardrail that already sets
skip_system_message_in_guardrail, so no per-provider initializer needs to
change.
* feat(guardrails): extend run_in_parallel opt-in to post_call guardrails
post_call_success_hook ran guardrails sequentially for the same reason
pre_call did: response-modifying guardrails thread the response forward. But
block-only output scanners (which read the response and reject on violation
without changing it) serialize for no benefit and add latency.
This reuses the existing run_in_parallel flag for the post_call hook. Opted-in
post_call guardrails are pulled out of the sequential loop and run concurrently
via asyncio.gather after the sequential (response-modifying) guardrails and
before the non-guardrail CustomLogger callbacks, so they inspect the final
response and still block it from reaching the client if any raises. Their
returned response is discarded since they are block-only.
The apply_guardrail path sets data["guardrail_to_apply"] immediately before
awaiting, and unified_guardrail pops it before its first suspension point, so
concurrent guardrails never race on that key under asyncio's cooperative
scheduling.
* fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes
Addresses review feedback on the run_in_parallel opt-in.
asyncio.gather propagated the first exception without cancelling or awaiting
the siblings, so a block at t=0 left the other guardrails running as
unobserved background tasks (wasted external calls plus event-loop warnings),
and a fast SensitiveDataRouteException/ModifyResponseException could return a
reroute or passthrough before a slower block finished, letting crafted input
bypass the block. Both the pre_call and post_call parallel batches now gather
with return_exceptions=True so every guardrail runs to completion, then raise
any blocking exception ahead of a flow-changing one.
The registry choke point wrote bool(None)==False onto every instance when the
config omitted run_in_parallel, silently disabling a constructor-set default;
it now only writes when the config provides an explicit value.
* fix(guardrails): record lifecycle logs for every concurrently-run guardrail
The log_guardrail_information decorator skipped its auto-record when it saw
that the count of standard_logging_guardrail_information entries in the shared
request_data had grown during the wrapped call, taking that as proof the
wrapped function had recorded its own richer entry. That heuristic breaks the
moment guardrails run concurrently (parallel pre_call/post_call, during_call):
a sibling guardrail's append inflates the shared count, so a guardrail that did
not self-record wrongly concludes it already did and drops its own entry. The
result is that enabling run_in_parallel silently loses per-guardrail lifecycle
logs, so the Admin UI Request Lifecycle timeline and downstream loggers
(Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent
guardrails.
Replace the shared-count heuristic with a ContextVar flag set when a guardrail
records its own entry. asyncio copies the context into each gathered task, so
the flag is isolated per concurrent guardrail while still catching the
self-record-then-skip-auto-record case within a single invocation.
* test(guardrails): declare run_in_parallel on post_call guardrail mocks
The post_call partition reads run_in_parallel on every CustomGuardrail
callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is
set in __init__, not on the class) so the attribute access raised, and even
a class-level default would return a truthy child mock that wrongly routes
the double into the parallel batch. Declare the flag False on the shared
mock factories so these pre-existing hook tests exercise the sequential
path they assert on.
* fix(guardrails): harden run_in_parallel reads and address review feedback
Read run_in_parallel via getattr(..., False) in the pre_call and post_call
partitions so a third-party CustomGuardrail subclass that overrides __init__
without chaining super().__init__() no longer raises AttributeError on a path
that previously worked. Drop the redundant in-function GuardrailEventHooks
import in _run_parallel_post_call_guardrails (already imported module-level).
Remove the flaky wall-clock upper-bound assertions from the two concurrency
tests; the all-start-before-any-end overlap assertion is the timing-independent
signal that actually proves concurrency.
Replace list(set(...)) dedupe with dict.fromkeys so callback insertion
order is preserved deterministically instead of being randomized by set
iteration order (influenced by PYTHONHASHSEED). Applies to both the
Logging and ProxyLogging implementations.
Fixes#33003
Co-authored-by: Yassin Kortam <yassin@berri.ai>
When a guardrail blocked a request through a flow-builder policy pipeline, the
proxy discarded the guardrail's own exception and synthesized a generic
guardrail_pipeline_error response, so the same guardrail produced a different
HTTP response and trace span depending on whether it was attached directly or
via a policy. The pipeline now carries the guardrail's original exception and
re-raises it verbatim on block, enriching it with the blocking guardrail's name
and mode exactly as the direct path does, so the two attachment methods are
indistinguishable to clients and tracing. The generic pipeline error remains
only as a fallback for blocks with no underlying exception (e.g. a guardrail
that could not be found).
Resolves LIT-4041
litellm_guardrail_latency_seconds was only emitted for pre-call guardrails.
during_call_hook and post_call_success_hook ran guardrails without recording
any latency, so during-call and post-call guardrail time was invisible in the
metric and leaked into litellm_overhead_latency_metric, making the documented
"subtract guardrail latency from overhead" workaround under-report total
guardrail time.
Extract the find-the-PrometheusLogger-and-record step into _emit_guardrail_metrics
and add _run_guardrail_with_metrics, a single wrapper that times a guardrail
coroutine, classifies its outcome (success / intervened / error), enriches any
raised HTTPException, and records the latency under the given hook_type. Route
the pre-call emit, during_call_hook, and post_call_success_hook through it so
every guardrail phase contributes to the metric the same way.
Resolves LIT-3999
* test(proxy/utils): pin ProxyLogging behavior
Add behavior-pinning tests for the ProxyLogging cluster in
litellm/proxy/utils.py under tests/test_litellm/proxy/utils/proxy_logging/.
Covers InternalUsageCache, _CallbackCapabilities, top-of-file helpers
(print_verbose, _get_email_logger_class, _accepts_litellm_call_info,
_enrich_http_exception_with_guardrail_context), the full ProxyLogging
class (lifecycle, MCP-LLM bridging, capability probes, guardrail
pipeline, pre/during/post/streaming hooks, alerting), plus the
bottom-of-region helpers (on_backoff, jsonify_object, _lookup_deprecated_key).
Each pinned symbol has happy-path and error-path coverage; happy paths
use direct dict-equality with three or more keys (or HiddenParams /
Pydantic model_validate where the surface is a Pydantic shape). The
subdirectory carries a local _pin_check.py and _coverage_check.py that
enforce the gate without surfacing numeric thresholds in CI logs.
Wires tests/test_litellm/proxy/utils into the existing test-path block
in .github/workflows/test-unit-proxy-endpoints.yml.
* test(proxy/utils): drop unused mock_httpx_client fixture
Declared in conftest.py but never referenced by any test. Removing
the dead fixture per Greptile P2 feedback.
* test(proxy/utils): drop local-only gate scripts from PR
_pin_check.py and _coverage_check.py are local stopping signals (not
wired into CI, consume a gitignored .pin_list.txt). They served their
purpose telling the engineer when to stop writing tests; the pytest
suite is the artifact that belongs in the repo.
---------
Co-authored-by: Claude <noreply@anthropic.com>
