Every repository handed its `.table` back untyped, so a dozen modules had
each grown a private `_PrismaTableActions` Protocol to paper over it. They
had drifted: some declared `update` as returning the row, others the row or
None, and none agreed on whether `find_many` was covariant
Replace all of them with a single `TableActions[RowT_co]` in
`litellm/repositories/prisma_protocols.py`, keyed to the prisma row each
repository is bound to. Query inputs stay `Mapping[str, object]` so callers
keep passing plain dicts, and `find_many` returns `Sequence` so the row type
stays covariant
Typing the nullable returns honestly surfaced paths that were already
crashing. A team admin could never edit or delete a memory entry owned by
their team: the write-auth check fed a raw prisma row to a helper that
expects the domain model, so `members_with_roles` arrived as plain dicts and
the request died as a 500 instead of applying the edit. Non-admin members hit
the same 500 in place of the 403 they were owed, so refusal and breakage were
indistinguishable. `/v2/model/info?user_models_only=true` dereferenced a
missing user row rather than returning the 400 the route already had, three
team routes dereferenced a team deleted between the read and the write, and
the agent registry dereferenced a missing agent instead of naming it
basedpyright drops 2,132 errors, 1,454 of them reportAny and 73
reportExplicitAny. The dashboard's generated types pick up `string[]` where
they had `unknown[]` for a team's members, admins and models
The Presidio guardrail masks messages in place inside pre_call_hook, but three
paths independently persisted or emitted the raw pre-guardrail data: the
SpendLogs proxy_server_request body snapshot (taken before the hook runs),
a verbose_proxy_logger.debug dump of the raw request, and logging_only mode's
async_logging_hook, which never masked the model's response before it reached
external logging callbacks.
Resolves LIT-6015
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
Twenty-three tests across eleven files opened with litellm.set_verbose = True
and never put it back, so the flag stayed on for everything that ran after them
in the same process. None of those files read the output it produces: no
caplog, no capsys, no assertion on a log line, so the flag was left over from
debugging. Deleting it beats restoring it, since restoring keeps the noise.
Ten of the eleven stop leaving the flag on. test_volcengine_embedding.py still
ends with it set, from something it exercises rather than from the test itself,
which is worth its own look.
Onyx, prompt security, hiddenlayer, repelloai and deepkeep all write straight to
os.environ and unset again at the bottom of each test. None of the five has a
try/finally, so the moment a test fails it returns to the runner with the keys
still set and whatever runs next in that worker inherits them.
Raising inside test_onyx_guard_with_custom_timeout_from_kwargs on the current
files leaves ONYX_API_BASE and ONYX_API_KEY behind; doing the same in
test_hiddenlayer_config_saas leaves HIDDENLAYER_API_BASE. Both come back clean
after this.
89 raw writes and the hand-rolled deletes become monkeypatch calls. The
class-level setup_method and teardown_method pair in the onyx file, sweeping the
same three keys twice, becomes one autouse fixture. The sys.path.insert lines
and their now-unused imports go too, and litellm.set_verbose = True, which only
turned global debug logging on for whatever ran next, is dropped rather than
restored.
test_onyx_guard_config and test_prompt_security_guard_config asserted nothing at
all, so they could only fail by raising. Each now pins what init_guardrails_v2
produces: exactly one guardrail of the right class on litellm.callbacks,
carrying the configured name, default_on and hook. The zero-assert tests in the
other three are left alone; those are a judgement about each guardrail rather
than a mechanical sweep.
tests/test_litellm/proxy/guardrails passes at 2873.
* test: use monkeypatch.setenv for env writes in tests/test_litellm
`os.environ["X"] = v` inside a test leaks the value into every test that runs
after it in the same worker, so ordering decides the result. 262 of those
writes across 40 files now go through pytest's `monkeypatch` fixture, which
restores the previous value at teardown.
The rewrite skips any test that a mock.patch-family decorator wraps, any test
with defaulted positional parameters, any test whose own name is called
directly elsewhere, and rebinds nothing inside nested defs, because in each of
those cases appending a fixture parameter changes what pytest or mock binds.
Ratchets the TQ004 ceiling from 768 to 506.
* fix(test): delete the key through monkeypatch instead of popping it first
Five tests popped a key straight out of `os.environ`, ran, then restored it with
`monkeypatch.setenv`. By the time monkeypatch saw the name it was already gone,
so it recorded "absent" as the value to go back to and deleted the key at
teardown. On a worker that inherited a real `RESEND_API_KEY`, `SENDGRID_API_KEY`,
`UI_PASSWORD`, `LITELLM_SALT_KEY` or `OPENAI_API_KEY`, every test after the first
one ran without it.
`monkeypatch.delenv(..., raising=False)` removes the key and restores whatever
was there, so the try/finally the manual restore needed goes with it.
* chore(test): leave the two cost-calc files to the PR that rewrites them fully
Both files are also in #37815, which converts the module-global writes as well
as the env writes and folds them into one fixture. Two PRs rewriting the same
lines differently is a conflict nobody benefits from resolving, so this one
drops back to staging on those two and keeps the other 39.
TQ004 clears 200 here instead of 275; the rest moves with #37815.
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* test: enforce PT012 so a pytest.raises block cannot hide dead assertions
`with pytest.raises(...)` stops at the first statement that raises. Anything
sequenced after it inside the block never runs, so an assertion written there is
never checked and the test still reports green.
Two sites were doing exactly that, and both assertions turned out to be wrong
once they started running. tests/llm_translation/test_prompt_factory.py asserted
the bedrock rejection names "requires at least one non-system message", which
holds. tests/proxy_unit_tests/test_proxy_server.py asserted the prisma startup
failure mentions "httpx.ConnectError", which never appears: the failure is an
httpx.ConnectError whose message is "All connection attempts failed", so that
test now asserts the type. Its DATABASE_URL override moves to monkeypatch, since
the old restore sat below the assertion and leaked the invalid URL into every
later DB test the moment the assertion started being able to fail.
The remaining 72 sites are rewritten without changing what they exercise: setup
that cannot raise moves above the block, a nested `patch` moves outside it, and
bodies with real control flow (a stream drain, an if/else on sync_mode, a
retry loop) move into a local closure the block calls.
Fixing PT012 unmasked two B017s, since ruff only reports a blind
pytest.raises(Exception) once the block holds a single statement.
tests/proxy_unit_tests/test_auth_checks.py narrows to the ProxyException
can_key_call_model actually raises. tests/local_testing/test_completion_cost.py
was asserting vertex_ai/medlm-medium has no cost entry, which stopped being true
at some point; that dead first half is gone and the rest of the test, which
checks medlm pricing resolves above zero, now runs instead of being skipped.
* chore(ci): ratchet TQ004 to 768 after the prisma test moved to monkeypatch
* test(lint): ban blind pytest.raises(Exception) with ruff B017
A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError
a refactor introduces satisfies it exactly as well as the rejection the test was
written for, so the crash reads as a pass and the test never goes red.
All 111 existing sites are narrowed here. A runtime probe recorded the concrete
exception each one actually catches, and each site now names that type. Where
the code under test genuinely raises a bare Exception, the site pins a stable
slice of the message with match= instead.
Two sites tell on themselves. The shared responses-API cancel test raises
"custom_llm_provider is required but passed as None" rather than talking to a
provider at all, because cancel_responses takes a provider, not a model. And
test_bedrock_guardrails_with_streaming was the only test in its file still
passing without AWS credentials, because the NoCredentialsError boto3 raised
long before the guardrail ran satisfied the blind raises.
* fix(test): widen the openai batch-dispatch assertion to OpenAIError
The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one
the SDK raises OpenAIError while building the client, long before any 404, so CI
went red. OpenAIError covers both and still rejects a TypeError from a refactor.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
ruff.toml excludes tests/* from `ruff check`, so nothing has ever checked the
test tree for names that do not exist. That matters more in tests than in
product code: a NameError inside a test whose body is wrapped in
`except Exception: pass` is swallowed, and the test reports green forever.
Adds ruff-tests.toml selecting F821 alone, wired into the lint workflow and
`make lint-ruff`, and clears every existing violation:
- 4 tests interpolated an unbound `e` into a `pytest.fail` message reached only
on the failure path, so the NameError, not the assertion, is what ran.
test_llm_guard_error_raising is the worst: it passes today with content
safety disabled entirely. It now asserts the 400 and its detail body.
- 5 sites construct BaseExceptionGroup, a 3.11 builtin, in a tree that still
supports 3.10. Guarded behind the exceptiongroup backport that anyio already
pulls in below 3.11.
- 9 missing imports (json, openai, Any, Final, HTTPException), including one in
a helper that catches HTTPException by a name it never imported, so the
challenge path it exists to detect raises NameError instead.
- 5 annotations naming types imported inside the function body, hoisted to
module scope or TYPE_CHECKING.
- 2 blocks of dead code: everything after a pytest.fail in
test_claude_agent_sdk, and an unused helper in test_end_users calling a
function defined in a different module.
- 1 error-path f-string in the router-settings doc test that masked the real
FileNotFoundError behind a NameError.
Only F821 for now. Widening the select list means ratcheting thousands of
pre-existing findings, so rules go in one at a time with their violations
already fixed.
* feat(proxy): redact or drop individual batch records instead of rejecting the file
A single record tripping a guardrail rejected the whole upload, which is unusable for a file
holding thousands of rows. A record a guardrail rewrites is now submitted in its rewritten
form, a record it blocks is left out, and the create response reports every changed record by
both custom_id and line so a caller can reconcile against the file it sent. The same outcome
is written to the proxy log and to request metadata, so it is not visible only to the caller.
A rewritten record goes straight to a spool and only its offset is carried, so a masking
guardrail touching most rows of a large upload does not build a second copy of the file on the
heap, and the rewrite runs off the event loop the way the sibling full-file validation does.
Both proxy-injected metadata keys are captured from the record and restored exactly, including
an explicit null, so a masked row keeps the tags that decide how it is attributed.
A record is dropped only when a guardrail judged its content. `GuardrailRaisedException` now
carries `blocked_content` for that, because half its raise sites in the repo signal an
unreachable or unparseable backend under a fail-closed policy, and treating those as blocks
would turn "refuse this request" into "drop this record and submit the rest". The default is
off, so a raise that does not say what it means aborts the upload instead of silently
shrinking the file.
* fix(proxy): only drop a batch record on a verdict the guardrail actually reached
A guardrail that reports a technical failure as an HTTPException carrying a block status was
read as a content block, so an unreachable backend under a fail-closed policy quietly shrank the
file instead of failing the upload. Two in-tree integrations do exactly that, and one of them
defaults to fail-closed, so the broken configuration was the default one. Such an exception is
raised `from` the underlying error, which is a deliberate statement that something else caused
it, and no content verdict in the repo is raised that way, so the chain now settles it. Implicit
context is left alone, since a block raised inside an unrelated `except` would read as a failure.
Two annotation errors in the same family: the one GuardrailRaisedException subclass in tree never
opted into blocked_content, so a real block took the whole upload down with it, and straiker's
block helper is reached both from its verdict and from its fail-closed handler, so it claimed a
verdict for an outage. The helper now takes the flag from its caller.
A record could also opt itself out of the chain. Guardrail selection reads a body-level
`guardrails` key ahead of the proxy-injected list, and online that key can only add to the key
and team selection, never replace it, so a batch record naming an empty list skipped every
guardrail that was not default_on and was still reported as scanned. Every injected key is now
stripped before dispatch and restored afterwards.
A guardrail that reroutes a record to another model is honoured on the online path by rewriting
the model, which the scan read as a rewrite and submitted in the same file, sending content to
the provider the reroute existed to avoid. Every record of a batch file goes to one provider, so
the upload is refused instead, naming the line.
The scan spool is closed on the paths that never read it back.
* fix(proxy): give the scan the metadata bag guardrails actually read, and close its spools
The narrowed request metadata was installed under `litellm_metadata` only, but a record is
scanned as the chat request it describes, and the guardrails that pick a policy from a request
header read `metadata` instead. Noma choosing an application and Aim choosing a user both look
there, so the header allowlist added for them did not reach either one and a batch record was
still evaluated under the fallback policy. The scan metadata now goes into both bags, which are
both stripped and restored, so neither survives into the record that ships.
The scan spool was closed on the paths that abort, which are exactly the paths where it is
empty, and left open on the one path where it holds the rewritten records. Nothing closed the
rewrite output either, where before this feature the uploaded handle belonged to Starlette. The
upload now owns both and closes them however it exits.
* fix(proxy): register the scan spool before the rewrite can fail
The scan spool was added to the request's cleanup list only after the rewrite returned, so a
rewrite that raised, which for a spilled file can be as ordinary as the disk filling up, jumped
to the handler with the list still empty and left the scan's own handle open. The rewrite also
left its half-written output behind on that path, since nothing owns that handle until it is
returned. Both now close.
A terminal HTTP failure partway through chunking now logs the summed usage
and cost of the ApplyGuardrail calls AWS already billed, mirroring the
blocked-chunk path.
A blocked chunk now logs the summed usage and cost of every ApplyGuardrail
call AWS billed for the logical request, not just the blocking chunk.
Client-supplied metadata.standard_logging_guardrail_information is stripped
at the proxy boundary so callers cannot forge (even negative) guardrail
cost into spend, and guardrail_information_cost ignores negative or
non-finite entry costs as defense in depth. The cost map schema test now
allows guardrail_cost_per_unit and the guardrail mode.
A model with a long time-to-first-token leaves the proxy's response completely
idle, so any hop with an idle read timeout (AWS ALB and nginx both default to
60s) drops a connection that is perfectly healthy and would have delivered its
tokens shortly after.
The keepalive engines LiteLLM already ships wrap the response object, so they
fill a gap once the upstream has answered and then gone quiet. They cannot fill
the gap before it answers at all, and that is where the whole wait is spent:
measured against api.openai.com/v1/chat/completions with gpt-5.6 at
reasoning_effort high, the response headers and the first body byte both arrive
at 37.90s. Nothing has entered the ASGI response phase by then.
The upstream call is now raced against the keepalive interval, and when it
loses, the SSE response is opened immediately and ": ping" comments, which every
conformant SSE client ignores, fill the wire until the real response is ready to
be replayed onto it. One seam per funnel: base_process_llm_request covers every
native route, create_pass_through_route covers every passthrough route.
Committing the status line that early is the cost. A failure discovered after
the first ping reaches the client as an SSE error frame under a 200 rather than
as an HTTP error status, and LiteLLM's own x-litellm-* response headers are not
yet known. keepalive_ping_has_fired already documents the same trade-off for the
existing engines. Both are why this stays off until an operator sets
litellm_settings.sse_keepalive_ping_interval_seconds.
Separately, the passthrough relay reached neither engine even for mid-stream
gaps, which is the shape of #32491 and #24929, so the relayed bytes get the same
treatment, gated on the upstream declaring text/event-stream and only emitted
between complete frames so a binary transport (AWS event streams on /bedrock)
and a stall halfway through a frame are both left alone.
Fixes#34819
Price ApplyGuardrail usage units recorded by PR #37225 with a new
bedrock/guardrails entry in the model cost map (regional override via
bedrock/{region}/guardrails), add the per-request guardrail_cost to the
standard logging payload's response_cost and CostBreakdown, surface it in
the x-litellm-response-cost header, and bill blocked requests through the
failure hook so key and team budgets see what AWS bills
GET /guardrails/usage/overview and GET /guardrails/usage/detail/{id} 500ed on a
database that has not applied 20260817143646_add_daily_guardrail_usage_units yet
(pip installs on litellm-proxy-extras 0.4.86 with DISABLE_SCHEMA_UPDATE=true).
Both endpoints now return their metrics with empty units and log one warning
until the migration lands.
The daily guardrail metrics and usage-unit upserts are non-idempotent
increments, but the retry loop re-sent every failed row on any exception.
An ambiguous post-send failure such as a read timeout after the write had
already committed therefore stacked a second increment and inflated the
billable unit totals served by the guardrail usage endpoints.
Retry only DB_RETRY_SAFE_ERROR_TYPES (httpx.ConnectError), the same rule
the spend writer and autorouter rollup use for increment upserts, and log
any other failure once as terminal for that row while the rest of the
batch still lands.
Follows up #37225
A transient DB error during the spend log flush dropped that batch's guardrail
metrics and usage unit rows for good. Retry only the rows that failed, up to 3
times with 1s/2s/4s backoff, mirroring the daily spend writer, and inject the
sleep so tests stay fast. Lowers the lint budgets the refactor freed up
The flush and the usage endpoints summed units with a scan per distinct key,
quadratic in rows times keys; group sorted rows instead. Skip payloads without
a request_id like the metrics path, type the flush key as a NamedTuple, and drop
the (guardrail_id, date) index that the primary key already covers
Per-row guards in the daily metrics and usage unit flush so a single DB error no longer drops the rest of the batch, plus removal of narrating comments flagged in review
The Noma guardrail sends the conversation to the scanner in `inputs`. It
also forwarded `request_data` whole, which repeats that same conversation
under `messages` (or `input` on the responses API), and attached
`logging_obj.model_call_details`, which repeats it a third time.
For image-heavy calls that duplication is most of the request. A
production scan of a request carrying base64 images measured 100MB total,
of which 94.8MB was `request_data` against 5.1MB of `inputs` - the proxy
was uploading ~95% redundant bytes, and paying to serialize them.
Drop `messages` and `input` from `request_data` and from
`model_call_details`. This is a denylist rather than an allowlist on
purpose: every other key is still forwarded untouched, so a scanner-side
change that starts reading a new `request_data` key needs no matching
release of this hook. The removed keys are ones the scanner never reads -
it takes context only from metadata, litellm_metadata,
provider_specific_header, litellm_session_id/trace_id/call_id, stream,
response/responses ids, and litellm_logging_obj.complete_streaming_response,
all of which still pass through.
The conversation still reaches the scanner in full via `inputs`, so no
detection coverage changes.
Trimming happens before serialization, so the duplicate is never encoded.
Existing payload tests asserted the duplication; they now assert the trim
while keeping what they originally guarded - deep-copy semantics and the
unpicklable-object (uvloop.Loop) regression.
* fix(guardrails): scan text on /guardrails/apply_guardrail for Azure Content Safety
The two Azure Content Safety guardrails never implemented apply_guardrail, so the
endpoint fell through to the base no-op and answered 200 with the caller's text
echoed back, having scanned nothing.
Implementing that method also flips the proxy's unified-vs-native dispatch, which
would move request traffic off these guardrails' own hooks. Add an opt-out that
keeps every lifecycle event on the native hooks, so only the endpoint changes.
* test(guardrails): cover the remaining native-hook opt-out dispatch sites
Adds regression tests for the parallel post-call path, the MCP post-call hook, and
the policy engine step, so every read of the opt-out flag fails when removed.
* fix(panw_prisma_airs): scan tool call args as plain text, not a tool_event
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(panw_prisma_airs): type the tool call argument extractor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(panw_prisma_airs): cover tool call error fallback and dict masking paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): scan tool names with args and tolerate custom tool calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): scan tool call arguments that arrive already parsed
The tool call slice types arguments as a string, so a client posting parsed JSON
failed validation and the whole tool call, name included, read as unscannable and
was skipped without ever reaching AIRS. The OpenAI request path forwards
client-supplied tool_calls verbatim, so that shape is reachable.
Coerce non-string arguments instead of rejecting them, so the content is scanned.
* fix(panw_prisma_airs): route tool-block masked data by scan side, not by key name
Merging #37036 (already on staging) with this PR produces no conflict and a
silent bug. #37036 withholds prompt_masked_data on response-side tool blocks,
which was right while tool calls went out as a request-side tool_event: AIRS
reported the model's arguments under that key. This PR scans tool calls as
ordinary prompt/response text, so the side of the scan now decides which key
holds what. The model's arguments arrive under response_masked_data, already
covered by _CLIENT_HIDDEN_SCAN_FIELDS, and prompt_masked_data goes back to
being the caller's own input -- one of the audit fields LIT-5638 asks for.
Left as merged, a response-side tool block drops that field with nothing to
flag it.
- Tool-path block branch calls _build_error_detail without also_hide
- also_hide parameter removed; after this change it has no callers
- Regression test asserts both directions: model output withheld, caller
input preserved. It fails against the auto-merged combination.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(panw_prisma_airs): a wrong-typed tool name must not suppress the scan
_ToolCallFunctionSlice types name as str, and _get_tool_call_function turns any
ValidationError into (None, None), which _scan_tool_calls_for_guardrail reads as
an unscannable tool call and skips. So a client posting "name": 123 keeps its
arguments off the wire to AIRS entirely -- no error, no log, no block. The
OpenAI request path forwards client tool_calls verbatim, so this is reachable by
any caller holding a valid key.
_coerce_arguments already existed for exactly this failure mode on the sibling
field. Widening it to cover name closes the gap:
name='transfer_funds' AIRS called: 1x args scanned: True
name=123 (int) AIRS called: 0x args scanned: False <- before
name=123 (int) AIRS called: 1x args scanned: True <- after
Reported by Cursor Bugbot on fd9f6396e5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(guardrails): return the full PANW AIRS scan response on blocked requests
The blocked-request error detail was assembled from a hardcoded allowlist, so audit fields like prompt_detection_details, prompt_masked_data, source, transaction_id and session_id never reached the client even though AIRS returned them.
Resolves LIT-5638
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(guardrails): drop redundant comment in AIRS error detail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): withhold response_masked_data from the blocked-response error
The full AIRS passthrough also reached the response-side block path, where
response_masked_data carries the model's own generation. That branch is only
reached when mask_response_content is False, so the operator had explicitly
declined to deliver that text, and the error body handed it back anyway.
Withhold response_masked_data from the client-visible detail. prompt_masked_data
stays: it is the caller's own input and one of the fields the ticket asks for.
Every other AIRS field, including prompt_detection_details, source,
transaction_id and session_id, is unchanged.
* fix(panw_prisma_airs): withhold generated tool args from response-side blocks
_scan_tool_calls_for_guardrail calls AIRS with is_response=False because
tool_event is request-side in the AIRS schema, so AIRS returns the scanned
tool arguments under prompt_masked_data. When the tool calls being scanned
are the model's own output, that key holds generated content, and the
_CLIENT_HIDDEN_SCAN_FIELDS default (response_masked_data, empty on this
path) does not cover it. With the default mask_response_content=False the
block branch then shipped the model's masked tool arguments in the 400 --
the same content channel this PR closed for response_masked_data.
_build_error_detail takes an extra_hidden_fields argument so the withholding
stays in one place, and the tool-call block branch passes prompt_masked_data
when is_response is True. Request-side blocks are unchanged and still carry
prompt_masked_data, which is what LIT-5638 asks for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* style(panw_prisma_airs): apply ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(panw_prisma_airs): surface scan_id and scan metadata on allowed requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format panw guardrail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(panw_prisma_airs): expose scan id header only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(panw_prisma_airs): inject http client instead of patching private api
Adds an http_client seam so the scan-id tests drive the real AIRS request/parse path through a mock transport, plus direct coverage for the scan-id header helper.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): expose guardrail scan id header to browser clients
Keeps the panw optional_fields block untouched to avoid a needless conflict with a sibling PR that deletes it.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): scan and re-emit raw Anthropic SSE streams in the bedrock post-call hook
* fix(guardrails): keep upstream id and model on a blocked Anthropic stream
* fix(guardrails): deliver a blocked Anthropic stream as an error frame
* fix(guardrails): deliver an unscannable Anthropic stream as an error frame
* fix(guardrails): emit the guardrail block detail as JSON in the stream error frame
* fix(guardrails): deliver an Anthropic block through the shared block-SSE builder
* fix(guardrails): keep the shared SSE assembler behavior-identical for existing callers
* fix(guardrails): keep the stream error message a string and drop an unreachable branch
* chore(guardrails): drop a comment that repeated its own docstring
* fix(guardrails): let bedrock service failures keep their status instead of framing them as blocks
* fix(guardrails): key the streamed block decision on status, not detail shape
InvokeGuardrailChecks details a Mapping on its 500 for an unparseable response,
so a detail-shape test read that outage as a policy block and framed it as a 200
guardrail_error. Both block sites raise 400, so gate on the status too.
* refactor(guardrails): narrow the SSE error-frame helper to the input it actually takes
Both callers pass a string, so the Mapping overload and its json.dumps branch
were unreachable. Folds the block branch's narrative comment into the rebind
suppressions that already carry a reason.
Three groups, all verified by running the suite rather than by inspection.
18 files whose every test function carries an unconditional @pytest.mark.skip,
39 test functions in total. They are collected on every CI run and always skip,
so they advertise coverage the suite does not have. Reasons on the marks include
"AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to
using 'otel' for logging"; 26 of the marks predate 2025.
30 test functions with a byte-identical body and identical decorators to a
sibling in the same file and class, differing only in name. Deleting one of each
pair removes no coverage. Four further candidates were excluded because they
override an inherited test, where deleting the override un-shadows the base
class implementation instead of removing a duplicate.
9 test functions that a later definition of the same name shadows, so Python
never binds them and pytest cannot collect them.
One file that is a demo script rather than a test; its own docstring says to run
it with python.
Verification: collecting the 26 edited files gives 2,492 node IDs before and
2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9
shadowed deletions account for 0 (confirming at runtime that they were never
collectable), nothing unexplained disappeared, and nothing new appeared. No
other test or module imports any deleted symbol.
* feat(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing
AWS's ApplyGuardrail API rejects requests whose content exceeds the
account's per-request "maximum input size in text units" quota with a
400 ValidationException. That cap is account/region/policy-dependent
and cannot be predicted from config, so it can only be reacted to.
_make_apply_guardrail_request now tries the whole-content call first
(no behavior change for requests that already fit). On a too-large
ValidationException it bisects the flat content list and retries each
half sequentially, recursing until every piece fits or cannot be split
further, then merges the per-chunk responses (action, assessments,
outputs, usage) into one so callers cannot tell chunking happened. A
real guardrail block on any (sub-)chunk still raises immediately.
Contextual-grounding requests are never chunked: grounding scores the
response holistically against the whole reference source, so
fragmenting it would produce misleading scores.
Each chunk call also gets a small exponential backoff retry on AWS
ThrottlingException (429), since chunking increases the number of
per-second API calls and can trade a 400 for a 429.
All new state is local to a single request's call stack (no shared
cache, no cross-process coordination), so this is safe for
single-pod, multi-pod, and cache-less LiteLLM proxy deployments alike.
* fix(guardrails): address Bedrock ApplyGuardrail chunking review feedback
Fixes three issues flagged in review of the chunking fallback: a single
oversized content item couldn't be split (only list-length bisection was
supported), a chunked request that got recovered still logged a stray
failure telemetry entry alongside the real outcome, and flattening chunk
outputs without positional bookkeeping could misalign masked text onto
the wrong original message once a chunk had nothing to mask.
* test(guardrails): add regression test for multi-level Bedrock guardrail chunking
Confirms the too-large bisection recursion isn't capped at a single split:
a payload that is still oversized after the first halving keeps splitting
until every piece fits, converging on however many chunks it takes rather
than only ever producing two.
* fix(guardrails): hybrid bin-pack+bisection chunking, whitespace-safe splits
Rework Bedrock ApplyGuardrail chunking from pure reactive bisection to a
hybrid strategy: bin-pack content into fixed-budget batches up front as
the fast path, falling back to the existing recursive bisection only for
a batch AWS still rejects as too large. Avoids paying O(log n) round
trips on every oversized request when a single pass would do.
Also switch single-item text splitting from a raw character midpoint to
the nearest whitespace boundary, so a fragment never starts or ends
mid-word. Closes the accidental-severing case from review; the residual
gap (a multi-word denied phrase deliberately straddling the boundary) is
documented as an accepted limitation, since fixing it would require an
overlap window reconciled against masked output with no documented
length-preservation guarantee from AWS.
* chore(ui): regenerate dashboard API types
* fix(guardrails): don't retry an oversized Bedrock guardrail call as a throttle
AWS reports an ApplyGuardrail request that exceeds the per-request
text-unit cap as a ThrottlingException (429), not only as the documented
ValidationException (400). Verified against a live guardrail with an
active content-filter policy: a 3273-text-unit request comes back as
"Input text size (3273 text units) exceeds the maximum allowed (1000 text
units) for the content filter policy (Classic tier)".
The throttle retry keyed off status 429 alone, so every oversized chunk
burned the full backoff-retry budget - each attempt a billed AWS call
preceded by a sleep - before the bisection fallback got a chance, at every
level of the recursion. A size error is not transient; re-posting the same
content can never succeed. It now short-circuits straight to bisection.
Also rename _is_input_too_large_validation_error to
_is_input_too_large_error (it never keyed off the status code, and the
error is not always a ValidationException), correct the docstrings that
asserted a 400, and log at warning level when a split happens so the
recovery is visible without --detailed_debug.
* Revert "chore(ui): regenerate dashboard API types"
This reverts commit ebf8ba2fd57f13bccf7aa6c5dfcac41c74db1ed9.
* fix(guardrails): group all fragments of one item and stop double-logging
Two defects found in review, both invisible to the existing tests.
Fragment grouping assumed a split content item always produces exactly two
adjacent fragments. That holds for one bisection level but not two: an item
split twice yields four fragments, which were regrouped in fixed pairs into
two output entries for a single message. Since masking walks the merged
outputs by a running index across the original, unchunked message list, that
message was written back truncated to its first half and every later message
shifted. Fragments now carry the size of the group they belong to, so any
number of them collapse back into exactly one output entry.
Telemetry was also double-counted. AsyncHTTPHandler.post calls
raise_for_status(), so every non-200 from Bedrock reaches _sign_and_post's
error path, which logged guardrail_failed_to_respond before re-raising as an
HTTPException that the consolidating caller then logged again. A request
recovered by chunking reported one failure per rejected attempt plus a
success. The ApplyGuardrail path now opts out of that per-attempt logging,
since it owns consolidated per-request logging; the connection-level branch
still logs, as nothing else records it.
The existing tests missed both because their mocks return a non-200 response
object, while the real client raises. Added a helper that raises a genuine
httpx.HTTPStatusError so these paths are covered the way production hits
them, plus a case asserting an unrecoverable failure still logs exactly once
rather than zero times.
* refactor(guardrails): move Bedrock chunking rationale into docstrings
The chunking work explained itself with inline comment blocks, which this
repo's conventions do not want. Folded that reasoning into the docstrings of
the functions it describes and dropped the comments, including the
module-level constant blocks and the test-file banner.
No behavior change. The banner also claimed AWS rejects an oversized request
with a 400 ValidationException, which live testing disproved, so removing it
drops a stale claim as well as an internal ticket reference from a public repo.
* feat(guardrails): match AWS default chunk budget and make it configurable
ApplyGuardrail's default quota is 25 text units, roughly 25,000 characters,
per second. Chunking has to respect that throughput limit rather than just the
per-request size, otherwise splitting an oversized request trades a size error
for a throttle. The budget now defaults to 25,000 to match that default for
every user, up from an arbitrary 20,000.
Accounts with raised quotas can spend fewer calls by setting
chunk_budget_chars on the guardrail. A value AWS still rejects as too large is
bisected automatically, so an over-large setting costs an extra round trip
rather than failing the request.
* fix(guardrails): never split a Bedrock text into an empty fragment
_nearest_whitespace_split_index could return len(text) when the only space at
or after the midpoint was the final character, so the first fragment came back
identical to the text AWS had just rejected as too large and the second came
back empty. AWS rejects the unchanged fragment again, and each retry re-splits
it into the same fragment, so an oversized single item shaped like a long
unbroken token with one trailing space exhausted the stack with a
RecursionError instead of scanning or surfacing Bedrock's error.
Candidate boundaries that would leave either side empty are now discarded, and
the raw midpoint is used when none remain. The midpoint is always safe because
_split_bedrock_content only calls this for text of at least two characters.
* style(guardrails): move chunking rationale out of comments and into docstrings
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Restores the source changes intended for a08e4cf309, which landed with only the
test. Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* style(guardrails): sort the constants import into the first-party block
* refactor(guardrails): bring the Bedrock chunking path under the LIT lint budgets
Annotates never-rebound locals with Final, replaces the retry counter and the two
branch-assigned locals with single bindings, and moves the internal chunking chain
to Sequence parameters and tuple returns. Collections that reach the logged payload
stay lists on purpose: redact_nested_match_and_regex_keys only traverses dict and
list, so a tuple would carry PII past redaction. The remaining constructions are
contract-bound and carry inline reasons
* fix(guardrails): keep the pre-chunking contract for failures reported inside a 200
Reverts the 500 this branch introduced for an AWS 200 whose body carries an
Output.__type exception marker: the request proceeds as it did before chunking
existed. The logged status is now derived from the merged response instead of
being hardcoded to success, so that shape is still reported as
guardrail_failed_to_respond. The consolidated failure logger also goes back to
logging a dict rather than a bare string, matching both the pre-chunking code and
the InvokeGuardrailChecks path in this file
* docs(guardrails): correct the docstring for failures reported inside a 200 body
The raise was reverted, so the docstring no longer describes the code. Records that
the request proceeds by design and points at LIT-5338 for closing the fail-open path
behind the existing unreachable_fallback setting
---------
Co-authored-by: spencer-burridge <265588760+spencer-burridge@users.noreply.github.com>
Prompt Security drops tool and function rows unless check_tool_results
is on, so it now reports scan-only support from that setting and the
registry refuses the pairing at boot. Pairing scan_only_tool_results
with skip_tool_message_in_guardrail excludes every message, so guardrail
initialization now rejects that combination too.
An SSE stream that cannot be positively identified as Anthropic (no
parseable message_start event) now blocks instead of passing through
unscanned, closing the bypass where any raw-SSE backend skipped tool
permission checks entirely. Buffered chunks are joined back into one
stream before parsing, so events split across network chunk boundaries
assemble correctly instead of being silently dropped. Rewrite mode now
resets finish_reason to stop when no tool call survives, so the
re-encoded Anthropic stream reports stop_reason end_turn and clients do
not wait for a tool result that never comes
Guardrails silently skipped three surfaces on the Anthropic Messages
path, so an agent loop driven by /v1/messages ran unguarded:
- The Anthropic input translation never walked tool_result blocks, so
content returned by a local tool (a curl, a file read, an MCP call)
reached the model unscanned in both the string and list content
shapes, images inside a tool_result included.
- tool_permission only understood ModelResponse, so an Anthropic
non-streaming response or a raw SSE stream carrying tool_use blocks
passed through with no rule ever evaluated.
- ContentFilterGuardrail scanned inputs["texts"] but never
inputs["tool_calls"], so the arguments a model proposes for a tool
call went unchecked.
Tool call arguments are parsed as JSON before filtering so a MASK
action rewrites the value and leaves the payload valid JSON; non-JSON
arguments fall back to scanning the raw string. Denied tool_use blocks
are dropped from the Anthropic content array and replaced with a text
block, and stop_reason resets to end_turn when nothing tool-shaped
survives.
