* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rename fork-flag to unit-flag now that it applies on every event
* test: move tests/test_litellm root and small trees into tests/unit
Pure renames, no content changes. Follow-up commits in this PR fix
references, merge the three files that already existed in tests/unit,
keep live-provider tests in tests/test_litellm and wire CI.
* test: carry tests/test_litellm conftest isolation into tests/unit
Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS,
proxy-URL and keychain env, and session-end client cleanup now reset for
unit tests too. The environment isolation owns its MonkeyPatch so a test's
own monkeypatch is undone before the model-cost teardown runs.
* test: merge, split and prune the moved root and small-tree tests
Merge batches/test_batch_utils.py and the chat_completions and messages
dispatch tests into the files that already existed in tests/unit. Keep
the live Gemini interactions tests, the async image-fetch format test and
the OpenAI embedding scorer test in tests/test_litellm since they need
real network or keys. Put test_router.py under tests/unit/test_router so
the existing package no longer shadows it. Delete eight tests the audit
found superseded by stronger ones kept in this move.
* ci: run the moved root and small-tree tests under their legacy flags
Add the misc and responses-caching-types flags to unit_selection.sh and
CircleCI, extend enterprise-routing and mcp-integration, and point the
legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest
and change classifier at the new paths.
* test: make the new tests/unit directories packages
tests/unit/test_package_layout.py requires every directory to carry an
__init__.py, and without one the moved and retained
test_litellm_responses_bridge.py modules collide on import.
* test: scope the unit socket block to tests/unit in shared sessions
The GHA shards collect the legacy test-path and the unit selection in one
pytest session. The unit conftest's loopback-only block leaked into legacy
modules that reach the network at import. The legacy conftest now lifts the
restriction at collect and setup time, and the unit conftest re-applies it
when collecting its own modules.
* test: move tests/test_litellm/llms into tests/unit/llms
Rename-only. Moves the provider tests and the fine-tuning fixtures they
load, mirroring the old paths. Follow-up commits merge, split and wire them.
* test: merge, split and prune the moved llms tests
Merges the Databricks chat transformation tests into the existing unit
file, keeps the tests that need real keys or the network in
tests/test_litellm, deletes the audited tests a stronger unit test
already covers, and points imports at tests.unit.llms.
* ci: run the moved llms tests under their legacy flags
The Vertex AI and All Other Providers shards keep their legacy test-path
for the retained files and add the llm-vertex-ai and llm-other-providers
unit selections. CircleCI gets matching unit jobs.
* test: make the tests/unit/llms directories packages
Adds __init__.py to the moved dirs and drops the legacy ones whose
directories no longer hold tests.
* test: drop script runners and path hacks the llms split left dangling
The __main__ runners in the split openai_like files and the Databricks e2e
runner called tests that now live in the other half of the split or were
deleted. The retained legacy halves also no longer need sys.path edits.
* test: give the shard-script tests their own GITHUB_OUTPUT
They only passed where the runner set it. The CircleCI unit job's env
allowlist drops it, so the script's redirect failed there.
* test: point the router and module-deletion checks at tests/unit
router_code_coverage and code_qa_check_tests only searched tests/test_litellm,
so the moved router tests no longer counted. The two silent-experiment tests
the audit deleted were the only direct callers of those methods; they are
replaced with tests that assert the forwarded shadow request and the
recursion guard.
* test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path
The Databricks e2e file is a manual script whose main() calls the tests
that were pruned, so pruning them broke the documented run. It is back to
its main version. The SageMaker Nova docstring now points at the file's
real location in tests/local_testing.
* test: keep the job's UNIT_FLAG out of the shard-script tests
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(presidio): cover first-frame utf8 split, comment keepalive and data-less ping in streaming output masking
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): classify the streaming output shape on a frame with a data line and tolerate a split utf8 boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): relay leading data-less sse frames before classifying the stream shape
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(anthropic): placement policy for mid-conversation system messages
Pure functions over the OpenAI-format message list: split off the leading
system run, keep later system messages as role=system at a placement Anthropic
accepts on models flagged supports_mid_conversation_system (after a user turn,
before an assistant turn or the end, never adjacent), and convert them to user
turns in place elsewhere, keeping tool_result first in a merged user turn.
* fix(anthropic): keep mid-conversation system out of the chat completions system prompt
translate_system_message hoisted every role=system message, at any index, into
the top-level system block. On a conversation carrying a mid-session reminder
that rewrites the cached prefix, so the provider re-bills the whole history at
cache-write pricing on every turn (#36559). #36968 fixed this on /v1/messages;
the chat completions path, shared by first-party Anthropic, Vertex, Azure AI
and Bedrock Invoke, still hoisted.
Only the leading system run becomes the system prompt now. Later system
messages go through the placement policy, and anthropic_messages_pt emits a
system message instead of rejecting the role. The caller's message list is no
longer mutated. Tests pin the two-turn prefix invariant across all four chat
configs and both flag states.
* refactor(anthropic): single-source the converted system note
The /v1/messages pass-through and the chat completions path must prefix a
converted system turn with the same operator note.
* test(e2e): prove the prompt cache survives a mid-conversation system reminder on chat completions
Same priming and assertions as the /v1/messages cases, through
/v1/chat/completions with OpenAI-format messages, for first-party Anthropic
and Bedrock Invoke on a flagged (Opus 4.8) and an unflagged (Haiku 4.5) model.
The reminder sits between the assistant turn and the next user turn, the shape
OpenAI-style agent frameworks send, which is the placement the chat path has
to translate.
* test(anthropic): cover the cache_control rebuild shapes and type the test helpers
Codecov flagged the 5m ttl branch and the empty-system path of the wire
builder; both now have a test. Greptile asked for full typing on the new
test helpers.
* refactor(anthropic): read the mid-conversation flag through a public supports_ helper
supports_mid_conversation_system joins the other supports_* helpers in
litellm.utils, so the chat transformation stops importing the private
_supports_factory.
* chore(typing): declare the mid-conversation type aliases with TypeAlias
The Final sweep tightened LIT010, which exempts TypeAlias declarations but
counts a bare alias assignment as an unannotated binding.
* fix(anthropic): let add_code_execution_tool take the pass-through message union
The translator now emits role=system inside messages for models that accept it,
so anthropic_messages_pt returns the pass-through union. add_code_execution_tool
still declared the narrower user/assistant union while only ever reading
content, so upstream's strip_advisor_blocks_from_messages call in between made
the mismatch visible to the type checker.
* fix(bedrock): keep mid-conversation system messages in place on converse path
* fix: ruff format + multi tool_result order + regression test
* fix: satisfy type-discipline gate + update osv ignore for mlflow PYSEC-2026-3865
* fix(bedrock): restore role narrowing in hoisted system loop for basedpyright budget
* test(bedrock): cover mid-conversation system conversion branches
- non-dict guard in _opens_with_tool_result
- in-place conversion without tool context
- str/list cache_control preservation in mid-conversation path
- drop unreachable non-system guard in hoisted loop
* Place type-discipline suppressions on the lines the gate scans
* Narrow hoisted loop to system role so basedpyright sees the right TypedDict
* fix(anthropic): place mid-conversation system runs by their neighbours only
A run after an assistant turn now slides behind the user turn that
immediately follows it, and a run that ends the array or precedes an
assistant turn becomes a user turn in place. No later message can move
an earlier run, so a client that replays the conversation with more
turns appended sends a byte-identical prefix and preserved thinking
blocks keep their binding
* refactor(bedrock): share the converted system note with the anthropic module
Converse imports CONVERTED_SYSTEM_NOTE instead of carrying its own copy
of the same text, and the reordering helpers lose their comments
* test: pin the replayed request prefix across preserved-thinking turns
One test per audited feature, through the real entrypoint: the chat
transformations for anthropic, bedrock invoke, vertex and converse, the
modify_params dummy tool result, dotprompt with unchanged variables, and
Presidio masking against an in-process fake. Each serializes system,
tools and the earlier messages of turn N and N+1 and asserts they match.
The e2e mid-conversation system test imports its content blocks from
models.py again and is marked provider_live
* fix(anthropic): move mid-conversation system placement into prompt_templates
The prompt factory imported the placement helper from the Anthropic provider
package, whose common_utils reads a factory constant at import time, so loading
the factory first raised ImportError. The module now sits next to
anthropic_messages_pt and every consumer imports core utils
A user turn with content [] or None puts no block on the wire, so a system run
anchored to it landed first in messages or behind an assistant turn. Such a run
now converts in place; empty strings and empty text blocks still anchor because
the factory fills them with a placeholder
* fix(anthropic): anchor system messages only on user turns that reach the wire
* fix(bedrock): type the converse system-message helpers over the message TypedDicts
* fix(anthropic): read replayed pydantic messages in the Converse helpers and convert a system run whose assistant follower sends nothing
A history that replays the previous turn as the litellm.Message object
was invisible to the Converse system-message helpers, so a mid-conversation
system stayed between a tool call and its result or reached Converse as
role: system. The helpers now read fields through the shared
message_field and parts_of accessors and drop the local role predicate.
Flagged placement anchored a system run on any assistant follower, but
anthropic_messages_pt drops an assistant turn that puts no block on the
wire (content None, an empty list, an unsigned thinking part), so the
system landed directly before the next user turn, which Anthropic
rejects. Such a run now converts in place. An empty or whitespace text
turn still anchors, since the converter pads it with a placeholder.
* fix(anthropic): treat bridged encrypted reasoning as a vanishing assistant turn for system placement
An assistant turn whose only blocks carry Responses API encrypted reasoning is
dropped by anthropic_messages_pt, so a mid-conversation system run anchored
before it landed directly before the next user turn. The unsignable-thinking
predicate now lives in common_utils and both the factory and the placement
policy consult it.
* fix(anthropic): let an inline thinking part hide separate thinking_blocks in system placement
anthropic_messages_pt skips an assistant turn's separate thinking_blocks as soon
as its content list carries an inline thinking or redacted_thinking part, so a
turn whose inline part is unsigned puts nothing on the wire even when the
separate block is signed. The placement policy now mirrors that rule.
---------
Co-authored-by: Shifat Islam Santo <shifatislamsanto764@gmail.com>
Co-authored-by: ege-arhan <egearhany@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(presidio): stream non-Anthropic raw SSE through the post_call hook unbuffered
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(presidio): keep the pytest.raises block to a single await
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(presidio): move raw SSE format check into a helper to keep hook complexity flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): fold the raw SSE format check into the existing bytes branch to stay within the complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): decide raw SSE stream shape on a complete first frame, not a transport fragment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover presidio post_call streaming for native gemini passthrough and anthropic messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): cap first SSE frame coalescing at 64 KiB so an unterminated first event cannot buffer unbounded
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): name raw SSE passthrough in the skipped output masking warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): speak the Straiker v3 platform API (/api/v3/detect)
The Straiker guardrail posted a webhook envelope to /api/v1/detect/webhook.
The v3 platform exposes /api/v3/detect instead, and its integration keys
(sk_agt_…) are rejected by the v1 route with an empty 401, so a tenant on
the v3 platform could not run this guardrail at all. Measured on a
customer gateway on 2026-09-17 after they rotated to a v3 key.
v3 parses the gateway's own traffic server-side, the same contract as
Straiker's unified Kong plugin. So on v3 the guardrail relays: the
request phase posts the provider body LiteLLM received (Anthropic
Messages or OpenAI chat), the response phase posts
{straiker_phase, sse, model, request}, the answer beside the request it
answers, and Straiker derives prompt, answer, agent and archetype. Both
phases also carry the flat prompt / app_response pair: a gateway-mode
integration key scores only the flat pair and an api-mode key only the
relayed body, each ignoring the other, so one payload serves whichever
key the console issued and it is one turn either way (measured on tenant
123, both key modes, 2026-09-18).
- api_version: "v1" | "v3", unset follows the key prefix, so a v3 key
needs no extra configuration. Explicit override still wins.
- The relayed body is an allowlist of provider fields. The hook sees the
client body merged with proxy state: `deployment` carries the resolved
provider credential and `proxy_server_request` the client's own
Authorization header. Neither travels. Identity survives as the
metadata subset Straiker's LiteLLM adapter reads.
- Identity never sends a proxy placeholder. `default_user_id` and the
master-key alias were being forwarded as a user and became the
session's identity on the platform.
- Headers: x-tool: litellm (ingress), x-straiker-phase, x-straiker-user,
and x-claude-code-session-id forwarded when the client sent it.
- Verdict: hookSpecificOutput.permissionDecision on the gateway envelope,
`action` on the flat one; block on block/deny, and on a non-empty
blocked_by as a backstop. A detect-mode control reads NONE.
- An error status from Straiker is now a webhook failure. LiteLLM's HTTP
client raises on any non-2xx and the retry loop caught only connection
errors, so a 401 or 503 from Straiker escaped the guardrail as an
exception and was relayed raw to the client, bypassing fail_open /
fail_closed. Retryable statuses retry; the rest are final.
- v1 is unchanged: same envelope, same X-Straiker-Webhook-Format header.
Tests: 15 new, fixtures from the request dict a hook sees on 1.98.0 and
the verdict envelopes the v3 platform returned on 2026-09-18. Each fix
was mutation-checked (handling removed, the test fails). Live: the same
eight-case battery (chat, /v1/messages, streaming, tool call; benign,
injection, PII) passes on a gateway-mode and an api-mode key, blocks at
pre_call with the tenant's block message, and lands under the declared
agent with the end user attributed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(guardrails): name the agent per application on v3 (x-s6r-agent)
One integration key can front several applications. Straiker enumerates them
as separate agents when the turn names one, which is what the unified Kong
plugin sends as x-s6r-agent. Without it every application on a gateway
collapses onto a single agent.
- Forwards a client-supplied x-s6r-agent.
- New `agent_ref` config names one agent for a route when the client sends
nothing. The client wins, matching Kong's precedence.
- Neither set: no header, and the platform derives the agent from the traffic.
Verified live on tenant 123 against an integration whose connector is
`gateway`: three distinct values minted three observed agents, and a turn
with no hint derived one from the traffic shape. An integration whose
connector is `custom-agent` declares its agent, so every turn attributes to
that one agent and the hint is ignored (agent_ref_source: attested).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(guardrails): which v3 shape is scored depends on the connector, not the key mode
The earlier comment said a gateway-mode key scores only the flat pair. Re-measured
on tenant 123 across all three integration types with one injection prompt:
custom-agent connector (Add Agent) raw body ignored flat prompt scored
gateway connector raw body scored flat prompt scored
api mode raw body scored flat prompt ignored
Behaviour unchanged: the payload already carries both shapes, which is why it works
on every type. Comment only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(guardrails): send exactly what the unified Kong plugin sends on v3
The v3 platform parses the gateway's traffic itself and derives agent,
archetype and identity from it. The earlier commits added to the relayed
body (a flat prompt / app_response pair, source, user_name) and to the
headers (x-tool, x-straiker-phase, x-straiker-user). None of that is in
the Kong v0.12 contract, and traffic through this guardrail was not
classifying by shape the way the same traffic through Kong does. Match
Kong byte for byte and leave classification to the platform.
Request phase: the provider body, plus session_id and
original.processed.Meta.user. Response phase: {straiker_phase, sse,
model, request} plus the same two. No flat fields, no phase or user
headers, no x-tool.
Session id follows Kong's precedence: the client's x-claude-code-session-id,
then the session LiteLLM resolved, then an md5 of system prompt + first
message so a conversation that states no session still groups across its
replays.
Routing hints complete the Kong set: x-s6r-agent (client header, else
`agent_ref`), and new `client` (x-s6r-client) and `format_hint`
(x-s6r-format) config, both optional.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(guardrails): sort imports in the v3 session test
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): send a streamed Messages answer back in the Messages shape on v3
On a streamed /v1/messages call the proxy rebuilds the answer as a chat
completion before the post-call hook runs, and that is what the plugin put in
the response envelope's sse field. Straiker's coding-agent reader parses a
Messages answer, so a Claude Code turn relayed this way came back
coding_agent/claude with no session and zero events scored: the model's tool
calls were never screened on the response phase. Captured live on 2026-09-18
against tenant 123, a real Claude Code Bash tool call through the proxy.
The proxy's own Anthropic adapter turns the rebuilt answer back into a Messages
response when the call arrived on the anthropic_messages route, which is what a
transport relay forwards. Chat completions calls keep the chat completion shape
and a buffered Messages answer is relayed untouched.
The regression test's fixture is the chat completion the proxy actually built
for that captured turn. After the fix the same turn scores on the response
phase (session resolved, one event, the Bash tool_use block present).
* style(straiker): ruff format the v3 guardrail and its tests
* refactor(straiker): one attempt per call in the webhook retry loop
The HTTPStatusError branch added for v3 duplicated the non-200 branch and put
_post_webhook over the strict complexity ceiling. One attempt is now its own
method that returns the verdict or a failure marked retryable, and the loop only
decides whether to try again. Behaviour is unchanged: retryable statuses and
transport errors retry, everything else is final.
* fix(straiker): name Claude Code's client and agent on v3 so its session lands under one coding agent
Straiker types a gateway turn as a coding agent from the "You are Claude Code"
preamble, which only the main agent turns carry. Claude Code's title and
topic-detection sidecars have their own system prompts, so they resolved by
shape as autonomous, and because they share the session id with the main turns
the whole session was filed under Autonomous rather than under a coding agent.
Kong does not hit this because its plugin config names the client and agent on
every call.
The User-Agent (claude-cli/...) is on every call including the sidecars, so the
plugin now reads it and sends x-s6r-client: claude plus, when the route names no
agent, x-s6r-agent: "Claude (LiteLLM)". A client-supplied x-s6r-agent or the
agent_ref config still wins. Verified live on tenant 123: a real Claude Code
session now lands as one coding_agent labelled "Claude (LiteLLM)" with its turns
scored, where before it split across Autonomous.
Identity: the key's own user (email then id) now outranks the end user the
request named. LiteLLM resolves Claude Code's hashed metadata.user_id as the end
user when nothing better is set, so a per-user key was being shadowed by a
session token. The key is the authenticated principal, the way a Kong consumer
is, so it wins; the request end user is the fallback.
* refactor(straiker): build the v3 request, envelope and headers as frozen mappings
The v3 builders seeded dicts and grew them, which the type-discipline gate
counts as mutable accumulators. Each is now one expression over a tuple of
pairs, frozen with MappingProxyType, and the JSON encoder unwraps a frozen
mapping through a default. The session seed and the verdict parser no longer
rebind locals. The wire is unchanged: 36 live calls through the proxy on this
commit carry the same fields, shapes, headers and identities as before, with
no mappingproxy text in any body.
* fix(straiker): satisfy basedpyright on the v3 builders
The frozen-mapping refactor left a shadowed headers local, a Mapping handed to
an HTTP client that takes a dict, an unguarded optional response, a turn id
typed object, and a redundant isinstance on already-typed texts. No behaviour
change: 4 live calls (chat, Messages, Bedrock, injection) return 200 with the
expected verdicts on this commit.
* fix(straiker): type the v3 config fields at the initializer and keep the verbose log as JSON
The four v3 routing fields (api_version, agent_ref, client, format_hint)
travelled through the untyped kwargs passthrough, which basedpyright counts
against the budget. They are now validated through a small Pydantic model at
the initializer and passed by name.
The verbose log serialized the frozen payload with default=str, which printed
a Python repr instead of JSON once the builders returned MappingProxyType.
Every serializer now unwraps a frozen mapping first. A test asserts the logged
payload parses as JSON and carries the identity; mutating the log site back to
default=str fails it.
* fix(straiker): address review findings on the v3 relay
Text completions relay their prompt: `prompt`, `suffix`, `echo` and `best_of`
join the provider allowlist, so /v1/completions traffic is screened.
The route's `agent_ref` now outranks the caller's `x-s6r-agent` header. The
header is caller-supplied, and letting it beat a pinned route would let any key
file its traffic under another application's agent and controls. On a route
that names nothing the header still names the application, which is how
several applications enumerate behind one key.
Credentials inside `tools` and `mcp_servers` (an OpenAI `mcp` tool's `headers`,
Anthropic's `authorization_token`) are replaced with `[redacted]` before the
body leaves the proxy, on both phases and in the verbose log. Detection reads
tool names, descriptions and schemas, never these.
A 200 whose body is valid JSON but not an object now reports an invalid
schema and follows the failure policy instead of raising out of the hook.
Comments that restated a constant are gone. Tests cover each change and the
failure paths (unreadable error body, client exceptions, missing response,
unmodellable request, session seeds from Anthropic block shapes); every fix
fails its test when reverted.
* fix(straiker): scrub tool credentials one level deep, without recursion
* fix(straiker): scrub only the fields that carry a credential, never a schema
The credential set is now the three fields that actually hold one on a tools
or mcp_servers entry (headers, authorization, authorization_token), read one
level deep. A function tool whose parameter schema defines a token, headers or
api_key property is relayed exactly as sent; a test pins that, and fails
against the recursive version.
* test(straiker): use example.com identities; drop a comment that restated its branch
* fix(straiker): present a legacy completion as the chat exchange it is
Straiker scores chat on both phases of a gateway turn but has no reader for a
text_completion answer: the request phase of a /v1/completions call was
scored and the response phase was refused with 501, whether or not the call
named an agent. A completion is one user turn and one assistant turn, so both
phases now present that exchange: the prompt becomes the single user message
and the TextCompletionResponse becomes a chat completion. Measured through the
proxy on this commit, both phases return 200 and score, and the derived
session is shared between them.
The derived session seed accepts the tuple the conversion produces; the test
pins the session on both phases and fails against the list-only check. The
unreachable "parsed is None" branch is folded into the failure branch, and a
malformed tools value is shown to relay as sent.
* fix(straiker): screen a completions prompt as the text the model receives
LiteLLM's /v1/completions accepts a string, a list of strings, a list of
token ids or a list of token-id lists, and decodes token ids with the
text-davinci-003 tokenizer before calling the model. The relay now renders
the prompt the same way, one user message per prompt, so a pre-tokenized
prompt is screened as the text it stands for rather than as digit strings.
A prompt in a shape this cannot render (empty, mixed, or with no tokenizer
available) is relayed untouched instead of being replaced with something
else. Tests cover all four accepted shapes and six unrenderable ones.
* fix(straiker): seed the derived session on the preamble and the first user turn
An OpenAI chat body carries its system prompt as messages[0], and the derived
session seeded on the Anthropic `system` field plus messages[0] with no role
check. For that shape the seed was the system prompt twice and the first user
turn never counted, so every unnamed conversation behind one system prompt
collapsed into one Straiker session. The seed now takes the preamble from
wherever the API puts it (`system`, `instructions`, or a leading system or
developer message) and the first message with role `user`, else a Responses
`input` string, else `prompt`. Two conversations sharing a system prompt are
two sessions again; a replayed conversation stays one.
* fix(straiker): seed the derived session on every text block of the first turn
A user turn that opens with an image or a document block and carries its
text later seeded the session on an empty string, so two different
conversations under the same preamble shared one Straiker session. Read
every text block of the turn instead of only the first block. A plain
string or a single text block seeds exactly as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(straiker): cover the tokenizer fallback, a textless first turn and Responses instructions
Three branches of the v3 relay had no test: a token-id prompt relayed as
sent when the tokenizer cannot be fetched, a first user turn with no text
seeding the session on the preamble alone, and a Responses API body
seeding on its instructions and first input turn. Each test fails when
its branch is mutated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): seed the derived session on the principal as well as the conversation
Straiker skips turns it has already scored for a session. The derived
session hashed the system prompt and the first user turn alone, so two
users who opened a conversation with the same words shared one session,
and the second user's copy of an attack came back as a replay: unscored
and allowed. Measured live on 2026-09-20: the first user's SSN turn was
blocked (`social_security_number`, scored=2), the second user's identical
turn was allowed (`controls: []`, replayed=2).
The principal now joins the seed. Explicit session ids, the Claude Code
header and LiteLLM's own session are unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): derive the session id with sha256 and drop comments that restated constants
The derived session now hashes the principal, and CodeQL flags MD5 over an
identity as a weak hash on sensitive data. SHA-256 truncated to the same
32 hex characters keeps the id shape. Comments that only labelled the
allowlist groups or restated a constant are removed; the two that explain
a non-obvious choice stay.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): keep a blocked conversation blocked when it is replayed
Straiker de-duplicates turns it has already scored per session and
answers a replay `allow`, whatever the first verdict was. A client that
resends a blocked request, or grows the conversation past the blocked
turn, was let through: measured on 2026-09-20, `block` then `allow,
events_replayed=2` for the same session and body, and Claude Code's
automatic retry after the 400 turned a blocked poisoned-file read into
a pass.
The guardrail now remembers, per session, a fingerprint of every
conversation it blocked (a bounded, day-long in-memory cache) and blocks
a request that repeats or extends one without asking again. A different
session with the same words is a new conversation and is scored afresh.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): scope the block memory by session or principal, never by content alone
A request with no derivable session keyed the replay memory on the
conversation fingerprint alone, so one caller's block could answer
another caller's identical request. The memory is now scoped by the
session, else by the principal, and a request with neither is not
remembered at all.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): remember only a block that names a control, never one that comes from state
The replay memory kept every block, including one the platform returns
because a kill switch is engaged (`action: block` with `blocked_by: []`).
An administrator lifting the kill switch then left the conversation
refused by the remembered copy: measured on 2026-09-21, traffic stayed
blocked after `POST /inventory/agents/{id}/restore` returned `engaged:
false`.
The same words are the same attack tomorrow, so a control-named block is
still worth remembering; state is not ours to cache. The parsed verdict
now carries `blocked_by` so the two can be told apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Phimmasone Phonpaseuth <PhimmStraiker@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(mcp): apply async_post_mcp_tool_call_hook content changes to the tool result
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): drop structuredContent when a post-call hook rewrites tool content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): satisfy type discipline and result contract
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): document internal logging patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): avoid Final assignments inside callback loops
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): run every post-call hook and chain the rewritten content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(mcp): credit the original fix from #33403
Co-authored-by: eric <mitrecx@163.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): cover post-call logging fallback paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): cover proxy hook logging context
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): preserve native structured guardrail replacements
* fix(mcp): invalidate stale structure after direct content edits
* fix(mcp): reconcile direct edits after callback exceptions
* fix(mcp): preserve successful in-place callback rewrites
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: eric <mitrecx@163.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(presidio): mask PII in streaming /v1/messages output
Raw Anthropic SSE frames were passed through the post_call output masking
callback untouched, and ProxyLogging rerouted the callback to the unified
apply_guardrail path on /v1/messages because mask_response_content was
false. Buffer the raw frames, assemble them with the shared Anthropic SSE
helpers, mask through Presidio, and re-emit the masked frames.
Resolves LIT-8288
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): replay raw SSE frames when masking fails mid-stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): propagate upstream stream errors instead of returning an empty stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(presidio): extract buffered stream masking to satisfy complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): let BLOCK on generated PII refuse the streaming /v1/messages response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(presidio): fold the BLOCK re-raise into the existing except to stay within the complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(presidio): move the blocked stream consumption into a helper so pytest.raises holds one statement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): fail closed when output masking of a raw SSE stream errors
A Presidio outage on streaming /v1/messages replayed the unscanned frames
to the caller. Propagate the error instead, matching the non streaming
path and the merge base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(presidio): cover structured chat stream output masking and trailing bytes passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Addresses Greptile review: typed RewritingGuardrail override, dropped routine docstrings, and the e2e waits for the key guardrail to sync via /chat/completions before its single /v1/videos call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
/v1/videos dispatches call_type avideo_generation, which CallTypes did not
know and no guardrail translation handler covered, so the unified guardrail
hook returned the request unscanned. Add the video call types and an OpenAI
video guardrail translation package that scans the prompt for create, remix,
edit and extension requests
Resolves LIT-6685
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Guardrails created through POST /guardrails on older releases have api_version "v1" saved in the database, because the writer persists every default. Azure Content Safety never accepts that value, so those guardrails kept answering 404 after the default moved to None. The Azure base now resolves "v1" to 2024-09-01 the same way it resolves a missing value. Also restores the OpenAPI snapshot line that a Python 3.14 regeneration had dedented
LitellmParams mixes every provider config model into one class, so the
Javelin api_version default of "v1" reached the Azure Content Safety
guardrails whenever config.yaml omitted api_version and Azure answered 404.
The shared field now defaults to None, Javelin keeps filling in "v1" itself,
and the Azure guardrails fall back to the documented 2024-09-01 at request
time so a DB update that omits api_version stays on the default too.
A stream_options.include_usage usage chunk (empty delta plus usage) was folded into the final
transform round and rebuilt without its usage, so token counts and cost vanished from clients.
Metadata-only chunks are now replayed after the final text flush.
A terminal tool-call chunk arriving while earlier text was still held back carried
finish_reason=tool_calls ahead of that text. The finish_reason is now deferred to the final
text chunk whenever the choice has held text.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Forward streaming_transform_mode from guardrail litellm_params into PromptSecurityGuardrail so incremental_diff is reachable from config; the default stays block_only. In incremental_diff the guardrail now returns stream_holdback_chars alongside the rewritten texts so that a value split across streamed chunks (or across an abbreviation period) is never partially released before the vendor rewrite arrives. Each response text gets its own protect call so modified_text maps back to the right choice when n > 1, and custom_guardrail no longer logs a clean response as mask just because the guardrail attached holdback metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Trimming the streamed buffer to the retained tail could drop a category
exception phrase that suppresses a later keyword, or the identifier word
of an unfinished sentence that a conditional category pairs with a later
block word. Refuse the cut while either would leave the buffer so the
bounded scan masks and blocks exactly like a scan of the full text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The streaming post-call hook rescanned the whole accumulated choice buffer on every chunk, so scan cost grew quadratically with output length. Keep a bounded per-choice buffer instead: once it exceeds twice the scan context, drop the head when masking the head and tail separately yields the same output as masking the whole buffer, so no pattern, phrase or exception straddles the cut. Detections from the dropped head are kept and merged, deduplicated, into the final log row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Removes the Singulr async_logging_hook and logging_hook overrides so logging_only runs through CustomGuardrail.async_logging_hook: the response scope reaches Singulr as an assistant message instead of a raw ModelResponse dump, a vendor timeout is recorded as guardrail_failed_to_respond, a request-scope block ends the scan, and the sync success callback thread makes no Singulr call.
Decides MCP versus LLM by the proxy logging object's call_type (then the call_type or server-only markers in request_data), never by name, arguments or mcp_tool_name keys a client can put in a chat body. REST /mcp-rest/tools/call pre-scans reach Singulr as mcp_request and a non-mapping arguments value is forwarded as tool_arguments instead of raising.
should_block is a strict bool defaulting to false so a null verdict is an invalid response that block_on_error decides; payload fields drop Any for Sequence, Mapping and AssistantMessage types; metadata carries only the keys present; docstrings and section comments removed per the repo comment policy.
Squash of BerriAI/litellm#37464 (head da298ca7) by @aniket-kardile, adopted onto main: v2 gateway payload contract with request, response, mcp_request and mcp_response scopes, typed payload models, proxy user, org and team metadata forwarded to Singulr, and the logging_only, pre_mcp_call and post_mcp_call modes.
