* fix(guardrails): add /v1/messages support for Straiker plugin
- Pass prepared response data to Anthropic Messages streaming post-call hooks (litellm/llms/anthropic/chat/guardrail_translation/handler.py)
- Normalize Straiker request, tool, finish-reason, and mode fields across Chat Completions, Messages, and Responses APIs
* fix(guardrails): gate cross-surface message resolution and cover streaming request data
Resolve request messages only for surfaces that have a mapped translation
handler. The unguarded fallback tried every registered handler in turn, which
raised AttributeError out of the guardrail's error handling on list-shaped
`input` bodies, and synthesized a chat message that was never sent for bodies
it happened to parse.
Prepare request data on the mid-stream Anthropic branch as well, matching the
terminal branch and the OpenAI handler, so guardrails that scan before
end-of-stream still receive identity metadata.
Read usage from Anthropic dict responses so non-streaming /v1/messages reports
token counts instead of null.
Add regression coverage for the streaming request data on both the terminal and
mid-stream branches; reverting either now fails.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
The http_handler pair only received its full mutation verdict after the
first removal batch landed; these five tests pass unchanged when every
function they execute is mutated and the owning file killed none of
their scored mutants. The ssl tests excluded from mutation scoring are
untouched.
* fix: handle explicit outputInfo: null in Vertex AI batch response
Vertex AI can return HTTP 200 for a create_batch/get_batch call with an
explicit "outputInfo": null body (the output directory is assigned
asynchronously and may not be populated yet at response time).
_get_output_file_id_from_vertex_ai_batch_response did:
response.get("outputInfo", OutputInfo()).get("gcsOutputDirectory", "")
dict.get(key, default) only substitutes default when the key is absent,
not when it is present but explicitly None, so this crashed with:
AttributeError: 'NoneType' object has no attribute 'get'
surfaced to callers as an opaque openai.InternalServerError 500 from
litellm.create_batch()/retrieve_batch() for any Vertex AI batch job,
regardless of whether the job ultimately succeeds.
Fixed by guarding with `response.get("outputInfo") or OutputInfo()`,
matching the existing null-safe pattern already used by the sibling
_get_input_file_id_from_vertex_ai_batch_response for inputConfig. The
existing outputConfig fallback branch (a few lines below) already
handles this case correctly once it's reachable - it just never was.
Added 2 regression tests covering outputInfo: null with and without an
outputConfig fallback available.
* test: drop explanatory comment from regression test
---------
Co-authored-by: htourinho-clgx <htourinho@cotality.com>
The accumulated-JSON fallback ran json.loads over the whole buffer after every fragment and, on failure, kept the buffer without resetting it. A buffer that ever held more than one concatenated JSON value could never parse (json raises on trailing data), so it returned None on every subsequent chunk while growing without bound - an unrecoverable per-request CPU spin. Parse one value at a time from the front with raw_decode and keep the remainder, draining trailing values on later calls and at end of stream.
The async streaming error paths fired the sync failure_handler in a thread
and the async_failure_handler via create_task at the same time, so both
mutated the shared logging object concurrently and could crash pydantic-core.
Route failure logging through a single guarded dispatch_failure_handlers, so
the sync handler only runs after the async one completes.
25 test functions across three files pass unchanged when every function
they execute is mutated; the owning file killed zero of their scored
mutants. Four zero-kill tests tied to the fix in #31288 are kept for
rewrite instead of removal.
Extract the inline sync streaming post/decode into make_sync_call so it can be
exercised with an injected client, mirroring make_async_call, and add a sync
regression test that each token is forwarded after exactly one pulled frame.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Mirror the sagemaker_chat fix on the native sagemaker/ streaming path: the
sync and async completion handlers read the invocations-response-stream body
with iter_bytes(chunk_size=1024) / aiter_bytes(chunk_size=1024), so httpx
withholds bytes until 1024 accumulate and tokens arrive in gap-then-burst
waves. Drop the fixed chunk size so each decoded event is forwarded as its
bytes arrive.
Also add a boundary-agnostic decoder test proving frames reassemble correctly
regardless of where transport reads split the stream.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The huggingface embedding test fixture reloaded
litellm.llms.custom_httpx.http_handler, creating a new HTTPHandler class
object. llm_http_handler keeps the class captured at import time, so any
test running later in the same process that injects a client built from
the reloaded class fails the isinstance check and the mock is silently
discarded, causing a real network call. Under pytest-xdist loadscope this
surfaced as a deterministic failure of
test_accept_header_in_completion_request_jwt whenever an unrelated PR
shifted worker distribution.
Also removes the same reload pattern from the vertex rerank integration
test (both were previously removed in a6df01caec and resurrected by a
merge conflict resolution) and hardens the agentcore victim test by
dropping the bare except that swallowed the real error
* fix(anthropic): strip uniqueItems + other unsupported array/object constraints from output_format schema
Anthropic's structured outputs (`output_format`) validate the JSON schema
against a strict subset and reject cross-element / count constraints that a
constrained-decoding grammar cannot enforce, returning a 400
`invalid_request_error`.
`filter_anthropic_output_schema` already stripped the numeric / string /
item-count constraints (minimum, maximum, exclusiveMinimum/Maximum, minLength,
maxLength, minItems, maxItems) but still let these through:
- uniqueItems
- contains / minContains / maxContains
- minProperties / maxProperties
so a request using them fails with e.g. "output_format.schema: For 'array'
type, property 'uniqueItems' is not supported".
This is provider-visible: newer Claude models on the native `output_format`
path (e.g. `azure_ai`) 400, while `vertex_ai` is unaffected because it is
forced onto the permissive tool-use path (#18625 / #19201).
Add the missing keywords to the unsupported-field set and the description map,
and skip the advisory description note for a disabled boolean constraint
(`uniqueItems: false`) so it isn't misdescribed as required.
* fix(anthropic): serialize contains sub-schema in output_format advisory note
Address Greptile review: the `contains` advisory note previously discarded the
sub-schema, so the description only said an item must match "a schema" without
saying which. It now serializes the sub-schema as JSON (e.g. "array must
contain an item matching: {\"type\": \"integer\", \"const\": 1}"), matching the
other stripped constraints which carry their value. Sub-schema (dict/list)
values are json.dumps'd; scalar constraints are unchanged.
* style(anthropic): apply ruff format to output_format filter change
* style(test): ruff format anthropic schema filter tests
* test(anthropic): cover output_format array/object constraint filtering in test_litellm tree
Mirrors the schema-filter tests under tests/test_litellm/ so the coverage
job exercises the new uniqueItems/contains/min-maxProperties handling and the
uniqueItems: false branch.
---------
Co-authored-by: Darien Kindlund <darien@kindlund.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guardrails): add only_scan_new_messages for per-session incremental scanning
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): use fixed TTL constant and revert unrelated test formatting
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): run only_scan_new_messages in the unified apply_guardrail path
The initial wiring lived in BedrockGuardrail.async_pre_call_hook, but the proxy
routes Bedrock through the unified apply_guardrail interface, so the flag had no
effect live. Move incremental selection into apply_guardrail: filter the flat
texts list against per-session scanned hashes, skip the Bedrock call when nothing
is new, and mark hashes only after a successful (non-blocked) scan. Full-context
fallback is preserved when there is no session id, the cache is unavailable, or a
masking guardrail is configured.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover session-id fallbacks and mark_texts_scanned guards
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): fall back to full scan when incremental guardrail masks content, use shared cache
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover generic agent multi-turn incremental scan
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover incremental scan cache resolver fallbacks
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover flag interactions and /v1/messages incremental scan semantics
* feat(guardrails): make GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS env configurable
* test(guardrails): prove skip_system/skip_tool are enforced upstream of incremental scan
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted
Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400
Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag
Closes the Vertex/Azure gap from the customer RCA
Caller-supplied bedrock_tags land as AWS resource tags under the proxy's
AWS identity, letting an authenticated caller forge ownership or
cost-allocation labels. Add bedrock_tags to _BANNED_REQUEST_BODY_PARAMS
so per-request tags need general_settings.allow_client_side_credentials
or configurable_clientside_auth_params on the deployment, matching the
aws_bedrock_project_id precedent. Deployment-level bedrock_tags in
litellm_params are unaffected.
Also stop an explicit empty bedrock_tags list in litellm_params from
falling through to optional_params
* fix(fireworks_ai): set Content-Type application/json in validate_environment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(fireworks_ai): delegate chat validate_environment to OpenAIGPTConfig
Instead of re-adding the JSON Content-Type default inside FireworksAIMixin,
FireworksAIConfig now delegates header construction to OpenAIGPTConfig and only
layers the Fireworks-specific x-session-affinity header on top, so the
Content-Type default can no longer drift away from the OpenAI base and reintroduce
the 415.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fireworks_ai): cover missing api key error path in chat validate_environment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
glm-5p2 (and its fireworks_ai/glm-5p2 alias) carried cache_read_input_token_cost
of 2.6e-07, the GLM 5.1 rate; the entry was seeded from the wrong row. Fireworks'
standard serverless rate for GLM 5.2 is $0.14/1M = 1.4e-07, so every prompt-cache
hit was billed at nearly double the real rate.
Corrects the value in both the canonical map and the bundled backup. The existing
fireworks cost-calculator test now reads the cached rate from the map instead of
hardcoding it, so it tracks the shipped value.
The /v1/messages handler resolved only the auth object and the trace id, so tool
listing and tool execution ran without the caller's MCP auth headers. That fails
quietly rather than loudly: the tool still executes, just with no credentials, so
every server behind interactive OAuth, a bearer token or per-user env vars returns
nothing while the model reports it has no access. Only a no-auth server looks
healthy, which is exactly what the first proof used.
Threading the missing arguments would have left the real problem in place. Each
gateway surface rebuilds the same context by hand (responses/main.py twice,
chat_completions_handler, mcp_streaming_iterator), which is why a new surface
drops fields; this adds a fifth that dropped six of eight. Resolve it once into a
frozen MCPRequestContext and have the handlers take that, so a field cannot be
forgotten at a call site. chat_completions_handler now uses it too, and the
resolver reads user_api_key_auth from both metadata keys because
LITELLM_METADATA_ROUTES carry it in litellm_metadata while chat uses metadata.
Also stop the loop when every tool call was skipped. tool_results is empty then,
and the tool_result message built from it has empty content, which Anthropic
rejects; the caller saw a 400 from mid-loop instead of the model's own answer.
Tests pin both: dropping the headers from either listing or execution fails, and
so does removing the empty-results guard.
