Exercises the streaming field-fill heuristics, model_construct fallbacks,
and the get/cancel/delete/list request and response transforms that had no
tests.
Cache web identity STS credentials in the shared IAM cache (restores the
pre-v1.85.0 behavior removed by #27125) and cap the Google OIDC token cache
TTL at the token's own exp claim minus a 60s margin, never caching an
already-expired token
* fix(router): don't cool down parent deployment on advisor sub-call failure
Advisor orchestration issues a sub-call to a different provider/credentials than the selected deployment. When that sub-call fails (e.g. a 401 because no advisor API key is configured), the exception propagates up and the router's deployment_callback_on_failure attributes it to the healthy parent deployment's model_info.id, cooling it down and rejecting unrelated callers to the same model group.
Tag advisor sub-call failures on the exception and skip cooldown for them in deployment_callback_on_failure. The exception is tagged rather than wrapped so its type is preserved and retry/fallback classification and the client-facing error are unchanged. Genuine executor/deployment failures are untagged and still cool down as before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): tag advisor orchestration failures via provider-neutral util
Address review on LIT-4565: move the cooldown-exemption marker into
litellm/router_utils/cooldown_handlers.py so the router imports it at
module top instead of an in-function anthropic import, and extend the
exemption to AdvisorMaxIterationsError so a max-iterations orchestration
failure no longer cools down the healthy executor deployment.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
translate_thinking_for_model duplicated the same summary/auto_summary
wrapping logic as _translate_thinking_to_openai without the
disabled-thinking guard, so it could still wrap "none" into an
{effort, summary} dict when reasoning_auto_summary is enabled (caught
by Cursor Bugbot). Extract the wrapping rule into one shared
_apply_reasoning_summary_wrapping helper used by both call sites so
this invariant can't drift apart again.
Codecov flagged the empty-list early-return in
_translate_stop_sequences_to_openai as an uncovered line in the diff —
add a regression test asserting stop_sequences=[] does not set
new_kwargs["stop"].
Guard against reasoning_auto_summary wrapping "none" into a dict when
thinking is disabled — there's no reasoning trace to summarize, and
non-Claude providers (e.g. Fireworks) expect reasoning_effort as a
plain string.
Claude Code's auto-mode classifier sends stop_sequences and thinking:
{type: disabled} on /v1/messages. The Anthropic adapter passed
stop_sequences through unchanged instead of mapping it to OpenAI's stop,
which Fireworks' OpenAI-compatible endpoint rejects with HTTP 400. It also
dropped disabled thinking instead of mapping it to reasoning_effort: none,
so the model spent its output budget on reasoning it was told to skip.
Resolves LIT-4798
* fix(guardrails): add /v1/messages support for Straiker plugin
- Pass prepared response data to Anthropic Messages streaming post-call hooks (litellm/llms/anthropic/chat/guardrail_translation/handler.py)
- Normalize Straiker request, tool, finish-reason, and mode fields across Chat Completions, Messages, and Responses APIs
* fix(guardrails): gate cross-surface message resolution and cover streaming request data
Resolve request messages only for surfaces that have a mapped translation
handler. The unguarded fallback tried every registered handler in turn, which
raised AttributeError out of the guardrail's error handling on list-shaped
`input` bodies, and synthesized a chat message that was never sent for bodies
it happened to parse.
Prepare request data on the mid-stream Anthropic branch as well, matching the
terminal branch and the OpenAI handler, so guardrails that scan before
end-of-stream still receive identity metadata.
Read usage from Anthropic dict responses so non-streaming /v1/messages reports
token counts instead of null.
Add regression coverage for the streaming request data on both the terminal and
mid-stream branches; reverting either now fails.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
The http_handler pair only received its full mutation verdict after the
first removal batch landed; these five tests pass unchanged when every
function they execute is mutated and the owning file killed none of
their scored mutants. The ssl tests excluded from mutation scoring are
untouched.
* fix: handle explicit outputInfo: null in Vertex AI batch response
Vertex AI can return HTTP 200 for a create_batch/get_batch call with an
explicit "outputInfo": null body (the output directory is assigned
asynchronously and may not be populated yet at response time).
_get_output_file_id_from_vertex_ai_batch_response did:
response.get("outputInfo", OutputInfo()).get("gcsOutputDirectory", "")
dict.get(key, default) only substitutes default when the key is absent,
not when it is present but explicitly None, so this crashed with:
AttributeError: 'NoneType' object has no attribute 'get'
surfaced to callers as an opaque openai.InternalServerError 500 from
litellm.create_batch()/retrieve_batch() for any Vertex AI batch job,
regardless of whether the job ultimately succeeds.
Fixed by guarding with `response.get("outputInfo") or OutputInfo()`,
matching the existing null-safe pattern already used by the sibling
_get_input_file_id_from_vertex_ai_batch_response for inputConfig. The
existing outputConfig fallback branch (a few lines below) already
handles this case correctly once it's reachable - it just never was.
Added 2 regression tests covering outputInfo: null with and without an
outputConfig fallback available.
* test: drop explanatory comment from regression test
---------
Co-authored-by: htourinho-clgx <htourinho@cotality.com>
The accumulated-JSON fallback ran json.loads over the whole buffer after every fragment and, on failure, kept the buffer without resetting it. A buffer that ever held more than one concatenated JSON value could never parse (json raises on trailing data), so it returned None on every subsequent chunk while growing without bound - an unrecoverable per-request CPU spin. Parse one value at a time from the front with raw_decode and keep the remainder, draining trailing values on later calls and at end of stream.
The async streaming error paths fired the sync failure_handler in a thread
and the async_failure_handler via create_task at the same time, so both
mutated the shared logging object concurrently and could crash pydantic-core.
Route failure logging through a single guarded dispatch_failure_handlers, so
the sync handler only runs after the async one completes.
25 test functions across three files pass unchanged when every function
they execute is mutated; the owning file killed zero of their scored
mutants. Four zero-kill tests tied to the fix in #31288 are kept for
rewrite instead of removal.
Extract the inline sync streaming post/decode into make_sync_call so it can be
exercised with an injected client, mirroring make_async_call, and add a sync
regression test that each token is forwarded after exactly one pulled frame.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Mirror the sagemaker_chat fix on the native sagemaker/ streaming path: the
sync and async completion handlers read the invocations-response-stream body
with iter_bytes(chunk_size=1024) / aiter_bytes(chunk_size=1024), so httpx
withholds bytes until 1024 accumulate and tokens arrive in gap-then-burst
waves. Drop the fixed chunk size so each decoded event is forwarded as its
bytes arrive.
Also add a boundary-agnostic decoder test proving frames reassemble correctly
regardless of where transport reads split the stream.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The huggingface embedding test fixture reloaded
litellm.llms.custom_httpx.http_handler, creating a new HTTPHandler class
object. llm_http_handler keeps the class captured at import time, so any
test running later in the same process that injects a client built from
the reloaded class fails the isinstance check and the mock is silently
discarded, causing a real network call. Under pytest-xdist loadscope this
surfaced as a deterministic failure of
test_accept_header_in_completion_request_jwt whenever an unrelated PR
shifted worker distribution.
Also removes the same reload pattern from the vertex rerank integration
test (both were previously removed in a6df01caec and resurrected by a
merge conflict resolution) and hardens the agentcore victim test by
dropping the bare except that swallowed the real error
* fix(anthropic): strip uniqueItems + other unsupported array/object constraints from output_format schema
Anthropic's structured outputs (`output_format`) validate the JSON schema
against a strict subset and reject cross-element / count constraints that a
constrained-decoding grammar cannot enforce, returning a 400
`invalid_request_error`.
`filter_anthropic_output_schema` already stripped the numeric / string /
item-count constraints (minimum, maximum, exclusiveMinimum/Maximum, minLength,
maxLength, minItems, maxItems) but still let these through:
- uniqueItems
- contains / minContains / maxContains
- minProperties / maxProperties
so a request using them fails with e.g. "output_format.schema: For 'array'
type, property 'uniqueItems' is not supported".
This is provider-visible: newer Claude models on the native `output_format`
path (e.g. `azure_ai`) 400, while `vertex_ai` is unaffected because it is
forced onto the permissive tool-use path (#18625 / #19201).
Add the missing keywords to the unsupported-field set and the description map,
and skip the advisory description note for a disabled boolean constraint
(`uniqueItems: false`) so it isn't misdescribed as required.
* fix(anthropic): serialize contains sub-schema in output_format advisory note
Address Greptile review: the `contains` advisory note previously discarded the
sub-schema, so the description only said an item must match "a schema" without
saying which. It now serializes the sub-schema as JSON (e.g. "array must
contain an item matching: {\"type\": \"integer\", \"const\": 1}"), matching the
other stripped constraints which carry their value. Sub-schema (dict/list)
values are json.dumps'd; scalar constraints are unchanged.
* style(anthropic): apply ruff format to output_format filter change
* style(test): ruff format anthropic schema filter tests
* test(anthropic): cover output_format array/object constraint filtering in test_litellm tree
Mirrors the schema-filter tests under tests/test_litellm/ so the coverage
job exercises the new uniqueItems/contains/min-maxProperties handling and the
uniqueItems: false branch.
---------
Co-authored-by: Darien Kindlund <darien@kindlund.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guardrails): add only_scan_new_messages for per-session incremental scanning
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): use fixed TTL constant and revert unrelated test formatting
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): run only_scan_new_messages in the unified apply_guardrail path
The initial wiring lived in BedrockGuardrail.async_pre_call_hook, but the proxy
routes Bedrock through the unified apply_guardrail interface, so the flag had no
effect live. Move incremental selection into apply_guardrail: filter the flat
texts list against per-session scanned hashes, skip the Bedrock call when nothing
is new, and mark hashes only after a successful (non-blocked) scan. Full-context
fallback is preserved when there is no session id, the cache is unavailable, or a
masking guardrail is configured.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover session-id fallbacks and mark_texts_scanned guards
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): fall back to full scan when incremental guardrail masks content, use shared cache
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover generic agent multi-turn incremental scan
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover incremental scan cache resolver fallbacks
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover flag interactions and /v1/messages incremental scan semantics
* feat(guardrails): make GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS env configurable
* test(guardrails): prove skip_system/skip_tool are enforced upstream of incremental scan
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted
Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400
Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag
Closes the Vertex/Azure gap from the customer RCA
Caller-supplied bedrock_tags land as AWS resource tags under the proxy's
AWS identity, letting an authenticated caller forge ownership or
cost-allocation labels. Add bedrock_tags to _BANNED_REQUEST_BODY_PARAMS
so per-request tags need general_settings.allow_client_side_credentials
or configurable_clientside_auth_params on the deployment, matching the
aws_bedrock_project_id precedent. Deployment-level bedrock_tags in
litellm_params are unaffected.
Also stop an explicit empty bedrock_tags list in litellm_params from
falling through to optional_params
