* fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it
* fix(bedrock): retry with the json tool when Bedrock refuses the native output schema
Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller
* fix(bedrock): make the native output retry send what an opted-out deployment sends
The retry after a native output rejection now reruns the same optional param
mapping completion() ran, with the deployment model_info opted out of
supports_bedrock_native_structured_output, so tool_choice, effort and the json
tool schema match an opted-out deployment instead of a hand-built tool. Each
config keeps only the predicate that decides when native output came from the
opt-in key.
Bedrock batch records for Anthropic models map with the opt-in turned off,
since a batch job has no retry. Models on supports_native_structured_output
keep native output in batch
* fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged
* fix(bedrock): give sync converse stream errors the upstream body as text
* fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising
A Converse json_schema request whose caller outputConfig failed litellm's
shape check raised a pydantic error before the request was sent, so the
caller got a 500 where the merge base passed Bedrock's 400 through. The
opt-in check now treats that outputConfig as carrying no native output.
Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of
native output, and extends the Bedrock native output integration tests.
* fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported
A caller can send its own outputConfig on Converse. When Bedrock refused that
schema on an opt-in deployment, the retry still carried the caller's textFormat,
so Bedrock got the same request twice and the warning told the operator to turn
off an opt-in that was not the cause. The retry is now skipped when the opted-out
request would still carry a native textFormat, so Bedrock's 400 reaches the
caller after one call as on the merge base. The opt-out advice is logged only for
the account-level "Extra inputs are not permitted" rejection.
* fix(bedrock): keep the caller's continuation message on the native output retry
The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt.
* test(bedrock): pin additional_drop_params on the /v1/messages native output retry
* fix(bedrock): give the Converse native output retry the fake stream timeout
* fix(bedrock): use native structured output on every Claude model whose Bedrock profiles accept it
Bedrock accepts outputConfig structured output on every reachable profile of
Claude Opus 4.7, 4.8, 5, 5.5, Sonnet 5, 5.5, Haiku 5.5, and Fable 5.1, so
json_schema now goes out natively there instead of as a forced tool, which kept
Bedrock from returning adaptive thinking. Fable 5 rejects outputConfig, so its
keys go back to the tool path. GovCloud keys keep their flags, and the drift
tests let GovCloud lag its commercial base but never lead it.
* fix(bedrock): clean Claude schemas for native structured output and keep recursive schemas on the old path
Bedrock native output rejects minimum/maximum, minItems, uniqueItems, oneOf,
additionalProperties true, and recursive or unresolved $ref, all of which the
forced tool accepted. Claude schemas on the Bedrock native paths now go through
the cleaner the Anthropic native path already uses, and a schema with a
recursive or unresolved $ref keeps the forced tool, inline, or drop path it had.
* fix(bedrock): keep older installs on the json tool with a Bedrock-only native output key
Older installs fetch the model map from main, so flipping supports_native_structured_output
moved them onto native output with their old schema handling. The new
supports_bedrock_native_structured_output key is ignored by older code. Schemas native output
cannot hold (recursive refs, open objects such as dict fields) keep the json tool on the new
key, models on the old key keep their current behavior for those shapes, and the Anthropic
API, Vertex AI, and Azure keep their current behavior.
* chore(cost-map): regenerate the model prices schema for the Bedrock native output key
* fix(bedrock): keep native output to profiles every account accepts and route schemas Bedrock rejects to the json tool
Only Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 keep the Bedrock native output key, since
Opus 4.8 and Sonnet 5 answer 400 on some accounts and Opus 5 garbles escaped characters.
Lookaround patterns and string formats outside Bedrock's list now keep the json tool, objects
typed as a union with null are closed, and the cleaner keeps minItems of 0 or 1.
* fix(bedrock): keep Opus 5.5 and Fable 5.1 on the json tool, since native output stops their adaptive thinking
* fix(bedrock): keep shared $defs in native schemas and send refused schemas to the json tool on every native model
Native output now keeps $defs instead of inlining each use, so a submodel reused across a schema counts once against Bedrock's optional and union limits. Schemas native output cannot hold use the json tool on Converse and Invoke chat for both native keys, and minItems is only kept when it is an integer 0 or 1
* fix(bedrock): keep the json tool unless thinking is requested and bound native output schemas
The Bedrock native output key now applies only when the call asks for thinking, so calls without
thinking keep the json tool as before. Schemas past Bedrock's documented union and optional counts,
or past a grammar size estimate fitted to live probes, keep the json tool on both map keys. Open
objects keep the json tool on the Anthropic API, Vertex AI and Azure AI, where native output returned
them empty. The $ref check no longer expands $defs, so doubling definitions no longer stall the request.
* fix(bedrock): resolve root-path $defs refs so nested Pydantic models keep native output on Invoke
* test(bedrock): cover native structured output and json tool routing through the proxy
Integration cells drive the proxy against a scripted Bedrock peer that enforces Bedrock's structured output schema rules, covering thinking with native output on Converse, Converse streaming, Invoke, /v1/responses and /v1/messages, the deployment opt-in, Fable 5, schema cleaning on every native entry, open objects, and malformed schemas. Vertex, Anthropic and Azure AI cells cover open objects on native-capable models and the cleaner's minItems and nullable object handling. The unit table now covers reasoning_effort none and disabled thinking
* fix(bedrock): keep native output for numeric formats and wide old-key schemas, and keep thinking in json mode
Check string formats only on nodes that can be strings, apply the 64-field cap only to the Bedrock-only key, carry thinking blocks into the json mode message, and send malformed json tool schemas as object schemas.
* fix(anthropic): keep native output for dict fields unless the json tool can be forced
* fix(anthropic): keep native output for dict fields when the caller sends tools
* test(bedrock): close the dangerous-tool-use test schema, since Bedrock native output requires closed objects
* test(anthropic): cover forced and unforced open-object routing, json mode thinking, numeric formats and wide schemas
Add proxy cells for open-object schemas that keep native output when the json tool cannot be forced (a model that refuses forced tools on the Anthropic API and Vertex AI, thinking, caller tools), for the thinking block a json mode answer keeps on Vertex AI and Bedrock Invoke, and for integer and number formats and 65-field schemas on each Bedrock map key. Point the forced open-object cells at the Anthropic API and Azure AI only, since no Vertex AI model in the cost map takes both native output and a forced tool
* fix(bedrock): keep the native output opt-in on the deployment that sets it, and route effort none open objects to native output
* test(bedrock): cover deployment-scoped native output opt-in, register_model and effort none open objects
The #46342 backport carried block_binding through every subclass of the
Anthropic and Bedrock Invoke configs, so Bedrock Mantle, Claude Platform and
OpenAI-compatible Anthropic endpoints started sending it. Each of the six chat
and messages configs now opts in by exact class, and the Bedrock Invoke
display updates and clear_thinking rebuilds carry block_binding only when the
config opts in. Every other config sends the same thinking as 1.105.1
* fix(proxy): authorize the public name of a team-scoped model on batch retrieve/cancel
A unified batch id resolves to the deployment's internal model_name
(model_name_{team_id}_{uuid}), which is in no key or team allowlist, so
authorize_model_for_key always denied and team keys got 403 retrieving or
cancelling their own batches. Authorize against model_info
team_public_model_name when the resolved deployment is team-scoped, falling
back to the resolved name otherwise.
Fixes#46958
* fix(proxy): authorize team-scoped batch models by their public name on both auth paths
* fix(proxy): accept the owning team's public name for its own team deployment on every auth path
Managed batch, file, fine-tuning, vector store, character and video ids resolve to the
team deployment's internal name again, so teams that granted the internal name keep access.
The model access check now also accepts the owning team's public name and its legacy team
aliases for that internal name, and only when the caller's team owns the deployment.
* fix(proxy): match a team deployment's public and alias names against the caller's own grants
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit e979bd0ecf)
* fix(anthropic): treat adaptive thinking as enabled when mapping response_format
is_thinking_enabled only recognised thinking.type=enabled, so a request that
paired thinking.type=adaptive with response_format took the synthetic JSON
tool path with a forced tool_choice, which Anthropic rejects alongside
thinking. The same check also gates the Bedrock Converse and Databricks
forced tool_choice downgrades, so they now see adaptive thinking too.
* fix(anthropic): keep signed thinking blocks with empty text on chat completions
Models that default to display: omitted return signed thinking blocks whose
text is empty, and Anthropic rejects the next turn when they are not replayed
unchanged. is_unsignable_thinking_block dropped them along with unsigned
blocks, and the inline content path required non-empty text, so a tool-use
turn on Opus 4.7+ or the 5 families lost its thinking blocks and 400ed on the
tool-result turn. Only a missing signature now drops a block, and the
mid-conversation system placement mirrors that rule. The /v1/messages strip
helper is left to #41362.
* fix(anthropic): skip the thinking-strip retry when block_binding asks for an error
The /v1/messages retry loop strips thinking blocks and resends on any
invalid-thinking 400, including the binding mismatch a caller opted into
with thinking.block_binding.prefix_mismatch_behavior: error. The retry
decision now sees the request body and surfaces that 400 instead of
overriding the caller's choice. #42888 covers the litellm_params side.
* fix(bedrock): carry thinking.block_binding through the Invoke thinking rebuilds
Bedrock Invoke rebuilt the thinking param from scratch when display: updates
was requested, on both the /v1/messages and chat completions paths, and the
legacy enabled-to-adaptive translation plus the clear_thinking injection did
the same, so thinking.block_binding never reached Bedrock. Every rebuild now
spreads the caller's thinking and only overrides the keys it owns, and
AnthropicThinkingParam types block_binding. The thinking-binding-controls
beta was already on the Bedrock allowlist.
* fix(anthropic): keep forcing the response_format tool under adaptive thinking
This reverts ad9e861ae7. Anthropic, Bedrock Invoke and Vertex all accept a
forced tool_choice next to thinking.type=adaptive and only reject it with
type=enabled, so counting adaptive as enabled fixed no live failure. It did
drop the forced JSON tool, so Claude answered response_format requests in
prose, and it downgraded a required tool_choice to auto on Bedrock Converse
* test(e2e): drop the adaptive thinking tool round trip case
With adaptive thinking the model often skips thinking on the weather prompt, so
turn 1 carries no signed empty block and the case fails before the replay it is
meant to cover. Anthropic also answers the replay with a 200 whether or not the
block is forwarded, so the case could not tell the fix from main. The factory
unit tests pin the outgoing shape instead
* fix(anthropic): only skip the thinking-strip retry for a binding mismatch 400
prefix_mismatch_behavior: error asks Anthropic to reject a block bound to
another conversation. It says nothing about unrelated bad blocks such as a
null signature, so those still get the strip-and-retry self-heal. Also trims
the empty-thinking docstrings to what the code guarantees and types the new
test parameters
* refactor(anthropic): narrow thinking fields with the mapping validator instead of typing.cast
typing.cast is banned by the strict ruff gate (TID251), so the three new casts pushed that rule one over its merge-base count. The existing pydantic mapping validator gives the same typed view without an unchecked cast, and it now has a generic name since it is no longer only used for Claude Code payloads
* refactor(anthropic): read thinking settings through the validated mapping so basedpyright stays at its merge-base count
The Bedrock clear_thinking rewrite still narrowed with a cast and an untyped dict fallback, which added two reportUnknownVariableType errors. It now uses the same pydantic mapping validator, made public for that reuse, and the retry hook validates the request body itself so the handler passes a typed dict instead of an untyped one
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
(cherry picked from commit f470e418ea)
The request-side sanitizer on /v1/messages dropped every thinking block whose
text is empty, even with a valid signature. Claude Opus 4.8 / Fable 5 / Fable 5.1
return exactly that shape by default (thinking.display omitted): the signature
is the preserved reasoning. Replaying the assistant turn as returned therefore
lost the model's reasoning on every multi-turn request, silently (the provider
answers 200 without it), and left preserved-thinking enforcement
(thinking.block_binding) nothing to bind.
Anthropic accepts an empty thinking block that carries a signature and binds it
to its prefix; only an empty block with no signature (the LIT-6357 case, a turn
a non-Anthropic model served through the bridge) is rejected with "each
thinking block must contain thinking". Use the signature-aware predicate
is_empty_unsigned_thinking_block, which #38809 already applied on the emit
side, for the request side too. The unsigned case is still dropped.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 897bfea38d)
* fix(responses): fold bridged Responses output into one chat choice
A Chat Completions request bridged to /responses got one choice per
Responses message item, so a reply shaped [message, message,
function_call] put the tool call on choices[1] and clients reading
only choices[0] never saw it. Fold every message text, annotation,
reasoning item and tool call into choices[0]
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop mutable annotations and the extra cast in the choice fold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): remove a cast-ok marker that suppressed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): join folded message texts with a newline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(responses): join folded message texts with no separator so stream and non-stream match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 4b7b8efd4c)
* revert(responses): revert "fix(responses): keep gpt-5.4/5.5 tool calls on chat and merge bridged tool calls into one choice" (#44295)
This reverts commit ca1994e403.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): merge bridged tool calls into the same choice as the text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 5724117116)
* fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): streamed alias matching a capability rule bills the deployment price
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): assert every streamed chunk carries the client alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): log the client alias on the priced streamed response, the same as non-streamed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 7b432d78d2)
* fix(proxy): stop queued registry read-throughs spending the resync budget
RegistryReadThrough.attempt serializes misses behind one lock, but every
request that was queued behind the first one still spent a unit of the
20-per-5s resync budget and re-ran the DB resync, even though the first
request had already loaded the object. A burst of more than 20 requests
for a model created on another worker therefore exhausted the budget and
the rest got 400 "Invalid model name".
attempt now checks whether the key is already loaded once it holds the
lock and returns early without touching the budget. Models check the
router's model names and deployment ids, guardrails and agents reuse
their existing registry lookups.
* test(proxy): gate the queued read-through test on events and cover each registry's loaded check
The queued-requests test now holds the first resync on an asyncio.Event instead of
a timed sleep and records calls in a recorder with tuple and frozenset state. New
tests show the model, guardrail and agent read-throughs each answer an object that
is already loaded without reading the database, so rewiring any registry's loaded
check now fails a test
* test(proxy): keep the queued read-through recorder inside its test and type the agent registry fixture
(cherry picked from commit b877a38e5f)
The decision model guardrail backport (#45894) landed on the line after this branch
forked and imports supported_decisions_providers from litellm.decisions.main, which
#45846 replaces here. Match main by exposing it from litellm.decisions.call and point
the guardrail endpoint and its test at that module. schema.d.ts and the lazy OpenAPI
snapshot are regenerated on the merged tree
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Ports the pytest_runtest_call hook from tests/unit/conftest.py on main (#44146). Without it the asystemone call type test picks up adecisions callbacks queued by earlier tests in the same worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
schema.d.ts is regenerated with npm run gen:api instead of hand-merged, the Strands logo the Add Model list imports is copied from main, and the picked tests get the import order and Final import this line's ruff asks for
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): add litellm.systemone and trim litellm.decisions to the OpenAI shape
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run tests/unit/systemone in the core-utils unit shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): keep litellm.decisions(state=...) working with a DeprecationWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make tests/unit/systemone a package
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): surface the litellm.decisions(state=...) deprecation as a UserWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 25d9b05386)
PR #45667 removed openrouter/typesafe/jev-1.13 from the cost map after #45640 moved OpenRouter decisions billing to usage.cost. The OpenAI-format wire test still read the removed row and raised KeyError on CircleCI integration-providers
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 5894968c5d)
* docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI
Add summaries, descriptions, inlined request body schemas with examples and typed 200 responses to the
decision routes and their aliases, so the OpenAPI document and the lazy snapshot show both request formats
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): cap _inlined schema walk depth for the recursion check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit bc2cc42340)
* feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN
A strands_decider deployment whose api_base is a bedrock-agentcore runtime ARN now
calls InvokeAgentRuntime on that runtime, signed with SigV4 from the deployment's aws_*
params or the default chain, reuses one session per runtime, and maps the runtime's
200 error envelopes to 400, 503, or 500
* ci: run the strands_decider unit tests in the llms job
* fix(strands_decider): take the AgentCore session id from the deployment, never the caller
A caller could pick the runtime session through extra_headers, and every new id starts another microVM. The session id now comes from the deployment's agentcore_runtime_session_id (banned in request bodies) or the ARN default, and caller-supplied host, x-amz-*, and x-amzn-bedrock-agentcore-runtime-* headers are not forwarded
* fix(strands_decider): map runtime error replies only on AgentCore ARN calls
* fix(strands_decider): sign plain decisions calls inline, off the AWS signing executor
* test(strands_decider): cover AgentCore runtime ARN calls in integration wire tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit ed9c31cfa8)
* feat(decisions): add the OpenAI Decisions spec types and the System One translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): share one DecisionsModel config and require model and usage on responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): rename the shared pydantic parent to DecisionsObjectBase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): switch /v1/decisions to the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): build the usage the OpenAI response schema now requires
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lens): ask signal questions through the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): refuse safety_identifier for System One providers unless drop_params
* feat(decisions): refuse safety_identifier on providers that cannot take it unless drop_params drops it
* test(decisions): cover hosted_vllm in the safety_identifier and per-provider wire tests
* fix(decisions): check a System One request's safety_identifier against the provider too
* fix(decisions): drop a non-string safety_identifier under drop_params
A malformed safety_identifier now follows the drop_params convention in both body shapes: it answers 400 without drop_params and is dropped before the provider call with it. A string identifier on OpenAI stays on the wire either way.
* fix(decisions): let /v1/decisions drop a non-string safety_identifier under drop_params
The route checked the whole body before routing, so a non-string
safety_identifier answered 400 even when the deployment or the body set
drop_params. The route now leaves that field to the Decisions call, which
knows every drop_params source
* refactor(decisions): return the unsupported safety_identifier as a value and raise it in _prepare_call
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 546402c98c)
* feat(azure_ai): support Microsoft-Decision-1 on the decisions API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): strip project path and API suffix together in decisions base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 2fcc780498)
