* fix(bedrock): forward Codex agent messages that carry encrypted content intact
Codex multi-agent v2 sends worker messages as agent_message items whose
encrypted_content block only the model server can decrypt. Bedrock Mantle and
Bedrock Runtime used to 400 on agent_message, so LiteLLM rewrote every one into
assistant text, pasting the ciphertext into the prompt and dropping the sender
and recipient. Bedrock now decrypts these items itself, so the rewrite left the
recipient model reading an opaque token instead of its task.
Agent messages with an encrypted_content block now go to Bedrock untouched on
both endpoints. The Bedrock configs declare supports_encrypted_agent_messages,
and the shared normalizer keys off that flag so the two cannot drift. Mantle's
/v1/responses route for open-weight models still rejects agent_message in any
form, so it keeps the old rewrite. Plaintext-only agent messages keep the
rewrite on every route.
* fix(bedrock): only keep worker messages locked for OpenAI GPT models
Gemma and Grok on Mantle's OpenAI route never lock Codex worker messages, so
Codex puts plain text in the locked slot and Bedrock rejected it with
invalid_encrypted_content once the route alone decided to forward it intact.
Decide from the model instead, and restore the full-value rewrite assertion.
* fix(bedrock): run realtime guardrails on Nova Sonic sessions
* fix(bedrock): quote the spoken violation message and skip it after a blocked Nova Sonic tool output
* fix(bedrock): screen every message role, end Nova Sonic sessions on blocked transcripts, drop unparseable frames
* fix(realtime): apply guardrail masks, keep block reasons out of spoken notices, fail closed on guardrail crashes
* fix(realtime): screen joined text parts, keep the documented notice, honor key opt-outs
Text parts of one message are joined before screening again, so a phrase split
across parts is still blocked, and a mask collapses them into one part. The
spoken notice is realtime_violation_message or the block reason, quoted only on
Nova Sonic. Client-written output_text and text parts are screened. Key and team
guardrail opt-outs reach regular and translation realtime sessions. A non-string
upstream transcript skips screening as before.
* fix(realtime): honor guardrail role scope, clear stale response.create drops, screen split keywords, report Nova guardrail outages
* fix(realtime): keep every per-part mask when a joined text masks differently
* fix(realtime): screen one joined text per item and pass the session's tenant context to guardrails
Multi-part items reach each guardrail once as the space-joined text, as on main, and only an item a
guardrail rewrites gets its parts screened again to keep per-part masks. Guardrails now receive the
session request data with its metadata, so key and team policies apply, and a ModifyResponseException
block is reported as a block. Nova scopes every message item as the user turn it sends to Bedrock.
* fix(realtime): catch phrases split mid-word across text parts without re-screening single-part items
* fix(realtime): screen each joined text in its own guardrail call and keep a Nova opening message for the retry while it is screened
* fix(realtime): screen text parts in a loop so large messages keep their masks
* fix(realtime): fail closed on a non-string transcript under a guardrail and give each turn its own guardrail metadata
* fix(realtime): fail a non-string transcript only when a guardrail would screen the user's turn
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(bedrock): enable native structured output by default for selected Claude models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep govcloud and bare model rows off native output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected native schemas with the JSON tool on default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep the rejection retry on the unforceable-tool native path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected schemas on the Messages path for default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format Messages native output retry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): allow profile-only native output metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): only disable default-native output on schema-rejection retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type fixture parameters in native output retry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop strict tools after a native-output retry hits the grammar limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format native output fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool
A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too.
* fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action
A role allowed only bedrock:InvokeModel got a 403 on schema streams once these
models went native, since native streams call converse-stream while the JSON
tool path buffered through converse. The retry now sends that request again
as a buffered JSON tool stream, the shape the base sent.
The wire test moves its neither-key and opt-in cells to Opus 4.7, its
Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the
bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native.
The batch wire test now checks a thinking schema record on both sides of the
contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5
and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8
cells assumed those models were opt-in.
* fix(bedrock): move the Claude 5 native output default to a key released proxies do not read
Released proxies load the cost map from main and have none of the native
output retries, so setting supports_native_structured_output on the Opus
5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide
or min/max schemas into 400s on v1.108.1.
Those rows keep supports_native_structured_output false and gain
bedrock_native_structured_output_by_default true, which get_model_info
reports as supports_native_structured_output, so this version behaves as
before while released versions keep the JSON tool.
* fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default
The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it
* fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key
A registered supports_bedrock_native_structured_output false now beats the
native output default, and batch uploads read the deployment's model_info, so
an opted-out deployment keeps the JSON tool on every record. The Opus 5 and
Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default,
so released proxies on main's cost map keep the JSON tool on them. The 64
property batch cutoff now applies only to rows with that key, so models that
were already native keep their batch output shape.
* fix(proxy): report a deployment's Bedrock native output opt-out on /model/info
/model/info merged the deployment's model_info over the backend row, so a
deployment that set only supports_bedrock_native_structured_output: false
still showed the backend's derived supports_native_structured_output: true.
The generic flag is now derived from the merged info unless the deployment
set it itself, through the same helper get_model_info uses.
* fix(bedrock): keep native output when a denied schema stream is resent buffered
A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output.
* fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend
* fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment
The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model
* fix(bedrock): answer a buffered invoke stream from the non-streaming reply
* fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected
* fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info
A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main.
/model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): do not surface server_tool_use as client tool_calls
Anthropic server-executed tools (web search, tool search, code execution)
were returned in chat completions tool_calls with srvtoolu_ ids, so
OpenAI-protocol agent clients treated them as pending work and re-invoked
the model until their recursion limit.
Move them to provider_specific_fields.server_tool_calls on both the
non-streaming and streaming paths. History replay, the /v1/responses
bridge and the /v1/messages streaming guardrail read that field, so
multi-turn reconstruction, web_search_call output items and the
guardrail's fail-closed check on server tool blocks keep working.
* fix(responses): return Claude web_fetch calls as web_search_call open_page items
* fix(responses): keep Claude server web calls without a result as hosted items
A non-streamed Claude turn that calls web_search or web_fetch next to a
client tool can return the server call before its result. Build an
in_progress web_search_call for it, so the Responses bridge no longer
returns it as a client function_call
* fix(anthropic): keep pending server calls native and hide server calls from client tool paths
Replaying a server call that has no result yet now sends it back as
server_tool_use, so Anthropic accepts the next turn. The tool permission
guardrail checks server calls again and rewrite mode removes a denied one.
The /v1/messages adapter keeps Claude's server blocks, and the Responses
bridge never returns a server call as a client function_call.
* fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls
Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls,
so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to
output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage
and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending
Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own
server_tool_use instead of running again
* fix(anthropic): let guardrails and loggers read Claude's server tool calls again
Guardrails, tool policies, tracing loggers and the Responses guardrail
translation read only client tool calls after server calls moved to
provider_specific_fields. They now read server calls first, then client
calls, through one shared helper, and the Responses guardrail treats
Claude's web_search_call items as web_search or web_fetch calls.
Non-streaming splits server calls by block type, as streaming does.
* fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing
A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy,
streamed or not. Redaction with message logging off hides server call arguments and hosted web
call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming
guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls.
* fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge
Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use.
* fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses
Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy.
* test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions
* test(guardrails): type the fake Pillar client like its Protocol
* fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool
Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or
web_search still needs that call and its thinking replayed, or Claude loses the fetch result
* fix(guardrails): scan replayed Claude server calls and hosted web calls
Request guardrails now see and rewrite server_tool_calls replayed in chat
messages. Cato and Purview scan Claude web_search and web_fetch calls on
Responses output, and a function tool call with an extra custom key is no
longer dropped from Cisco, Purview, Repello and PANW scans.
* fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks
* fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows
* test: annotate the new server tool call test locals with Final and type their parameters
* test: type the new guardrail test helpers with litellm's tool call types instead of Any
* test: narrow the remaining new test helper parameters from bare dict and object
* fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs
Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls
* fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera
* fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock/converse): omit reasoning signature for non-Anthropic models (#49969)
* test(bedrock): cover signed reasoning replay in mapped tests
* fix(bedrock): drop another model's signed thinking when replaying to a non-Claude Converse target
A signed thinking block from Claude replayed to a Converse model known not to be Claude, such as GPT-6 Luna or Nova 2, now leaves the request instead of going out as reasoningContent. Bedrock rejects that field on those models, so the whole request answered 400
A target that may still be Claude, such as an application inference profile ARN with no base_model, keeps the signed block, since Claude needs it on a tool-use continuation. Unsigned blocks keep the existing text path
The thinking-block error matcher added earlier on this branch is removed, because Converse and non-Claude Bedrock requests never reach that retry
* fix(bedrock): keep signed thinking for Claude behind a region prefix litellm does not list
ca. and in. inference profiles name Claude after a region prefix get_bedrock_cross_region_inference_regions does not have yet, so they counted as non-Claude and lost their signed thinking. The thinking-only turn test now pins the exact merged and continue-message shapes.
* fix(bedrock): limit the region-prefix Claude match to signed thinking and drop a dropped block's cache point
* fix(bedrock): drop Claude's bridged reasoning items before a Bedrock Responses API replay
* fix(bedrock_mantle): drop litellm-minted reasoning items before a Mantle Responses call
* test(integration): expect Bedrock and Bedrock Mantle Responses to drop minted reasoning
Bedrock and Bedrock Mantle answer 400 on a reasoning item litellm minted, so the
wire test now expects both to drop it, like OpenAI and Azure
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The primary stream's mid-stream stall reaches failure_exits as a mapped litellm.Timeout (408), which cools the deployment on the Rust core. Keep that behavior and update the integration test that expected the Python core's exemption. Fallback hops stay exempt through is_stream_stall.
* feat(interactions): continue requires_continuation long decodes with continuation_token hops
A foreground Interactions request stops at Google's per-hop output cap with
status requires_continuation and a continuation_token. litellm now re-posts the
request with previous_interaction_id and continuation_token until the status
moves on, merging the hops into one interaction (last id, every hop's steps,
summed usage with initial_input_tokens from hop 1), for non-streaming and
streaming calls. continuation_token is accepted as a request param, and
requires_continuation is a known status for billing.
* refactor(gemini interactions): validate streaming chunks with model_validate
* fix(interactions): keep the held completion across stream hops, stop on a repeated continuation_token, follow the spec redirect
* fix(interactions): add grounding_tool_count per tool type, stop on any seen continuation_token, keep streaming past a [DONE] frame when a hop is pending
* ci(interactions): point the spec canary at aistudio.google.com, where Google moved the Interactions OpenAPI spec
* test(interactions): scripted Google ends each long decode hop with a [DONE] frame
* refactor(interactions): pass requires_continuation and continuation_token through instead of looping hops in the proxy
The caller drives the continuation, the same as against Google directly. /interactions accepts continuation_token in place of input so a resume request needs no input.
* test(proxy): assert the accepted /interactions bodies pass the required-param check
* fix(proxy): keep the router default fallback for input on /interactions, continuation_token stands in for it
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): stop forcing the response_format json tool when Claude will think
A forced tool call skips Claude's thinking. The json_schema emulation forced
json_tool_call unless thinking.type was enabled, so adaptive thinking and the
models that think by default (Opus 5, Sonnet 5, Haiku 5.5, and the always-on
families) lost their thinking on every structured-output request.
AnthropicModelInfo.response_format_tool_can_be_forced is now the one check
the Anthropic, Bedrock Converse (including the native output fallback) and
Databricks paths use. It leaves the tool unforced when thinking is requested
in any form or the model thinks by default, and keeps forcing it when the
request turns thinking off (thinking disabled or reasoning_effort none) on a
model that accepts forced tools.
The new thinking_on_by_default cost map key marks Opus 5, Sonnet 5 and
Haiku 5.5 on the providers that reach these paths, and a miss-only fallback
rule covers unmapped Claude 5 and later ids.
* fix(bedrock): route native output and thinking token counts by whether Claude will think
Bedrock's native output opt-in now uses the same will-think check as the json tool, so Claude 5 models that think by default with no thinking param go native. Converse also asks for /usage/output_tokens_details on those models, so reasoning tokens are reported for them.
* fix(anthropic): tell Claude to answer through the unforced response_format tool
Moves the integration tests that check the forced JSON tool from Claude 5 models to Opus 4.8, since Claude 5 models now keep their default thinking and take the unforced tool or native output
* fix(anthropic): keep open-object schemas on the json tool when Claude will think
Native output closes every object, so an open object came back empty. Also declare thinking_on_by_default in the Rust model catalog and expect the Converse thinking-token field in test_main
* fix(anthropic): hold streamed prose until the JSON tool decides the answer
In json mode with the unforced JSON tool, Claude can write its working before it calls the tool. The stream now holds that text, drops it when the JSON tool answers, and sends it when Claude calls a caller tool or skips the JSON tool, matching the non-streamed reply.
* test(anthropic): build the stream test events without nested comprehensions
* fix(anthropic): keep always-on thinking models unforced, open objects native when thinking, and the tool instruction Claude-only
* test(anthropic): keep main's open-object integration cases and cover the unforced tool on a default-thinking model
* fix(anthropic): treat thinking False as thinking off so the JSON tool stays forced
Cover thinking False on the Anthropic and Converse forcing tests, and pin that a Converse
json_schema stream with the unforced JSON tool never sends Claude's working text as content
* fix(anthropic): drop the thinking False forcing case that completion never reaches
litellm.completion turns thinking False into no thinking param before any route sees it,
so the case was dead on the request path and False keeps meaning the model's default
* test(bedrock): collect the Converse json_schema stream chunks in one assignment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(nvidia_riva): send the api key once per call by talking to Riva over grpc directly
Drops nvidia-riva-client from the stt-nvidia-riva extra. Depending on its version it sent
the bearer token zero times on plaintext or twice on TLS, so token-checking Riva deployments
rejected every request (#50059). The provider now declares the few Riva messages it uses,
opens its own grpc channel and puts authorization and function-id on each call exactly once
* test(nvidia_riva): run the provider against a real grpc server built from nvidia's own definitions
* fix(nvidia_riva): validate endpointing settings with pydantic before building the riva message
* test(nvidia_riva): check the key is sent once over tls, and drop a comment that restated the code
* feat(decisions): add EmpirioLabs AI as a decisions provider
empiriolabs/aplomb-1 now works with litellm.decisions and litellm.systemone, the proxy /v1/decisions and /v1/systemone routes, the decision model guardrail and the Admin UI. Requests go to POST /v1/decisions on the EmpirioLabs AI API with the System One body, keyed by EMPIRIOLABS_API_KEY with an optional EMPIRIOLABS_API_BASE
Chat models keep routing through the existing OpenAI-compatible JSON provider, and a test pins that
* ci: run the EmpirioLabs AI decisions tests in the llms unit shard
* fix: align EmpirioLabs registration with CI gates
---------
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages
Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.
An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.
* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping
* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame
The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529
* test(proxy): expect the top-level Anthropic envelope from count_tokens errors
count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.
* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets
* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status
An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.
* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status
* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload
* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases
* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status
* fix(proxy): walk nested mid-stream fallbacks without recursion
* fix(anthropic): preserve 503s and answer upstream 529s on messages routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use integration coverage for keepalive errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost-map): remove the 52 rows whose provider retired them before 2026
* test(utils): pin the proxy function-calling parity case to azure/gpt-4.1
* test(batches): price the Azure batch-logging regression case against gpt-4.1
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API
* fix(logging): log bridged chat completions with the chat response they returned
* fix(logging): log bridged text completions and /v1/messages as served, with the request's options
Bridged rows were translated without the request's options, so structured
output and replay fields could differ from what the client got. Text
completions and non-streamed /v1/messages through the chat bridge also kept
a Responses API object or the encoded id.
* fix(logging): redact reasoning summaries on bridged rows and log bridged text completions as text completions
Redaction now drops reasoning_items on chat messages and deltas, so a bridged row keeps no reasoning summary when message logging is off. A non-streamed bridged text completion logs the text completion the client got, priced like the chat response it came from. The bridge handler's input TypedDict fields are ReadOnly.
* fix(logging): redact bridged text completions for opted-out callbacks and pin the sync logged shape and usage ack
* test(integration): look up bridged chat spend rows by the id the client got
Bridged chat spend rows are keyed by the client-facing id and logged with the caller's call type, so the helpers that decoded the old resp_<base64> key or pinned call_type responses now assert the new contract
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry
Two deployments that share one model string share one key in the live
litellm.model_cost map, and router registration writes each deployment's
supports_prompt_cache_breakpoint into that key. The cache control hook read
the live map, so a flagless deployment picked up a sibling's opt-in and sent
prompt_cache_options and prompt_cache_breakpoint upstream, which the provider
rejects with a 400
When a call carries the routed deployment's model_info, the prompt cache
checks now use its declared supports_prompt_cache_breakpoint, and fall back
to the loaded catalog instead of the live map. This covers the hook on
completion, acompletion, Responses and the /v1/messages route, the azure_ai
chat transform, the OpenAI Responses transform under drop_params, and the
/v1/messages to Responses adapter
* fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers
A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's
prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a
provider other than openai, such as azure_ai OpenAI models and bedrock_mantle.
Bedrock on every route and providers that build Anthropic messages keep
cache_control, which Bedrock Converse turns into cachePoint
The openai chat reader now passes the routed model_info and checks base_model.
The Bedrock cache point reader takes the deployment's flag first and, for a
routed call with no flag, reads the catalog as loaded instead of the live map
that sibling deployments write into. model_info is threaded from the converse
transform, the factory message and tool builders, and the responses bridge
* fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls
A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations.
* fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways
* fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases
* test(anthropic): cover auto caching on an unmapped Claude deployment
* fix(cache_control): keep the merge base's reads where the per-deployment flag changed them
A register_model entry under a provider-prefixed name reaches routed calls again, since the
overlay now resolves the bare and prefixed names the way runtime registrations are stored.
The hook and the openai chat reader no longer judge base_model, as at the merge base, so a
client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint
and get a 400.
A Bedrock deployment flag that is not a real true reads as false again, as at the merge base.
An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again,
as at the merge base. Other wildcards still leave both cache flags to the catalog.
The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag
on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps
its own flag and that a flagless unmapped ARN keeps its cache points after the delete.
* fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name
Routed calls read register_model entries through an overlay that merged the bare
name after the provider-prefixed one whatever order they were registered in, so a
later opt-out under one name lost to an earlier opt-in under the other. Each
registration now moves its key to the end of the kept registrations, and the
overlay merges both names oldest first, which matches what SDK calls read from
the live model map.
Also lists the wildcard cache flag helper in the router coverage script, since
the hook's wildcard tests drive it rather than calling it by name.
* fix(cache_control): keep register_model entries visible while a cost map reload replays them
The previous commit moved a re-registered name to the end of the saved
registrations by removing and re-adding it, and the overlay walked that dict to
order a bare and a provider-prefixed name. A reload replays every saved entry,
so a request on another thread could briefly miss an entry or fail on a dict
that changed size mid-walk. Entries are written in place again, a per-name
sequence records which registration came last, and both the overlay and the
reload replay order by it, so the later registration still wins after a reload.
* fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix
A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos
A provider registered in custom_provider_map with a BaseVideoConfig handler now resolves on every
video call type (create, status, content, list, remix, edit, extension, characters), the same way
CustomLLM handlers already resolve on /v1/images/generations. The video entry points and the HTTP
handler stop casting the provider name to LlmProviders, which raised before any lookup ran, and a
provider without a video config gets a 400 instead of a 500.
* fix(videos): type the custom provider lookups so the type gate holds
Pick the CustomLLM for a custom provider through ProviderConfigManager.get_custom_llm_handler,
which narrows the now wider custom_provider_map handler union, read custom_provider_map instead
of the private _custom_providers list, and annotate the untyped custom_llm_provider parameters
on the video entry points
* test(videos): cover a CustomLLM shadowing a built-in video provider name and type the new tests
Mark custom_handler ReadOnly on CustomLLMItem, type the new tests, and assert that a CustomLLM
registered under openai still leaves openai video on the built-in config
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): reproduce comparison response ID ownership and wire failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): guard stale retries and malformed comparison envelopes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): authorize and unwrap comparison response ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): require comparison IDs to follow response security policy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): deny raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): reject raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): enforce managed ownership with unmanaged ID opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool
Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking
sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model
map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The
GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows
* fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool
Native output without thinking now applies to every Bedrock Claude model that cannot
force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5
needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose.
The invoke chat path keeps the native output_format it built instead of inlining it, and
GovCloud region-path ids read their own false rows.
* fix(bedrock): clean caller output_format schemas before sending them natively on invoke
* fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke
* fix(bedrock): keep the faked Converse stream for native output that replaces the json tool
A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture.
* test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): gemini tool-call signature rejected by the backend is resent once with the placeholder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): keep the placeholder resend inside the cache rebuild so either rejection order ends in an accepted body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): resend a cached-prefix signature inline and cover native generateContent
When the rejected signature sits inside a cachedContents prefix, the
placeholder resend now carries the history inline, since replacing the
signature in the request body cannot reach the cache. The native
generateContent handlers get the same one-time resend.
* fix(gemini): send the history inline when Vertex rejects a signature while creating the cache
* fix(gemini): keep the context cache on a signature resend and go inline only when that is rejected too
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): log the encrypted response id on streamed /v1/responses spend rows
With response id security on, the stream hook encrypts the resp_ id on every
frame, but logging had already received the SDK's own copy of the terminal
response, so the spend row, the standard logging payload and callbacks got the
managed id (or the chatcmpl id on a bridged stream) the client never saw. The
hook now records the exact id it advertises on the request's logging object,
and the logging layer stamps that id on the logged streamed response
* fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id
Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the
id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new
session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes
to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response,
and a raw provider id still gets a managed id
Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or
<id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's
request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The
background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id
* fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner
A GET or cancel of a stored response now logs a new encrypted id of its own,
so callbacks and spend rows never reuse the create's id, while the owner of
the id still gets back the exact id they sent. An admin reading someone
else's response gets an id bound to the admin, as before. The spend row id
suffix for reads and cancels is gone, so with response id security off the
rows and session history match the default branch. A streamed GET follows
the same rules.
* fix(responses): echo the sent id to every reader that passes the response access check
A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed
the access check, not only when the caller is the user or team the id was issued to. An admin, the master key
or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not
reach the first turn from it, so a bridged follow-up through that id lost the conversation history
Every read and cancel still logs a fresh id bound to the reader, never the create's id
* fix(responses): keep a read or cancel row in session history only when it brings new output
A GET, streamed GET or cancel sent with a session header joins the
create's session, and its stored response repeats the create's output.
The replay then sent the same assistant turn once per read.
Session history now keeps an aget_responses or acancel_responses row
only when its stored Responses output has an item id that no earlier
kept row carried. A polled background response keeps the first poll
that returns the finished output, since its queued create row has none,
and later polls of the same output are dropped. Every other row keeps
today's rule
* fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart
A create row now owns its output items in session history, and a read or cancel row adds only items no
create and no earlier read carried, in one linear pass, so a background response polled while it grows,
or a read that ends before its streamed create, replays the answer once and after its prompt
DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id
like a read does
Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row
id, so a streamed background create and its cost event, which now share the client's id, keep separate
files
The integration tests that decoded the managed id now match the exact id the client received
* fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events
A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh
encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET
Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage
file and unbatched GCS object, so batch retrieves and events with no id keep main's names
An item a session read replays takes the newest read's copy at the first read's place, so an item a
poll caught half written replays finished
Tests pin the client's id over the router's continued response id, and the storage test no longer reads
the clock after the upload
* fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items
A streamed background create and its later cost event share the client's
id, so OpenMeter dropped the second event and DynamoDB overwrote the
create's item. Both now key the cost event as <id>_background_cost, like
its spend row and storage file.
Session history again replays a read's output items that carry no id,
as main did.
* fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras
* fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id
A streamed background create and its later cost event now share the id the
client received, so ClickHouse and DeepEval saw both events under one id.
Route both through logged_event_key, the way the Azure storage, GCS,
OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse
request_id matches its spend row's `<id>_background_cost` and its DeepEval
span gets its own uuid. ClickHouse response_id stays the raw provider id.
* test(deepeval): annotate the background cost span test's locals with Final
* test(responses): run the DELETE wire test that pins the echoed client id
DELETE /v1/responses/{id} now answers with the id the caller sent, so the
skip that marked it as a known bug no longer applies. The test passes on
the /v1, /openai/v1 and bare routes.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests like their chat templates
Render grok-4-1, grok-4.3 and DeepSeek V4.1 tool schemas the way each chat
template does, and add a per-deployment token_count_tool_framing next to
token_count_framing for the hidden tool preamble a host bills. The Azure AI
Foundry grok-4-1, xAI grok-4.3 and Fireworks DeepSeek V4.1 cost map rows carry
the measured framing, overridable in model_info
* fix(count_tokens): count Azure grok-4-1 Responses requests with their own framing and use the tool framing in place of the base framing
* fix(count_tokens): keep operator framing on tool and Responses counts, frame bridged Responses like chat, follow base_model for tool rendering, count grok tool_choice
* fix(token_counter): count Azure Responses strict default, skip false strict and built-in-only tools, per-kind operator framing
* fix(token_counter): count an explicit null strict like false on Azure Responses
* fix(count_tokens): follow the chat completions bridge flag and Azure's strict default on azure/ deployments
Keep Azure grok-4-1 chat tool counts at or over the bill by raising its tool framing to 302
* fix(token_counter): keep strict false for non-grok templates, coerce the bridge flag, frame DeepSeek API counts
* fix(proxy): frame Responses counts for the api_base router defaults send inference to
* fix(proxy): keep the api_base helper's name so main's /v1/messages count caller resolves
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): resolve Invoke Claude default max_tokens from the base model id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): build Invoke Claude request params without mutating
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): type tool-search normalizer return to clear unknown-type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): resolve Invoke and Mantle Claude default max_tokens from the deployment id, ARN base_model, and Mantle row
* fix(bedrock): default Invoke Claude max_tokens from the deployment's own cap and honor AnthropicConfig
Read the called deployment's model_info.max_output_tokens before the base model's
cost row, skip the shared regional key the router fills from every deployment, and
leave the default to litellm.AnthropicConfig(max_tokens) when it is set.
* fix(bedrock): send the mapped default max_tokens on bedrock_mantle Claude chat
* fix(bedrock): bound a global AnthropicConfig max_tokens by the deployment's own cap
* test(bedrock): annotate the new Mantle default max_tokens tests with Final
* refactor(bedrock): inject the AnthropicConfig max_tokens reader into the Invoke Claude config
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(databricks): join plain text blocks for non-Claude json_object requests
* fix(databricks): join only the last message for json_object and drop cache_control under drop_params
Databricks checks for "json" only in the last message, so earlier turns now go out as sent and only the
last message, plus any system messages merged into it, is joined into a string. Under drop_params a
text block's cache_control is dropped so that message can be joined too, and without it the request
still goes out as a list.
* fix(databricks): skip dropped assistant turns and blank parts when joining for json_object
The json_object join picked the last message before the request transform drops bare assistant turns,
so a trailing thinking-only or reasoning-only turn (a Responses input ending in a reasoning item included)
left the real last user message as a list and Databricks still returned its 400. The pick now skips the
turns that transform drops.
The join also ran when only one part had text, which sent whitespace-only parts the existing empty block
filter used to drop on requests that already worked. It now joins only two or more non-blank parts, so a
request with one non-blank part goes out exactly as before. Without drop_params, a message with a
message-level cache_control now goes out as sent instead of being rewritten into one cached block.
* fix(databricks): join json_object text parts only on models Databricks hosts
Databricks checks for the word "json" only on the models it hosts
(databricks-* pay per token names and system.ai.* AI Gateway names). An
external model endpoint skips that check and takes a list of text parts
today, so its request now goes out as sent instead of joined.
The drop_params read now goes through drops_unsupported_params, and a
test pins that a text block with a truthy non-string text is kept.
* fix(databricks): reject a text part holding a non-string locally with a clear 400
* refactor(databricks): reject a non-string text part at the transform_request boundary
* fix(databricks): read pydantic message objects in the non-string text check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): trim a malformed prompt_cache_breakpoint under drop_params on chat completions
* fix(openai): leave a non-object content part for OpenAI to reject under drop_params
* fix(openai): trim an unsupported prompt_cache_breakpoint ttl on every path and on azure_ai chat under drop_params
* fix: trim prompt_cache_breakpoint to what Azure and Mantle accept under drop_params
Azure OpenAI chat now gets the same content part trim as OpenAI chat, and
the Azure and Foundry gates also read a deployment's declared base_model.
Each Responses config keeps only the ttl its API takes: Azure keeps 30m,
and Mantle keeps no ttl. A trimmed ttl is logged at debug.
* fix(responses): trim prompt_cache_breakpoint only on APIs known to read it
Other Responses providers get the marker whole or not at all, as on main.
Azure chat now trims for a deployment with no base_model unless the cost
map marks the model as not reading breakpoints, and Azure AI Foundry
Responses keeps Azure's 30m-only shape.
* fix(azure): trim prompt cache breakpoints only for deployments known to read them
Azure and Azure AI Responses keep main's whole-or-nothing rule for unmapped deployment names, an opted-in custom OpenAI host trims the ttl and keeps the mode, and Azure chat strips every marker for a known non-reader under drop_params
* fix(providers): keep main's marker handling where the target won't trim it
A /v1/responses model with unknown breakpoint support on OpenAI's host gets a marker whole or not at all, as on main. The chat to Responses bridge keeps main's whole-marker check when a non-bool global drop_params leaves the target's trim off. A hyphenated name like gpt-5-6-sol now reads as gpt-5.6.
* fix(providers): read a non-version hyphen suffix like -32k as part of the model name
* fix(providers): keep main's reading of a dotted minor version like gpt-5.6o
* fix(providers): keep main's breakpoint shape for unmapped Mantle models and skip the Azure lookup without drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept
* test(e2e): send the azure_ai beta check without the code execution tool
* test(anthropic): require a vertex_ai and azure_ai decision for every anthropic beta
* fix(anthropic): forward the betas Claude Code sends that anthropic, vertex_ai, and azure_ai accept
* test(e2e): send Vertex's header-only betas in the beta acceptance case
* fix(anthropic): keep forwarding Vertex computer-use betas and point beta tests at platforms that still lack them
A live probe found each Claude 4.5 and 4.6 model on Vertex needs one of the two
computer-use betas for the computer tool, so dropping both broke it. The e2e
acceptance test leaves those two out of its all-at-once Vertex call, and the
integration and unit tests that relied on a beta being dropped now use one
the platform still lacks.
* test(anthropic): point unsupported-beta tests at betas the platforms still drop
The Azure AI handler test used claude-code-20250219 as the beta Azure drops, and the
Rust parse test used it as an unknown value. This PR forwards it on Azure AI and maps it,
so both now use values that stay dropped or unknown.
* fix(anthropic): add computer-use-2025-11-24 for the computer_20251124 tool
The computer_20251124 tool fell through to the default computer-use-2024-10-22 beta, which the
provider rejects for that tool version. Map it to computer-use-2025-11-24, which the beta map now
forwards on anthropic and azure_ai
* fix(anthropic): send no computer use beta for an unknown computer tool version
An unknown computer tool version fell back to computer-use-2024-10-22, which the anthropic and azure_ai columns now forward. It now gets no beta, as every platform effectively sent before. Every computer tool version also skips defer_loading and allowed_callers, computer_20251124 included
* test(azure_ai): read the branch's beta map in the auto-injected beta wire test
* fix(anthropic): keep defer_loading and allowed_callers on the computer_20251124 chat tool
* test(anthropic): read the branch's beta map in the azure_ai safeguards beta wire test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(caching): return the server tool call id instead of a TypeGuard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Gemini cache write carries the system prompt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(watsonx): read passthrough credentials from use_in_pass_through deployments
/watsonx/{endpoint} always called the provider config with api_key=None and api_base=None, so it only ever
worked from WATSONX_API_KEY / WATSONX_API_BASE. It now asks PassthroughEndpointRouter for the proxy-wide
watsonx deployment flagged use_in_pass_through, like the other passthrough routes, and keeps the env
fallback when none is flagged.
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(watsonx): keep env key precedence when no deployment is flagged
Resolve only use_in_pass_through deployments in the route. get_proxy_wide_credentials also falls back to
WATSONX_API_KEY, which put it ahead of WX_API_KEY in generate_iam_token
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(anthropic): map pause_turn and return refusal explanations on chat and responses
pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw
stop_reason in provider_specific_fields. Responses reports a paused turn as
incomplete with max_output_tokens and shows the unrun server search as an
in_progress web_search_call instead of a client function_call.
A refusal's stop_details.explanation now fills message.refusal and the final
stream delta.refusal on chat, and becomes a refusal content part on Responses,
streamed or not.
* fix(anthropic): let a chat client resend a paused turn so Claude continues it
* fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message
Chat cannot say where Claude wrote each thinking block in a paused web search turn, and
Anthropic answers 400 when one moves but accepts the turn without them. A new message after
a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped.
* fix(responses): send the streamed refusal part's type and text in content_part.added
* fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text
A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does
* fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses
* fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going
* fix(responses): mark a paused turn's message item completed
* fix(responses): mark every item of a paused turn completed
* fix(responses): keep a refusal's explanation when a refused turn is sent back
A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed
* fix(responses): scan refusal parts in Responses guardrails
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(decisions): honor a deployment's own supports_audio_input and supports_video_input flags
A Decisions deployment's model_info flags for audio and video now win over
the cost map, per media kind, the same way supports_box_questions already
does. An unlisted model flagged true gets its clips sent upstream, and a
false flag refuses them (or drops them under drop_params). OpenAI-format
providers still refuse clips since their request shape has no field for
them, and the SystemOne request builder no longer re-reads the cost map.
* fix(decisions): ignore media flags on a deployment whose own mode is not decisions
A chat-mode deployment's supports_audio_input describes chat input, so it no longer
turns on audio or video clips for Decisions. The cost map decides for it, as before.
Adds a proxy integration test for flagged, partly flagged, unflagged, and drop_params
deployments on both /v1/decisions and /v1/systemone, and for caller-sent model_info.
* fix(decisions): follow the cost map's mode for a flagged deployment without its own mode
A chat deployment flagged for audio or video usually leaves mode unset, so
its flags described chat input yet still let clips through on /v1/decisions.
When the deployment sets no mode, the cost map's mode for the model now
decides whether its media flags apply, and an unlisted model still takes
them. The audio and video lookups also run only when the request carries a
clip.
* fix(decisions): read the mode a flagged deployment falls back to from the shipped cost map
The router copies each deployment's model_info into the model's shared cost map row, so another deployment's mode there could switch off a flagged deployment's media flags. The fallback now reads the cost map as loaded, untouched by router registrations
* fix(decisions): route clips in a load-balanced group to a deployment that takes them
A Decisions or SystemOne request carrying audio or video now skips group deployments whose own flags or the cost map refuse that media, and keeps the whole group when none takes it, so the 400 still names the media.
* fix(decisions): keep deployments that drop or cannot resolve clips in the media filter
* fix(router): skip clip-refusing Decisions deployments after cooldown and tag filtering
Running the media filter before cooldown turned the 400 a group answered
when its only clip-taking deployment was cooling down into a 429. It now
runs next to the image filter, just before order filtering, in both the
async and sync selection paths.
* fix(router): honor router default drop_params when skipping clip-refusing Decisions deployments
* test(router): list the router default params helper in the router coverage check
* test(bedrock): stop reloading the cost map module in the GovCloud fixture
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): forward the client's own metadata on Anthropic and custom pass-through routes
* fix(proxy): forward only a valid metadata.user_id to Anthropic Messages on pass-through routes
The whole-object forward sent keys Anthropic refuses and reached other
providers, so a request that used to pass could start failing with a 400.
Read the client's user_id after hooks, drop it unless it is a string with no
email or phone number, and send only that key when the upstream URL path ends
in /v1/messages. Share the validator with the translated route, whose email
check now matches any embedded address as Anthropic's does.
* fix(proxy): forward metadata.user_id only to Anthropic upstreams no guardrail scans
Limit the pass-through user_id to Messages create on api.anthropic.com or the
configured Anthropic base, mirror the checks Anthropic answers 400 for, and keep
the field off any request a pre-call guardrail inspects. The translated route
goes back to its own user_id filter.
* fix(proxy): judge user_id forwarding on the whole request and cap it for configured Anthropic bases
* refactor(proxy): pass the user_id check a plain mapping instead of an optional body
* fix(proxy): drop an unprintable metadata.user_id and warn when a refused id is dropped
Anthropic's whitespace is not Python's, so a control character before the at sign hid an email from the filter and turned a 200 into a 400. Fireworks and Moonshot also fail on newline ids. Requiring a printable id covers both, and a refused id now logs a warning that never carries the value.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): cover text verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity as chat verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses): type the bridged text verbosity lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity validation on the codex tool search follow-up turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text.verbosity on previous_response_id client tool_search follow-up
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Seven integration tests fail on main for reasons that are not product bugs. Two are order or race bugs in the tests, one reads rows other tests wrote, one passed on an error body, and three still expect behavior that #48644, #39864 and #39540 changed on purpose
* fix(proxy): persist sent model_info fields on POST /model/update
POST /model/update rebuilt litellm_params from the request but dropped the
request's model_info, so changing base_model (or mode, tier, etc.) on an
existing DB model was silently ignored. Clients that only use this endpoint,
like the Terraform provider, could never fix a deployment's base_model
without deleting and recreating it.
Persist the model_info fields the caller actually sent, on top of the stored
ones. ModelInfo defaults never overwrite stored values, server-owned, team,
PTU and pricing fields stay on their dedicated write paths, and values echoed
from a /model/info response are not pinned.
* fix(proxy): keep echoed and blank model_info values out of POST /model/update writes
Blank strings from typed clients no longer overwrite stored values. Values echoed from
/model/info (the stored base model's cost map entry and discovered limits) are not
pinned, and an echoed catalog value resets a differing stored override like PATCH does.
direct_access and access_via_team_ids stay display-only. The tests stop patching the
auth check class and give the team lookup a real row instead.
* fix(proxy): lock the model row and skip unchanged model_info on POST /model/update
* fix(proxy): treat only keyed /model/update bodies as echoes and write the stored model_info as saved
A typed model_info value that matched the cost map, or the previous base model's entry,
was dropped and could erase a stored override. Echo detection now follows PATCH, only a
body carrying the /model/info key counts, compared against the base model the row has
after the write. The locked write patches the stored JSON instead of dumping the hydrated
Deployment, so litellm_params pricing is no longer copied into model_info. Tests run the
real Router with discovery over a mock upstream
* docs(terraform): note that model_info updates now take effect and overwrite values set outside Terraform
* test: expect POST /model/update to save the cache point flag, and wrap long test lines
test_x06 pinned the old contract, where POST /model/update answered 200 and ignored model_info.
This PR makes POST save model_info the way PATCH does, so x06 now asserts what its PATCH sibling
x01 asserts, the stored flag flips to true and cache points come back on.
* fix(proxy): answer 400 when POST /model/update gets an access_groups that is not a list of strings
* fix(proxy): read discovered model info through the router registry entry point
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): route unmapped converse-only family model ids to converse
#42713 stopped Bedrock-native OpenAI ids that are not in the cost map yet from
falling to the invoke route. The same gap exists for the other families that
have no invoke transformation at all: minimax, nvidia, writer, zai, google and
xai. An id from one of them that is newer than the cost map hits
get_bedrock_route -> "invoke" -> get_bedrock_invoke_provider() -> None ->
BedrockError, when Converse would have served it
Add a second family regex for those six and default them to converse, the same
way the openai family is handled. _OPENAI_FAMILY_MODEL_RE is left alone because
the prompt-caching rule is openai-specific
Every shipped id of these families already resolves to a converse entry via its
base model (checked against the cost map), so this only changes behaviour for
ids that are not in the map yet
Tests: six unmapped ids route to converse, and an unmapped meta.* id still
routes to invoke since meta has an invoke handler
* fix(bedrock): route a model id with no Invoke provider to Converse
A bedrock/<model> id missing from the cost map fell to the Invoke route,
where an id that names no Invoke provider (a new vendor's model, a bare
provisioned-model, imported-model, or custom-model-deployment ARN) fails
locally with "Unknown provider=None" before any request is sent. Such ids
now go to Converse, which replaces the per-family regex with the general
rule. Ids that name an Invoke provider and explicit route prefixes are
unchanged.
* test(bedrock): expect Converse for ids with no Invoke provider in the nova ARN and not-found tests
* fix(bedrock): accept tools on Converse for a model with no cost-map row or Invoke provider, and keep a 404 for an invalid id
* fix(bedrock): answer 404 for an invalid model id instead of AWS's 400
AWS answers a mistyped Bedrock model id with a 400 ValidationException. The mapping already raised NotFoundError, but passing AWS's 400 response through made the openai base class report 400, so the proxy answered 400 and the router skipped its cooldown
* fix(bedrock): drop lookaround regexes from tool schemas for a model the cost map does not list
* fix(bedrock): apply the unknown-model defaults only to bare ids LiteLLM knows nothing about
A bare Bedrock id with no cost-map row and no Invoke vendor gets Converse's open params, its reasoning_effort as sent, lookaround-free tool schemas, and a 404 for AWS's invalid-identifier reply, which keeps AWS's response headers. An explicit route like converse/, an ARN, or an id LiteLLM recognizes keeps main's params, regexes and error status, so drop_params still drops tools there
* fix(bedrock): judge lookaround support by model id on every route, ignore price-only rows
The regex check now strips route and region prefixes from the model and its
base_model before asking whether LiteLLM knows the model, so
bedrock/converse/<id> gets the same answer in a direct transform call as on the
completion path. A cost-map row counts only when it names a provider, so a
price-only register_model row leaves an unlisted id unknown, as on the proxy.
AmazonInvokeConfig now inherits the Invoke provider lookup instead of keeping
an identical copy.
* fix(bedrock): treat a region path like the bare id when the cost map does not list it
A deployment like bedrock/us-east-1/<unlisted id> now gets the same tools,
reasoning_effort and typo 404 as bedrock/<unlisted id>, so Claude Code works
on it and a mistyped one still answers 404.
* fix(bedrock): keep today's 404 for ids that reach Converse only by default, and keep an empty id local
* fix(bedrock): answer 404 for an invalid id only when Converse rejected it, and keep a region path with no id local
* fix(bedrock): keep a path-shaped id local, and map /v1/completions errors with the model's provider
An id with a slash that is not an ARN, like bedrock/../../converse or an unknown region path, keeps Invoke and its local 404 instead of reaching AWS Converse.
atext_completion mapped every error as OpenAI, so a Bedrock typo on /v1/completions answered 400 while chat answered 404. It now resolves the provider from the request's model, as the other async entry points do.
* fix(text completion): keep /v1/completions error mapping as it is on main
Mapping async /v1/completions errors with each provider's rules is wider than this Bedrock routing change. It moves classes, statuses, cooldown and fallbacks for most providers, and some lose context-window fallbacks. So this restores main's mapping here and leaves that change to its own PR. A Bedrock typo on /v1/completions answers AWS's 400, while chat keeps the 404.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): keep a deployment's own model_id over the request body's
A request body model_id overrode the deployment's pinned model_id, so a client could move SageMaker chat and embedding calls to another inference component. The router now drops the request's model_id when the deployment sets one.
* fix(router): let deployments opt back in to a client model_id and stop client component headers on SageMaker chat
Drop the request's model_id after the router defaults are applied, so a default_litellm_params model_id cannot win
over the deployment either, and apply the same precedence on the sync text_completion path. A deployment that lists
model_id in configurable_clientside_auth_params keeps the client's value. SageMaker chat drops any case variant of a
client's X-Amzn-SageMaker-Inference-Component header when the deployment sets a component
* fix(router): keep a SageMaker deployment's component when it pins it through extra_headers
* fix(router): read the SageMaker component header as a pin only on SageMaker deployments
* fix(sagemaker): send the inference component only as a header, never as a body model_id
SageMaker picks the component from X-Amzn-SageMaker-Inference-Component, so the chat
container body and legacy parameters no longer carry model_id. Nova endpoints accept a
fixed body field list, so a pinned Nova deployment could reject the extra field.
* fix(health): probe header-pinned SageMaker deployments at their component
Health checks bypass the router, so a legacy sagemaker/ deployment pinned only
through its inference component header sent no component on /health while routed
requests now send it. Also update the two SageMaker integration tests that still
asserted the old precedence and the dropped body field.
* fix(health): apply the router's model_id pin rule to health probes
Health probes now share the router's pin function, so a deployment that lets
clients set model_id gets no header pin and its own model_id wins without
reading the header. Header keys that are not strings are skipped when
reading the inference component header instead of failing the call.
* fix(sagemaker): keep header keys that aren't strings and report a string opt-in in health checks
The chat transformation now keeps extra_headers keys that aren't strings, so such a deployment fails the way it did before instead of answering 500, and the model_id pin checks configurable_clientside_auth_params with isinstance, so a string there reaches the health probe's own error capture again.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply
Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its
default audio setup at connect, so a client's session.update (GA output_modalities or
beta modalities) was dropped and every answer came back as audio. The setup now goes
out on the client's first message: a session.update becomes that setup, anything else
gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a
provider hook. Text setups label deltas response.output_text.delta, and downgrading
TEXT to AUDIO on an audio-only model logs a warning
* fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup
* fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup
Gemini and Vertex Live accept one setup per connection, so the bridge used to
turn only the first session.update into that setup and drop the rest without
telling the client. Now every session.update that arrives before the first
content frame is held, deep-merged (later values win per field), acknowledged
with a session.updated built from the setup it would produce, and the merged
session becomes the one setup sent with the first conversation item, audio
frame, or tool result. The session events also surface the systemInstruction
text as instructions, which the old string check never matched because the
setup carries it as a Content object.
* fix(realtime): cap the session.update merge depth so the recursive detector accepts it
_merge_session_fields recurses once per nested mapping level, so CI's
recursive_detector flagged it. Bound the descent at eight levels (the OpenAI
session object nests four deep at most), replace anything deeper wholesale
with the later session.update's value, and list the helper with the other
depth-capped functions the detector allows.
* fix(gemini-live): accept a string output modality and drop values Live cannot produce
A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once.
* fix(gemini realtime): answer an error for session updates Gemini Live cannot apply
An output modality the model cannot produce, or a session.update sent after the
first content, now gets an invalid_request_error event instead of a silent swap
or drop, unless drop_params is set. The synthetic session.created reports the
model's default modality, setupComplete no longer adds an extra session.updated,
and merging nested session updates is linear in their size
* fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns
A pre_call guardrail blocking the first typed message now goes out after the
setup, so Gemini and Vertex no longer close the session with 1007.
Push-to-talk sessions (turn_detection null or create_response false) wrap each
audio turn in empty activityStart and activityEnd messages, which Gemini needs
when automatic activity detection is off.
A session with no content after 60 seconds sends its setup, since Vertex
refuses a setup 300 seconds after connect. Held session.update frames past
10 MB answer session_update_too_large, and deferred mode now errors on a
second session.update unless drop_params is set.
* fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection
An update that leaves the applied setup unchanged now gets session.updated
instead of session_update_not_applied, so clients that resend their session
with the first response keep working. Deferred setup mode merges held updates
like the default mode, turn_detection null turns Gemini's activity detection
off, drop_params drops a voice outside the documented Live voices, and the
lazy setup deadline moves to 240 seconds
* fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content
* fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices
A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name.
* fix(realtime): hold first content so a following session.update joins the Gemini setup
Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too
* fix(realtime): check AI Studio Live voices like Vertex
* fix(realtime): forward any voice to AI Studio Live models that accept it
* fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config
The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup.
* fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock
The deadline and grace tests waited on real 50 ms timers, and the merge cost
check compared wall-clock runs. The session now reads an injected clock,
time.monotonic by default, the two tests step it, and the merge cost check
moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes
the backend through wrap_backend, which main added after the test was written
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126)
The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests.
* test(proxy): pin provider metadata and key billing on /cursor/chat/completions
A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land
* fix(proxy): pick the Cursor route's metadata field from the body shape
A Responses-style body on /cursor/chat/completions keeps proxy metadata in
litellm_metadata, the same as /v1/responses, while a Chat Completions body
keeps metadata, the same as /v1/chat/completions. Adding the whole route to
the internal-metadata list let a client's chat-body tags replace the key's
tags in spend logs and skip tag budgets.
* fix(proxy): keep get_metadata_variable_name callable with only the request
* fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies
A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field
* fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies
* fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string
* fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(azure_ai): keep OpenAI prompt cache breakpoints on the Foundry chat completions path
* fix(azure_ai): keep only OpenAI content part types when preserving prompt cache breakpoints
* fix(azure_ai): share the breakpoint check with the cache-mark step and keep only text parts
A deployment's model_info opt-in is registered under its provider-prefixed cost-map key without a litellm_provider, so that prefix now names the provider the flag speaks for.
* refactor(azure_ai): read text parts through a walrus and mark new test values Final
* fix(azure_ai): read a router-registered breakpoint opt-in only for azure_ai and keep a message mark off dropped file parts
* fix(azure_ai): read the breakpoint opt-in from each deployment's own model_info and base_model
A router deployment without its own supports_prompt_cache_breakpoint flag now
reads the cost map as loaded, so a sibling's opt-in on the same model string no
longer turns its requests into 400s. The shared hook no longer reads router
registered keys for azure_ai.
* fix(azure_ai): send breakpoints in Foundry's shape under drop_params and fall back to the model name
* test(integration): expect the injected breakpoint on the Foundry GPT-6 chat wire
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
