346 commits

Author	SHA1	Message	Date
ryan-crabbe-berri	fa254b9346	fix(responses): surface mid-stream upstream errors that arrive without an event type (#50466) ... * fix(responses): surface mid-stream upstream errors that arrive without an event type Bedrock's native Responses route can end a stream with a bare {"error": {...}} chunk. It parsed as a typeless GenericEvent and crashed the iterator with AttributeError, so clients saw that instead of the upstream error. Such chunks are now read as error events, which raise the mapped upstream error and reach the client as response.failed. * refactor(responses): inline the untyped error chunk check to keep LIT001 at the base count	2026-11-07 14:53:07 -08:00
Mateo Wang	709b27e268	fix(anthropic): preserve streamed tool argument fragments verbatim (#50470) ... * fix(anthropic): preserve streamed tool argument fragments verbatim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): send the schema-less stream test the null properties form providers accept * fix(anthropic): unwrap values when a schema-less stream sends its whole tool input in one piece * fix(anthropic): unwrap only a one-piece values input and never emit Infinity in schema-less streams * fix(anthropic): keep JSON mode answers verbatim when they cannot be written back as strict JSON A tool input holding a number past a float or an integer past Python's digit limit now passes through as the model wrote it on the stream and non-stream paths, instead of coming back with Infinity or raising. One-piece values rows for false, null, a plain object and too-deep nesting pin the remaining unwrap branches. * fix(anthropic): reject non-finite floats through pydantic instead of a raising parse hook * fix(anthropic): keep split JSON mode answers whole when a field cannot be written back as strict JSON * fix(anthropic): keep out-of-range numbers as the model wrote them in JSON mode answers Decode tool input with parse hooks that keep a number past a float or an integer past the digit limit as its source text, and write it back unchanged, so held, wrapped and name and value split answers stay whole instead of falling back to raw blocks. * fix(anthropic): keep numbers that round to zero as the model wrote them in JSON mode answers * refactor(anthropic): encode JSON mode answers without a recursive helper * fix(anthropic): decode deeply nested JSON mode answers without slow recursive validation * fix(anthropic): type the decoded JSON mode answer as object and test non-object answers json.loads output now goes through the existing TypeAdapter(object), which returns it unchanged, so no Any reaches the object guard. New stream and non-stream cases check that a list, string or number answer comes back as written. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-07 01:44:48 -08:00
HootRock	8f19607259	fix(proxy)!: preserve Responses SSE event framing (#47980) ... * fix(proxy): preserve Responses SSE event framing * style(proxy): format Responses stream terminator condition * fix(proxy): validate Responses guardrail event names * fix(proxy): keep Responses first-event errors and keepalive failures in Responses framing * test(proxy): use controlled clocks for Responses keepalive failures * fix(proxy): end late Responses failures with an error event the SDK raises on Errors that arrive after keepalive pings, and guardrail error strings yielded mid-stream, now end /v1/responses with a named `error` event carrying the same error object chat sends, so the OpenAI SDK still raises APIError with a clean message and provider_specific_fields. Background polling passes the same framing and records an `error` event as a failed response. The first data line of a frame is found by splitting on newlines only, so a U+2028 in an error message no longer hides a first-event error. * fix(proxy): match OpenAI's Responses error event and stream late passthrough blocks as block responses * fix(proxy): keep the reason when a background Responses request fails A background poll of a request a post-call guardrail blocked saved a null code and message, because the guardrail's error event nests them. A request whose first chunk was an error polled as completed with no error. Both now poll failed with the same code and message a late failure gets. * test(proxy): type the first-chunk poll test parameters * fix(proxy): keep first-chunk error detection to lone error frames and handle string poll errors * fix(proxy): keep the HTTP status on late Responses errors and end Responses streams on error events A late error body without a code now takes the response's status, an upstream error event counts as the stream's terminal event, and a Responses stream that fails before its first chunk ends with an error event instead of data: [DONE]. * test(proxy): type the first-chunk failure test's parameters * fix(proxy): keep cursor chat framing and JSON-encode Responses hook errors * fix(proxy): encode non-dict Mapping Responses chunks as JSON objects * fix(proxy): encode mappings and models nested in Responses stream chunks as JSON objects * test(proxy): freeze the hook-output fixture * fix(proxy): keep masked mappings and odd keys safe in Responses stream frames Only read-only dict views and pydantic models nested in a hook chunk or error event expand into JSON objects. Other mappings, such as httpx.Headers, keep their own string form so masked values stay masked. Keys JSON cannot take become strings, and a chunk JSON still cannot encode goes out as the plain data frame it was before. * fix(proxy): never fail a Responses stream frame on a value JSON cannot encode Only dicts and read-only dict views expand into JSON, at the top level too, so a header mapping yielded as a whole chunk keeps its masked string form. Any failure while encoding a chunk falls back to the plain data frame the base sent, and an error event that cannot encode still goes out with its code and message, so a late block never drops the connection. * fix(proxy): keep what a nested value's repr hides out of Responses stream frames * fix(proxy): print values in Responses stream frames the way each base path did * fix(proxy): send string and number subclasses whose repr shows their value as that value in Responses stream frames * fix(proxy): print container subclasses and SDK models in Responses stream frames the way base did * fix(proxy): read hook container storage the way repr does and send error-event models as str() * fix(proxy): frame streamed Responses reads and send unreadable hook values as base did Track pre-framed and raw SSE frames so a late failure never follows a terminal event, accept CRLF and CR line breaks in a first named error, frame GET /v1/responses/{id}?stream=true as Responses events, print a whole hook proxy chunk once and fall back to its text, ignore a non-integer status_code in a hook error frame, read a late-block dict subclass through items() when that view differs from its storage, and keep the order and inline form base's repr printed for nested mappings * fix(proxy): match base on Responses stream edge cases Read a decimal-string or integral-float status_code as its integer again, so "429" and 429.0 still map to rate_limit_exceeded. Print each hook value's repr once and reuse it, so a repr that changes between calls cannot reach the client. Track only pre-framed sequence numbers from 0 to 2**63-1, so a huge one cannot break the failure event. Build nested JSON with one frame per level again, which restores the nesting depth sent as JSON. Count a pre-framed frame as terminal from its data type, and a named error frame only when its data carries an error, the way the OpenAI SDKs read them. * fix(proxy): print hook proxies once and keep base's chunk tracking Read-only mappings in hook chunks now emit the text from their first print, hook chunks are read without their error field, only a non-empty error object makes a framed error event terminal, and framed sequence numbers stay below 2**63-1 while hook chunks keep base's unbounded tracking * fix(proxy): print each read-only mapping item once and poll framed error events as failed A MappingProxyType in a hook chunk or error detail now expands from the mapping it wraps, so each item is printed once, the way base printed it. A value whose repr changes between calls stays masked, a value whose repr works once is kept, and deep nesting keeps one stack frame per level The background reader takes each frame's type from the same terminal rule the stream uses, so an event: error frame with an error object polls failed even when its data names another type or none * fix(proxy): print self-holding hook values once and poll how the stream ended A container that holds itself now goes out as the marker its repr prints, like {...}, [...] or mappingproxy({...}), and the dict behind a read-only mapping joins the cycle check, so no item is printed a second time. A dict whose item repr adds a key keeps going after the items already shown, the way dict's repr does, instead of printing the whole dict again A whole read-only mapping chunk that cannot go out as JSON is printed once by the formatter, so a print that fails fails the stream as it did on the base The background poll reads the event type from the block it parses, as on the base, and keeps the first error frame the stream treats as terminal anywhere in a chunk. That frame fails the poll only when no response.completed, response.failed or response.incomplete follows it * fix(proxy): name hook chunk events from the sent JSON and keep polls on frames they cannot read A mapping chunk now takes its event: line from the JSON it sends, so a masked type value never reaches that line, and a chunk the event model cannot read goes out as a plain data frame instead of failing the stream A dict whose item repr deletes or swaps keys keeps printing every key not yet shown, and an error code past the integer digit limit reads as no status The background poll skips a frame whose shape it cannot read, as base skipped frames it could not parse, stops reading a chunk at [DONE], and lets only the last terminal frame in a chunk decide its status A Responses stream's first chunk answers as a JSON error only for codes 400 and up * fix(proxy): read every frame of a polled chunk and name model chunks from the sent JSON A background poll parsed only the first data frame of each chunk, so a terminal event behind an error or delta frame in the same chunk was lost, and the poll stored a status its own stream never ended with. It now reads every frame up to [DONE] A terminal event whose response is a string or null kept no status and polled completed. Its status now comes from the event type, and final output items that are not objects are skipped, so the rest of the final output is still stored Model chunks now take their event line from the JSON they send, as mapping chunks already do, so a serializer that drops type no longer sends event: response.completed over {} * fix(proxy): read polled chunks as SSE events and keep draining past a mid-chunk [DONE] A background poll split a chunk into data lines, so one event with two data lines parsed as two payloads, and a frame with an integer past the digit limit raised a ValueError the parse guard did not catch, which failed the poll and dropped its spend row. The poll now joins each event's data lines the way an SSE reader does and skips any frame it cannot parse A [DONE] behind another frame in one chunk ended the poll and left the stream unbilled. Only a chunk that opens with [DONE] ends the poll now, as on main, and the poll stops reading a chunk at its first terminal event, so two terminal events in one chunk poll the first one as main did * fix(proxy): JSON-encode only plain hook chunks on Responses streams A hook chunk whose whole tree is builtin JSON types streams as its named event. Any other chunk keeps the base data frame, so nothing a custom repr hides reaches the wire. Error event details encode the way the chat error frame does. * fix(proxy): skip containers already visited when checking hook chunks for JSON * fix(responses): keep streaming past an event with no type The Responses stream iterator read event.type directly when it noted a yielded event, so an event whose JSON had no type, like a custom hook's dict that the proxy now sends as JSON, raised AttributeError and ended the stream. Read it with getattr, as the iterator's other checks do. * fix(proxy): keep hook dicts with no type on the base Responses frame A hook dict with no type is not a Responses event, so it gets no event name either way. Sending it as JSON only made litellm SDKs that read type on every event stop the stream. Keep such dicts on the frame the base sends, so every SDK version reads them as before. * fix(responses): keep a background poll's first terminal status and take a first-chunk error code from its HTTP status --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 20:17:57 -08:00
ryan-crabbe-berri	6b745635c8	fix(bedrock): forward Codex agent messages that carry encrypted content intact (#50457) ... * fix(bedrock): forward Codex agent messages that carry encrypted content intact Codex multi-agent v2 sends worker messages as agent_message items whose encrypted_content block only the model server can decrypt. Bedrock Mantle and Bedrock Runtime used to 400 on agent_message, so LiteLLM rewrote every one into assistant text, pasting the ciphertext into the prompt and dropping the sender and recipient. Bedrock now decrypts these items itself, so the rewrite left the recipient model reading an opaque token instead of its task. Agent messages with an encrypted_content block now go to Bedrock untouched on both endpoints. The Bedrock configs declare supports_encrypted_agent_messages, and the shared normalizer keys off that flag so the two cannot drift. Mantle's /v1/responses route for open-weight models still rejects agent_message in any form, so it keeps the old rewrite. Plaintext-only agent messages keep the rewrite on every route. * fix(bedrock): only keep worker messages locked for OpenAI GPT models Gemma and Grok on Mantle's OpenAI route never lock Codex worker messages, so Codex puts plain text in the locked slot and Bedrock rejected it with invalid_encrypted_content once the route alone decided to forward it intact. Decide from the model instead, and restore the full-value rewrite assertion.	2026-11-06 17:47:43 -08:00
tavren27	2d2290067c	fix(bedrock): run realtime guardrails on Nova Sonic sessions (#49833) ... * fix(bedrock): run realtime guardrails on Nova Sonic sessions * fix(bedrock): quote the spoken violation message and skip it after a blocked Nova Sonic tool output * fix(bedrock): screen every message role, end Nova Sonic sessions on blocked transcripts, drop unparseable frames * fix(realtime): apply guardrail masks, keep block reasons out of spoken notices, fail closed on guardrail crashes * fix(realtime): screen joined text parts, keep the documented notice, honor key opt-outs Text parts of one message are joined before screening again, so a phrase split across parts is still blocked, and a mask collapses them into one part. The spoken notice is realtime_violation_message or the block reason, quoted only on Nova Sonic. Client-written output_text and text parts are screened. Key and team guardrail opt-outs reach regular and translation realtime sessions. A non-string upstream transcript skips screening as before. * fix(realtime): honor guardrail role scope, clear stale response.create drops, screen split keywords, report Nova guardrail outages * fix(realtime): keep every per-part mask when a joined text masks differently * fix(realtime): screen one joined text per item and pass the session's tenant context to guardrails Multi-part items reach each guardrail once as the space-joined text, as on main, and only an item a guardrail rewrites gets its parts screened again to keep per-part masks. Guardrails now receive the session request data with its metadata, so key and team policies apply, and a ModifyResponseException block is reported as a block. Nova scopes every message item as the user turn it sends to Bedrock. * fix(realtime): catch phrases split mid-word across text parts without re-screening single-part items * fix(realtime): screen each joined text in its own guardrail call and keep a Nova opening message for the retry while it is screened * fix(realtime): screen text parts in a loop so large messages keep their masks * fix(realtime): fail closed on a non-string transcript under a guardrail and give each turn its own guardrail metadata * fix(realtime): fail a non-string transcript only when a guardrail would screen the user's turn --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 13:18:48 -08:00
Mateo Wang	a94118c6cd	test(bedrock): expect native output on Opus 4.8 profiles after #50222 (#50343) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-06 11:04:32 -08:00
Mateo Wang	36c64ce85d	feat(bedrock): default to native structured output on Opus 5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 (#50221) ... * feat(bedrock): enable native structured output by default for selected Claude models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep govcloud and bare model rows off native output Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected native schemas with the JSON tool on default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep the rejection retry on the unforceable-tool native path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected schemas on the Messages path for default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format Messages native output retry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): allow profile-only native output metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): only disable default-native output on schema-rejection retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): type fixture parameters in native output retry tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): drop strict tools after a native-output retry hits the grammar limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format native output fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too. * fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action A role allowed only bedrock:InvokeModel got a 403 on schema streams once these models went native, since native streams call converse-stream while the JSON tool path buffered through converse. The retry now sends that request again as a buffered JSON tool stream, the shape the base sent. The wire test moves its neither-key and opt-in cells to Opus 4.7, its Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native. The batch wire test now checks a thinking schema record on both sides of the contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5 and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8 cells assumed those models were opt-in. * fix(bedrock): move the Claude 5 native output default to a key released proxies do not read Released proxies load the cost map from main and have none of the native output retries, so setting supports_native_structured_output on the Opus 5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide or min/max schemas into 400s on v1.108.1. Those rows keep supports_native_structured_output false and gain bedrock_native_structured_output_by_default true, which get_model_info reports as supports_native_structured_output, so this version behaves as before while released versions keep the JSON tool. * fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it * fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key A registered supports_bedrock_native_structured_output false now beats the native output default, and batch uploads read the deployment's model_info, so an opted-out deployment keeps the JSON tool on every record. The Opus 5 and Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default, so released proxies on main's cost map keep the JSON tool on them. The 64 property batch cutoff now applies only to rows with that key, so models that were already native keep their batch output shape. * fix(proxy): report a deployment's Bedrock native output opt-out on /model/info /model/info merged the deployment's model_info over the backend row, so a deployment that set only supports_bedrock_native_structured_output: false still showed the backend's derived supports_native_structured_output: true. The generic flag is now derived from the merged info unless the deployment set it itself, through the same helper get_model_info uses. * fix(bedrock): keep native output when a denied schema stream is resent buffered A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output. * fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend * fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model * fix(bedrock): answer a buffered invoke stream from the non-streaming reply * fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected * fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main. /model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-06 09:38:00 -08:00
JeongCheolLee	febbe38326	fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385) ... * fix(anthropic): do not surface server_tool_use as client tool_calls Anthropic server-executed tools (web search, tool search, code execution) were returned in chat completions tool_calls with srvtoolu_ ids, so OpenAI-protocol agent clients treated them as pending work and re-invoked the model until their recursion limit. Move them to provider_specific_fields.server_tool_calls on both the non-streaming and streaming paths. History replay, the /v1/responses bridge and the /v1/messages streaming guardrail read that field, so multi-turn reconstruction, web_search_call output items and the guardrail's fail-closed check on server tool blocks keep working. * fix(responses): return Claude web_fetch calls as web_search_call open_page items * fix(responses): keep Claude server web calls without a result as hosted items A non-streamed Claude turn that calls web_search or web_fetch next to a client tool can return the server call before its result. Build an in_progress web_search_call for it, so the Responses bridge no longer returns it as a client function_call * fix(anthropic): keep pending server calls native and hide server calls from client tool paths Replaying a server call that has no result yet now sends it back as server_tool_use, so Anthropic accepts the next turn. The tool permission guardrail checks server calls again and rewrite mode removes a denied one. The /v1/messages adapter keeps Claude's server blocks, and the Responses bridge never returns a server call as a client function_call. * fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls, so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own server_tool_use instead of running again * fix(anthropic): let guardrails and loggers read Claude's server tool calls again Guardrails, tool policies, tracing loggers and the Responses guardrail translation read only client tool calls after server calls moved to provider_specific_fields. They now read server calls first, then client calls, through one shared helper, and the Responses guardrail treats Claude's web_search_call items as web_search or web_fetch calls. Non-streaming splits server calls by block type, as streaming does. * fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy, streamed or not. Redaction with message logging off hides server call arguments and hosted web call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls. * fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use. * fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy. * test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions * test(guardrails): type the fake Pillar client like its Protocol * fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or web_search still needs that call and its thinking replayed, or Claude loses the fetch result * fix(guardrails): scan replayed Claude server calls and hosted web calls Request guardrails now see and rewrite server_tool_calls replayed in chat messages. Cato and Purview scan Claude web_search and web_fetch calls on Responses output, and a function tool call with an extra custom key is no longer dropped from Cisco, Purview, Repello and PANW scans. * fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks * fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows * test: annotate the new server tool call test locals with Final and type their parameters * test: type the new guardrail test helpers with litellm's tool call types instead of Any * test: narrow the remaining new test helper parameters from bare dict and object * fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls * fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera * fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 03:54:34 -08:00
Siddharth	2a4f8b1d19	fix(bedrock): drop another model's signed thinking when replaying to a non-Claude Converse target (#50073) ... * fix(bedrock/converse): omit reasoning signature for non-Anthropic models (#49969) * test(bedrock): cover signed reasoning replay in mapped tests * fix(bedrock): drop another model's signed thinking when replaying to a non-Claude Converse target A signed thinking block from Claude replayed to a Converse model known not to be Claude, such as GPT-6 Luna or Nova 2, now leaves the request instead of going out as reasoningContent. Bedrock rejects that field on those models, so the whole request answered 400 A target that may still be Claude, such as an application inference profile ARN with no base_model, keeps the signed block, since Claude needs it on a tool-use continuation. Unsigned blocks keep the existing text path The thinking-block error matcher added earlier on this branch is removed, because Converse and non-Claude Bedrock requests never reach that retry * fix(bedrock): keep signed thinking for Claude behind a region prefix litellm does not list ca. and in. inference profiles name Claude after a region prefix get_bedrock_cross_region_inference_regions does not have yet, so they counted as non-Claude and lost their signed thinking. The thinking-only turn test now pins the exact merged and continue-message shapes. * fix(bedrock): limit the region-prefix Claude match to signed thinking and drop a dropped block's cache point * fix(bedrock): drop Claude's bridged reasoning items before a Bedrock Responses API replay * fix(bedrock_mantle): drop litellm-minted reasoning items before a Mantle Responses call * test(integration): expect Bedrock and Bedrock Mantle Responses to drop minted reasoning Bedrock and Bedrock Mantle answer 400 on a reasoning item litellm minted, so the wire test now expects both to drop it, like OpenAI and Azure --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 03:15:49 -08:00
devin-ai-integration[bot]	fc8f614a03	test(bedrock_mantle): expect the generic unsupported-param 400 for temperature on fixed-sampling Claude (#50344) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-05 21:53:06 -08:00
devin-ai-integration[bot]	d96da7267b	test(copilot): route direct-token fixtures through authenticated peers (#50307) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-05 17:42:46 -08:00
Nate Armstrong	58896e2009	test(router): expect a primary stream's stall to cool its deployment (#50261) ... The primary stream's mid-stream stall reaches failure_exits as a mapped litellm.Timeout (408), which cools the deployment on the Rust core. Keep that behavior and update the integration test that expected the Python core's exemption. Fallback hops stay exempt through is_stream_stall.	2026-11-06 00:06:59 +00:00
Kerry Lu	3b20b48473	feat(interactions): forward continuation_token and type requires_continuation so callers can resume a long decode (#50151) ... * feat(interactions): continue requires_continuation long decodes with continuation_token hops A foreground Interactions request stops at Google's per-hop output cap with status requires_continuation and a continuation_token. litellm now re-posts the request with previous_interaction_id and continuation_token until the status moves on, merging the hops into one interaction (last id, every hop's steps, summed usage with initial_input_tokens from hop 1), for non-streaming and streaming calls. continuation_token is accepted as a request param, and requires_continuation is a known status for billing. * refactor(gemini interactions): validate streaming chunks with model_validate * fix(interactions): keep the held completion across stream hops, stop on a repeated continuation_token, follow the spec redirect * fix(interactions): add grounding_tool_count per tool type, stop on any seen continuation_token, keep streaming past a [DONE] frame when a hop is pending * ci(interactions): point the spec canary at aistudio.google.com, where Google moved the Interactions OpenAPI spec * test(interactions): scripted Google ends each long decode hop with a [DONE] frame * refactor(interactions): pass requires_continuation and continuation_token through instead of looping hops in the proxy The caller drives the continuation, the same as against Google directly. /interactions accepts continuation_token in place of input so a resume request needs no input. * test(proxy): assert the accepted /interactions bodies pass the required-param check * fix(proxy): keep the router default fallback for input on /interactions, continuation_token stands in for it --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-05 15:38:44 -08:00
Mateo Wang	4679b9f642	test(copilot): align login guards with direct access tokens (#50285) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-05 15:38:33 -08:00
devin-ai-integration[bot]	796f751603	test(providers): expect 503 server_is_overloaded for an upstream overload on the responses bridge (#50166) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-05 04:40:22 +00:00
quietjules	5a392b9e5f	fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) ... * fix(anthropic): stop forcing the response_format json tool when Claude will think A forced tool call skips Claude's thinking. The json_schema emulation forced json_tool_call unless thinking.type was enabled, so adaptive thinking and the models that think by default (Opus 5, Sonnet 5, Haiku 5.5, and the always-on families) lost their thinking on every structured-output request. AnthropicModelInfo.response_format_tool_can_be_forced is now the one check the Anthropic, Bedrock Converse (including the native output fallback) and Databricks paths use. It leaves the tool unforced when thinking is requested in any form or the model thinks by default, and keeps forcing it when the request turns thinking off (thinking disabled or reasoning_effort none) on a model that accepts forced tools. The new thinking_on_by_default cost map key marks Opus 5, Sonnet 5 and Haiku 5.5 on the providers that reach these paths, and a miss-only fallback rule covers unmapped Claude 5 and later ids. * fix(bedrock): route native output and thinking token counts by whether Claude will think Bedrock's native output opt-in now uses the same will-think check as the json tool, so Claude 5 models that think by default with no thinking param go native. Converse also asks for /usage/output_tokens_details on those models, so reasoning tokens are reported for them. * fix(anthropic): tell Claude to answer through the unforced response_format tool Moves the integration tests that check the forced JSON tool from Claude 5 models to Opus 4.8, since Claude 5 models now keep their default thinking and take the unforced tool or native output * fix(anthropic): keep open-object schemas on the json tool when Claude will think Native output closes every object, so an open object came back empty. Also declare thinking_on_by_default in the Rust model catalog and expect the Converse thinking-token field in test_main * fix(anthropic): hold streamed prose until the JSON tool decides the answer In json mode with the unforced JSON tool, Claude can write its working before it calls the tool. The stream now holds that text, drops it when the JSON tool answers, and sends it when Claude calls a caller tool or skips the JSON tool, matching the non-streamed reply. * test(anthropic): build the stream test events without nested comprehensions * fix(anthropic): keep always-on thinking models unforced, open objects native when thinking, and the tool instruction Claude-only * test(anthropic): keep main's open-object integration cases and cover the unforced tool on a default-thinking model * fix(anthropic): treat thinking False as thinking off so the JSON tool stays forced Cover thinking False on the Anthropic and Converse forcing tests, and pin that a Converse json_schema stream with the unforced JSON tool never sends Claude's working text as content * fix(anthropic): drop the thinking False forcing case that completion never reaches litellm.completion turns thinking False into no thinking param before any route sees it, so the case was dead on the request path and False keeps meaning the model's default * test(bedrock): collect the Converse json_schema stream chunks in one assignment --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-05 04:07:11 +00:00
devin-ai-integration[bot]	5c60824024	test(providers): expect 503 server_is_overloaded for an upstream overload (#50154) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-05 03:28:30 +00:00
ryan-crabbe-berri	42b4cfc5bf	fix(nvidia_riva): send the api key exactly once by calling riva over grpc directly (#50140) ... * fix(nvidia_riva): send the api key once per call by talking to Riva over grpc directly Drops nvidia-riva-client from the stt-nvidia-riva extra. Depending on its version it sent the bearer token zero times on plaintext or twice on TLS, so token-checking Riva deployments rejected every request (#50059). The provider now declares the few Riva messages it uses, opens its own grpc channel and puts authorization and function-id on each call exactly once * test(nvidia_riva): run the provider against a real grpc server built from nvidia's own definitions * fix(nvidia_riva): validate endpointing settings with pydantic before building the riva message * test(nvidia_riva): check the key is sent once over tls, and drop a comment that restated the code	2026-11-04 18:07:31 -08:00
Kerry Lu	b662f3f7a4	feat(decisions): add EmpirioLabs AI as a decisions provider (#49977) ... * feat(decisions): add EmpirioLabs AI as a decisions provider empiriolabs/aplomb-1 now works with litellm.decisions and litellm.systemone, the proxy /v1/decisions and /v1/systemone routes, the decision model guardrail and the Admin UI. Requests go to POST /v1/decisions on the EmpirioLabs AI API with the System One body, keyed by EMPIRIOLABS_API_KEY with an optional EMPIRIOLABS_API_BASE Chat models keep routing through the existing OpenAI-compatible JSON provider, and a test pins that * ci: run the EmpirioLabs AI decisions tests in the llms unit shard * fix: align EmpirioLabs registration with CI gates --------- Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 10:41:16 -08:00
devin-ai-integration[bot]	527d882508	test(responses): isolate cancel wire checks from background polling (#50002) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 05:12:01 +00:00
Mateo Wang	d5ca88900c	fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063) ... * fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte. An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError. * test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping * fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529 * test(proxy): expect the top-level Anthropic envelope from count_tokens errors count_tokens now answers a ProxyException as a JSONResponse carrying the Anthropic error envelope instead of raising HTTPException, which FastAPI wrapped in {"detail": ...}. The three legacy error-format tests now read the response body and check that no detail wrap is left. * test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets * fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers, and the OpenAI-shaped routes behave as on main. The error now carries provider_status_code, and the /v1/messages routes answer 529 overloaded_error from it. * refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status * fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload * test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases * fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status * fix(proxy): walk nested mid-stream fallbacks without recursion * fix(anthropic): preserve 503s and answer upstream 529s on messages routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): use integration coverage for keepalive errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 17:37:09 -08:00
Kerry Lu	ce73cf8b9d	chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) ... * chore(cost-map): remove the 52 rows whose provider retired them before 2026 * test(utils): pin the proxy function-calling parity case to azure/gpt-4.1 * test(batches): price the Azure batch-logging regression case against gpt-4.1 --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 16:23:06 -08:00
zoveki	ff182c194e	fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API (#49710) ... * fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API * fix(logging): log bridged chat completions with the chat response they returned * fix(logging): log bridged text completions and /v1/messages as served, with the request's options Bridged rows were translated without the request's options, so structured output and replay fields could differ from what the client got. Text completions and non-streamed /v1/messages through the chat bridge also kept a Responses API object or the encoded id. * fix(logging): redact reasoning summaries on bridged rows and log bridged text completions as text completions Redaction now drops reasoning_items on chat messages and deltas, so a bridged row keeps no reasoning summary when message logging is off. A non-streamed bridged text completion logs the text completion the client got, priced like the chat response it came from. The bridge handler's input TypedDict fields are ReadOnly. * fix(logging): redact bridged text completions for opted-out callbacks and pin the sync logged shape and usage ack * test(integration): look up bridged chat spend rows by the id the client got Bridged chat spend rows are keyed by the client-facing id and logged with the caller's call type, so the helpers that decoded the old resp_<base64> key or pinned call_type responses now assert the new contract --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 10:02:54 +00:00
tessellark	5b546259d5	fix(cache_control)!: keep one deployment's prompt-cache opt-in from reaching siblings on the same model (#49685) ... * fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry Two deployments that share one model string share one key in the live litellm.model_cost map, and router registration writes each deployment's supports_prompt_cache_breakpoint into that key. The cache control hook read the live map, so a flagless deployment picked up a sibling's opt-in and sent prompt_cache_options and prompt_cache_breakpoint upstream, which the provider rejects with a 400 When a call carries the routed deployment's model_info, the prompt cache checks now use its declared supports_prompt_cache_breakpoint, and fall back to the loaded catalog instead of the live map. This covers the hook on completion, acompletion, Responses and the /v1/messages route, the azure_ai chat transform, the OpenAI Responses transform under drop_params, and the /v1/messages to Responses adapter * fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a provider other than openai, such as azure_ai OpenAI models and bedrock_mantle. Bedrock on every route and providers that build Anthropic messages keep cache_control, which Bedrock Converse turns into cachePoint The openai chat reader now passes the routed model_info and checks base_model. The Bedrock cache point reader takes the deployment's flag first and, for a routed call with no flag, reads the catalog as loaded instead of the live map that sibling deployments write into. model_info is threaded from the converse transform, the factory message and tool builders, and the responses bridge * fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations. * fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways * fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases * test(anthropic): cover auto caching on an unmapped Claude deployment * fix(cache_control): keep the merge base's reads where the per-deployment flag changed them A register_model entry under a provider-prefixed name reaches routed calls again, since the overlay now resolves the bare and prefixed names the way runtime registrations are stored. The hook and the openai chat reader no longer judge base_model, as at the merge base, so a client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint and get a 400. A Bedrock deployment flag that is not a real true reads as false again, as at the merge base. An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again, as at the merge base. Other wildcards still leave both cache flags to the catalog. The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps its own flag and that a flagless unmapped ARN keeps its cache points after the delete. * fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name Routed calls read register_model entries through an overlay that merged the bare name after the provider-prefixed one whatever order they were registered in, so a later opt-out under one name lost to an earlier opt-in under the other. Each registration now moves its key to the end of the kept registrations, and the overlay merges both names oldest first, which matches what SDK calls read from the live model map. Also lists the wildcard cache flag helper in the router coverage script, since the hook's wildcard tests drive it rather than calling it by name. * fix(cache_control): keep register_model entries visible while a cost map reload replays them The previous commit moved a re-registered name to the end of the saved registrations by removing and re-adding it, and the overlay walked that dict to order a bare and a provider-prefixed name. A reload replays every saved entry, so a request on another thread could briefly miss an entry or fail on a dict that changed size mid-walk. Entries are written in place again, a per-name sequence records which registration came last, and both the overlay and the reload replay order by it, so the later registration still wins after a reload. * fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:55:51 +00:00
devin-ai-integration[bot]	3da6029977	test(bedrock): assert native structured output when the model rejects forced tools (#49799) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 06:10:00 +00:00
Kerry Lu	fcef9cd0a3	feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos (#49785) ... * feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos A provider registered in custom_provider_map with a BaseVideoConfig handler now resolves on every video call type (create, status, content, list, remix, edit, extension, characters), the same way CustomLLM handlers already resolve on /v1/images/generations. The video entry points and the HTTP handler stop casting the provider name to LlmProviders, which raised before any lookup ran, and a provider without a video config gets a 400 instead of a 500. * fix(videos): type the custom provider lookups so the type gate holds Pick the CustomLLM for a custom provider through ProviderConfigManager.get_custom_llm_handler, which narrows the now wider custom_provider_map handler union, read custom_provider_map instead of the private _custom_providers list, and annotate the untyped custom_llm_provider parameters on the video entry points * test(videos): cover a CustomLLM shadowing a built-in video provider name and type the new tests Mark custom_handler ReadOnly on CustomLLMItem, type the new tests, and assert that a CustomLLM registered under openai still leaves openai video on the built-in config --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-02 21:28:46 -08:00
ryan-crabbe-berri	be50cbe9d8	fix(responses)!: authorize and decode comparison response IDs (#49402) ... * test(responses): reproduce comparison response ID ownership and wire failures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): guard stale retries and malformed comparison envelopes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): authorize and unwrap comparison response ids Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): require comparison IDs to follow response security policy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): deny raw comparison response ids by default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): reject raw comparison response ids by default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): enforce managed ownership with unmanaged ID opt-in Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 03:40:46 +00:00
tavren27	94b5b6386a	fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool (#49703) ... * fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows * fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool Native output without thinking now applies to every Bedrock Claude model that cannot force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5 needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose. The invoke chat path keeps the native output_format it built instead of inlining it, and GovCloud region-path ids read their own false rows. * fix(bedrock): clean caller output_format schemas before sending them natively on invoke * fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke * fix(bedrock): keep the faked Converse stream for native output that replaces the json tool A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture. * test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 01:56:52 +00:00
tavren27	640a87f70d	fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature (#49697) ... * fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): gemini tool-call signature rejected by the backend is resent once with the placeholder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): keep the placeholder resend inside the cache rebuild so either rejection order ends in an accepted body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): resend a cached-prefix signature inline and cover native generateContent When the rejected signature sits inside a cachedContents prefix, the placeholder resend now carries the history inline, since replacing the signature in the request body cannot reach the cache. The native generateContent handlers get the same one-time resend. * fix(gemini): send the history inline when Vertex rejects a signature while creating the cache * fix(gemini): keep the context cache on a signature resend and go inline only when that is rejected too --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 01:24:55 +00:00
devin-ai-integration[bot]	0bd677afef	test(integration): look up streamed Responses spend rows by the client-facing id (#49752) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-02 17:13:41 -08:00
Mateo Wang	4e268c2c55	test(gemini): assert vertex lists on content chunks (#49734) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-02 15:40:32 -08:00
simonp9	250af8f027	fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) ... * fix(responses): log the encrypted response id on streamed /v1/responses spend rows With response id security on, the stream hook encrypts the resp_ id on every frame, but logging had already received the SDK's own copy of the terminal response, so the spend row, the standard logging payload and callbacks got the managed id (or the chatcmpl id on a bridged stream) the client never saw. The hook now records the exact id it advertises on the request's logging object, and the logging layer stamps that id on the logged streamed response * fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response, and a raw provider id still gets a managed id Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or <id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id * fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner A GET or cancel of a stored response now logs a new encrypted id of its own, so callbacks and spend rows never reuse the create's id, while the owner of the id still gets back the exact id they sent. An admin reading someone else's response gets an id bound to the admin, as before. The spend row id suffix for reads and cancels is gone, so with response id security off the rows and session history match the default branch. A streamed GET follows the same rules. * fix(responses): echo the sent id to every reader that passes the response access check A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed the access check, not only when the caller is the user or team the id was issued to. An admin, the master key or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not reach the first turn from it, so a bridged follow-up through that id lost the conversation history Every read and cancel still logs a fresh id bound to the reader, never the create's id * fix(responses): keep a read or cancel row in session history only when it brings new output A GET, streamed GET or cancel sent with a session header joins the create's session, and its stored response repeats the create's output. The replay then sent the same assistant turn once per read. Session history now keeps an aget_responses or acancel_responses row only when its stored Responses output has an item id that no earlier kept row carried. A polled background response keeps the first poll that returns the finished output, since its queued create row has none, and later polls of the same output are dropped. Every other row keeps today's rule * fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart A create row now owns its output items in session history, and a read or cancel row adds only items no create and no earlier read carried, in one linear pass, so a background response polled while it grows, or a read that ends before its streamed create, replays the answer once and after its prompt DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id like a read does Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row id, so a streamed background create and its cost event, which now share the client's id, keep separate files The integration tests that decoded the managed id now match the exact id the client received * fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage file and unbatched GCS object, so batch retrieves and events with no id keep main's names An item a session read replays takes the newest read's copy at the first read's place, so an item a poll caught half written replays finished Tests pin the client's id over the router's continued response id, and the storage test no longer reads the clock after the upload * fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items A streamed background create and its later cost event share the client's id, so OpenMeter dropped the second event and DynamoDB overwrote the create's item. Both now key the cost event as <id>_background_cost, like its spend row and storage file. Session history again replays a read's output items that carry no id, as main did. * fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras * fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id A streamed background create and its later cost event now share the id the client received, so ClickHouse and DeepEval saw both events under one id. Route both through logged_event_key, the way the Azure storage, GCS, OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse request_id matches its spend row's `<id>_background_cost` and its DeepEval span gets its own uuid. ClickHouse response_id stays the raw provider id. * test(deepeval): annotate the background cost span test's locals with Final * test(responses): run the DELETE wire test that pins the echoed client id DELETE /v1/responses/{id} now answers with the id the caller sent, so the skip that marked it as a known bug no longer applies. The test passes on the /v1, /openai/v1 and bare routes. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 14:13:17 -08:00
yuneng-jiang	044bd824c0	test(integration): align remaining release qualification checks (#49714) ... * test(integration): align remaining release qualification checks * test(integration): scope offline resolution to the restricted runner	2026-11-02 12:43:15 -08:00
tavren27	a4de720144	fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests the way their hosts bill them (#49074) ... * fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests like their chat templates Render grok-4-1, grok-4.3 and DeepSeek V4.1 tool schemas the way each chat template does, and add a per-deployment token_count_tool_framing next to token_count_framing for the hidden tool preamble a host bills. The Azure AI Foundry grok-4-1, xAI grok-4.3 and Fireworks DeepSeek V4.1 cost map rows carry the measured framing, overridable in model_info * fix(count_tokens): count Azure grok-4-1 Responses requests with their own framing and use the tool framing in place of the base framing * fix(count_tokens): keep operator framing on tool and Responses counts, frame bridged Responses like chat, follow base_model for tool rendering, count grok tool_choice * fix(token_counter): count Azure Responses strict default, skip false strict and built-in-only tools, per-kind operator framing * fix(token_counter): count an explicit null strict like false on Azure Responses * fix(count_tokens): follow the chat completions bridge flag and Azure's strict default on azure/ deployments Keep Azure grok-4-1 chat tool counts at or over the bill by raising its tool framing to 302 * fix(token_counter): keep strict false for non-grok templates, coerce the bridge flag, frame DeepSeek API counts * fix(proxy): frame Responses counts for the api_base router defaults send inference to * fix(proxy): keep the api_base helper's name so main's /v1/messages count caller resolves --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 12:37:35 -08:00
zoveki	6be98d8c75	fix(bedrock)!: give Invoke and Mantle Claude the model's own default max_tokens (#49692) ... * fix(bedrock): resolve Invoke Claude default max_tokens from the base model id Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): build Invoke Claude request params without mutating Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): type tool-search normalizer return to clear unknown-type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): resolve Invoke and Mantle Claude default max_tokens from the deployment id, ARN base_model, and Mantle row * fix(bedrock): default Invoke Claude max_tokens from the deployment's own cap and honor AnthropicConfig Read the called deployment's model_info.max_output_tokens before the base model's cost row, skip the shared regional key the router fills from every deployment, and leave the default to litellm.AnthropicConfig(max_tokens) when it is set. * fix(bedrock): send the mapped default max_tokens on bedrock_mantle Claude chat * fix(bedrock): bound a global AnthropicConfig max_tokens by the deployment's own cap * test(bedrock): annotate the new Mantle default max_tokens tests with Final * refactor(bedrock): inject the AnthropicConfig max_tokens reader into the Invoke Claude config --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 11:42:45 -08:00
yuneng-jiang	f8f12a270a	test(integration): repair stale contract fixtures (#49700)	2026-11-02 11:23:11 -08:00
zoveki	508a98586e	fix(databricks): join plain text blocks for non-Claude json_object requests (#49385) ... * fix(databricks): join plain text blocks for non-Claude json_object requests * fix(databricks): join only the last message for json_object and drop cache_control under drop_params Databricks checks for "json" only in the last message, so earlier turns now go out as sent and only the last message, plus any system messages merged into it, is joined into a string. Under drop_params a text block's cache_control is dropped so that message can be joined too, and without it the request still goes out as a list. * fix(databricks): skip dropped assistant turns and blank parts when joining for json_object The json_object join picked the last message before the request transform drops bare assistant turns, so a trailing thinking-only or reasoning-only turn (a Responses input ending in a reasoning item included) left the real last user message as a list and Databricks still returned its 400. The pick now skips the turns that transform drops. The join also ran when only one part had text, which sent whitespace-only parts the existing empty block filter used to drop on requests that already worked. It now joins only two or more non-blank parts, so a request with one non-blank part goes out exactly as before. Without drop_params, a message with a message-level cache_control now goes out as sent instead of being rewritten into one cached block. * fix(databricks): join json_object text parts only on models Databricks hosts Databricks checks for the word "json" only on the models it hosts (databricks-* pay per token names and system.ai.* AI Gateway names). An external model endpoint skips that check and takes a list of text parts today, so its request now goes out as sent instead of joined. The drop_params read now goes through drops_unsupported_params, and a test pins that a text block with a truthy non-string text is kept. * fix(databricks): reject a text part holding a non-string locally with a clear 400 * refactor(databricks): reject a non-string text part at the transform_request boundary * fix(databricks): read pydantic message objects in the non-string text check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 11:17:40 -08:00
tavren27	a0b1edc0a0	fix(providers): trim prompt_cache_breakpoint to what each API takes under drop_params (#49064) ... * fix(openai): trim a malformed prompt_cache_breakpoint under drop_params on chat completions * fix(openai): leave a non-object content part for OpenAI to reject under drop_params * fix(openai): trim an unsupported prompt_cache_breakpoint ttl on every path and on azure_ai chat under drop_params * fix: trim prompt_cache_breakpoint to what Azure and Mantle accept under drop_params Azure OpenAI chat now gets the same content part trim as OpenAI chat, and the Azure and Foundry gates also read a deployment's declared base_model. Each Responses config keeps only the ttl its API takes: Azure keeps 30m, and Mantle keeps no ttl. A trimmed ttl is logged at debug. * fix(responses): trim prompt_cache_breakpoint only on APIs known to read it Other Responses providers get the marker whole or not at all, as on main. Azure chat now trims for a deployment with no base_model unless the cost map marks the model as not reading breakpoints, and Azure AI Foundry Responses keeps Azure's 30m-only shape. * fix(azure): trim prompt cache breakpoints only for deployments known to read them Azure and Azure AI Responses keep main's whole-or-nothing rule for unmapped deployment names, an opted-in custom OpenAI host trims the ttl and keeps the mode, and Azure chat strips every marker for a known non-reader under drop_params * fix(providers): keep main's marker handling where the target won't trim it A /v1/responses model with unknown breakpoint support on OpenAI's host gets a marker whole or not at all, as on main. The chat to Responses bridge keeps main's whole-marker check when a non-bool global drop_params leaves the target's trim off. A hyphenated name like gpt-5-6-sol now reads as gpt-5.6. * fix(providers): read a non-version hyphen suffix like -32k as part of the model name * fix(providers): keep main's reading of a dotted minor version like gpt-5.6o * fix(providers): keep main's breakpoint shape for unmapped Mantle models and skip the Azure lookup without drop_params --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 10:00:00 -08:00
tavren27	1046716b6d	fix(anthropic)!: sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept (#49409) ... * fix(anthropic): sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept * test(e2e): send the azure_ai beta check without the code execution tool * test(anthropic): require a vertex_ai and azure_ai decision for every anthropic beta * fix(anthropic): forward the betas Claude Code sends that anthropic, vertex_ai, and azure_ai accept * test(e2e): send Vertex's header-only betas in the beta acceptance case * fix(anthropic): keep forwarding Vertex computer-use betas and point beta tests at platforms that still lack them A live probe found each Claude 4.5 and 4.6 model on Vertex needs one of the two computer-use betas for the computer tool, so dropping both broke it. The e2e acceptance test leaves those two out of its all-at-once Vertex call, and the integration and unit tests that relied on a beta being dropped now use one the platform still lacks. * test(anthropic): point unsupported-beta tests at betas the platforms still drop The Azure AI handler test used claude-code-20250219 as the beta Azure drops, and the Rust parse test used it as an unknown value. This PR forwards it on Azure AI and maps it, so both now use values that stay dropped or unknown. * fix(anthropic): add computer-use-2025-11-24 for the computer_20251124 tool The computer_20251124 tool fell through to the default computer-use-2024-10-22 beta, which the provider rejects for that tool version. Map it to computer-use-2025-11-24, which the beta map now forwards on anthropic and azure_ai * fix(anthropic): send no computer use beta for an unknown computer tool version An unknown computer tool version fell back to computer-use-2024-10-22, which the anthropic and azure_ai columns now forward. It now gets no beta, as every platform effectively sent before. Every computer tool version also skips defer_loading and allowed_callers, computer_20251124 included * test(azure_ai): read the branch's beta map in the auto-injected beta wire test * fix(anthropic): keep defer_loading and allowed_callers on the computer_20251124 chat tool * test(anthropic): read the branch's beta map in the azure_ai safeguards beta wire test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 05:57:51 -08:00
Mateo Wang	3155aacda4	fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests (#49468) ... * fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): return the server tool call id instead of a TypeGuard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the Gemini cache write carries the system prompt Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-31 12:27:17 -07:00
Kerry Lu	948445c828	fix(watsonx): read pass-through credentials from use_in_pass_through deployments (#49462) ... * fix(watsonx): read passthrough credentials from use_in_pass_through deployments /watsonx/{endpoint} always called the provider config with api_key=None and api_base=None, so it only ever worked from WATSONX_API_KEY / WATSONX_API_BASE. It now asks PassthroughEndpointRouter for the proxy-wide watsonx deployment flagged use_in_pass_through, like the other passthrough routes, and keeps the env fallback when none is flagged. Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com> * fix(watsonx): keep env key precedence when no deployment is flagged Resolve only use_in_pass_through deployments in the route. get_proxy_wide_credentials also falls back to WATSONX_API_KEY, which put it ahead of WX_API_KEY in generate_iam_token --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>	2026-10-31 11:53:25 -07:00
paveln83	27fbcd9895	fix(anthropic): map pause_turn and return refusal explanations on chat and responses (#49192) ... * fix(anthropic): map pause_turn and return refusal explanations on chat and responses pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw stop_reason in provider_specific_fields. Responses reports a paused turn as incomplete with max_output_tokens and shows the unrun server search as an in_progress web_search_call instead of a client function_call. A refusal's stop_details.explanation now fills message.refusal and the final stream delta.refusal on chat, and becomes a refusal content part on Responses, streamed or not. * fix(anthropic): let a chat client resend a paused turn so Claude continues it * fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message Chat cannot say where Claude wrote each thinking block in a paused web search turn, and Anthropic answers 400 when one moves but accepts the turn without them. A new message after a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped. * fix(responses): send the streamed refusal part's type and text in content_part.added * fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does * fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses * fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going * fix(responses): mark a paused turn's message item completed * fix(responses): mark every item of a paused turn completed * fix(responses): keep a refusal's explanation when a refused turn is sent back A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed * fix(responses): scan refusal parts in Responses guardrails --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 02:53:59 -07:00
mika-lt	c4b01b2ac8	fix(decisions)!: honor a deployment's own supports_audio_input and supports_video_input flags (#49207) ... * fix(decisions): honor a deployment's own supports_audio_input and supports_video_input flags A Decisions deployment's model_info flags for audio and video now win over the cost map, per media kind, the same way supports_box_questions already does. An unlisted model flagged true gets its clips sent upstream, and a false flag refuses them (or drops them under drop_params). OpenAI-format providers still refuse clips since their request shape has no field for them, and the SystemOne request builder no longer re-reads the cost map. * fix(decisions): ignore media flags on a deployment whose own mode is not decisions A chat-mode deployment's supports_audio_input describes chat input, so it no longer turns on audio or video clips for Decisions. The cost map decides for it, as before. Adds a proxy integration test for flagged, partly flagged, unflagged, and drop_params deployments on both /v1/decisions and /v1/systemone, and for caller-sent model_info. * fix(decisions): follow the cost map's mode for a flagged deployment without its own mode A chat deployment flagged for audio or video usually leaves mode unset, so its flags described chat input yet still let clips through on /v1/decisions. When the deployment sets no mode, the cost map's mode for the model now decides whether its media flags apply, and an unlisted model still takes them. The audio and video lookups also run only when the request carries a clip. * fix(decisions): read the mode a flagged deployment falls back to from the shipped cost map The router copies each deployment's model_info into the model's shared cost map row, so another deployment's mode there could switch off a flagged deployment's media flags. The fallback now reads the cost map as loaded, untouched by router registrations * fix(decisions): route clips in a load-balanced group to a deployment that takes them A Decisions or SystemOne request carrying audio or video now skips group deployments whose own flags or the cost map refuse that media, and keeps the whole group when none takes it, so the 400 still names the media. * fix(decisions): keep deployments that drop or cannot resolve clips in the media filter * fix(router): skip clip-refusing Decisions deployments after cooldown and tag filtering Running the media filter before cooldown turned the 400 a group answered when its only clip-taking deployment was cooling down into a 429. It now runs next to the image filter, just before order filtering, in both the async and sync selection paths. * fix(router): honor router default drop_params when skipping clip-refusing Decisions deployments * test(router): list the router default params helper in the router coverage check * test(bedrock): stop reloading the cost map module in the GovCloud fixture --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 01:09:46 -07:00
mika-lt	e5aa3d32cd	fix(proxy): forward a valid metadata.user_id to Anthropic Messages on pass-through routes (#49333) ... * fix(proxy): forward the client's own metadata on Anthropic and custom pass-through routes * fix(proxy): forward only a valid metadata.user_id to Anthropic Messages on pass-through routes The whole-object forward sent keys Anthropic refuses and reached other providers, so a request that used to pass could start failing with a 400. Read the client's user_id after hooks, drop it unless it is a string with no email or phone number, and send only that key when the upstream URL path ends in /v1/messages. Share the validator with the translated route, whose email check now matches any embedded address as Anthropic's does. * fix(proxy): forward metadata.user_id only to Anthropic upstreams no guardrail scans Limit the pass-through user_id to Messages create on api.anthropic.com or the configured Anthropic base, mirror the checks Anthropic answers 400 for, and keep the field off any request a pre-call guardrail inspects. The translated route goes back to its own user_id filter. * fix(proxy): judge user_id forwarding on the whole request and cap it for configured Anthropic bases * refactor(proxy): pass the user_id check a plain mapping instead of an optional body * fix(proxy): drop an unprintable metadata.user_id and warn when a refused id is dropped Anthropic's whitespace is not Python's, so a control character before the at sign hid an email from the filter and turned a 200 into a 400. Fireworks and Moonshot also fail on newline ids. Requiring a printable id covers both, and a refused id now logs a warning that never carries the value. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 00:16:42 -07:00
ryan-crabbe-berri	5950af8fb5	fix(responses): preserve text verbosity when bridging to chat completions (#49049) ... * test(responses): cover text verbosity on the chat completions bridge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): send text.verbosity as chat verbosity on the chat completions bridge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses): type the bridged text verbosity lookup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): cover text verbosity on the client tool search follow-up bridge turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): cover text verbosity validation on the codex tool search follow-up turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): send text.verbosity on the client tool search follow-up bridge turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): cover text.verbosity on previous_response_id client tool_search follow-up Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 21:41:20 -07:00
yuneng-jiang	fbd126e518	test: fix integration tests that fail on test order, races and intended changes (#49310) ... Seven integration tests fail on main for reasons that are not product bugs. Two are order or race bugs in the tests, one reads rows other tests wrote, one passed on an error body, and three still expect behavior that #48644, #39864 and #39540 changed on purpose	2026-10-30 13:25:47 -07:00
tin-berri	632f71b850	fix(router): recover streamed context window errors before output (#49293)	2026-10-30 13:19:58 -07:00
Maxime Boudier	b20acb4869	fix(proxy)!: persist sent model_info fields on POST /model/update (#46902) ... * fix(proxy): persist sent model_info fields on POST /model/update POST /model/update rebuilt litellm_params from the request but dropped the request's model_info, so changing base_model (or mode, tier, etc.) on an existing DB model was silently ignored. Clients that only use this endpoint, like the Terraform provider, could never fix a deployment's base_model without deleting and recreating it. Persist the model_info fields the caller actually sent, on top of the stored ones. ModelInfo defaults never overwrite stored values, server-owned, team, PTU and pricing fields stay on their dedicated write paths, and values echoed from a /model/info response are not pinned. * fix(proxy): keep echoed and blank model_info values out of POST /model/update writes Blank strings from typed clients no longer overwrite stored values. Values echoed from /model/info (the stored base model's cost map entry and discovered limits) are not pinned, and an echoed catalog value resets a differing stored override like PATCH does. direct_access and access_via_team_ids stay display-only. The tests stop patching the auth check class and give the team lookup a real row instead. * fix(proxy): lock the model row and skip unchanged model_info on POST /model/update * fix(proxy): treat only keyed /model/update bodies as echoes and write the stored model_info as saved A typed model_info value that matched the cost map, or the previous base model's entry, was dropped and could erase a stored override. Echo detection now follows PATCH, only a body carrying the /model/info key counts, compared against the base model the row has after the write. The locked write patches the stored JSON instead of dumping the hydrated Deployment, so litellm_params pricing is no longer copied into model_info. Tests run the real Router with discovery over a mock upstream * docs(terraform): note that model_info updates now take effect and overwrite values set outside Terraform * test: expect POST /model/update to save the cache point flag, and wrap long test lines test_x06 pinned the old contract, where POST /model/update answered 200 and ignored model_info. This PR makes POST save model_info the way PATCH does, so x06 now asserts what its PATCH sibling x01 asserts, the stored flag flips to true and cache points come back on. * fix(proxy): answer 400 when POST /model/update gets an access_groups that is not a list of strings * fix(proxy): read discovered model info through the router registry entry point --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 12:10:13 -07:00
Siddhant Badola	b71491c6c9	fix(bedrock): route a Bedrock model id with no Invoke provider to Converse (#43290) ... * fix(bedrock): route unmapped converse-only family model ids to converse #42713 stopped Bedrock-native OpenAI ids that are not in the cost map yet from falling to the invoke route. The same gap exists for the other families that have no invoke transformation at all: minimax, nvidia, writer, zai, google and xai. An id from one of them that is newer than the cost map hits get_bedrock_route -> "invoke" -> get_bedrock_invoke_provider() -> None -> BedrockError, when Converse would have served it Add a second family regex for those six and default them to converse, the same way the openai family is handled. _OPENAI_FAMILY_MODEL_RE is left alone because the prompt-caching rule is openai-specific Every shipped id of these families already resolves to a converse entry via its base model (checked against the cost map), so this only changes behaviour for ids that are not in the map yet Tests: six unmapped ids route to converse, and an unmapped meta.* id still routes to invoke since meta has an invoke handler * fix(bedrock): route a model id with no Invoke provider to Converse A bedrock/<model> id missing from the cost map fell to the Invoke route, where an id that names no Invoke provider (a new vendor's model, a bare provisioned-model, imported-model, or custom-model-deployment ARN) fails locally with "Unknown provider=None" before any request is sent. Such ids now go to Converse, which replaces the per-family regex with the general rule. Ids that name an Invoke provider and explicit route prefixes are unchanged. * test(bedrock): expect Converse for ids with no Invoke provider in the nova ARN and not-found tests * fix(bedrock): accept tools on Converse for a model with no cost-map row or Invoke provider, and keep a 404 for an invalid id * fix(bedrock): answer 404 for an invalid model id instead of AWS's 400 AWS answers a mistyped Bedrock model id with a 400 ValidationException. The mapping already raised NotFoundError, but passing AWS's 400 response through made the openai base class report 400, so the proxy answered 400 and the router skipped its cooldown * fix(bedrock): drop lookaround regexes from tool schemas for a model the cost map does not list * fix(bedrock): apply the unknown-model defaults only to bare ids LiteLLM knows nothing about A bare Bedrock id with no cost-map row and no Invoke vendor gets Converse's open params, its reasoning_effort as sent, lookaround-free tool schemas, and a 404 for AWS's invalid-identifier reply, which keeps AWS's response headers. An explicit route like converse/, an ARN, or an id LiteLLM recognizes keeps main's params, regexes and error status, so drop_params still drops tools there * fix(bedrock): judge lookaround support by model id on every route, ignore price-only rows The regex check now strips route and region prefixes from the model and its base_model before asking whether LiteLLM knows the model, so bedrock/converse/<id> gets the same answer in a direct transform call as on the completion path. A cost-map row counts only when it names a provider, so a price-only register_model row leaves an unlisted id unknown, as on the proxy. AmazonInvokeConfig now inherits the Invoke provider lookup instead of keeping an identical copy. * fix(bedrock): treat a region path like the bare id when the cost map does not list it A deployment like bedrock/us-east-1/<unlisted id> now gets the same tools, reasoning_effort and typo 404 as bedrock/<unlisted id>, so Claude Code works on it and a mistyped one still answers 404. * fix(bedrock): keep today's 404 for ids that reach Converse only by default, and keep an empty id local * fix(bedrock): answer 404 for an invalid id only when Converse rejected it, and keep a region path with no id local * fix(bedrock): keep a path-shaped id local, and map /v1/completions errors with the model's provider An id with a slash that is not an ARN, like bedrock/../../converse or an unknown region path, keeps Invoke and its local 404 instead of reaching AWS Converse. atext_completion mapped every error as OpenAI, so a Bedrock typo on /v1/completions answered 400 while chat answered 404. It now resolves the provider from the request's model, as the other async entry points do. * fix(text completion): keep /v1/completions error mapping as it is on main Mapping async /v1/completions errors with each provider's rules is wider than this Bedrock routing change. It moves classes, statuses, cooldown and fallbacks for most providers, and some lose context-window fallbacks. So this restores main's mapping here and leaves that change to its own PR. A Bedrock typo on /v1/completions answers AWS's 400, while chat keeps the 404. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 06:27:59 -07:00
simonp9	d1758cda91	fix(router)!: keep a deployment's own model_id over the request body's (#49045) ... * fix(router): keep a deployment's own model_id over the request body's A request body model_id overrode the deployment's pinned model_id, so a client could move SageMaker chat and embedding calls to another inference component. The router now drops the request's model_id when the deployment sets one. * fix(router): let deployments opt back in to a client model_id and stop client component headers on SageMaker chat Drop the request's model_id after the router defaults are applied, so a default_litellm_params model_id cannot win over the deployment either, and apply the same precedence on the sync text_completion path. A deployment that lists model_id in configurable_clientside_auth_params keeps the client's value. SageMaker chat drops any case variant of a client's X-Amzn-SageMaker-Inference-Component header when the deployment sets a component * fix(router): keep a SageMaker deployment's component when it pins it through extra_headers * fix(router): read the SageMaker component header as a pin only on SageMaker deployments * fix(sagemaker): send the inference component only as a header, never as a body model_id SageMaker picks the component from X-Amzn-SageMaker-Inference-Component, so the chat container body and legacy parameters no longer carry model_id. Nova endpoints accept a fixed body field list, so a pinned Nova deployment could reject the extra field. * fix(health): probe header-pinned SageMaker deployments at their component Health checks bypass the router, so a legacy sagemaker/ deployment pinned only through its inference component header sent no component on /health while routed requests now send it. Also update the two SageMaker integration tests that still asserted the old precedence and the dropped body field. * fix(health): apply the router's model_id pin rule to health probes Health probes now share the router's pin function, so a deployment that lets clients set model_id gets no header pin and its own model_id wins without reading the header. Header keys that are not strings are skipped when reading the inference component header instead of failing the call. * fix(sagemaker): keep header keys that aren't strings and report a string opt-in in health checks The chat transformation now keeps extra_headers keys that aren't strings, so such a deployment fails the way it did before instead of answering 500, and the model_id pin checks configurable_clientside_auth_params with isinstance, so a string there reaches the health probe's own error capture again. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 02:40:01 -07:00