mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-26 06:03:19 +00:00
24 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Mateo Wang
|
897dda7920
|
fix(router)!: stop a chat-mode Mantle deployment from rewriting the shared Responses routing row (#46263)
* fix(router): stop a chat-mode Mantle deployment from rewriting the shared Responses routing row A deployment with an explicit custom_llm_provider and model_info.mode: chat used to overwrite the shared litellm.model_cost row, moving every other deployment and plain SDK calls onto Chat Completions until restart, because the downgrade guard read a double-prefixed key and never fired. The guard now reads the real key, and a deployment's own mode decides its routing in responses_api_bridge_check, except on chatgpt and github_copilot where Chat Completions is dead for responses-mode models. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): keep the bridge-check mode tests in the sharded bridge flag file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): let a deployment mode pin replace only the shared-row mode so the bridge capability rules still run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep Responses-only backends bridged under a deployment chat pin and satisfy the type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): give the bridge casts a reason and drop the bare noqa so the type discipline gate passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep a responses row with no supported_endpoints bridged under a deployment chat pin Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): read the downgrade guard's mode from the row registration writes A chat-mode openai/<name> deployment merged into the bare <name> row while the guard read openai/<name>, so the guard never fired and the shared responses row turned to chat. An explicit responses/ prefix now also outranks a deployment's chat pin. * fix(router): let an explicit responses/ prefix outrank a registered chat row The router now registers a chat-pinned openai/responses/<model> deployment under its own name, so the lookup found mode chat and the request went to Chat Completions as responses/<model>. * fix(router): keep the shared row's mode for every deployment pin and honor the pin on /v1/messages A deployment's mode pin now never rewrites the shared cost-map row in either direction, including the responses/-stripped twin key, so siblings and SDK calls keep routing by the registered mode. The backend key formula goes back to main's, which keeps capability fields off rows that registration never wrote at base. The /v1/messages adapter and handler read the deployment's model_info, so a chat-pinned deployment gets a plain reasoning effort on chat completions. * fix(router): keep copilot responses opt-in, make unmapped pins order-free, send Mantle auto summary as auto A responses pin on a github_copilot model or a model missing from the cost map still marks the shared row, since that is how those models opt into the Responses API. A chat pin never marks a row, so registration order no longer matters. Mantle's OpenAI path gets summary auto in place of the auto summary detailed. The Azure o3-pro and o3-deep-research rows drop Chat Completions, which Azure does not serve for them. * fix(router): skip a non-string model_info.mode when keeping the registered mode * fix(router): keep chat-pinned Mantle tool calls and summaries on the Responses API A Mantle row that also serves the Responses API now follows the GPT-5 tool and reasoning summary bridge rules, since Mantle Chat Completions rejects function tools on gpt-5.4+ unless reasoning is off and rejects the summary aliases. Mantle chat gets a dict reasoning_effort as its effort level. The opt-in auto summary asks each Responses config for its level, auto on Mantle's OpenAI path, so a client's own detailed summary follows drop_params as at base. No mode replaces a registered responses mode, and chatgpt and github_copilot rows take any pin until they are responses, which restores the Copilot chat opt-out. * fix(responses): ask the target for its auto summary level on the chat bridge and reasoning_effort * test(integration): run the Mantle region-prefixed v1 base case again This PR routes that deployment by its own chat pin, so the case passes and no longer needs the skip * fix(bedrock_mantle): keep web search and reasoning summaries on the Responses API under a chat pin A chat-pinned Mantle GPT-5 deployment now sends web_search_options and a dict reasoning_effort carrying a summary to the Responses API, which serves them. Mantle Chat Completions drops n above 1 and a reasoning summary under drop_params and answers 400 without it. The /v1/messages adapter probes the bridge with only the client's own summary, so an auto-injected one never moves the request * fix(bedrock_mantle): check n and reasoning summaries only on the Chat Completions request * fix(router): keep a non-string deployment mode off the shared routing row A deployment whose model_info.mode was an int, a list, or an object still reached the shared cost-map row, so an unpinned sibling on the same backend stopped routing to the Responses API. The shared registration now drops a mode that is not a string, and the deployment keeps its own raw value. Adds integration cells for per-deployment routing across the OpenAI and Anthropic SDKs, raw httpx, wildcard routes, malformed modes, an upstream outage, config changes under traffic, and a worker kill plus restart. * test(integration): find each routing cell's spend row by the response it logged A bridged call logs its row under the wrapped Responses id, which the caller never sees on a streamed Messages answer, so look the row up by the answer's own ids and the upstream ids for its marker. Int and list modes 500 their own model group on main too, so those cells check only the sibling, and the summary cell sends the auto level Mantle's Responses path accepts * fix(bedrock_mantle): run extra_body fields through the chat checks A chat-pinned Mantle request that nests n or reasoning_effort in extra_body now gets the same 400, drop, and effort-level handling as the top-level fields. The bridge test injects a MockTransport client instead of patching HTTPHandler.post. * test(router): cover nested reasoning_effort routing, messages thinking summaries and per-target auto summary levels * fix(router): probe a pinned deployment's own API on /health and keep a requested summary beside an extra_body effort * fix(router): keep a custom-priced deployment's mode pin off the shared backend row on each request * fix(router): run per-request custom pricing through the shared mode guard so a health probe keeps the row's mode * fix(router): drop the debug log that wrote a request's model name from keep_registered_mode * test(router): cover custom-priced chat pins, their health probes and summary aliases beside an extra_body effort * fix(proxy): pass only the mode pin to a /health probe so a failing probe never cools its deployment down * test(router): cover failing health probes leaving their deployment out of cooldown, with and without background checks * test(router): kill the worker that holds burst calls, so the worker kill cell never depends on how the kernel spread them * fix(router): route a fine-tune the same whichever deployment registers first Main's register_model now seeds an unmapped fine-tune's row from its base model's built-in row, mode included, on the first registration. The mode guard read only the live row, so a responses pin marked the shared row only when it registered before its siblings. The guard now reads the row the registration merges onto, the live row or else the row it would seed. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d6e6c6f26c
|
fix(responses): map Anthropic tool search to OpenAI server tool search (#46441)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models * fix(responses): define the tool_search_call output item locally for older openai SDKs * fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500 * fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out A replayed Claude tool search call that the caller answered with a tool message kept its tool_result in the user turn while its tool_use was dropped from the assistant turn, so Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when no tool message answers it. The Responses bridge now strips defer_loading from every tool whenever no tool search tool is sent, whether the caller sent none or the route cannot run it, since a deferred tool with no search tool can never be loaded. Namespace members had started carrying the flag to every bridge provider, which providers that reject unknown tool keys answered 422. The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit. * fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried * fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator * fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic A function tool whose allowed_callers is not a list of strings now answers 400 instead of 500. The guardrail scan reads tools without malformed flags, so the route decides between a 400 and a drop under drop_params. When the caller sends a native Anthropic search tool beside tool_search, only the native one goes out. Databricks Claude no longer counts as an Anthropic request route. A previous_response_id follow-up off Anthropic also drops a search call the client replayed as a function_call with a srvtoolu_ id, and its output. The answered tool id scan in the Anthropic message factory runs once per assistant run instead of once per message. A null created_by on tool_search_call and tool_search_output output items is left out of the serialized response, so a cache hit returns the same keys as the first call. * fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan The scan translated tool_search into Anthropic's regex search tool on every route, so a guardrail returning only function tools removed tool_search from native OpenAI requests and OpenAI answered 400 for the deferred tools * fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai * fix(anthropic): keep answered tool search replays paired with their answers * fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails * fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object * fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent * fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id * fix(responses): answer 400 to a tool_search execution that is not server or client OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed execution is dropped like the other malformed tool flags, which leaves a server search * test(responses): cover the 400 and drop_params paths for a malformed tool_search execution * fix(responses): map Anthropic tool search to OpenAI server tool search Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): sync generated tool search API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): type tool search mapping predicates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): validate tool-search mappings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): use the openai SDK ToolSearchToolParam for the mapped tool search * fix(responses): keep deferred chat tools beside a tool search on the responses bridge Restore the hand-rolled OpenAIToolSearchToolParam, since openai 2.20.0 (the supported floor) has no ToolSearchToolParam, and carry defer_loading from chat function tools onto the Responses request when a tool search tool is present, which OpenAI requires for tool_search * fix(responses): leave the bridge's defer_loading carry to its own PR #46439 owns keeping defer_loading on chat function tools sent through the Responses bridge, so this PR keeps only the tool search mapping and the openai 2.20.0 compatible tool type * fix(openai): keep tool search mapping to the OpenAI API and drop it where the model or history cannot take it * fix(openai): narrow the tool search drop to Anthropic's search tool, drop it when nothing is deferred, and run the wire tests against the OpenAI host * fix(openai): pass malformed tools to OpenAI as sent and keep a caller's own tool_search under drop_params * perf(openai): scan Responses history for Claude search calls only when Anthropic's search tool is sent * test(responses): cover the Anthropic tool search mapping to OpenAI Responses across routes, settings, and chaos * test(integration): define the fallback cell's deployments in the gateway config so both workers load them * test(integration): keep the worker-kill burst held past the replacement wait * fix(responses bridge): count an Anthropic search tool as hosted tool search on the chat bridge main's #46439 refuses defer_loading when no tool_search tool is in the request, and the chat bridge only counted {"type": "tool_search"}, so the Anthropic bm25 or regex tool this PR maps to OpenAI's tool search was refused before the mapping ran Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect the chat bridge to map the search tool now that defer_loading survives --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
52fc37888b
|
fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444)
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): give the defer_loading casts a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): read replayed tool-search and namespace fields through typed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep tool_search items on a turn with no message or tool call Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep every reasoning item on a search-only turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): leave content unset on a search-only turn so replay adds no empty message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): replay tool search items only with tool_search, count replayed tools in the allowlist, keep namespace off guardrails * fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses Under drop_params, defer_loading is dropped when no tool_search tool is sent, so the 400 it would cause upstream goes away as before. A non-bool value answers 400 without drop_params and is dropped with it. A null tool-level value falls back to the nested function value. A request carrying a tool_search tool now bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort none, since chat completions rejects the tool_search type. * fix(responses bridge): replay tool search items in output order and count replayed tools in tool policy Store the ordered output prefix through the last tool search item, reasoning included, so a second reasoning item no longer makes OpenAI reject the replay. The shared chat-to-Responses converter now replays tool search items and namespace only when the bridge opts in, so Noma, the complexity router and guardrail payloads keep their shape. The tool policy guardrail and the allowlist count replayed tools only when the request can send them. * fix(responses bridge): forward defer_loading only beside a tool_search tool defer_loading only means something with tool_search, and backends without it reject the key (xAI answers 403). A true value with no tool_search tool now answers 400 naming drop_params and is dropped under drop_params; a false value there is a no-op and is dropped. * fix(tool_loop): keep tool search items and namespace between rounds * fix(responses bridge): check replayed tool search loads when a deployment adds tool_search * fix(chat): drop litellm response fields before strict OpenAI-compatible providers A chat reply from the Responses bridge carries provider_specific_fields on the message and on each tool call, plus reasoning_items. Groq, Fireworks, Mistral and Together reject unknown fields, so sending that reply back to them as history failed with a 400 or 422. Strip them in the shared helper and use it in those providers. * fix(caching): keep reasoning_items and tool search items on response cache hits A cached bridge response lost reasoning_items on a non-streaming hit and lost both reasoning_items and provider_specific_fields (tool_search_items) on a streamed hit, so the next turn could not replay the search or the reasoning around it * test(integration): cover tool search item replay on the chat to Responses bridge Proxy, SDK, authorization and chaos cells for returning and replaying tool_search_call and tool_search_output items and the tool call namespace * test(integration): add response cache twins for the tool search guardrail cells Each guardrail cell now has a twin where the answer is already in the Redis response cache: a key opted out of global guardrails caches a replay that loads a policy-blocked tool, the same replay without guardrails is cached before block mode sees it, rewrite mode is not served the unrewritten conversation's cached answer, and an allowed replay is served from the cache. * fix(responses bridge): forward defer_loading beside a client-executed tool_search The bridge now replays a client tool_search_call and the client's tool_search_output through provider_specific_fields.tool_search_items, so a deferred function beside a client tool_search keeps defer_loading like it does beside a hosted one * test(responses bridge): pin the client tool_search turn and its replay, streamed and not * fix(responses bridge): strip nested namespace for strict providers and keep streamed reasoning_items on text replies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep dropping defer_loading beside a client-executed tool_search Forwarding defer_loading beside a client tool_search made GPT stop with a search call the chat client has to answer by hand, so a plain question that used to get tool calls got none. Keep main's behavior of dropping it there. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
c70dd4ece6
|
fix(responses bridge): keep defer_loading on function tools sent to the Responses API (#46439)
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): give the defer_loading casts a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses Under drop_params, defer_loading is dropped when no tool_search tool is sent, so the 400 it would cause upstream goes away as before. A non-bool value answers 400 without drop_params and is dropped with it. A null tool-level value falls back to the nested function value. A request carrying a tool_search tool now bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort none, since chat completions rejects the tool_search type. * fix(responses bridge): forward defer_loading only beside a tool_search tool defer_loading only means something with tool_search, and backends without it reject the key (xAI answers 403). A true value with no tool_search tool now answers 400 naming drop_params and is dropped under drop_params; a false value there is a no-op and is dropped. * fix(responses bridge): drop defer_loading beside a client-executed tool_search Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
6d1a3bce2c
|
fix(responses): return the first message item as content for structured output on the chat bridge (#46508)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b6df3d7fe5
|
fix(responses): number bridged chat tool calls 0, 1 and pin the remaining bridge shapes (#46109)
* test(responses): pin bridged commentary plus final answer as one chat choice Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): pin reasoning, two-message and two-tool-call bridge shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): number bridged chat tool calls 0, 1 instead of all 0 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4b7b8efd4c
|
fix(responses): fold bridged Responses output into one chat choice (#46099)
* fix(responses): fold bridged Responses output into one chat choice A Chat Completions request bridged to /responses got one choice per Responses message item, so a reply shaped [message, message, function_call] put the tool call on choices[1] and clients reading only choices[0] never saw it. Fold every message text, annotation, reasoning item and tool call into choices[0] Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): drop mutable annotations and the extra cast in the choice fold Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): remove a cast-ok marker that suppressed nothing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): join folded message texts with a newline Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(responses): join folded message texts with no separator so stream and non-stream match Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
bbe5595a99
|
fix(responses): keep a flagged hosted deployment's prompt cache breakpoint on the chat bridge (#45500)
* fix(responses): keep a flagged hosted deployment's prompt cache breakpoint on the chat bridge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): read the serving provider's row for prompt cache breakpoints on the chat bridge and base_model * fix(anthropic_cache_control_hook): rename the module-level provider resolver so the recursion check passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): follow the credential dialog's new name field and auth method id in the federation e2e spec Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
42d2158894
|
fix(responses): map input_audio blocks in the chat-to-Responses bridge (#45224)
* fix(responses): map input_audio blocks in the chat-to-Responses bridge * fix(responses): honor base_model when deciding to drop input_audio under drop_params * fix(responses): carry a dropped audio part's cache breakpoint to the preceding part Dropping an input_audio part under drop_params lost the prompt_cache_breakpoint riding on it, so an injected system marker landing on a trailing audio block left prompt_cache_options explicit with no breakpoint on the wire. The marker now moves to the nearest preceding kept part when that part has none. Update the CircleCI-only Responses bridge wire tests that pinned input_audio to its stringified input_text shape: video_url becomes the stringified kind so the marker-on-stringified-block coverage survives, and dedicated audio tests assert the forwarded part and the drop_params carry-over. * refactor(responses): carry dropped audio markers in one pass Replace the per-block slice that re-read the rest of the content list for every retained part with a single reverse scan, and skip the copy for a message that carries no audio part. Type the unit test helper's messages as a sequence of mappings instead of bare lists. * test(responses): add integration cells for input_audio parts through the bridge Cover the chat-to-Responses bridge's input_audio handling on the real proxy: the part is forwarded as input_audio without drop_params and dropped under drop_params unless the model or its base_model supports audio input, across httpx, the OpenAI SDK (sync stream and async), the Anthropic SDK, native /v1/responses, tool and assistant messages, hostile input_audio values, duplicate parts, an audio-only message, marker carry-over, a null drop_params, a global litellm_settings.drop_params, a mixed burst, and a burst with dropped upstream connections * test(responses): check every upstream attempt of a dropped connection The dropped-connection cell required exactly one upstream POST per call, which tied it to the retry policy instead of the fix. It now groups the recorded POSTs by marker, requires one attempt per answered call, and checks that every attempt of a doomed call carries the dropped audio shape. |
||
|
Mateo Wang
|
9d3bf29d6d
|
fix(responses): keep cache breakpoints on blocks the bridge stringifies (#40032)
* fix(responses): keep cache breakpoints on requests that reach the Responses API The chat-completions bridge rebuilt every content block without the prompt_cache_breakpoint marker the cache-control hook had just placed on it, so the request went out with prompt_cache_options set to explicit mode and nothing actually marked. Explicit mode caches only what is marked, so those deployments lost the implicit caching they were getting before the injection point was added The native surface had a second miss. The bridging check looked the provider config up with the still-prefixed model while the config keys on the bare one, so every bedrock_mantle model read as having no native Responses support, and role-targeted injection points were deferred to a chat-completions pass that never runs for them * fix(responses): keep prompt cache breakpoint on stringified content blocks * test: cover the responses/ routing prefix in the native cache point case * fix(responses): keep only the fallback-branch marker carry, main already routes the hook #42281 landed the text, image_url and file branch carry and the implicit default on main, and the Responses routing lookup this branch changed has no observable effect there, so the PR shrinks to the unknown-block fallback branch and its regression test * fix(responses): drop a malformed prompt_cache_breakpoint under drop_params in the chat-to-Responses bridge * fix(responses): accept the 30m prompt_cache_breakpoint ttl under drop_params OpenAI's Responses API takes a ttl of 30m on an explicit breakpoint marker, so the drop_params validation keeps it instead of stripping it as an unknown key. Trims the bridge test docstrings to the dated vendor citation. * test(integration): audit cells for prompt_cache_breakpoint through the chat-to-Responses bridge Adds the /audit cells for the bridge's prompt_cache_breakpoint carry: valid and malformed markers on every carrying block kind and on tool and assistant content, the drop_params on and off contract, the hook-injected system marker, the Anthropic SDK path through the Responses adapter, upstream errors, idempotent spend rows, and the chaos cells (mixed burst, upstream outage, slow streams, worker SIGKILL and a proxy restart mid burst), all against a scripted Responses endpoint with the request it received read back by response id * test(integration): give every scripted responses reply its own id The prompt-cache-breakpoint audit's scripted upstream minted the response id from the request's marker, so three identical marked requests shared one id. The spend-log writer skips rows whose request_id already landed, and the idempotent-logging cell saw one row for three calls on every leg. The upstream now mints a unique id per reply like a real provider, and the cells match a response to its request by the marker inside that id. |
||
|
devin-ai-integration[bot]
|
d910653b33
|
fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge (#44119)
* fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge Preserve cache-breakpoint markers through the bridge for supported models and drop them for models without breakpoint support Co-authored-by: Simon Sorg <simonsorg13@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): honor base_model when gating bridge cache breakpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): avoid recursive cache-breakpoint stripping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): justify bridge stripping casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep cache breakpoints out of non-bridge converter callers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): read prompt_cache_breakpoint without validating content blocks The marker read ran every chat content block through TypeAdapter(dict[str, object]).validate_python, which rejects dict blocks with non-string keys that chat completion callers passing Python dicts could previously send; the request then failed with a pydantic ValidationError on the bridge keep path, the strip path, and the image/file conversions alike. item is already isinstance-narrowed to a dict at every read site, so read the marker with dict.get directly and drop the adapter. Adds a regression test covering text/image_url/file blocks carrying non-string keys on both the keep (gpt-5.6) and strip (gpt-4o) paths. * fix(responses): cast content block before reading prompt_cache_breakpoint basedpyright flags the raw dict.get read as reportUnknownArgumentType (+2 against the error budget); cast the isinstance-narrowed block to dict[str, object] first, matching the strip helpers' cast-ok idiom. * style(responses): ruff-format the marker-read cast * fix(responses): hoist one cast-ok content block read for the type gates A Final assignment inside the conversion loop trips reportGeneralTypeIssues, and per-site casts trip the LIT006 budget; read the marker through one cast-narrowed local instead. --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Simon Sorg <simonsorg13@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Simon
|
48c88bae04
|
fix(responses): preserve prompt cache reuse in chat bridge (#42281)
* fix(responses): preserve prompt cache breakpoints in chat bridge * fix(responses): preserve multimodal cache breakpoints * fix(cache): retain implicit lookup with injected breakpoints * test(cache): expect implicit responses lookup * test(cache): expect implicit chat lookup |
||
|
devin-ai-integration[bot]
|
c3a23fe499
|
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3242bdfed2
|
fix(responses): honor request cache controls on chat completions bridged to the Responses API (#44676)
* fix(responses): honor request cache controls on chat completions bridged to the Responses API Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): assert spend and cache-hit status on bridged no-cache rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): unskip LIT-9196 openai_responses basic translation cases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): restore azure attribution check on bridged no-cache rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): exercise bridged cache controls through a real local cache instead of patched responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
5724117116
|
fix(responses): merge bridged tool calls into the same choice as the text (#44346)
* revert(responses): revert "fix(responses): keep gpt-5.4/5.5 tool calls on chat and merge bridged tool calls into one choice" (#44295)
This reverts commit
|
||
|
devin-ai-integration[bot]
|
d96e56c76f
|
revert(responses): revert "fix(responses): keep gpt-5.4/5.5 tool calls on chat and merge bridged tool calls into one choice" (#44295) (#44344)
This reverts commit
|
||
|
devin-ai-integration[bot]
|
ca1994e403
|
fix(responses): keep gpt-5.4/5.5 tool calls on chat and merge bridged tool calls into one choice (#44295)
* fix(responses): keep gpt-5.4/5.5 tool calls on chat and merge bridged tool calls into one choice Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): preserve deferred logging bridge coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): cover gpt-6 family bridge routing and merged tool calls in integration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
fe910889f7
|
fix(responses): drop bridge-minted reasoning items from OpenAI replays (#44132)
* fix(responses): drop bridge-minted reasoning items from OpenAI replays * fix(responses): send id-less stored reasoning items without a made-up id The chat-to-Responses bridge gave a stored reasoning item with no id an rs_<n> id that OpenAI rejects (404 without encrypted content, 400 with it); an id-less item is accepted and verified by OpenAI itself. Decoding encrypted_content now keeps the verifiable thinking blocks of a mixed array instead of rejecting the whole array, and the verifiable-block rule lives in the shared module. * test(integration): audit minted reasoning item replay across responses, chat bridge and messages --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e814532033
|
fix(streaming): keep the served service_tier on streamed chunks and spend rows (#42870)
* fix(streaming): keep the provider's served service_tier on streamed chunks and spend rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): satisfy type-discipline and strict ruff budgets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): stamp the served service_tier on every Responses bridge chunk Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic-adapter): expose streamed chunks so disconnects bill partial spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(service-tier): cover anthropic and responses served-tier billing paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic-adapter): return a chunks-exposing stream so disconnects bill partial spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(service-tier): bill disconnects through the router's anthropic stream wrapper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: apply ruff format to the anthropic stream changes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coverage): ignore delegating properties the ast scan cannot see Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: keep the cast-ok reasons on the cast call line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover served service_tier billing for streamed chat and messages, complete and disconnected Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic-cache): delegate chunks/messages/model through the messages stream cache writer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): keep service_tier on OpenAI-compatible parsed chunks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(streaming): parameterize delegated chunks and messages types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): follow the anthropic pass_through rename after merging main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): drain the logging worker between response cache tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): cover azure, databricks, responses bridge and gemini served tiers in the stream billing integration test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(databricks): keep the served service_tier on streamed chunks and bill it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(databricks): type the served service_tier chunk without a loose kwargs dict Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): bill the served service_tier over the requested one Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost): drop explanatory comment from the tier resolution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: kerry <kerry@berri.ai> |
||
|
yuneng-jiang
|
f6882246d4
|
test: move tests/test_litellm root and small trees into tests/unit (#43186)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng
|
d086f8574d |
test: add sync and async custom_llm_provider bridge propagation tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng
|
30a422087c |
test: add __init__.py to migrated tests/unit packages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng
|
72abd11b4c |
test: assert the responses bridge forwards aws_region_name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng
|
03a650c8a9 |
test: migrate wave 1 phase 1 legacy tests to tests/unit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |