mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
307 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Mateo Wang
|
d76ec3a47e
|
fix(proxy)!: answer 400 for malformed tools on count_tokens instead of crashing (#46583)
* fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing A non-list tools value crashed the vector store lookup during auth and the built-in tool cost lookup in the failure logging path, so the proxy answered an unhandled 500 and closed the keep-alive connection. count_tokens now answers a 400 invalid_request_error for a malformed body, drops malformed tools when drop_params is on, and both lookups skip values of the wrong shape. * fix(proxy): skip non-list tools in the tool allowlist check Also validate the count_tokens body without recursion, which the recursion detector rejects. * test(integration): restore the int tools case on count_tokens The proxy now answers a non-list tools value with a 400, so the case is deterministic again. * fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids * fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids Check tools after request setup so the Claude Code default and the logged call type apply, read the request, global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off * fix(proxy): keep the request's call type when logging a gate rejection after request setup The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses rejection as aembedding, so it now only applies when the logging object carries no known call type * fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry * fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket * refactor(proxy): leave the responses websocket out of the request-level tool check * fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections * fix(proxy): read router default drop lists and deployment ids in the malformed tools check * fix(proxy): leave malformed tools to the router when it can pick another deployment Fallbacks, auto-routers, model strings, comma batches and team model names let the router pick a deployment after the proxy checks the request, so the proxy now forwards a malformed tools field there as main does, and only rejects it when the deployment it checks is the one that serves. Token counting never reroutes, so count_tokens and input_tokens keep reading the group they count with. additional_drop_params names are read the way core reads them, so a list with a non-string entry still names its strings. * test(integration): cover malformed tools on reroutable models, router defaults and deployment ids * fix(proxy): resolve malformed tools at the deployment the router picks * fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells. * test(integration): expect a 400 for an empty-string tools field on /v1/messages Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test * fix(router): keep web search routing for tools passed as a tuple * fix(router): keep a deployment's own tools when drop_params drops a malformed request tools * test(router): list _merged_tools as covered through the deployment tools merge tests * fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them * fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError * fix(prometheus): keep deployment health metrics clean on a malformed tools rejection * test: type the new malformed tools tests * test: mark new test locals Final and type kwargs dicts |
||
|
AxelRay
|
388251a5b8
|
fix(bedrock): drop dangerous-tool-use beta without safeguards (#44471)
* fix(bedrock): drop dangerous beta without safeguards * fix(bedrock): drop the dangerous-tool-use beta only next to structured outputs, keep it on Mantle Bedrock rejects the pair with 'invalid beta flag' on Claude models before Opus 4.7, while requests without structured outputs and Bedrock Mantle accept the beta, so the drop now only applies to Invoke requests with output_config.format or a strict tool and no safeguards. * fix(anthropic): keep the AnthropicThinkingParam import that main now uses * fix(bedrock): drop the client's dangerous-tool-use beta whenever safeguards is absent Bedrock rejects the beta with 'invalid beta flag' on Sonnet 4.5, Opus 4.5, and Opus 4.6 even without structured outputs, so dropping it only next to structured outputs left those beta-only requests failing. The beta does nothing without safeguards, so Invoke now drops it in every such request. Mantle keeps it. The thinking display helper goes back to main's version. * fix(bedrock): drop the client's dangerous-tool-use beta on Invoke chat without safeguards * test(bedrock): wire cells for the dangerous-tool-use beta on Invoke messages, chat and responses * test(bedrock): type the new dangerous-tool-use beta tests * test(bedrock_mantle): type the dangerous-tool-use beta Mantle test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
f8217d2a03
|
fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count (#46843)
* fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count /v1/messages runs clear_tool_uses context edits before it sends an Anthropic body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted the body unedited, so a request with older tool results cleared counted about 2.8x what inference bills. The count now runs the same history and context-edit step inference runs, honors additional_drop_params on the deployment, and skips compact edits so a count never makes a paid summary call. * fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs Run compact edits in the count without the summary call so trigger checks and history slicing match /v1/messages, honor a request-level additional_drop_params, keep the full body when any deployment in the model group drops context_management, and answer 400 naming a block Gemini cannot count when an edit trigger fails on it * fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check * fix(count_tokens): log a failed edit in the count at debug, since the send already logs it * fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does * fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits. * fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name * fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to * fix(messages): drop an invalid context_management spec under drop_params so the send matches the count * fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes. * test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code |
||
|
devin-ai-integration[bot]
|
859529ccbc
|
fix(realtime)!: keep turn_detection.type when a transcript guardrail disables auto-response (#44939)
* fix(realtime): keep turn_detection.type when a transcript guardrail disables auto-response * test(realtime): type the GA session.update helper and mark its locals Final * test(realtime): expect typed turn_detection on the voice guardrail session updates * test(realtime): accept either gate path for the client's first voice session.update * fix(gemini): drop VAD tuning when the guardrail turns auto-response off * fix(gemini): check create_response without a redundant isinstance * fix(vertex_ai): keep the client's VAD tuning on Vertex when auto-response is off * fix(realtime): keep the transcript-guardrail gate typeless on xAI so it does not auto-answer * fix(realtime): replace any client turn_detection with the untyped gate on xAI xAI answers on its own under a typed server_vad even with create_response off, so a client's typed turn_detection must not reach xAI while the transcript guardrail gates the turn. * fix(realtime): keep a client's early turn_detection and hide the guardrail cancel's error A session.update sent before session.created reached OpenAI ahead of the proxy's own gate, so the gate replaced the client's turn_detection. The proxy now re-sends the client's gated turn_detection after the gate when the two differ. The guardrail's response.cancel now carries a litellm_guardrail_ event_id, and the error OpenAI returns for it (response_cancel_not_active when nothing is answering) no longer reaches the client. * fix(realtime): send the guardrail gate before a client's early session.update and let Muse and Chirp fall back to server_vad under drop_params * fix(realtime): hide only the cancel errors that echo the guardrail's own response.cancel ids Track the event ids of the guardrail cancels this session sent and drop a backend error only when its error.event_id is one of them, so a client's own event id with the same prefix still gets its error. Cover the scripted response.cancel reply in the integration upstream, update the stale list-mode test to the typed gate, and mark the xAI cancel error cell as an expected failure, since xAI replies with its own event id. * test(integration support): deliver the disconnect a real server sends after the client goes away The scripted upstream now keeps reading frames after a reply hits a closed socket, so the unit test's fake ASGI receive has to end with the websocket.disconnect message a real server delivers. * fix(realtime): delete the blocked transcript item before the guardrail's refusal turn OpenAI GA answered the blocked question in some refusal turns because the blocked user item stayed in the conversation. The raw OpenAI-protocol path now sends conversation.item.delete for it right after the cancel. Provider config backends have no such event, so that path no longer passes an item id. Also bound the guardrail integration file by the shared graceful-stop timeout and make the xAI cancel-error xfail require an AssertionError. * fix(realtime): keep the blocked item on xAI, which cannot delete an audio input item * test(realtime): assert the xAI voice block sends no conversation.item.delete upstream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
2dc6aafa6b
|
fix(proxy)!: count prompt-only token_counter requests with the provider when call_endpoint is set (#46839)
* fix(proxy): count prompt-only token_counter requests with the provider when call_endpoint is set * fix(proxy): keep the local count for a prompt-only token_counter request when the provider count fails * fix(proxy): log a prompt-only token count fallback as a one-line warning * fix(proxy): send a prompt-only token_counter request to the provider only on Gemini and Vertex AI * fix(proxy): count a token_counter prompt sent next to an empty messages or contents list * fix(proxy): count a prompt next to messages on the local token count instead of answering 500 * fix(proxy): answer 400 for a token count prompt next to a conversation unless drop_params is set With drop_params the conversation is counted without the prompt. On the local count, a prompt sent with system or tools is counted as a user message with them, like the messages form. * test(proxy): count tools with a prompt in the token counter fallback test * test(proxy): cover prompt counting on /utils/token_counter in the Gemini count wire tests * test(proxy): name the token counter fallback test by what it checks |
||
|
Marcus
|
642a2ad934
|
fix(prompt-templates): treat initial_prompt_value and final_prompt_value as optional (#39762)
* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(model_hub): surface model_info.description in model group info and Model Hub UI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): aggregate model group description without in-place mutation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prompt-templates): treat initial_prompt_value and final_prompt_value as optional completion() only stores a custom prompt template key when its value is truthy, and custom_prompt() already defaults both prompt values to "". Seven call sites read them back with direct indexing, so a template that sets only roles raises KeyError instead of rendering, on ollama, petals, vllm, predibase, codestral, anthropic text completion, and response_schema_prompt. Five other providers already read defensively, so the same config works on sagemaker and 500s on ollama. Read both keys with .get(key, ""), matching those five providers and the defaults custom_prompt() already declares. Templates that set all three keys render exactly as before. * test(prompt-templates): drop the explanatory docstrings from the optional-keys regression test * fix(health): skip background health check DB writes when the latest-row read fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prompt-templates): read every custom prompt template field through one typed helper Templates that omit roles, initial_prompt_value, or final_prompt_value now render with empty defaults across every provider that reads them, instead of failing with KeyError or TypeError. A field with the wrong type fails with a validation error naming it. * fix(prompt-templates): keep templates that rendered on main working and fail bad fields with a 400 Only the roles a request uses are checked, null fields mean unset, and extra role keys are ignored, so every template main rendered renders the same. A wrong-type field now raises a BadRequestError naming it. initial_prompt_value is declared litellm-owned so it stays out of provider request bodies, like final_prompt_value. The template tests move into the mapped factory test file, and the predibase test asserts the real rendered prompt * fix(prompt-templates): keep bad-template failover and render non-mapping role entries like main A wrong-type template field now raises a 500 InternalServerError naming the field, so the router still retries and cools down the bad deployment. A roles value or role entry that is not a mapping renders unwrapped as it did on main, and role keys are only checked for roles a request uses. * fix(prompt-templates): warn when a custom prompt template roles value is not a mapping * fix(prompt-templates): keep deployments in rotation on a bad template and reject a template with nothing to render * test(integration): audit custom prompt templates across endpoints, providers, and chaos --------- Co-authored-by: milan <milan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
debanshd
|
1b1ddef1b2
|
fix: preserve non-null anyOf branches when stripping null types for Vertex (#41683)
* fix: preserve non-null anyOf branches when stripping null types for Vertex * refactor: build anyOf branches immutably and drop explanatory comments * style: apply ruff format * fix: annotate anyOf list construction for the LIT002 budget * fix(vertex_ai): send a null-only anyOf as one nullable null instead of raising * test(vertex_ai): pin empty anyOf passthrough * fix(vertex_ai): forward a non-list anyOf unchanged and add wire cells for the null fold * fix(vertex_ai): forward a non-object anyOf branch for Google to judge instead of crashing * fix(vertex_ai): fold null out of type arrays and union branches before sending to Gemini --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4e49394ade
|
fix(anthropic): keep trailing user messages with None or empty list content (#41397)
* fix(anthropic): keep user messages with None or empty list content so requests do not end on an assistant turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): avoid mutable empty-list construction in sanitizer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): avoid rebinding sanitizer input Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): keep a trailing user turn whose content key was dropped litellm.completion strips None fields from every message before the provider transform, so a user message sent with content null reaches the Anthropic converter with no content key and raised KeyError. Read the content with get so that turn gets the same placeholder as an empty list. * fix(anthropic): only fill user turns with empty content and keep their cache_control * fix(anthropic): fill each empty user message in place like an empty string * fix(anthropic): add the placeholder only for an empty turn or a cache breakpoint, in linear time * test(anthropic): cover the tool-result guard with cache_control and widen the linear-time bound * test(anthropic): cover empty user turns through the proxy on chat, responses and messages * test(anthropic): type the empty user turn tests and move the linear-time check to a benchmark --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
b21e659c24
|
fix(proxy)!: count endpoints return errors in their source API shape instead of silent local counts (#46815)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): reject unconvertible blocks nested inside a tool_result * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system * fix(proxy): count endpoints return missing-key and request errors in their source API shape * fix(proxy): drop imports the count error change left unused * fix(proxy): let Entra-authenticated Azure AI Claude count tokens and keep Google count errors in the Google envelope * fix(proxy): send Azure environment keys as x-api-key on Azure AI Claude counts and log unexpected Google count failures * fix(proxy): count with the global OpenAI and Anthropic keys chat already uses * fix(proxy): resolve the OpenAI count base with the key so a global key stays on its gateway * fix(proxy): keep unexpected Google count failures generic so internal details stay out of the response * fix(proxy): resolve the Azure AI Claude count base from the same settings chat reads * fix(proxy): count OpenAI tokens with the workload identity bearer chat already uses * fix(proxy): mint the OpenAI workload identity bearer inside the count error handling * fix(proxy): report a failed OpenAI workload identity exchange on the count endpoints as a 401 --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
yucheng-berri
|
b5870df967
|
fix(bedrock-passthrough)!: return AWS-shaped errors so AWS SDKs read the code and message (#46522)
* fix(bedrock-passthrough): return AWS-shaped errors so AWS SDKs read the code and message A guardrail block on a /bedrock route now answers with x-amzn-errortype ValidationException and a top-level message, and an error Bedrock returned (such as a 429 ThrottlingException) is passed through with its status, x-amzn-errortype and body * fix(bedrock-passthrough): keep 5xx guardrail failures and failure-hook replacements unchanged * test(bedrock-passthrough): guardrail vendor failures and unknown models keep the litellm error * fix(bedrock-passthrough): only answer guardrail exceptions that blocked content as ValidationException * fix(bedrock-passthrough): pass Bedrock's content-type through with its error body * refactor(bedrock-passthrough): wrap the guardrail ProxyException instead of copying its fields Unit tests drive the error response with the real request processor and ProxyLogging * test(bedrock-passthrough): cover the direct model route, invoke, and auth, budget and rate-limit errors * fix(bedrock-passthrough): pass through only errors a Bedrock model operation returned A guardrail service's HTTP error that carries x-amzn-errortype keeps the litellm error * refactor(bedrock-passthrough): mark Bedrock's errors where the passthrough raises them instead of matching the URL A guardrail service's HTTP error never goes through the passthrough call, so it keeps the litellm error whatever its URL --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
79ca820933
|
fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert (#46772)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): reject unconvertible blocks nested inside a tool_result * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
Mateo Wang
|
dba86bec28
|
fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them (#46767)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
Kerry Lu
|
5ed3629242
|
fix(responses): write one spend-log row per native WebSocket turn (#46768)
* fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn * test(responses): mock the websocket slot release and mark the per-turn start time naive * style(responses): keep the per-turn start time line within ruff format * refactor(responses): type websocket turn events as read-only mappings * fix(responses): settle the websocket budget reservation on the first turn only Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly * fix(proxy): keep the legacy limiter's websocket slot until disconnect A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d25b58f744
|
fix(count_tokens): count Anthropic-format requests for OpenAI the way /v1/messages inference converts them (#46759)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * test(count_tokens): expect the converted Responses input in the budgeted-key count integration test --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
Mateo Wang
|
b47f7a4712
|
fix(responses): preserve explicit websocket API keys (internal copy of #40853) (#46724)
* fix(responses): preserve explicit websocket API keys Refresh the one-line fix from BerriAI/litellm#34253 against the current default branch Co-authored-by: puffer <389053104@qq.com> * test(responses): verify websocket authorization Cover explicit deployment keys, precedence over global keys, and the existing global fallback at the upstream WebSocket transport boundary * test(responses): type the authorization capture Replace the mutable header list with a typed, final Future carrying the upstream authorization value, addressing the review on #40853 * Refactor test_openai_websocket_authorization method * fix(responses): resolve websocket keys with the deployment's provider Pass the deployment custom_llm_provider to get_llm_provider so an api_base host never swaps in another provider's env key, and drop the OpenAI-only global key fallbacks so non-OpenAI websocket deployments resolve their own env keys the way the HTTP route does * fix(responses): let a deployment's own websocket key beat env keys A deployment with an api_base on a host litellm recognizes, and no declared provider, got that host's env key ahead of its own api_key. The explicit key now always wins, matching the HTTP Responses route. Drop the litellm_params api_key fallback, which could never be set because api_key is a named argument * fix(responses): keep openai deployments on other providers' known hosts on the HTTP stream An openai deployment whose api_base is a known OpenAI-compatible provider host (DeepSeek, Groq, ...) opened a native Responses WebSocket there and got a 405, because those hosts serve /responses over HTTP only. Route it through the managed HTTP stream with the deployment's own key instead. * fix(responses): resolve openai deployments on known provider hosts as that provider An openai deployment whose api_base is a known provider host (DeepSeek, Mistral, ...) now resolves to that host's provider on the Responses WebSocket again, as before this branch, so it keeps that provider's chat completions bridge, pricing and env key fallback. Its own api_key still wins over the env key. The known-host test now mocks the upstream HTTP route and checks the route, the bearer and response.completed. * test(responses): prove websocket deployment keys and declared providers reach the provider socket * test(responses): use the reported bare model shape for the websocket deployment key cell --------- Co-authored-by: Ysh <734645956@qq.com> Co-authored-by: puffer <389053104@qq.com> |
||
|
mrinal-berri
|
bd60cc8608
|
fix(bedrock): forward Nova Sonic realtime tool calls with their arguments (#46214) | ||
|
yuneng-jiang
|
b602209ca6
|
test(integration): hand every harness server a port from one per-worker block (#46761)
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit
|
||
|
Mateo Wang
|
d489c04554
|
fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking (#46741)
* fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking * fix(bedrock): drop unknown thinking keys under drop_params on display updates rebuilds Bedrock Invoke copied every caller thinking key when adding display updates, so an unknown key still reached the provider with drop_params on. Filter that copy the same way, and cover each chat call site plus the global litellm.drop_params path in tests * fix(anthropic): drop budget_tokens from rebuilt adaptive thinking under drop_params * test(anthropic): cover drop_params sources for rebuilt adaptive thinking on /v1/messages and Bedrock Invoke |
||
|
Mateo Wang
|
7090588bdf
|
feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240)
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions * fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel * feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing early-access row, carrying the same limits and capabilities. The AI Studio row mirrors the early-access entry; the Vertex row differs only in provider and pricing source. Prices follow once Google publishes them. Extend the mapped metadata test to cover all three ids, that each routes to its provider, that its backup copy matches, and that the public rows carry the same limits as the early-access row. * feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops * chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap * fix(gemini): follow continuation hops in a loop instead of recursion The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls * fix(gemini): bill Google Search grounding from every continuation request Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain * fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch * chore(gemini): name the type errors the dual-mode stream iterator trips The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up * fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment * fix(gemini): keep one response id across a continued answer A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key * test(gemini): add integration cells for long decode continuation Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once * fix(vertex_ai): fetch the access token again before each continuation request A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged * fix(vertex_ai): start each continuation stream with an empty parser buffer When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async * fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way * fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes * fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not * fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0 A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer Adds the audit's wire tests for both, which fail 24 cells before this change * test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed * fix(vertex_ai): log a continued answer's raw response as one object A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch * test(integration): check a continued answer logs one merged raw response A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch * test(vertex_ai): cover a follow-up tool call and its thought signature in long decode * test(gemini): pin that a one-request answer logs Google's body unchanged |
||
|
Mateo Wang
|
84ff9f84a9
|
fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses (#46720)
* fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses * fix(ui): send detail on playground Responses image parts and code snippets * test(messages): cover image detail on the Responses bridge wire |
||
|
Mateo Wang
|
62af39aaf8
|
fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it (#46591)
* fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it With modify_params on, a trailing tool search turn whose call has a kept result got a placeholder answer from the factory and went out as a marked tool_use, but the cache control hook counted marks before that placeholder existed and skipped the mark, so a request at the cap reached Anthropic with 5 marks and failed with a 400. The hook now finds client answers on the sanitized messages, the same list the factory reads. Also make the replay checks cheaper (prefix check before validation, early return without provider_specific_fields, and look up client answers only when a tool call carries cache_control), drop the two audit cells that cover old gaps outside the tool search fix, and derive the pass-through test timeout from the graceful stop budget. * fix(anthropic): keep the duplicate tool result warning out of the cache control hook The hook finds client answers with only the step of the modify_params sanitize pass that pairs tool calls with their results, so a request with a duplicate tool result logs the warning once again instead of once per mark count. Also restore the Snowflake non-stream and cached-stream tool search cells as known bugs with pytest.skip instead of deleting them, and put the Anthropic pass-through test timeout back to 240 seconds. * fix(anthropic): count cache marks next to a message the pairing step cannot read With modify_params on and a marked tool call, the cache control hook paired tool calls with their results before counting. A message without a get method (a stray string or null in extra_body.messages, or an openai SDK ChatCompletionMessage) made that pairing raise AttributeError before any provider call, where main counted the marks and let the request through. The hook now pairs only when every message can be read the way the pairing step reads it, and otherwise counts on the messages as sent, like main did. * fix(anthropic): count cache marks after placing a later system message the way the request builder does * test(anthropic): cover a leading system prompt and a system message before the answer in the cache mark census * test(anthropic): align the tool search replay cells with main and cover a system message after the search turn The malformed kept result cells now expect the unanswered search call to be dropped, as main does since #45882. The stream cut cell checks streamed tool call names, since the kept search result names get_weather as a tool reference. The streamed response cache hit cell runs again because #46104 fixed it. The modify_params cap cell expects the four client marks, since the placeholder answer sends the search call as a plain tool_use with its mark, and a new cell covers a system message after the trailing search turn in both stream modes * fix(anthropic): count marks on the raw messages when they cannot be placed, and find client answers once per request * test(anthropic): cover marked tool calls next to a malformed tool id and a later system message * perf(anthropic): look up client tool answers only when a marked tool call is a server tool call Answered ids only decide whether a marked srvtoolu_ call is rebuilt as server_tool_use, so a request whose marked tool calls are all client calls no longer places system messages or runs the modify_params pairing step to count marks * test(anthropic): reach the client answer lookup with a marked search call in the malformed tool id cell * perf(anthropic): place mid-conversation system messages in linear time for models that keep them * fix(anthropic): count cache marks with the model's own system message placement and pair tool results in linear time * test(anthropic): cover modify_params dropping a tool answer that answers no call of the latest assistant turn * fix(anthropic): count marks with in-place system placement when the model lookup cannot confirm the flag * test(anthropic): cover a bare model with custom_llm_provider filling the cache mark budget through litellm.completion and Router * fix(anthropic): give unmapped models the request builder's converting placement and survive any model lookup failure * test(anthropic): cover an unmapped model name counting cache marks with the request builder's converting placement * fix(anthropic): count server tool call cache marks the request builder's way only on routes that build Anthropic Messages requests Databricks, OpenRouter and Bedrock Converse send every marked tool call as is, so the hook keeps the census main uses there. The seeding step carries custom_llm_provider on the injection points when a server tool call is marked, so a bare model name still resolves its route. * test(anthropic): cover two answered marked server tool calls before a system message on Databricks, OpenRouter and Bedrock Converse * fix(anthropic): keep main's cache mark count on GitHub Copilot and ChatGPT routes without resolving them, so a sync call with no stored login never starts the device flow * test(anthropic): cover a marked server tool call reaching the GitHub Copilot and ChatGPT login hosts like an unmarked one |
||
|
mgeorgaklis
|
b3c9d4c1f7
|
fix(prompt_templates): do not mutate messages in-place in function_call_prompt (#41676)
* fix(prompt_templates): do not mutate messages in-place in function_call_prompt * refactor(test): avoid local list mutation and satisfy type discipline gate * style(prompt_templates): keep line length under 120 chars for ruff format * style(prompt_templates): type _with_function_prompt with AllMessageValues for ANN401 * style(prompt_templates): eliminate basedpyright errors in function_call_prompt * fix(prompt_templates): keep tuple and None system content when adding the function prompt Missing system content becomes the function prompt alone, a tuple keeps its parts, and the old test that pinned the caller-side mutation now asserts the caller's messages stay unchanged * fix(prompt_templates): reject unreadable system content without echoing it Only list and tuple content is read as parts. Any other non-string content raises a TypeError that names the type, so error logs never carry the system prompt * fix(prompt_templates): leave non-list system content failing as before None and tuple content fail the same way they did before this change, so the only behavior change is that the caller's messages are no longer mutated * fix(prompt_templates): answer 400 for unreadable system content, send the function prompt alone under drop_params * test(prompt_templates): pin the function prompt on every system message * test(integration): cover function prompt fallbacks, retries, cache and drop_params on ollama --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
da2f5a498f
|
feat(agents): support Databricks Agent Platform agents in the agent gateway (#45681)
* feat(agents): support Databricks Agent Platform agents in the agent gateway Add a databricks_agent provider that speaks the MLflow ResponsesAgent wire (Responses-shaped input, output_text items, output_text.delta SSE) to Model Serving endpoints (/invocations or the unified /serving-endpoints/responses) and Databricks Apps (/responses), with a PAT or OAuth M2M client credentials Register it through the agents config block or the Admin UI agent catalog and call it through /a2a/<agent> (message/send, message/stream) or model: a2a/<agent> chat completions. The chat route now runs every completion-bridge agent through its own provider, and attaches the minted Databricks OAuth token for both bridge and URL agents. Consumed OAuth settings never reach the provider call * fix(agents): harden databricks_agent input, output, and OAuth handling Non-text content parts answer 400 unless drop_params is set, a reply without an output list answers 502, flat OAuth fields resolve an os.environ api_base, the bridge debug log no longer echoes the request model, routing tests stub only the HTTP boundary, and the catalog description fits the dropdown * fix(agents): merge agent and client extra_headers on the a2a chat route, and reject an empty databricks_oauth block * fix(agents): bridge only databricks_agent agents on the a2a chat route, keep the card url over a client api_base, and require workspace_url for a Databricks App * refactor(agents): read the a2a chat bridge providers from a constants set instead of branching on a provider name * fix(databricks_agent): let the configured credential win, reject malformed client extra_headers, and keep a missing endpoint name empty * test(agents): drive the flat OAuth a2a send through the real bridge * fix(agents): keep an empty databricks_oauth block ignored on url agents, as before this change * fix(agents): accept bare App hosts, env service principals, and os.environ refs on Databricks agents * fix(databricks_agent): address review caveats on chat bridging, params, and auth Map max_tokens and max_completion_tokens to max_output_tokens and pass temperature, top_p, and user, so /v1/messages works without drop_params. Send A2A message metadata only with string values, answering 400 or dropping the rest under drop_params. Join message items with a blank line in both stream and non-stream replies. Cache the env service principal token until shortly before it expires. Keep databricks_agent replies out of the response cache. On the chat bridge, let request fields win over the agent's stored params while pinning the agent's api_base and api_key, and apply the agent's static_headers between client and agent extra_headers, as /a2a does. Resolve os.environ/ only for databricks_agent api_base and api_key instead of every bridge provider. Ignore an unusable databricks_oauth block on URL agents on chat, as at the merge base, instead of answering 400. * test(internal_context): list the databricks agent token cache as process-local * fix(databricks_agent): keep the agent's endpoint, input, and target out of client control extra_body model and input answer 400, or are dropped under drop_params. A client api_base or base_url never replaces the agent's card URL on the bridge. Chat to a URL agent keeps its merge-base call, and an incomplete databricks_oauth block on a URL agent keeps its merge-base ValueError. * test(databricks): expect the merge-base 400 for a client api_base on agent chat * fix(databricks): keep agent-owned inputs and identity headers out of a chat caller's reach * fix(databricks): bound the agent-over-caller merge to two levels so it needs no recursion --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
1938f537bf
|
fix(decisions): read a deployment's own supports_box_questions flag before the cost map (#46696)
* fix(decisions): read a deployment's own supports_box_questions flag before the cost map The box-question check only read the shared cost-map row and required a decisions mode there, so an OpenAI-format deployment whose model_info sets supports_box_questions was refused with a 400, which kept the wire test from #46633 red on main * fix(decisions): keep the box flag on its deployment and out of request bodies A model_info that is not a plain dict no longer fails the call, the proxy decisions endpoint drops a caller-sent model_info, and the router stops copying supports_box_questions onto the shared cost-map key. * fix(decisions): keep the box flag shared with same-model deployments like main Scoping supports_box_questions to one deployment turned box questions on an unflagged sibling of the same backend model from reaching the provider into a 400. Siblings inherit the flag through the shared cost-map row again, as they do on main. Also formats two new tests. * test(decisions): pin box flag inheritance by an unflagged same-model deployment * test(decisions): audit box-question deployment flag across providers, routes, and chaos cases |
||
|
Kerry Lu
|
fac0135d82
|
fix(search): bill an unpriced fallback search tool at its own price, not the failed primary's (#46734)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
3151c47960
|
fix(guardrails): resolve decision models in caller team scope (#46085)
* fix(guardrails): resolve decision and judge models in caller team scope * fix(guardrails): scope guardrail model calls to the caller's team only where it owns the deployment Read the team only from the auth object the proxy attached to the request, never from caller JSON, and use it only when the router's route to the guardrail model for that team reaches a deployment a teamless caller cannot. Shared names, aliases, legacy team tags and wildcards keep resolving and billing as before. Decision calls pass the team in litellm_metadata only when it is used, so the logged model group stays intact. Adds integration cells for chat, responses and messages, forged team ids, a team deployment that shares a name with a shared model or wildcard, and billing, plus unit cases for legacy team tags and aliases. * fix(guardrails): route team guardrail calls to the team deployment without charging the team Guardrail calls on a team's own model now go to the deployment the router picks for that team, by its id, with no team on the call. Like every other guardrail call, they write no spend row and add nothing to the team's or the key's spend * fix(guardrails): pin the team deployment only for the judge guardrail, not shadow eval judge calls * fix(guardrails): keep the teamless route when only a team wildcard covers the guardrail model * refactor(guardrails): scope the team model fix to the decision model guardrail only * fix(guardrails): send decision calls on a team's own model with the team, and keep team wildcards out The decision call on a team's own model now carries the caller's team instead of being pinned to a deployment id, so it is charged to that team and goes through the team's normal routing: an access window that reserves the deployment for the team lets the call through, and pre-call checks see the input and skip a deployment too small for it. A team deployment listed under a wildcard public name, such as team_public_model_name openai/*, no longer captures a guardrail model the team never named, whatever model it points at. Adds unit and integration cells for the reserved deployment, mixed context windows under enable_pre_call_checks, a team wildcard name over a fixed model, and the owner team's spend. * fix(guardrails): log the owner team's name on decision calls charged to it The decision call on a team's own model now carries the team alias beside the team id, so the spend row and the Logs page name the team the call was charged to. * fix(decision_model): use the caller's team only for names nothing else serves A team model named like a shared deployment, a shared wildcard or an SDK public name took over the guardrail's decision calls for that team, so a team admin could point a proxy-wide guardrail at their own endpoint. The team is now used only when a caller without a team reaches no model under the name. * refactor(decision_model): drop the team route subset check the served check covers Once nothing serves the name without a team, every deployment a caller without a team can reach is a wildcard, so the subset check only ever rejected an empty team route. Pin team wildcards over a name nothing else serves. * fix(decision_model): use the caller's team only when judge_target finds nothing without a team The served check rebuilt the router's rules and missed a shared wildcard whose provider comes from custom_llm_provider, so a team model under the guardrail's name took over that team's decision calls. Ask judge_target, the same lookup the teamless dispatch makes, instead --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Taylor Hawkes
|
fd84e97f49
|
feat(providers): add Inference APIs as OpenAI-compatible provider (#41669)
* feat(providers): add Inference APIs as OpenAI-compatible provider Registers https://api.inferenceapis.com/v1 as a JSON-configured provider (chat completions and Responses API). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(providers): list Inference APIs in Add Model and the provider docs index, add request-level tests - provider_create_fields.json entry so the Add Model dropdown test passes - provider_endpoints_support.json (+ backup) entry so the provider docs check passes - tests now capture the outgoing HTTP request (URL, auth header, body) instead of only inspecting registrations - drop a redundant comment Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(providers): inject a recording HTTP session instead of patching httpx.Client Request-level tests now set litellm.client_session to an httpx.Client with a MockTransport, so nothing is monkeypatched on a class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * chore(providers): drop redundant comment on the inferenceapis entry Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * feat(inferenceapis): add cost map rows for the 12 chat models and test the responses route Prices and context windows come from https://inferenceapis.com/pricing and GET /v1/models. Moves the provider test to tests/unit and adds the interactions flag the endpoint matrix now carries. * feat(inferenceapis): price the vendor's chat model aliases and mark /v1/messages supported * fix(inferenceapis): serve /v1/messages natively and keep audio routes off the provider * fix(json providers): send provider-specific params in extra_body and report context overflow as ContextWindowExceededError * fix(exception mapping): match the Inference APIs overflow phrase instead of any context_length_exceeded substring * fix(exception mapping): map context overflow to ContextWindowExceededError for Inference APIs only Providers without their own mapper keep reporting an overflow as BadRequestError, since the router raises a ContextWindowExceededError instead of trying generic fallbacks when context_window_fallbacks has no entry for the group. Adds the integration cells for the provider, its fallbacks, and the extra_body and overflow families. * test(inferenceapis): hold upstream replies across the worker kill in the chaos cell A reply the killed worker sent before SIGKILL can lose its queued spend row, which the product never promised to keep. Holding the upstream until after the kill means only the surviving worker answers, so every 200 must log exactly once. * test(inferenceapis): make the chaos cell kill the quieter worker and always release held replies The survivor now must hold at least one in-flight request, so the burst check cannot pass as 0 == 0, and the held vendor is released on exit so a failed assertion cannot stall the wire server teardown --------- Co-authored-by: TaylorHawkes <857858+TaylorHawkes@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
17f621e116
|
fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836)
* fix(vertex_ai): route Model Garden models through the generateContent adapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vertex_ai): annotate generateContent route test parameters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): type litellm_params in get_vertex_ai_model_route * fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions Streamed Model Garden calls through the generateContent adapter now request stream usage where the deployment supports it, keep cached-token details, and look up the provider-relative cost key, so they bill like the non-stream call. The adapter forwards router metadata, passes upstream errors through with their status, countTokens falls back to local counting for non-Gemini routes, and the native Gemini config is limited to the Gemini routes. * fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing Streamed generateContent through the chat adapter now ends with the upstream usage, honors always_include_stream_usage false and the caller's stream_options, counts one failure per failed call, and routes Vertex deployments by their base_model like chat does * fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage * test(google_genai): expect the trailing usage chunk on bridged OpenAI streams The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone. * fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk * fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent * test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire * test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
08a24d391a
|
fix(count_tokens): count system and tools on Vertex Gemini, fall back to Mantle on Bedrock 404 (#46430)
* fix: align token counting with inference context Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve Vertex Gemini count message shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: forward Google countTokens tools in shapes every provider counter accepts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: add package marker for Vertex count-token unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): ignore deployment system_instruction when counting tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): forward system and tools on Google AI Studio, keep strict on Anthropic tools Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(count_tokens): ruff format gemini count handler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): return Mapping from count request body builders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): truncate long Anthropic tool names on Vertex count, update stale strict test expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): simplify Gemini count context and tests Put the Vertex Gemini count branch back in the same shape as main and the partner model branch, keeping only the system and tools context, the provider error mapping and the gemini tokenizer default Share one error response helper between the Vertex and AI Studio callers, drop the unread kind fields and the extra validation of map_function output, and say Gemini instead of Vertex AI in the error that AI Studio users also see Tighten the tests by removing a stray parametrize on a helper, sharing the SDK count args, asserting the exact forwarded system and tools, and folding the Anthropic tool shape tests into one table * fix(count_tokens): reuse inference tool translation for Vertex Gemini count Vertex and AI Studio count now convert tools with the same functions inference uses, translate_anthropic_tools_to_openai and the Responses tool converter, before VertexGeminiConfig.map_function. Unnamed and Anthropic hosted tools now count the same declarations inference sends instead of being dropped Tools that cannot be converted, such as a non list tools value, a non object tool or a schema that is not a JSON object, now return a 400 before any request instead of being counted without them --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
606ecf56de
|
fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals (#46633)
* fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove an OpenAI-format box answer reaches the caller as a box Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): give the box wire test a deployment that advertises supports_box_questions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Sushit Lal Pakrashy
|
958cba7016
|
fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results (#38284)
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
c00706218b
|
fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097)
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(vertex_ai): keep yield outside the JSON decode guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): surface BaseLLMException raised while streaming a request body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(http_handler): only follow __cause__ when unwrapping streamed body errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): avoid recursive streamed error unwrapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows Keep main's error path for a directly raised provider error and only unwrap one found behind a transport error. Add the parser column to invalid JSON errors, and cover the real aiohttp chain in the _handle_error test * fix(vertex_ai): report every row conversion error as a 400 naming its line A message with no role, a non-string role, a bad temperature, or a litellm BadRequestError raised while converting a row still returned a 500, and the async upload lost the 400 for litellm's own exceptions. The stream now turns any error from one row's conversion into a 400 naming that line. _handle_error only unwraps an httpx.TransportError, which is what the aiohttp transport raises around a failing request body, so other callers keep their own status. A file that starts with a UTF-8 byte order mark now uploads, and provider messages start with 'Batch input file line N' like the proxy check. * fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line A chat row whose body is missing or not an object now raises a 400 naming its line instead of uploading a blank prompt, and a non-object body on line 1 no longer crashes the GCS object name lookup. Rows too deeply nested or too long to parse return a 400 from both the SDK and the proxy's own batch file check. JSON error columns now count leading whitespace. * fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line * test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos * fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact * test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Raushan kumar
|
f9d45f8a43
|
fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)
* fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kolade Fajimi
|
35970c3eac
|
fix(vertex): report an unreadable credentials file as a read failure (#40173)
* fix(vertex): report an unreadable credentials file as a read failure Tell a path apart from inline JSON by shape, not os.path.exists(). Fixes #40166 * test(vertex): narrow pytest.raises to the expected message * fix(vertex): log the credentials path instead of returning it to the caller * refactor(vertex): log the credential source decision where it is made * fix(vertex): handle paths open() rejects with a bare ValueError * fix(vertex): redact the credentials path before it reaches the log * test(vertex): cover the brace fallback and non-text credentials file * test(vertex): exclude the unreachable exhaustiveness guard from coverage * refactor(vertex): read the failure's fields off the matched object CodeQL's py/uninitialized-local-variable does not follow keyword capture patterns in a match statement and reported `path` and `reason` as possibly unbound. Narrowing on the class and reading the attributes says the same thing without a suppression. * fix(vertex): accept a read-only mapping in extract_aws_params * fix(vertex): keep encoded credentials out of the log and cover the legacy route * fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust * fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file * fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message * test(vertex): integration cells for credentials file and inline value failures * test(vertex): drive the Rust messages cell through Vertex Claude --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
4496fe0ebc
|
test: cover response contracts across translation endpoints (#46632)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
a9d6123a43
|
test: expect the list-position index on streamed thinking blocks in the integration suite (#46618)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d6e6c6f26c
|
fix(responses): map Anthropic tool search to OpenAI server tool search (#46441)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models * fix(responses): define the tool_search_call output item locally for older openai SDKs * fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500 * fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out A replayed Claude tool search call that the caller answered with a tool message kept its tool_result in the user turn while its tool_use was dropped from the assistant turn, so Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when no tool message answers it. The Responses bridge now strips defer_loading from every tool whenever no tool search tool is sent, whether the caller sent none or the route cannot run it, since a deferred tool with no search tool can never be loaded. Namespace members had started carrying the flag to every bridge provider, which providers that reject unknown tool keys answered 422. The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit. * fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried * fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator * fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic A function tool whose allowed_callers is not a list of strings now answers 400 instead of 500. The guardrail scan reads tools without malformed flags, so the route decides between a 400 and a drop under drop_params. When the caller sends a native Anthropic search tool beside tool_search, only the native one goes out. Databricks Claude no longer counts as an Anthropic request route. A previous_response_id follow-up off Anthropic also drops a search call the client replayed as a function_call with a srvtoolu_ id, and its output. The answered tool id scan in the Anthropic message factory runs once per assistant run instead of once per message. A null created_by on tool_search_call and tool_search_output output items is left out of the serialized response, so a cache hit returns the same keys as the first call. * fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan The scan translated tool_search into Anthropic's regex search tool on every route, so a guardrail returning only function tools removed tool_search from native OpenAI requests and OpenAI answered 400 for the deferred tools * fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai * fix(anthropic): keep answered tool search replays paired with their answers * fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails * fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object * fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent * fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id * fix(responses): answer 400 to a tool_search execution that is not server or client OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed execution is dropped like the other malformed tool flags, which leaves a server search * test(responses): cover the 400 and drop_params paths for a malformed tool_search execution * fix(responses): map Anthropic tool search to OpenAI server tool search Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): sync generated tool search API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): type tool search mapping predicates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): validate tool-search mappings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): use the openai SDK ToolSearchToolParam for the mapped tool search * fix(responses): keep deferred chat tools beside a tool search on the responses bridge Restore the hand-rolled OpenAIToolSearchToolParam, since openai 2.20.0 (the supported floor) has no ToolSearchToolParam, and carry defer_loading from chat function tools onto the Responses request when a tool search tool is present, which OpenAI requires for tool_search * fix(responses): leave the bridge's defer_loading carry to its own PR #46439 owns keeping defer_loading on chat function tools sent through the Responses bridge, so this PR keeps only the tool search mapping and the openai 2.20.0 compatible tool type * fix(openai): keep tool search mapping to the OpenAI API and drop it where the model or history cannot take it * fix(openai): narrow the tool search drop to Anthropic's search tool, drop it when nothing is deferred, and run the wire tests against the OpenAI host * fix(openai): pass malformed tools to OpenAI as sent and keep a caller's own tool_search under drop_params * perf(openai): scan Responses history for Claude search calls only when Anthropic's search tool is sent * test(responses): cover the Anthropic tool search mapping to OpenAI Responses across routes, settings, and chaos * test(integration): define the fallback cell's deployments in the gateway config so both workers load them * test(integration): keep the worker-kill burst held past the replacement wait * fix(responses bridge): count an Anthropic search tool as hosted tool search on the chat bridge main's #46439 refuses defer_loading when no tool_search tool is in the request, and the chat bridge only counted {"type": "tool_search"}, so the Anthropic bm25 or regex tool this PR maps to OpenAI's tool search was refused before the mapping ran Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect the chat bridge to map the search tool now that defer_loading survives --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
d0fa05076d
|
fix(xai): name a completed xAI batch's results file and stop looking it up on xAI's files API (#45559)
* fix(xai): report a completed xAI batch's results as its output file details * fix(xai): keep a short results download and per-call timeouts out of the reported file details * fix(xai): describe a batch's results file from the batch alone and let the batch job fill its size Reading an xAI batch id as a file now costs one batch read and never downloads the results: the file reports a 0 byte placeholder, the results filename and the registration time. The managed proxy fills in the size the batch cost job measures when it downloads the results, keeps a known size when a provider reports none, and drops the per-process results download cache. * fix(managed_files): keep a measured batch output size when a refresh races the batch job A refresh that read the row before the batch cost job stored the measured size could save the provider's unsized details over it. The refresh now reads the row back from the writer and saves only while the row still holds what it read, retrying when another write landed in between. The new tests drive the real xAI and OpenAI retrieve paths through a fake HTTP upstream. * fix(managed_files): guard the refresh save with the row the request already read The conditional save first compares against the row the request read, so a refresh adds no database read unless another write landed in between; only then does it read the row back from the writer and try again. * fix(managed_files): save a refreshed file object with one guarded write and no extra database reads A refresh that loses its guarded write to a newer one now leaves that row alone and drops this worker's cached copy, so the next read loads the newer row instead of retrying against the database on the request path * fix(managed_files): retry the batch cost job's measured size after a read refreshes the row first The batch cost job re-reads the row from the writer and saves again when its guarded write loses to a concurrent refresh, so its measured size is never dropped. Reads without a measured size still make one write and no re-read * fix(managed_files): size a concurrently refreshed row instead of restoring the batch job's placeholder copy When the batch cost job loses its guarded write and the row it re-reads holds refreshed details, it applies its measured size to those details, so the results filename and any size the provider reported are kept * fix(managed_files): keep a batch output's earlier created_at when a refresh replaces its placeholder A refresh saved over a stored row now keeps the earlier of the two creation times, so an xAI placeholder keeps the time the proxy registered it instead of the read time, while a provider's own earlier creation time still wins * test(integration): an xAI placeholder row keeps its registration time after the proxy refreshes it * test(integration): drop the cached output row so the placeholder cell reads the seeded database row * test(integration): drop the cached output row without requiring it to still exist * fix(managed_files): limit the batch output size fill, guarded save and kept created_at to xAI rows * fix(xai): keep the batch id as the batch output filename Managed xAI batch output files keep the filename main already returns (the batch id) instead of renaming them to <batch id>_results.jsonl. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
c0663d5202
|
fix(anthropic)!: translate the OpenAI tool_search tool on /v1/chat/completions for Claude models (#46389)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models
* fix(responses): define the tool_search_call output item locally for older openai SDKs
* fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields
Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500
* fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out
A replayed Claude tool search call that the caller answered with a tool message kept its
tool_result in the user turn while its tool_use was dropped from the assistant turn, so
Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when
no tool message answers it.
The Responses bridge now strips defer_loading from every tool whenever no tool search tool is
sent, whether the caller sent none or the route cannot run it, since a deferred tool with no
search tool can never be loaded. Namespace members had started carrying the flag to every
bridge provider, which providers that reject unknown tool keys answered 422.
The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit.
* fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact
A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried
* fix(anthropic): translate the OpenAI tool_search tool on /v1/chat/completions for Claude models
* fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups
With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does
A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result
The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator
* fix(bedrock): answer 400 for the OpenAI tool_search tool on Converse unless drop_params is set
* fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic
A function tool whose allowed_callers is not a list of strings now answers 400
instead of 500. The guardrail scan reads tools without malformed flags, so the
route decides between a 400 and a drop under drop_params. When the caller sends
a native Anthropic search tool beside tool_search, only the native one goes out.
Databricks Claude no longer counts as an Anthropic request route.
A previous_response_id follow-up off Anthropic also drops a search call the
client replayed as a function_call with a srvtoolu_ id, and its output. The
answered tool id scan in the Anthropic message factory runs once per assistant
run instead of once per message. A null created_by on tool_search_call and
tool_search_output output items is left out of the serialized response, so a
cache hit returns the same keys as the first call.
* fix(anthropic): send one search tool and judge tool search on the real model for chat tool_search
Several OpenAI tool_search items now collapse into one tool_search_tool_regex
tool, since Anthropic answers 400 when two tools share a name. A native
Anthropic search tool the caller already sent wins over the OpenAI item.
Bedrock invoke and Vertex AI now decide tool search support on the real model
before tool-based response_format swaps in a stub model name, so a Claude 5.5
request with response_format no longer gets a 400 on a route that runs tool
search.
* fix(anthropic): name the proxy drop_params setting in the tool_search 400
* fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan
The scan translated tool_search into Anthropic's regex search tool on every
route, so a guardrail returning only function tools removed tool_search from
native OpenAI requests and OpenAI answered 400 for the deferred tools
* fix(anthropic): run tool search on azure_ai, gate it on Databricks Claude, and strip deferral without recursion
Anthropic's Foundry docs list no tool search exclusion for either hosting
option, so the claude-tool-search rule now fills azure_ai. Databricks Claude
models route their tools through the same unsupported-tool-search gate, so a
tool_search tool answers 400 or drops under drop_params instead of a 500.
_without_deferral strips two fixed levels instead of walking recursively.
* fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai
* fix(anthropic): keep answered tool search replays paired with their answers
* fix(anthropic): drop Claude tool search calls from history sent to other routes, name the fix in the 400
A second turn that carries Claude's srvtoolu_ search call and its result now has both removed
before completion() sends it to a route that does not build an Anthropic request, including the
responses bridge. The tool search 400 now names model_info supports_tool_search for a model the
cost map does not know, and bedrock/invoke for a Converse model that runs tool search there.
* fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails
* fix(chat): keep answered Claude tool search calls in history sent to other providers
Only a srvtoolu_ search call with no tool result is dropped before a
non-Anthropic route, so an answered pair keeps the caller's shape where
OpenAI and Gemini take it. The history check no longer copies the list
through a TypeAdapter.
* fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object
* fix(chat): drop unanswered Claude tool search calls only on routes that reject them
OpenAI, Azure OpenAI and Bedrock Converse answer 400 for a srvtoolu_ call
with no tool result, while Gemini answers 200. Keep the history as sent
everywhere else so routes that already work see what the caller sent.
* fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent
* test(integration): cover chat tool_search routing, history drop and load on the wire
Audit round 1 of 3 at
|
||
|
Mateo Wang
|
52fc37888b
|
fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444)
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): give the defer_loading casts a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): read replayed tool-search and namespace fields through typed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep tool_search items on a turn with no message or tool call Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep every reasoning item on a search-only turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): leave content unset on a search-only turn so replay adds no empty message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): replay tool search items only with tool_search, count replayed tools in the allowlist, keep namespace off guardrails * fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses Under drop_params, defer_loading is dropped when no tool_search tool is sent, so the 400 it would cause upstream goes away as before. A non-bool value answers 400 without drop_params and is dropped with it. A null tool-level value falls back to the nested function value. A request carrying a tool_search tool now bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort none, since chat completions rejects the tool_search type. * fix(responses bridge): replay tool search items in output order and count replayed tools in tool policy Store the ordered output prefix through the last tool search item, reasoning included, so a second reasoning item no longer makes OpenAI reject the replay. The shared chat-to-Responses converter now replays tool search items and namespace only when the bridge opts in, so Noma, the complexity router and guardrail payloads keep their shape. The tool policy guardrail and the allowlist count replayed tools only when the request can send them. * fix(responses bridge): forward defer_loading only beside a tool_search tool defer_loading only means something with tool_search, and backends without it reject the key (xAI answers 403). A true value with no tool_search tool now answers 400 naming drop_params and is dropped under drop_params; a false value there is a no-op and is dropped. * fix(tool_loop): keep tool search items and namespace between rounds * fix(responses bridge): check replayed tool search loads when a deployment adds tool_search * fix(chat): drop litellm response fields before strict OpenAI-compatible providers A chat reply from the Responses bridge carries provider_specific_fields on the message and on each tool call, plus reasoning_items. Groq, Fireworks, Mistral and Together reject unknown fields, so sending that reply back to them as history failed with a 400 or 422. Strip them in the shared helper and use it in those providers. * fix(caching): keep reasoning_items and tool search items on response cache hits A cached bridge response lost reasoning_items on a non-streaming hit and lost both reasoning_items and provider_specific_fields (tool_search_items) on a streamed hit, so the next turn could not replay the search or the reasoning around it * test(integration): cover tool search item replay on the chat to Responses bridge Proxy, SDK, authorization and chaos cells for returning and replaying tool_search_call and tool_search_output items and the tool call namespace * test(integration): add response cache twins for the tool search guardrail cells Each guardrail cell now has a twin where the answer is already in the Redis response cache: a key opted out of global guardrails caches a replay that loads a policy-blocked tool, the same replay without guardrails is cached before block mode sees it, rewrite mode is not served the unrewritten conversation's cached answer, and an allowed replay is served from the cache. * fix(responses bridge): forward defer_loading beside a client-executed tool_search The bridge now replays a client tool_search_call and the client's tool_search_output through provider_specific_fields.tool_search_items, so a deferred function beside a client tool_search keeps defer_loading like it does beside a hosted one * test(responses bridge): pin the client tool_search turn and its replay, streamed and not * fix(responses bridge): strip nested namespace for strict providers and keep streamed reasoning_items on text replies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep dropping defer_loading beside a client-executed tool_search Forwarding defer_loading beside a client tool_search made GPT stop with a search call the chat client has to answer by hand, so a plain question that used to get tool calls got none. Keep main's behavior of dropping it there. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
bbd58931fe
|
test(integration): follow #46101 in the breakpoint cache mark census expectation (#46509)
* test(integration): follow #44788 and #46101 in the openrouter translation and cache mark census expectations Expect prompt_cache_breakpoint for opted-in OpenAI and omit assistant tool-call marks from the capped census Match OpenRouter refusal, logprobs, and system fingerprint nulls Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): follow #46341 blank env and guardrail param handling in keepalive and scope expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): read the first stream chunk through the stream's iterator so MCP streams keep working Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): follow #46493 retry and fallback semantics for streams that fail before content Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop the openrouter expectations #46510 already landed on main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): restore blank lines lost in the main merge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
314f17986a
|
fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models (#45882)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models * fix(responses): define the tool_search_call output item locally for older openai SDKs * fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500 * fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out A replayed Claude tool search call that the caller answered with a tool message kept its tool_result in the user turn while its tool_use was dropped from the assistant turn, so Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when no tool message answers it. The Responses bridge now strips defer_loading from every tool whenever no tool search tool is sent, whether the caller sent none or the route cannot run it, since a deferred tool with no search tool can never be loaded. Namespace members had started carrying the flag to every bridge provider, which providers that reject unknown tool keys answered 422. The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit. * fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried * fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator * fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic A function tool whose allowed_callers is not a list of strings now answers 400 instead of 500. The guardrail scan reads tools without malformed flags, so the route decides between a 400 and a drop under drop_params. When the caller sends a native Anthropic search tool beside tool_search, only the native one goes out. Databricks Claude no longer counts as an Anthropic request route. A previous_response_id follow-up off Anthropic also drops a search call the client replayed as a function_call with a srvtoolu_ id, and its output. The answered tool id scan in the Anthropic message factory runs once per assistant run instead of once per message. A null created_by on tool_search_call and tool_search_output output items is left out of the serialized response, so a cache hit returns the same keys as the first call. * fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan The scan translated tool_search into Anthropic's regex search tool on every route, so a guardrail returning only function tools removed tool_search from native OpenAI requests and OpenAI answered 400 for the deferred tools * fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai * fix(anthropic): keep answered tool search replays paired with their answers * fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails * fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object * fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent * fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id * fix(responses): answer 400 to a tool_search execution that is not server or client OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed execution is dropped like the other malformed tool flags, which leaves a server search * test(responses): cover the 400 and drop_params paths for a malformed tool_search execution * test(responses): cover a Claude search call replayed to Bedrock Converse with cache marks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
7b7604f0ed
|
test(translation): share rich fixtures across Claude, GPT and Gemini providers (#46585)
* test(integration): add rich anthropic claude haiku 5.5, opus 5.5 and fable 5.1 translation cases Capture-backed basic cases on /v1/chat/completions, /v1/messages and /v1/responses for claude-haiku-5-5, claude-opus-5-5 and claude-fable-5-1, reusing the shared client templates and ANTHROPIC_MESSAGES_REQUEST/RESPONSE with model-specific id, content and usage overrides. The minimal Opus 5.5 hello cases are replaced by the rich conversation. Sonnet 5.5 and legacy models are unchanged * test(translation): rich OpenAI captures for gpt-6.1-sol, gpt-6-luna, gpt-6-astra and gpt-5.6-terra Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(translation): rich Azure gpt-6.1-sol, gpt-6-luna, gpt-6-astra and gpt-5.6-terra cases * test(translation): rich azure_ai Claude cases for haiku-5-5 and fable-5-1 Bind the shared chat_completions, messages and responses client templates and the anthropic provider templates to azure_ai/claude-haiku-5-5 and azure_ai/claude-fable-5-1 from real Azure AI Foundry captures, keeping the Azure dialect exact (/anthropic/v1/messages, x-api-key plus api-key on chat_completions and responses, system as a text block list on /v1/messages). Add prompt_cache_hit variants for both models from the second call inside the cache window. The two deployments need rows in tests/integration/proxy_config.yaml Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(translation): rich Gemini bases for gemini-3.8-flash, gemini-3.1-pro-preview and gemini-4-argon Add the shared generateContent provider templates in templates/gemini.py from real captures and bind the three target models on chat completions, messages and responses with explicit per-model response overrides. gemini-4-argon is registered in the three basic lists. gemini-3.5-flash is unchanged * test(translation): enrich vertex_ai claude and gemini cases from live captures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): rich Bedrock converse, invoke and mantle translation cases from real captures * test(integration): register the rich translation deployments and follow the rich Gemini 3.8 Flash cases Add the synthetic deployments the rich provider cases bind (anthropic, openai, azure, azure_ai, gemini, vertex_ai, bedrock converse, invoke and mantle) to tests/integration/proxy_config.yaml, all pointed at the fake provider on 127.0.0.1:8191 with synthetic keys. The Gemini omitted-temperature contract now takes its expected generationConfig and reply text from the rich GEMINI_3_8_FLASH_TEST_CASE literals instead of the retired minimal hello cases * test(translation): reuse bridged and native response templates --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: kerry <kerry@berri.ai> |
||
|
ryan-crabbe-berri
|
e889d9d653
|
fix(bedrock)!: replay redacted_thinking on Converse and keep its data as Claude's own string (#46532)
* fix(bedrock): send redacted_thinking back to Converse as reasoningContent.redactedContent Replaying an assistant turn that carried a redacted_thinking block to Bedrock Converse turned it into an empty unsigned reasoningText, which the assistant content builder then dropped as blank text. Bedrock returned 200 so callers never noticed. Map redacted_thinking to redactedContent in both the thinking_blocks and inline content paths of the sync and async builders * fix(bedrock): drop blank redacted_thinking data and keep non-str signatures signed on Converse replay * fix(bedrock): replay redacted_thinking to Converse only when the target is Claude Nova, Llama, DeepSeek and gpt-oss reject reasoningContent.redactedContent with a 400, so a Claude history replayed to them through a Router fallback or model switch keeps dropping the redacted block the way main does. Claude is resolved from the Bedrock base model of the target, and an application inference profile ARN counts as Claude when its deployment's base_model names a Claude model. Adds integration contracts for the replayed Converse request shape, stream and non-stream, with a tool call, and for non-Claude and ARN targets. * test(e2e): cover Bedrock Converse replaying redacted_thinking live through the proxy Turn 1 triggers redacted thinking on Claude Sonnet 4.6, turn 2 replays it, and a live edge between the proxy and Bedrock checks the replayed Converse body carries the same reasoningContent.redactedContent. Covers nonstream, stream and a tool call round trip * fix(bedrock): validate litellm_params before reading the deployment base_model litellm_params arrives as an untyped dict, so passing it straight into the typed helper grew reportUnknownArgumentType. Validating it into Mapping[str, object] first keeps the read typed without growing the type-check gate * fix(bedrock): wrap non-Converse redacted_thinking data and let a named model id win over base_model * test(bedrock): cover redacted_thinking minted outside Converse, model_id and the sync Router base_model path * fix(bedrock): match bare claude- base_model names without reading the untyped cost map * fix(bedrock): keep redacted_thinking data as Claude's own string on every route Converse hands redactedContent back base64-wrapped, and that wrapper used to leak out as data, so a Converse turn replayed to Invoke or the Anthropic API was rejected. Converse replies now decode it, Converse requests wrap it exactly once, and the Anthropic-shaped builders unwrap histories saved with the old wrapping. Responses API encrypted reasoning is no longer sent to Claude on Converse, which rejected it with a 400 |
||
|
Mateo Wang
|
9ba4cc4fb3
|
fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440)
* fix(anthropic): keep tool_search_tool_result so the search call replays with its result Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): drop a client tool_result for a search call replayed with its server result * fix(anthropic): replay a server tool call the client answered as the client sent it * fix(anthropic): keep a server tool result under modify_params, count client-answered marks, and replay it from run_tool_loop modify_params no longer adds a placeholder answer for a server tool call whose result is kept, unless the call ends the request. The cache control hook counts a mark on a server tool call the client answered, since that call goes out as a plain tool_use. run_tool_loop sends provider_specific_fields back and runs only the client's tool calls. * fix(anthropic): keep a server tool result when another tool answer follows, scope run_tool_loop's replay, and keep code_interpreter_results off tool search turns * test(anthropic): audit cells for tool search result round trips, replay, cache marks, pass-through logging and chaos --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
c70dd4ece6
|
fix(responses bridge): keep defer_loading on function tools sent to the Responses API (#46439)
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): give the defer_loading casts a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses Under drop_params, defer_loading is dropped when no tool_search tool is sent, so the 400 it would cause upstream goes away as before. A non-bool value answers 400 without drop_params and is dropped with it. A null tool-level value falls back to the nested function value. A request carrying a tool_search tool now bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort none, since chat completions rejects the tool_search type. * fix(responses bridge): forward defer_loading only beside a tool_search tool defer_loading only means something with tool_search, and backends without it reject the key (xAI answers 403). A true value with no tool_search tool now answers 400 naming drop_params and is dropped under drop_params; a false value there is a no-op and is dropped. * fix(responses bridge): drop defer_loading beside a client-executed tool_search Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
0ececab7a1
|
test(integration): update stale CircleCI integration expectations (#46570)
Brings integration tests in line with intentional behavior changes from #46331, #46291, #46341, #46355, #46368 and #46493, moves the Databricks non-Claude cases off llama-3 names so they no longer download a Hugging Face tokenizer, drops the unrealistic count_tokens tools=5 case and skips the Mantle region-prefixed v1 base case as unrealistic input |
||
|
Kerry Lu
|
6d1a3bce2c
|
fix(responses): return the first message item as content for structured output on the chat bridge (#46508)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
dba46c0513
|
fix(responses): keep the provider code and message on top-level stream error events (#46438)
* fix(responses): keep the provider code and message on top-level stream error events Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): log a mid-stream failure before the router falls back CustomStreamWrapper now waits for failure logging before it raises the error the router falls back on, so the cooldown lands on the deployment that failed instead of the fallback that replaced it. Also keeps numeric error codes and the top-level message on the WebSocket and ChatGPT paths. * fix(streaming): run sync failure callbacks before a failed stream falls back to another deployment The router cools a deployment down from a sync failure callback that reads the request's model_info. A failed stream submitted that callback to the logging executor without waiting, so it often ran after the router's fallback had overwritten model_info and cooled down the healthy fallback deployment instead, leaving the model group with no deployments * test(responses): cover top-level OpenAI stream error events end to end * test(streaming): read live worker pids so the second SIGKILL chaos variant sees two workers * fix(streaming): run waited sync failure logging off the shared logging pool * revert(streaming): keep failed-stream failure logging in the background Waiting for sync failure callbacks before a failed stream raises fixed the router cooldown race, but it slowed healthy calls under many failing streams with a slow callback, since the wait shares a thread pool with every call's start. The race is pre-existing on main and is tracked separately, so this PR goes back to main's logging timing and keeps only the top-level error event changes --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Tobias Mayr
|
d03923e6a9
|
fix(responses): propagate cache_control through responses->chat completion bridge (#32511)
* fix(responses): propagate cache_control through responses->chat completion bridge User/assistant messages, tool call outputs, and function calls built from Responses API input items were dropping cache_control, breaking prompt caching for Bedrock/Anthropic when routed through the litellm_completion transform. Also propagate cache_control onto the last content element of user/assistant messages, since Anthropic/Bedrock read it from the content block rather than (or in addition to) the message level. Adds the missing cache_control field to ChatCompletionToolMessage and ChatCompletionResponseMessage TypedDicts to keep the assignments type-safe. * chore(ui): regenerate schema.d.ts for ChatCompletionToolMessage.cache_control Adding cache_control to the ChatCompletionToolMessage TypedDict changed the proxy's OpenAPI spec; regenerated via `npm run gen:api`. * fix(responses): don't overwrite existing content-block cache_control; add list-content test Per Greptile review on the PR: - The message-level cache_control was unconditionally overwriting the last content block's cache_control when that block already carried its own (preserved by _transform_responses_api_content_to_chat_completion_content). Now only propagates onto the last block when it doesn't already have one. - Added the missing test coverage for the content-block propagation path with list content, which no existing test exercised (all used plain string content, so the isinstance(content, list) branch was never hit). * style(responses): extract cache_control into a named variable, trim comment Per style review: the message-building branch read cache_control 3 different ways in 5 lines (.get() twice, then [] indexing once), unlike the sibling tool-call-output/function-call functions this PR touches, which each extract it into a named variable up front (output_cache_control, function_call_cache_control). Matched that convention here too. Also trimmed the 3-line inline comment to one line, matching the file's usual comment length for similarly subtle behavior. * chore: refresh lazy OpenAPI snapshot * fix: type cache control propagation * fix: place cache control type suppressions * style: format cache control type conversion * refactor(responses): build bridged cache_control messages in one shot * fix(responses): keep cache_control marks on bridged tool calls and within the provider cap A function_call item's mark now rides on the tool call the bridge builds, an assistant text item merged behind a function_call keeps its own mark, a developer item keeps its mark when it is mapped to a system message, and a previous_response_id replay yields its stale marks to the newest ones so the provider cap holds. The Responses layer counts the marks on function_call and function_call_output items before it places an ordinal injection point, and the chat pass replaces that count with its own instead of adding to it. * fix(responses): count each cache_control mark once and keep the bridge within the provider's limits List content carries an item-level cache_control on its last block only, so the provider sees one cache point and the breakpoint census counts one. Marked function tools and SDK-typed input items are handled by the Responses-layer reserve, assistant text merged into a function call turn keeps its mark on its own block, and the previous_response_id replay budget drops marks per turn, newest kept, instead of stripping every older turn once one overshoots. The request's own marks now fall under the same budget, newest kept, so five marked items no longer answer the provider's 400. A cache_control value that is not an ephemeral mark with an optional ttl is left off the bridged message instead of reaching the provider as sent, and a developer message maps to the system role with its content, name, and cache_control only, so unknown keys never reach a strict provider. * fix(responses): send item cache_control only to cache-point targets and follow drop_params Item marks reach the provider only when the target turns them into cache points, and every other target keeps main's behavior of stripping them. Without drop_params a mark the provider would refuse reaches it as sent, so the client gets the same 400 as on /v1/chat/completions. With drop_params a malformed mark is dropped and only the newest 4 are kept. The shared input-to-messages helper and the developer mapping in base_utils go back to main, and a marked developer item reaches a cache-point target as a system message in the bridge. * fix(responses): budget top-level marks, follow drop_params on Nova tool marks, and match chat-keeping providers * fix(responses): validate replayed marks and count input tokens without item marks * fix(responses): leave unknown Bedrock ARNs unmarked under drop_params, match OpenRouter's list, and reserve only forwarded marks * test(responses): cover item cache_control parity per chat route, drop_params trimming, and replay budgets * fix(responses): keep item marks off inspection paths and match each chat route's keep rule Only the request the bridge sends gets item marks now. Guardrails, token counting, rate limiting, and input_tokens read the input with no item marks, as on main, so an off-spec mark cannot fail a guardrail call. Tool message and tool call marks are typed inside the bridge, leaving the shared chat types and guardrail schemas as on main. Databricks keeps marks only for Claude, Snowflake uses its chat route's Claude check, and every Anthropic deployment keeps them whatever its model name. The injection census skips SDK-typed tools, and the owned-proxy test expects the marked messages on /v1/responses. * test(responses): cover guardrail reads without item marks, Snowflake name parity, and SDK-typed tools in the census * fix(responses): leave a non-list input untouched when stripping item marks * test(responses): cover string, list, object and tuple input with and without item marks kept Each input shape now answers the same whether the unmapped Bedrock ARN keeps item marks or strips them under drop_params, through the proxy and the SDK. Object input also runs through the OpenAI-compatible chat bridge, /v1/responses/input_tokens and the project input-token limiter * test(cache): derive owned-proxy cell timeouts from the harness boot budget The capped-burst cells and the uncapped response-cache cells boot their own two-worker proxy, and a fixed 240 or 360 second timeout cut them during boot on a slow runner before the harness readiness budget could decide. They now carry the same derived budget as the sibling chaos and guardrail files, with four boot and stop budgets for the restart cell that boots twice --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |