mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-05 07:37:50 +00:00
544 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
David Bochenski
|
f072498304
|
fix(bedrock): grant sts:GetWebIdentityToken and sts:TagGetWebIdentityToken in web identity session policy for claude_platform (#46670)
* fix(bedrock): grant sts:GetWebIdentityToken in web identity session policy for claude_platform Claude Platform on AWS calls STS GetWebIdentityToken (outbound identity federation) with the caller's session before serving inference. The session policy passed to AssumeRoleWithWebIdentity is a permission ceiling and never granted it, so every bedrock/claude_platform request using OIDC web identity auth failed with 403 "no session policy allows the sts:GetWebIdentityToken action", even when the role's identity policy allowed it. Add a ClaudePlatformOutboundIdentityLiteLLM statement granting only sts:GetWebIdentityToken, plus tests that it is granted, scoped to its own statement, and that no sts:* wildcard is introduced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(bedrock): also grant sts:TagGetWebIdentityToken for claude_platform web identity Claude Platform on AWS tags the outbound web identity token it requests, so the caller also needs sts:TagGetWebIdentityToken. Found end-to-end on a deployment: with only GetWebIdentityToken granted, calls failed with "not authorized to perform: sts:TagGetWebIdentityToken on resource: arn:aws:sts::<account>:self"; granting both made /v1/chat/completions and /v1/messages (streamed and not) succeed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(bedrock): only let Claude Platform use the outbound identity STS grant Scope the ClaudePlatformOutboundIdentityLiteLLM session policy statement with aws:CalledViaLast = aws-external-anthropic.amazonaws.com, so a web identity session can no longer call sts:GetWebIdentityToken or sts:TagGetWebIdentityToken directly. Drop the code comment and fix the stale test messages. * test(bedrock): fail when the Claude Platform caller condition reaches other statements --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Dan Lemon
|
c8b73cde84
|
feat(soniox): decode proxy form fields for context, translation and flags (#40605)
* feat(soniox): decode proxy form fields for context, translation and flags Multipart form fields reach the proxy as strings, so Soniox received context and translation as JSON strings and booleans as "true". Decode them at the handler entry point and add typed shapes for the context and translation request fields. * fix(soniox): keep bracket-prefixed free-form context and move types under llms Free-form context such as "[inaudible] consultation" is valid Soniox input, so a string that fails JSON parsing is now forwarded unchanged instead of returning a 400. The Soniox request types live next to the provider code. * test(soniox): exercise SDK-typed context and translation shapes through the decoder * refactor(soniox): derive handler options and provider params without mutating a local dict * chore(soniox): drop explanatory comment on the non-forwarded param set * refactor(soniox): return invalid form bool as a value and raise once in the handler * fix(soniox): validate decoded form fields and keep language with hints Validate structured context before sending it as an object, answer 400 for a bad polling interval or flag unless drop_params is on, nest SDK bracketed extra_body fields, keep the OpenAI language at the head of language_hints, and return the translated text for one-way translation without tags. * fix(soniox): keep the source transcript, rebuild SDK list fields, and reject malformed JSON values * fix(soniox): keep cleanup strings whole and drop the unused translation type * fix(soniox): let explicit language_hints win over language and test the proxy form path --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
79ca820933
|
fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert (#46772)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): reject unconvertible blocks nested inside a tool_result * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
Mateo Wang
|
dba86bec28
|
fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them (#46767)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
Emmanuel Acheampong
|
de4d28653d
|
fix(crusoe): update base URL to current endpoint (#33121)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d25b58f744
|
fix(count_tokens): count Anthropic-format requests for OpenAI the way /v1/messages inference converts them (#46759)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * test(count_tokens): expect the converted Responses input in the budgeted-key count integration test --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
mrinal-berri
|
bd60cc8608
|
fix(bedrock): forward Nova Sonic realtime tool calls with their arguments (#46214) | ||
|
Mateo Wang
|
d489c04554
|
fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking (#46741)
* fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking * fix(bedrock): drop unknown thinking keys under drop_params on display updates rebuilds Bedrock Invoke copied every caller thinking key when adding display updates, so an unknown key still reached the provider with drop_params on. Filter that copy the same way, and cover each chat call site plus the global litellm.drop_params path in tests * fix(anthropic): drop budget_tokens from rebuilt adaptive thinking under drop_params * test(anthropic): cover drop_params sources for rebuilt adaptive thinking on /v1/messages and Bedrock Invoke |
||
|
Mateo Wang
|
7090588bdf
|
feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240)
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions * fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel * feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing early-access row, carrying the same limits and capabilities. The AI Studio row mirrors the early-access entry; the Vertex row differs only in provider and pricing source. Prices follow once Google publishes them. Extend the mapped metadata test to cover all three ids, that each routes to its provider, that its backup copy matches, and that the public rows carry the same limits as the early-access row. * feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops * chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap * fix(gemini): follow continuation hops in a loop instead of recursion The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls * fix(gemini): bill Google Search grounding from every continuation request Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain * fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch * chore(gemini): name the type errors the dual-mode stream iterator trips The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up * fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment * fix(gemini): keep one response id across a continued answer A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key * test(gemini): add integration cells for long decode continuation Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once * fix(vertex_ai): fetch the access token again before each continuation request A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged * fix(vertex_ai): start each continuation stream with an empty parser buffer When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async * fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way * fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes * fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not * fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0 A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer Adds the audit's wire tests for both, which fail 24 cells before this change * test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed * fix(vertex_ai): log a continued answer's raw response as one object A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch * test(integration): check a continued answer logs one merged raw response A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch * test(vertex_ai): cover a follow-up tool call and its thought signature in long decode * test(gemini): pin that a one-request answer logs Google's body unchanged |
||
|
Mateo Wang
|
84ff9f84a9
|
fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses (#46720)
* fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses * fix(ui): send detail on playground Responses image parts and code snippets * test(messages): cover image detail on the Responses bridge wire |
||
|
Fede Kamelhar
|
68759a239f
|
fix(oci): drop non-function tools under drop_params and gate reasoning_effort on declared levels (#40732)
* fix(oci): skip non-function tools and gate reasoning_effort on models that reject it Fixes #31449. Two OCI GENERIC request-building defects that made Grok-on-OCI unusable for tool-using coding-agent clients (Codex over the Responses API, Claude Code over the Anthropic Messages API): 1. `adapt_tool_definition_to_oci_standard` raised `OCIError("OCI only supports function tools")` for any tool whose type is not "function", aborting the whole request when a client mixed a built-in/server tool in with its function tools. Non-function tools are now skipped with a warning; if none remain, `tools`/`toolChoice` are omitted from the request instead of sending an empty array. 2. The GENERIC param map forwarded `reasoning_effort` to every non-Cohere model, but OCI rejects it for all served xAI Grok models (`Model grok-4.20 does not support parameter reasoningEffort`) and for OpenAI's non-reasoning models (`Unrecognized request argument supplied: reasoning_effort` on gpt-4o / gpt-4.1); gpt-5, the o-series, gpt-oss-* and Meta/Google models accept it. `_model_supports_reasoning_effort` encodes that, `get_supported_openai_params` stops advertising the param for those models, and `map_openai_params` follows the standard unsupported-param contract (dropped under drop_params, clear error otherwise) so `drop_params` / `additional_drop_params` now work as documented instead of the 400 surfacing from the service. Behaviour verified against OCI GenAI in us-chicago-1 (2026-09-11) per model. The two existing reasoning_effort normalisation tests move from a Grok model to openai.gpt-5, which is the case they exercise. 21 new tests; the OCI suite is 406 passed. * fix(oci): drive reasoning_effort gating from the model catalog and drop tool_choice for skipped tools Review follow-ups on #40732: - `_model_supports_reasoning_effort` now reads the `supports_reasoning` flag of the model's `oci/` catalog entry through `get_model_info` instead of hardcoding xAI / gpt-oss branches. Catalogued models without the flag (xAI Grok, Meta Llama) no longer receive `reasoningEffort`; models absent from the catalog keep the historical pass-through. - Catalog: flag `oci/google.gemini-2.5-{flash,pro,flash-lite}` as reasoning models and add `oci/openai.gpt-oss-120b` / `oci/openai.gpt-oss-20b` (pricing from Oracle's public price list, 128K context per OCI docs). - `_model_uses_max_completion_tokens` routes on the `openai.*` prefix only. OCI-hosted Gemini accepts `maxCompletionTokens` but ignores it (the output cap silently disappears) and Llama rejects it with 400, so the catalog's reasoning flag must not drive that switch. - After tool adaptation, a `FUNCTION` tool_choice naming a tool that was skipped (built-in / server tools) is dropped with a warning instead of being sent to OCI, which rejects the unmatched choice. Verified live against OCI GenAI (us-chicago-1): Grok with mixed tools plus a forced built-in now completes and still calls the function tool; Gemini and gpt-oss forward reasoningEffort with maxTokens honoured; Llama drops it under drop_params; gpt-5 keeps maxCompletionTokens. * style(oci): use an immutable set for the tool-name lookup Keeps the type-discipline gate (LIT002, mutable collections) within its budget; the lookup is read-only. * fix(oci): fail closed when a forced tool_choice names a tool OCI will not receive Review follow-ups on #40732: - When tool_choice forces a function that is not among the adapted tools (a skipped built-in or a misspelled name), send `toolChoice: NONE` instead of removing the choice. Falling back to the service default would let the model call any other supplied function even though the caller forced a specific one, which an agent processing untrusted content must not do. - Move the #31449 regression tests into the mapped test module (`test_oci_chat_transformation.py`) instead of a separate file. Verified live against OCI GenAI: Grok and Llama with a forced skipped built-in now complete with no tool call; a forced kept function is still honoured. * test(oci): point reasoning_effort comments at the in-module gating tests The referenced module was folded into this file. * fix(oci): drop non-function tools only under drop_params and gate reasoning_effort on declared levels Non-function tools now follow the unsupported-param contract: a 400 naming drop_params without it, and dropped with a warning under it, with a forced dropped tool turning tool calls off. reasoning_effort is gated on the model's reasoning_effort_levels catalog entry, so OCI Llama keeps forwarding it and the Grok 4.20 models OCI serves, now in the catalog with their Oracle prices, stop sending a param the service rejects * fix(oci): gate reasoning_effort on oci/xai.grok-4.20, which rejects it * fix(oci): disable tool calls when tool_choice forces a dropped custom tool by name * fix(oci): drop tool_choice that forces a dropped tool and keep a kept function's name * fix(oci): keep Gemini 2.5 Flash Lite unflagged for reasoning so released proxies keep its max_tokens cap * fix(oci): answer 400 when a request depends on a dropped tool and leave Gemini max_tokens routing to its own PR Under drop_params, non-function tools still drop when function tools remain and tool_choice does not force a dropped tool. A request whose every tool is dropped, or whose tool_choice forces one, now answers 400 instead of a tool-less reply the model can fill with invented results, or a swap to another tool. The maxTokens routing change for OCI Gemini goes back to base, since it is outside this fix and an open PR carries it. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
da2f5a498f
|
feat(agents): support Databricks Agent Platform agents in the agent gateway (#45681)
* feat(agents): support Databricks Agent Platform agents in the agent gateway Add a databricks_agent provider that speaks the MLflow ResponsesAgent wire (Responses-shaped input, output_text items, output_text.delta SSE) to Model Serving endpoints (/invocations or the unified /serving-endpoints/responses) and Databricks Apps (/responses), with a PAT or OAuth M2M client credentials Register it through the agents config block or the Admin UI agent catalog and call it through /a2a/<agent> (message/send, message/stream) or model: a2a/<agent> chat completions. The chat route now runs every completion-bridge agent through its own provider, and attaches the minted Databricks OAuth token for both bridge and URL agents. Consumed OAuth settings never reach the provider call * fix(agents): harden databricks_agent input, output, and OAuth handling Non-text content parts answer 400 unless drop_params is set, a reply without an output list answers 502, flat OAuth fields resolve an os.environ api_base, the bridge debug log no longer echoes the request model, routing tests stub only the HTTP boundary, and the catalog description fits the dropdown * fix(agents): merge agent and client extra_headers on the a2a chat route, and reject an empty databricks_oauth block * fix(agents): bridge only databricks_agent agents on the a2a chat route, keep the card url over a client api_base, and require workspace_url for a Databricks App * refactor(agents): read the a2a chat bridge providers from a constants set instead of branching on a provider name * fix(databricks_agent): let the configured credential win, reject malformed client extra_headers, and keep a missing endpoint name empty * test(agents): drive the flat OAuth a2a send through the real bridge * fix(agents): keep an empty databricks_oauth block ignored on url agents, as before this change * fix(agents): accept bare App hosts, env service principals, and os.environ refs on Databricks agents * fix(databricks_agent): address review caveats on chat bridging, params, and auth Map max_tokens and max_completion_tokens to max_output_tokens and pass temperature, top_p, and user, so /v1/messages works without drop_params. Send A2A message metadata only with string values, answering 400 or dropping the rest under drop_params. Join message items with a blank line in both stream and non-stream replies. Cache the env service principal token until shortly before it expires. Keep databricks_agent replies out of the response cache. On the chat bridge, let request fields win over the agent's stored params while pinning the agent's api_base and api_key, and apply the agent's static_headers between client and agent extra_headers, as /a2a does. Resolve os.environ/ only for databricks_agent api_base and api_key instead of every bridge provider. Ignore an unusable databricks_oauth block on URL agents on chat, as at the merge base, instead of answering 400. * test(internal_context): list the databricks agent token cache as process-local * fix(databricks_agent): keep the agent's endpoint, input, and target out of client control extra_body model and input answer 400, or are dropped under drop_params. A client api_base or base_url never replaces the agent's card URL on the bridge. Chat to a URL agent keeps its merge-base call, and an incomplete databricks_oauth block on a URL agent keeps its merge-base ValueError. * test(databricks): expect the merge-base 400 for a client api_base on agent chat * fix(databricks): keep agent-owned inputs and identity headers out of a chat caller's reach * fix(databricks): bound the agent-over-caller merge to two levels so it needs no recursion --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
f470e418ea
|
fix(anthropic): keep signed empty thinking blocks and honor thinking.block_binding (#46342)
* fix(anthropic): treat adaptive thinking as enabled when mapping response_format
is_thinking_enabled only recognised thinking.type=enabled, so a request that
paired thinking.type=adaptive with response_format took the synthetic JSON
tool path with a forced tool_choice, which Anthropic rejects alongside
thinking. The same check also gates the Bedrock Converse and Databricks
forced tool_choice downgrades, so they now see adaptive thinking too.
* fix(anthropic): keep signed thinking blocks with empty text on chat completions
Models that default to display: omitted return signed thinking blocks whose
text is empty, and Anthropic rejects the next turn when they are not replayed
unchanged. is_unsignable_thinking_block dropped them along with unsigned
blocks, and the inline content path required non-empty text, so a tool-use
turn on Opus 4.7+ or the 5 families lost its thinking blocks and 400ed on the
tool-result turn. Only a missing signature now drops a block, and the
mid-conversation system placement mirrors that rule. The /v1/messages strip
helper is left to #41362.
* fix(anthropic): skip the thinking-strip retry when block_binding asks for an error
The /v1/messages retry loop strips thinking blocks and resends on any
invalid-thinking 400, including the binding mismatch a caller opted into
with thinking.block_binding.prefix_mismatch_behavior: error. The retry
decision now sees the request body and surfaces that 400 instead of
overriding the caller's choice. #42888 covers the litellm_params side.
* fix(bedrock): carry thinking.block_binding through the Invoke thinking rebuilds
Bedrock Invoke rebuilt the thinking param from scratch when display: updates
was requested, on both the /v1/messages and chat completions paths, and the
legacy enabled-to-adaptive translation plus the clear_thinking injection did
the same, so thinking.block_binding never reached Bedrock. Every rebuild now
spreads the caller's thinking and only overrides the keys it owns, and
AnthropicThinkingParam types block_binding. The thinking-binding-controls
beta was already on the Bedrock allowlist.
* fix(anthropic): keep forcing the response_format tool under adaptive thinking
This reverts
|
||
|
devin-ai-integration[bot]
|
e344dbd8db
|
feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043)
* feat(proxy): split Azure PTU deployments across teams by share
A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to
ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on,
routing hides the deployment from teams holding no share, the v3 limiter enforces each
share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily
flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team
Usage page report PTU-hours next to tokens.
* fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name
The PTU flag is read from the process environment on every check instead of
through the secret-manager path, so a proxy with a hosted secret manager does
not pay a round trip per request. A team's ceiling now carries the model's
cached-input ratio and settlement charges cache reads at that ratio, matching
how Azure sizes a PTU. A deployment registered for one team is found by its
team_public_model_name for PTU-hours and the ceiling, since its model_name is
rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and
parsed_ptu_shares refuses booleans, strings, and non-string team ids, a
fractional ptu_count reserves nothing, and models_processed counts distinct
deployments rather than holdings.
* fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations
* fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id
* fix(ptu): refuse ptu_shares naming a team that does not exist
* refactor(ptu): keep Azure detection under llms/azure and check share teams with one query
* fix(proxy): drop the reverted TeamIdSearchFilter import after merging main
Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally
* fix(proxy): chunk the ptu_shares team lookup under the IN-list bound
* fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling
* fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling
* fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour
* fix(ui): seed total_ptu_hours in the empty daily activity metadata
* fix(ptu): size usage rows by the name the request used
* fix(ptu): size a team's usage rows by the deployment it is served from
* fix(ptu): never size a team's usage row by another team's reservation
* fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length
* fix(ptu): read a share map only through the terms registration requires
A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and
ptu_effective_from, or one that does not add up to ptu_count, is refused at
registration but was enforced once loaded: every other team got the 400 and
the share holder a ceiling while no flat cost accrued. _deployment_shares now
reads the shares through ptu_terms, so such a row reserves nothing, the same
as before the map was read.
* fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off
|
||
|
ryan-crabbe-berri
|
1f041a3265
|
fix(openai): strip prompt_cache_breakpoint under drop_params for OpenAI models that cannot read it (#46453)
A chat completions request to gpt-4.1 (or any OpenAI model older than 5.6) with drop_params on still sent the marker inside messages and tools. It is now removed there, while drop_params off keeps sending it through and models that read it keep it either way Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
2c4ec9c6fa
|
fix(responses): open faked stream items empty so tool arguments and text arrive once (#46533)
* fix(responses): open faked stream items empty so tool arguments and text arrive once * fix(responses): open faked stream items in progress with a fresh empty list, and open mid-stream guardrail blocks empty * fix(responses): open faked refusals, reasoning summaries, custom tool calls and MCP calls empty so each value arrives once * test(integration): cover the empty-open Responses stream items on every fake, replay and block path Integration cells for the faked /v1/responses stream, the chat and messages bridges, cache replay, guardrail block events and the MCP gateway call. Each opened item starts empty and each value arrives once, checked with raw httpx, the OpenAI SDK and the Anthropic SDK, plus upstream errors, 5 KB values, a client drop, a dropped upstream connection and a worker SIGKILL |
||
|
Taylor Hawkes
|
fd84e97f49
|
feat(providers): add Inference APIs as OpenAI-compatible provider (#41669)
* feat(providers): add Inference APIs as OpenAI-compatible provider Registers https://api.inferenceapis.com/v1 as a JSON-configured provider (chat completions and Responses API). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(providers): list Inference APIs in Add Model and the provider docs index, add request-level tests - provider_create_fields.json entry so the Add Model dropdown test passes - provider_endpoints_support.json (+ backup) entry so the provider docs check passes - tests now capture the outgoing HTTP request (URL, auth header, body) instead of only inspecting registrations - drop a redundant comment Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(providers): inject a recording HTTP session instead of patching httpx.Client Request-level tests now set litellm.client_session to an httpx.Client with a MockTransport, so nothing is monkeypatched on a class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * chore(providers): drop redundant comment on the inferenceapis entry Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * feat(inferenceapis): add cost map rows for the 12 chat models and test the responses route Prices and context windows come from https://inferenceapis.com/pricing and GET /v1/models. Moves the provider test to tests/unit and adds the interactions flag the endpoint matrix now carries. * feat(inferenceapis): price the vendor's chat model aliases and mark /v1/messages supported * fix(inferenceapis): serve /v1/messages natively and keep audio routes off the provider * fix(json providers): send provider-specific params in extra_body and report context overflow as ContextWindowExceededError * fix(exception mapping): match the Inference APIs overflow phrase instead of any context_length_exceeded substring * fix(exception mapping): map context overflow to ContextWindowExceededError for Inference APIs only Providers without their own mapper keep reporting an overflow as BadRequestError, since the router raises a ContextWindowExceededError instead of trying generic fallbacks when context_window_fallbacks has no entry for the group. Adds the integration cells for the provider, its fallbacks, and the extra_body and overflow families. * test(inferenceapis): hold upstream replies across the worker kill in the chaos cell A reply the killed worker sent before SIGKILL can lose its queued spend row, which the product never promised to keep. Holding the upstream until after the kill means only the surviving worker answers, so every 200 must log exactly once. * test(inferenceapis): make the chaos cell kill the quieter worker and always release held replies The survivor now must hold at least one in-flight request, so the burst check cannot pass as 0 == 0, and the held vendor is released on exit so a failed assertion cannot stall the wire server teardown --------- Co-authored-by: TaylorHawkes <857858+TaylorHawkes@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
17f621e116
|
fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836)
* fix(vertex_ai): route Model Garden models through the generateContent adapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vertex_ai): annotate generateContent route test parameters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): type litellm_params in get_vertex_ai_model_route * fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions Streamed Model Garden calls through the generateContent adapter now request stream usage where the deployment supports it, keep cached-token details, and look up the provider-relative cost key, so they bill like the non-stream call. The adapter forwards router metadata, passes upstream errors through with their status, countTokens falls back to local counting for non-Gemini routes, and the native Gemini config is limited to the Gemini routes. * fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing Streamed generateContent through the chat adapter now ends with the upstream usage, honors always_include_stream_usage false and the caller's stream_options, counts one failure per failed call, and routes Vertex deployments by their base_model like chat does * fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage * test(google_genai): expect the trailing usage chunk on bridged OpenAI streams The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone. * fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk * fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent * test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire * test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
08a24d391a
|
fix(count_tokens): count system and tools on Vertex Gemini, fall back to Mantle on Bedrock 404 (#46430)
* fix: align token counting with inference context Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve Vertex Gemini count message shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: forward Google countTokens tools in shapes every provider counter accepts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: add package marker for Vertex count-token unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): ignore deployment system_instruction when counting tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): forward system and tools on Google AI Studio, keep strict on Anthropic tools Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(count_tokens): ruff format gemini count handler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): return Mapping from count request body builders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): truncate long Anthropic tool names on Vertex count, update stale strict test expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): simplify Gemini count context and tests Put the Vertex Gemini count branch back in the same shape as main and the partner model branch, keeping only the system and tools context, the provider error mapping and the gemini tokenizer default Share one error response helper between the Vertex and AI Studio callers, drop the unread kind fields and the extra validation of map_function output, and say Gemini instead of Vertex AI in the error that AI Studio users also see Tighten the tests by removing a stray parametrize on a helper, sharing the SDK count args, asserting the exact forwarded system and tools, and folding the Anthropic tool shape tests into one table * fix(count_tokens): reuse inference tool translation for Vertex Gemini count Vertex and AI Studio count now convert tools with the same functions inference uses, translate_anthropic_tools_to_openai and the Responses tool converter, before VertexGeminiConfig.map_function. Unnamed and Anthropic hosted tools now count the same declarations inference sends instead of being dropped Tools that cannot be converted, such as a non list tools value, a non object tool or a schema that is not a JSON object, now return a 400 before any request instead of being counted without them --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
606ecf56de
|
fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals (#46633)
* fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove an OpenAI-format box answer reaches the caller as a box Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): give the box wire test a deployment that advertises supports_box_questions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Sushit Lal Pakrashy
|
958cba7016
|
fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results (#38284)
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Rick
|
71e508b65d
|
feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos (#43705)
* feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos MinimaxVideoConfig maps litellm's OpenAI Videos API onto MiniMax's V2 task API. Create, status, list, content and delete hit /v2/video_generation and /v2/query/video_generation. Remix is not exposed: MiniMax's /v2/video_regeneration only upscales a finished 768P task to 2K and ignores any prompt, so it raises instead of returning a video that silently drops the edit OpenAI params translate to MiniMax's contract: seconds becomes duration, size reduces via gcd to a valid ratio, and text-to-video requests default to ratio 16:9, resolution 768P and duration 5 when unset. input_reference becomes a first_frame content item, base64 encoded into a data URI for file inputs, and multimodal reference media passes through a content array in extra_body. Caller content is validated strictly as image, video or audio items, and the scanned prompt is always the one text item sent, so text can't reach MiniMax past the guardrails; anything else in content returns a 400. api_base shares MINIMAX_API_BASE with the chat config (a trailing /v1 or /v2 is stripped because the video API carries its own version). The list endpoint pages by number with no cursor or sort, so after and an ascending order are rejected with a 400 rather than ignored, and page_num passes through extra_query. Responses validate into frozen pydantic models at the boundary, video ids are provider-encoded so status and content route back to MiniMax, and create reports duration_seconds and video_resolution in usage for cost tracking End-to-end tests drive the real avideo_generation() entrypoint through an httpx MockTransport and assert on the encoded request body, which is what catches a mapper that drops input_reference before the request is built Both model_prices maps gain minimax/MiniMax-H3 and minimax/MiniMax-H3-Max, priced per output second by resolution tier from MiniMax's pay-as-you-go list (H3 $0.08 at 768P and $0.13 at 2K, H3-Max $0.05 at 480P and $0.08 at 768P). MiniMax also bills input images beyond a per-model free allowance (5 for H3, 2 for H3-Max), so each entry carries input_cost_per_image and provider_specific_entry.minimax_free_input_images, and the create response reports the full charge as provider_reported_cost_usd. Reference video input is rejected with a 400, because MiniMax bills its duration only after the create call that litellm bills Co-authored-by: AaronHowell <237895480@qq.com> * fix(minimax): fail closed on invalid video params and malformed task responses Addresses review on #43705: - A create, status or delete response without a task id (or a task) is a 502 instead of a queued object with an empty id - An unmapped task status is a 502 instead of defaulting to queued - seconds/duration must be a positive whole number and size a positive WIDTHxHEIGHT or W:H, otherwise 400; a 0x0 size no longer divides by zero and an unparseable duration no longer falls back to the billed default - characters and image are rejected with UnsupportedParamsError unless drop_params is set, instead of being dropped silently - Downloading a task that is not ready or failed is a 400 that keeps its message through the handler, instead of a ValueError surfaced as a retried 500 - The video download goes through safe_get / async_safe_get Co-authored-by: AaronHowell <237895480@qq.com> * test(minimax): cover the video download path and invalid-param branches The download now goes through safe_get, so the happy path and the finished-task-without-a-url branch are asserted, alongside the non-object parameters rejection and the remix response stub. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> * fix(minimax): return 400 for video remix and clear new lint gate counts * fix(minimax): send file-tuple images and keep input_reference next to content media * fix(minimax): answer 404 when an H3 video is downloaded before it finishes A download that lands while the task is still queued or running answered 400. That is a client error, which was the intent, but 404 is what the OpenAI videos API documents for content that is not ready, and it is the status the sibling DashScope and Volcengine video paths now answer, so a client polling all three reads one shape A failed task keeps its 400, since that is not a timing mistake the caller can retry away. The download test drives avideo_content and asserts the caller sees NotFoundError rather than an internal server error * fix(minimax): answer 400 for unsupported video params, early downloads and unknown ids A create param MiniMax cannot honor (seed, n, quality and the like) was dropped without a word. It now answers 400 unless drop_params is on, like other providers A download before the video is ready and a status call for an unknown task id now answer 400. The router cools a deployment down on a 404 and retries a 5xx, and neither is a deployment fault. DashScope on main answers 400 for the same case Input images past the free allowance are now billed through the shared video cost path, so a deployment with custom pricing bills them too. Status usage no longer carries MiniMax's token counts, list sets has_more to false since MiniMax pages by number, and the price map stops listing video as an input modality * fix(minimax): keep the polled video id, honor deployment drop settings, and match chat's key order Status re-encodes the id with the polled id's model when MiniMax omits it, drop_params and additional_drop_params never reach MiniMax and listed keys are dropped, MINIMAX_API_KEY wins over litellm.api_key, create sets created_at, and a content variant other than the video answers 400 * fix(minimax): list a task without a model under the routed group MiniMax leaves `model` off some listed tasks. The caller polls the ids a list hands back, and the proxy resolves a poll by the model in the id, so an id encoded with an empty model no longer routes back to the deployment that served the task. Fall back to the group the router was asked for. * fix(minimax): honor deployment drop keys inside parameters and bill images at the map rate when a deployment prices only seconds A field the deployment lists in additional_drop_params no longer reaches MiniMax when it arrives inside parameters, and a deployment that sets only its per-second price now bills reference images past the free allowance at the cost map's per-image rate instead of $0 * fix(minimax): keep deployment-dropped fields out of the body when they arrive through extra_body * fix(minimax): answer 400 for a video resolution with no price unless drop_params is on * fix(minimax): send the default resolution when a deployment drops resolution --------- Co-authored-by: AaronHowell <237895480@qq.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Happy <yesreply@happy.engineering> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Wang Yunzhuo
|
a2f792634e
|
fix: preserve explicit provider for async image, audio, and realtime routes (#36346)
* fix(images): preserve explicit provider for async generation * test(images): prove a bare model name reaches its explicit provider on the async path * fix(images): validate async provider hints instead of reading untyped kwargs Read custom_llm_provider and api_base in aimage_generation through a TypeAdapter over a small TypedDict, so the provider lookup gets typed values and the basedpyright gate stays within its merge-base ceiling * fix: answer bad provider hints with a 400 and keep the explicit provider for async speech and transcription * fix: keep the explicit provider for realtime and cover the audio bad-hint 400 * fix: keep the explicit provider on the realtime HTTP routes and read base_url in async audio * fix: stop async image generation from rejecting a base_url it never reads * fix(audio): skip base_url validation when api_base is set on async speech and transcription * test(integration): cover a declared provider with a bare model on image, audio, and realtime routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c00706218b
|
fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097)
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(vertex_ai): keep yield outside the JSON decode guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): surface BaseLLMException raised while streaming a request body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(http_handler): only follow __cause__ when unwrapping streamed body errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): avoid recursive streamed error unwrapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows Keep main's error path for a directly raised provider error and only unwrap one found behind a transport error. Add the parser column to invalid JSON errors, and cover the real aiohttp chain in the _handle_error test * fix(vertex_ai): report every row conversion error as a 400 naming its line A message with no role, a non-string role, a bad temperature, or a litellm BadRequestError raised while converting a row still returned a 500, and the async upload lost the 400 for litellm's own exceptions. The stream now turns any error from one row's conversion into a 400 naming that line. _handle_error only unwraps an httpx.TransportError, which is what the aiohttp transport raises around a failing request body, so other callers keep their own status. A file that starts with a UTF-8 byte order mark now uploads, and provider messages start with 'Batch input file line N' like the proxy check. * fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line A chat row whose body is missing or not an object now raises a 400 naming its line instead of uploading a blank prompt, and a non-object body on line 1 no longer crashes the GCS object name lookup. Rows too deeply nested or too long to parse return a 400 from both the SDK and the proxy's own batch file check. JSON error columns now count leading whitespace. * fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line * test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos * fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact * test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Raushan kumar
|
f9d45f8a43
|
fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)
* fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kolade Fajimi
|
35970c3eac
|
fix(vertex): report an unreadable credentials file as a read failure (#40173)
* fix(vertex): report an unreadable credentials file as a read failure Tell a path apart from inline JSON by shape, not os.path.exists(). Fixes #40166 * test(vertex): narrow pytest.raises to the expected message * fix(vertex): log the credentials path instead of returning it to the caller * refactor(vertex): log the credential source decision where it is made * fix(vertex): handle paths open() rejects with a bare ValueError * fix(vertex): redact the credentials path before it reaches the log * test(vertex): cover the brace fallback and non-text credentials file * test(vertex): exclude the unreachable exhaustiveness guard from coverage * refactor(vertex): read the failure's fields off the matched object CodeQL's py/uninitialized-local-variable does not follow keyword capture patterns in a match statement and reported `path` and `reason` as possibly unbound. Narrowing on the class and reading the attributes says the same thing without a suppression. * fix(vertex): accept a read-only mapping in extract_aws_params * fix(vertex): keep encoded credentials out of the log and cover the legacy route * fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust * fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file * fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message * test(vertex): integration cells for credentials file and inline value failures * test(vertex): drive the Rust messages cell through Vertex Claude --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ErenAta16
|
bc58a5857b
|
fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema (#34653)
* fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema `translate_anthropic_tools_to_openai` assigned the caller's `input_schema` straight into `function_chunk["parameters"]`, then merged the tool's remaining top-level keys into that same dict. Because the assignment aliased rather than copied, keys such as `display_width_px` on a computer tool were written back into the caller's schema. That matters because callers reuse the tool list across passes: the pass-through guardrail translates the tools, and the request is then translated again for the real call. The first pass pollutes the source and the second forwards a schema carrying non-schema keys. Shallow-copying fixes it. `translate_anthropic_output_format_to_openai` in the same file already copies its schema for this reason, so this brings the two into line. Rebased onto current litellm_internal_staging; the original branch had gone to conflict on surrounding drift while the line it fixes is unchanged. * fix(anthropic/adapter): drop the inline comment and test the strict tool shape that Anthropic rejects * fix(anthropic/adapter): deep copy input_schema so downstream schema edits never reach the caller's tool A shallow copy still shared nested properties with the caller, and the Gemini, vLLM and watsonx bridges strip nested additionalProperties in place, so a fallback to Claude sent a strict tool Anthropic rejects. A deep copy also keeps a null or malformed input_schema as sent, as before this PR * fix(anthropic/adapter): deep copy the incoming tools so downstream edits never reach the caller's tools Copying only input_schema left the extra keys merged into parameters, such as input_examples, and OpenAI-format function tools shared with the caller, so Gemini's strict strip still edited the caller's tool before a fallback to Claude * fix(anthropic/adapter): keep translating tools that deepcopy cannot copy A tool passed as a frozen mapping, or with a schema nested past Python's recursion limit, made copy.deepcopy raise, so the request failed before it reached the provider. Copy each tool on its own and keep the original object when the copy fails, which is how these tools were handled before * fix(anthropic/adapter): copy tools through JSON so the Responses bridge and frozen tools keep the caller's schema * test(anthropic/adapter): parse stub bodies with json and cover the tool copy fallback * test(anthropic/adapter): close the connection after the malformed tools request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Chloe Lu
|
897bfea38d
|
fix(anthropic): keep signed empty thinking blocks on /v1/messages requests (#41362)
The request-side sanitizer on /v1/messages dropped every thinking block whose text is empty, even with a valid signature. Claude Opus 4.8 / Fable 5 / Fable 5.1 return exactly that shape by default (thinking.display omitted): the signature is the preserved reasoning. Replaying the assistant turn as returned therefore lost the model's reasoning on every multi-turn request, silently (the provider answers 200 without it), and left preserved-thinking enforcement (thinking.block_binding) nothing to bind. Anthropic accepts an empty thinking block that carries a signature and binds it to its prefix; only an empty block with no signature (the LIT-6357 case, a turn a non-Anthropic model served through the bridge) is rejected with "each thinking block must contain thinking". Use the signature-aware predicate is_empty_unsigned_thinking_block, which #38809 already applied on the emit side, for the request side too. The unsigned case is still dropped. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
3fec5075fa
|
feat(typesafe): support Jev 1.14 images, box questions, and long-context headers (#46596)
* feat(typesafe): support Jev 1.14 images, box questions, and long-context headers * fix(cost): drop the typesafe/jev-1.14.0 entry main already prices * fix(typesafe): send images only with the images header and box questions only to models that answer them TypeSafe takes assets and input_image parts only when the request carries TypeSafe-Danger-Zone-Images: true, so without it they follow drop_params like any other provider that cannot take images. Box support now comes from a supports_box_questions cost-map flag set on jev-1.14.0 and jev-preview, so jev-1.13.0 and jev-latest refuse or drop a box question before the call. A malformed provider_specific_header answers 400 instead of 500. * fix(typesafe): allow null box confidence and validate asset references and base64 * fix(typesafe): gate box questions on the images header, drop unreadable assets under drop_params, forward opt-in headers on the pass-through --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
efd8274315
|
feat(xai): bill grok voice realtime sessions (#46600)
* feat(xai): bill grok voice realtime sessions * chore: revert unrelated formatting churn in test_cost_calculator.py * refactor(xai): tidy realtime usage meter and tests * fix(xai): read push-to-talk mode from client session.update only * fix(xai): clamp decoded audio bytes and lock push-to-talk before content --------- Co-authored-by: kerry <kerry@berri.ai> |
||
|
Kerry Lu
|
0d741a9816
|
fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them (#46611)
* fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them * fix: satisfy the type discipline and basedpyright gates for the thinking block index * chore: sync the OpenAPI snapshot and schema.d.ts with the thinking block index field --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d6e6c6f26c
|
fix(responses): map Anthropic tool search to OpenAI server tool search (#46441)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models * fix(responses): define the tool_search_call output item locally for older openai SDKs * fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500 * fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out A replayed Claude tool search call that the caller answered with a tool message kept its tool_result in the user turn while its tool_use was dropped from the assistant turn, so Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when no tool message answers it. The Responses bridge now strips defer_loading from every tool whenever no tool search tool is sent, whether the caller sent none or the route cannot run it, since a deferred tool with no search tool can never be loaded. Namespace members had started carrying the flag to every bridge provider, which providers that reject unknown tool keys answered 422. The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit. * fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried * fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator * fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic A function tool whose allowed_callers is not a list of strings now answers 400 instead of 500. The guardrail scan reads tools without malformed flags, so the route decides between a 400 and a drop under drop_params. When the caller sends a native Anthropic search tool beside tool_search, only the native one goes out. Databricks Claude no longer counts as an Anthropic request route. A previous_response_id follow-up off Anthropic also drops a search call the client replayed as a function_call with a srvtoolu_ id, and its output. The answered tool id scan in the Anthropic message factory runs once per assistant run instead of once per message. A null created_by on tool_search_call and tool_search_output output items is left out of the serialized response, so a cache hit returns the same keys as the first call. * fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan The scan translated tool_search into Anthropic's regex search tool on every route, so a guardrail returning only function tools removed tool_search from native OpenAI requests and OpenAI answered 400 for the deferred tools * fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai * fix(anthropic): keep answered tool search replays paired with their answers * fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails * fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object * fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent * fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id * fix(responses): answer 400 to a tool_search execution that is not server or client OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed execution is dropped like the other malformed tool flags, which leaves a server search * test(responses): cover the 400 and drop_params paths for a malformed tool_search execution * fix(responses): map Anthropic tool search to OpenAI server tool search Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): sync generated tool search API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): type tool search mapping predicates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): validate tool-search mappings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): use the openai SDK ToolSearchToolParam for the mapped tool search * fix(responses): keep deferred chat tools beside a tool search on the responses bridge Restore the hand-rolled OpenAIToolSearchToolParam, since openai 2.20.0 (the supported floor) has no ToolSearchToolParam, and carry defer_loading from chat function tools onto the Responses request when a tool search tool is present, which OpenAI requires for tool_search * fix(responses): leave the bridge's defer_loading carry to its own PR #46439 owns keeping defer_loading on chat function tools sent through the Responses bridge, so this PR keeps only the tool search mapping and the openai 2.20.0 compatible tool type * fix(openai): keep tool search mapping to the OpenAI API and drop it where the model or history cannot take it * fix(openai): narrow the tool search drop to Anthropic's search tool, drop it when nothing is deferred, and run the wire tests against the OpenAI host * fix(openai): pass malformed tools to OpenAI as sent and keep a caller's own tool_search under drop_params * perf(openai): scan Responses history for Claude search calls only when Anthropic's search tool is sent * test(responses): cover the Anthropic tool search mapping to OpenAI Responses across routes, settings, and chaos * test(integration): define the fallback cell's deployments in the gateway config so both workers load them * test(integration): keep the worker-kill burst held past the replacement wait * fix(responses bridge): count an Anthropic search tool as hosted tool search on the chat bridge main's #46439 refuses defer_loading when no tool_search tool is in the request, and the chat bridge only counted {"type": "tool_search"}, so the Anthropic bm25 or regex tool this PR maps to OpenAI's tool search was refused before the mapping ran Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect the chat bridge to map the search tool now that defer_loading survives --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
d0fa05076d
|
fix(xai): name a completed xAI batch's results file and stop looking it up on xAI's files API (#45559)
* fix(xai): report a completed xAI batch's results as its output file details * fix(xai): keep a short results download and per-call timeouts out of the reported file details * fix(xai): describe a batch's results file from the batch alone and let the batch job fill its size Reading an xAI batch id as a file now costs one batch read and never downloads the results: the file reports a 0 byte placeholder, the results filename and the registration time. The managed proxy fills in the size the batch cost job measures when it downloads the results, keeps a known size when a provider reports none, and drops the per-process results download cache. * fix(managed_files): keep a measured batch output size when a refresh races the batch job A refresh that read the row before the batch cost job stored the measured size could save the provider's unsized details over it. The refresh now reads the row back from the writer and saves only while the row still holds what it read, retrying when another write landed in between. The new tests drive the real xAI and OpenAI retrieve paths through a fake HTTP upstream. * fix(managed_files): guard the refresh save with the row the request already read The conditional save first compares against the row the request read, so a refresh adds no database read unless another write landed in between; only then does it read the row back from the writer and try again. * fix(managed_files): save a refreshed file object with one guarded write and no extra database reads A refresh that loses its guarded write to a newer one now leaves that row alone and drops this worker's cached copy, so the next read loads the newer row instead of retrying against the database on the request path * fix(managed_files): retry the batch cost job's measured size after a read refreshes the row first The batch cost job re-reads the row from the writer and saves again when its guarded write loses to a concurrent refresh, so its measured size is never dropped. Reads without a measured size still make one write and no re-read * fix(managed_files): size a concurrently refreshed row instead of restoring the batch job's placeholder copy When the batch cost job loses its guarded write and the row it re-reads holds refreshed details, it applies its measured size to those details, so the results filename and any size the provider reported are kept * fix(managed_files): keep a batch output's earlier created_at when a refresh replaces its placeholder A refresh saved over a stored row now keeps the earlier of the two creation times, so an xAI placeholder keeps the time the proxy registered it instead of the read time, while a provider's own earlier creation time still wins * test(integration): an xAI placeholder row keeps its registration time after the proxy refreshes it * test(integration): drop the cached output row so the placeholder cell reads the seeded database row * test(integration): drop the cached output row without requiring it to still exist * fix(managed_files): limit the batch output size fill, guarded save and kept created_at to xAI rows * fix(xai): keep the batch id as the batch output filename Managed xAI batch output files keep the filename main already returns (the batch id) instead of renaming them to <batch id>_results.jsonl. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
c0663d5202
|
fix(anthropic)!: translate the OpenAI tool_search tool on /v1/chat/completions for Claude models (#46389)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models
* fix(responses): define the tool_search_call output item locally for older openai SDKs
* fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields
Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500
* fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out
A replayed Claude tool search call that the caller answered with a tool message kept its
tool_result in the user turn while its tool_use was dropped from the assistant turn, so
Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when
no tool message answers it.
The Responses bridge now strips defer_loading from every tool whenever no tool search tool is
sent, whether the caller sent none or the route cannot run it, since a deferred tool with no
search tool can never be loaded. Namespace members had started carrying the flag to every
bridge provider, which providers that reject unknown tool keys answered 422.
The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit.
* fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact
A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried
* fix(anthropic): translate the OpenAI tool_search tool on /v1/chat/completions for Claude models
* fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups
With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does
A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result
The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator
* fix(bedrock): answer 400 for the OpenAI tool_search tool on Converse unless drop_params is set
* fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic
A function tool whose allowed_callers is not a list of strings now answers 400
instead of 500. The guardrail scan reads tools without malformed flags, so the
route decides between a 400 and a drop under drop_params. When the caller sends
a native Anthropic search tool beside tool_search, only the native one goes out.
Databricks Claude no longer counts as an Anthropic request route.
A previous_response_id follow-up off Anthropic also drops a search call the
client replayed as a function_call with a srvtoolu_ id, and its output. The
answered tool id scan in the Anthropic message factory runs once per assistant
run instead of once per message. A null created_by on tool_search_call and
tool_search_output output items is left out of the serialized response, so a
cache hit returns the same keys as the first call.
* fix(anthropic): send one search tool and judge tool search on the real model for chat tool_search
Several OpenAI tool_search items now collapse into one tool_search_tool_regex
tool, since Anthropic answers 400 when two tools share a name. A native
Anthropic search tool the caller already sent wins over the OpenAI item.
Bedrock invoke and Vertex AI now decide tool search support on the real model
before tool-based response_format swaps in a stub model name, so a Claude 5.5
request with response_format no longer gets a 400 on a route that runs tool
search.
* fix(anthropic): name the proxy drop_params setting in the tool_search 400
* fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan
The scan translated tool_search into Anthropic's regex search tool on every
route, so a guardrail returning only function tools removed tool_search from
native OpenAI requests and OpenAI answered 400 for the deferred tools
* fix(anthropic): run tool search on azure_ai, gate it on Databricks Claude, and strip deferral without recursion
Anthropic's Foundry docs list no tool search exclusion for either hosting
option, so the claude-tool-search rule now fills azure_ai. Databricks Claude
models route their tools through the same unsupported-tool-search gate, so a
tool_search tool answers 400 or drops under drop_params instead of a 500.
_without_deferral strips two fixed levels instead of walking recursively.
* fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai
* fix(anthropic): keep answered tool search replays paired with their answers
* fix(anthropic): drop Claude tool search calls from history sent to other routes, name the fix in the 400
A second turn that carries Claude's srvtoolu_ search call and its result now has both removed
before completion() sends it to a route that does not build an Anthropic request, including the
responses bridge. The tool search 400 now names model_info supports_tool_search for a model the
cost map does not know, and bedrock/invoke for a Converse model that runs tool search there.
* fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails
* fix(chat): keep answered Claude tool search calls in history sent to other providers
Only a srvtoolu_ search call with no tool result is dropped before a
non-Anthropic route, so an answered pair keeps the caller's shape where
OpenAI and Gemini take it. The history check no longer copies the list
through a TypeAdapter.
* fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object
* fix(chat): drop unanswered Claude tool search calls only on routes that reject them
OpenAI, Azure OpenAI and Bedrock Converse answer 400 for a srvtoolu_ call
with no tool result, while Gemini answers 200. Keep the history as sent
everywhere else so routes that already work see what the caller sent.
* fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent
* test(integration): cover chat tool_search routing, history drop and load on the wire
Audit round 1 of 3 at
|
||
|
Mateo Wang
|
52fc37888b
|
fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444)
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): give the defer_loading casts a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): read replayed tool-search and namespace fields through typed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep tool_search items on a turn with no message or tool call Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep every reasoning item on a search-only turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): leave content unset on a search-only turn so replay adds no empty message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): replay tool search items only with tool_search, count replayed tools in the allowlist, keep namespace off guardrails * fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses Under drop_params, defer_loading is dropped when no tool_search tool is sent, so the 400 it would cause upstream goes away as before. A non-bool value answers 400 without drop_params and is dropped with it. A null tool-level value falls back to the nested function value. A request carrying a tool_search tool now bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort none, since chat completions rejects the tool_search type. * fix(responses bridge): replay tool search items in output order and count replayed tools in tool policy Store the ordered output prefix through the last tool search item, reasoning included, so a second reasoning item no longer makes OpenAI reject the replay. The shared chat-to-Responses converter now replays tool search items and namespace only when the bridge opts in, so Noma, the complexity router and guardrail payloads keep their shape. The tool policy guardrail and the allowlist count replayed tools only when the request can send them. * fix(responses bridge): forward defer_loading only beside a tool_search tool defer_loading only means something with tool_search, and backends without it reject the key (xAI answers 403). A true value with no tool_search tool now answers 400 naming drop_params and is dropped under drop_params; a false value there is a no-op and is dropped. * fix(tool_loop): keep tool search items and namespace between rounds * fix(responses bridge): check replayed tool search loads when a deployment adds tool_search * fix(chat): drop litellm response fields before strict OpenAI-compatible providers A chat reply from the Responses bridge carries provider_specific_fields on the message and on each tool call, plus reasoning_items. Groq, Fireworks, Mistral and Together reject unknown fields, so sending that reply back to them as history failed with a 400 or 422. Strip them in the shared helper and use it in those providers. * fix(caching): keep reasoning_items and tool search items on response cache hits A cached bridge response lost reasoning_items on a non-streaming hit and lost both reasoning_items and provider_specific_fields (tool_search_items) on a streamed hit, so the next turn could not replay the search or the reasoning around it * test(integration): cover tool search item replay on the chat to Responses bridge Proxy, SDK, authorization and chaos cells for returning and replaying tool_search_call and tool_search_output items and the tool call namespace * test(integration): add response cache twins for the tool search guardrail cells Each guardrail cell now has a twin where the answer is already in the Redis response cache: a key opted out of global guardrails caches a replay that loads a policy-blocked tool, the same replay without guardrails is cached before block mode sees it, rewrite mode is not served the unrewritten conversation's cached answer, and an allowed replay is served from the cache. * fix(responses bridge): forward defer_loading beside a client-executed tool_search The bridge now replays a client tool_search_call and the client's tool_search_output through provider_specific_fields.tool_search_items, so a deferred function beside a client tool_search keeps defer_loading like it does beside a hosted one * test(responses bridge): pin the client tool_search turn and its replay, streamed and not * fix(responses bridge): strip nested namespace for strict providers and keep streamed reasoning_items on text replies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep dropping defer_loading beside a client-executed tool_search Forwarding defer_loading beside a client tool_search made GPT stop with a search call the chat client has to answer by hand, so a plain question that used to get tool calls got none. Keep main's behavior of dropping it there. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nate Armstrong
|
23025c92ba
|
fix(anthropic): emit one tool_use block per call when a /v1/messages stream chunk carries several (#46492)
Providers like Mistral put every parallel tool call, with full arguments, into one
chat-completions chunk. The /v1/messages adapter opened one tool_use block for the
first call and concatenated every call's arguments into it, so clients got invalid
JSON like {...}{...}. _CombinedChunkSplitter now splits such a chunk into one chunk
per call. The first call streams live and calls opened behind it are held, gathering
their fragments by index, until a non-fragment chunk arrives or the stream ends
Fixes #44029
|
||
|
devin-ai-integration[bot]
|
314f17986a
|
fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models (#45882)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models * fix(responses): define the tool_search_call output item locally for older openai SDKs * fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500 * fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out A replayed Claude tool search call that the caller answered with a tool message kept its tool_result in the user turn while its tool_use was dropped from the assistant turn, so Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when no tool message answers it. The Responses bridge now strips defer_loading from every tool whenever no tool search tool is sent, whether the caller sent none or the route cannot run it, since a deferred tool with no search tool can never be loaded. Namespace members had started carrying the flag to every bridge provider, which providers that reject unknown tool keys answered 422. The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit. * fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried * fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator * fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic A function tool whose allowed_callers is not a list of strings now answers 400 instead of 500. The guardrail scan reads tools without malformed flags, so the route decides between a 400 and a drop under drop_params. When the caller sends a native Anthropic search tool beside tool_search, only the native one goes out. Databricks Claude no longer counts as an Anthropic request route. A previous_response_id follow-up off Anthropic also drops a search call the client replayed as a function_call with a srvtoolu_ id, and its output. The answered tool id scan in the Anthropic message factory runs once per assistant run instead of once per message. A null created_by on tool_search_call and tool_search_output output items is left out of the serialized response, so a cache hit returns the same keys as the first call. * fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan The scan translated tool_search into Anthropic's regex search tool on every route, so a guardrail returning only function tools removed tool_search from native OpenAI requests and OpenAI answered 400 for the deferred tools * fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai * fix(anthropic): keep answered tool search replays paired with their answers * fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails * fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object * fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent * fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id * fix(responses): answer 400 to a tool_search execution that is not server or client OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed execution is dropped like the other malformed tool flags, which leaves a server search * test(responses): cover the 400 and drop_params paths for a malformed tool_search execution * test(responses): cover a Claude search call replayed to Bedrock Converse with cache marks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4428c4c78f
|
feat(cost-map): add scx-ai DeepSeek-V4.1-flash, GLM-5.3 and Kimi-K3 (#46496)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: bhuvan2134686 <bhuvankakkar@gmail.com> |
||
|
ryan-crabbe-berri
|
6c78089d34
|
fix(bedrock): use base_model for Claude capability checks on application inference profile ARNs (#46569) | ||
|
ryan-crabbe-berri
|
e889d9d653
|
fix(bedrock)!: replay redacted_thinking on Converse and keep its data as Claude's own string (#46532)
* fix(bedrock): send redacted_thinking back to Converse as reasoningContent.redactedContent Replaying an assistant turn that carried a redacted_thinking block to Bedrock Converse turned it into an empty unsigned reasoningText, which the assistant content builder then dropped as blank text. Bedrock returned 200 so callers never noticed. Map redacted_thinking to redactedContent in both the thinking_blocks and inline content paths of the sync and async builders * fix(bedrock): drop blank redacted_thinking data and keep non-str signatures signed on Converse replay * fix(bedrock): replay redacted_thinking to Converse only when the target is Claude Nova, Llama, DeepSeek and gpt-oss reject reasoningContent.redactedContent with a 400, so a Claude history replayed to them through a Router fallback or model switch keeps dropping the redacted block the way main does. Claude is resolved from the Bedrock base model of the target, and an application inference profile ARN counts as Claude when its deployment's base_model names a Claude model. Adds integration contracts for the replayed Converse request shape, stream and non-stream, with a tool call, and for non-Claude and ARN targets. * test(e2e): cover Bedrock Converse replaying redacted_thinking live through the proxy Turn 1 triggers redacted thinking on Claude Sonnet 4.6, turn 2 replays it, and a live edge between the proxy and Bedrock checks the replayed Converse body carries the same reasoningContent.redactedContent. Covers nonstream, stream and a tool call round trip * fix(bedrock): validate litellm_params before reading the deployment base_model litellm_params arrives as an untyped dict, so passing it straight into the typed helper grew reportUnknownArgumentType. Validating it into Mapping[str, object] first keeps the read typed without growing the type-check gate * fix(bedrock): wrap non-Converse redacted_thinking data and let a named model id win over base_model * test(bedrock): cover redacted_thinking minted outside Converse, model_id and the sync Router base_model path * fix(bedrock): match bare claude- base_model names without reading the untyped cost map * fix(bedrock): keep redacted_thinking data as Claude's own string on every route Converse hands redactedContent back base64-wrapped, and that wrapper used to leak out as data, so a Converse turn replayed to Invoke or the Anthropic API was rejected. Converse replies now decode it, Converse requests wrap it exactly once, and the Anthropic-shaped builders unwrap histories saved with the old wrapping. Responses API encrypted reasoning is no longer sent to Claude on Converse, which rejected it with a 400 |
||
|
Mateo Wang
|
9ba4cc4fb3
|
fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440)
* fix(anthropic): keep tool_search_tool_result so the search call replays with its result Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): drop a client tool_result for a search call replayed with its server result * fix(anthropic): replay a server tool call the client answered as the client sent it * fix(anthropic): keep a server tool result under modify_params, count client-answered marks, and replay it from run_tool_loop modify_params no longer adds a placeholder answer for a server tool call whose result is kept, unless the call ends the request. The cache control hook counts a mark on a server tool call the client answered, since that call goes out as a plain tool_use. run_tool_loop sends provider_specific_fields back and runs only the client's tool calls. * fix(anthropic): keep a server tool result when another tool answer follows, scope run_tool_loop's replay, and keep code_interpreter_results off tool search turns * test(anthropic): audit cells for tool search result round trips, replay, cache marks, pass-through logging and chaos --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
litellm-stampbot
|
4d5291885f
|
fix(cost): price AgentCore web search at $7 per 1,000 queries (#46571)
* fix(cost): price AgentCore web search at $7 per 1,000 queries * feat(search): honor input_cost_per_query set on a search tool (#46560) * feat(search): honor input_cost_per_query set on a search tool * feat(search): record the search tool price on proxy-created logging objects * chore(search): type the search tool price lookups and shard tests/unit/search * chore(openapi): regenerate snapshot and ui types for search tool input_cost_per_query --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Kerry Lu <klu@berri.ai> |
||
|
Kerry Lu
|
44e94eb985
|
fix(cost): split xai spend rows across input and output instead of booking the total as output (#46320)
* fix(cost): keep the input/output split on streamed xai spend rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): split xAI's reported total across input and output by token math Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
dba46c0513
|
fix(responses): keep the provider code and message on top-level stream error events (#46438)
* fix(responses): keep the provider code and message on top-level stream error events Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): log a mid-stream failure before the router falls back CustomStreamWrapper now waits for failure logging before it raises the error the router falls back on, so the cooldown lands on the deployment that failed instead of the fallback that replaced it. Also keeps numeric error codes and the top-level message on the WebSocket and ChatGPT paths. * fix(streaming): run sync failure callbacks before a failed stream falls back to another deployment The router cools a deployment down from a sync failure callback that reads the request's model_info. A failed stream submitted that callback to the logging executor without waiting, so it often ran after the router's fallback had overwritten model_info and cooled down the healthy fallback deployment instead, leaving the model group with no deployments * test(responses): cover top-level OpenAI stream error events end to end * test(streaming): read live worker pids so the second SIGKILL chaos variant sees two workers * fix(streaming): run waited sync failure logging off the shared logging pool * revert(streaming): keep failed-stream failure logging in the background Waiting for sync failure callbacks before a failed stream raises fixed the router cooldown race, but it slowed healthy calls under many failing streams with a slow callback, since the wait shares a thread pool with every call's start. The race is pre-existing on main and is tracked separately, so this PR goes back to main's logging timing and keeps only the top-level error event changes --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Tobias Mayr
|
d03923e6a9
|
fix(responses): propagate cache_control through responses->chat completion bridge (#32511)
* fix(responses): propagate cache_control through responses->chat completion bridge User/assistant messages, tool call outputs, and function calls built from Responses API input items were dropping cache_control, breaking prompt caching for Bedrock/Anthropic when routed through the litellm_completion transform. Also propagate cache_control onto the last content element of user/assistant messages, since Anthropic/Bedrock read it from the content block rather than (or in addition to) the message level. Adds the missing cache_control field to ChatCompletionToolMessage and ChatCompletionResponseMessage TypedDicts to keep the assignments type-safe. * chore(ui): regenerate schema.d.ts for ChatCompletionToolMessage.cache_control Adding cache_control to the ChatCompletionToolMessage TypedDict changed the proxy's OpenAPI spec; regenerated via `npm run gen:api`. * fix(responses): don't overwrite existing content-block cache_control; add list-content test Per Greptile review on the PR: - The message-level cache_control was unconditionally overwriting the last content block's cache_control when that block already carried its own (preserved by _transform_responses_api_content_to_chat_completion_content). Now only propagates onto the last block when it doesn't already have one. - Added the missing test coverage for the content-block propagation path with list content, which no existing test exercised (all used plain string content, so the isinstance(content, list) branch was never hit). * style(responses): extract cache_control into a named variable, trim comment Per style review: the message-building branch read cache_control 3 different ways in 5 lines (.get() twice, then [] indexing once), unlike the sibling tool-call-output/function-call functions this PR touches, which each extract it into a named variable up front (output_cache_control, function_call_cache_control). Matched that convention here too. Also trimmed the 3-line inline comment to one line, matching the file's usual comment length for similarly subtle behavior. * chore: refresh lazy OpenAPI snapshot * fix: type cache control propagation * fix: place cache control type suppressions * style: format cache control type conversion * refactor(responses): build bridged cache_control messages in one shot * fix(responses): keep cache_control marks on bridged tool calls and within the provider cap A function_call item's mark now rides on the tool call the bridge builds, an assistant text item merged behind a function_call keeps its own mark, a developer item keeps its mark when it is mapped to a system message, and a previous_response_id replay yields its stale marks to the newest ones so the provider cap holds. The Responses layer counts the marks on function_call and function_call_output items before it places an ordinal injection point, and the chat pass replaces that count with its own instead of adding to it. * fix(responses): count each cache_control mark once and keep the bridge within the provider's limits List content carries an item-level cache_control on its last block only, so the provider sees one cache point and the breakpoint census counts one. Marked function tools and SDK-typed input items are handled by the Responses-layer reserve, assistant text merged into a function call turn keeps its mark on its own block, and the previous_response_id replay budget drops marks per turn, newest kept, instead of stripping every older turn once one overshoots. The request's own marks now fall under the same budget, newest kept, so five marked items no longer answer the provider's 400. A cache_control value that is not an ephemeral mark with an optional ttl is left off the bridged message instead of reaching the provider as sent, and a developer message maps to the system role with its content, name, and cache_control only, so unknown keys never reach a strict provider. * fix(responses): send item cache_control only to cache-point targets and follow drop_params Item marks reach the provider only when the target turns them into cache points, and every other target keeps main's behavior of stripping them. Without drop_params a mark the provider would refuse reaches it as sent, so the client gets the same 400 as on /v1/chat/completions. With drop_params a malformed mark is dropped and only the newest 4 are kept. The shared input-to-messages helper and the developer mapping in base_utils go back to main, and a marked developer item reaches a cache-point target as a system message in the bridge. * fix(responses): budget top-level marks, follow drop_params on Nova tool marks, and match chat-keeping providers * fix(responses): validate replayed marks and count input tokens without item marks * fix(responses): leave unknown Bedrock ARNs unmarked under drop_params, match OpenRouter's list, and reserve only forwarded marks * test(responses): cover item cache_control parity per chat route, drop_params trimming, and replay budgets * fix(responses): keep item marks off inspection paths and match each chat route's keep rule Only the request the bridge sends gets item marks now. Guardrails, token counting, rate limiting, and input_tokens read the input with no item marks, as on main, so an off-spec mark cannot fail a guardrail call. Tool message and tool call marks are typed inside the bridge, leaving the shared chat types and guardrail schemas as on main. Databricks keeps marks only for Claude, Snowflake uses its chat route's Claude check, and every Anthropic deployment keeps them whatever its model name. The injection census skips SDK-typed tools, and the owned-proxy test expects the marked messages on /v1/responses. * test(responses): cover guardrail reads without item marks, Snowflake name parity, and SDK-typed tools in the census * fix(responses): leave a non-list input untouched when stripping item marks * test(responses): cover string, list, object and tuple input with and without item marks kept Each input shape now answers the same whether the unmapped Bedrock ARN keeps item marks or strips them under drop_params, through the proxy and the SDK. Object input also runs through the OpenAI-compatible chat bridge, /v1/responses/input_tokens and the project input-token limiter * test(cache): derive owned-proxy cell timeouts from the harness boot budget The capped-burst cells and the uncapped response-cache cells boot their own two-worker proxy, and a fixed 240 or 360 second timeout cut them during boot on a slow runner before the harness readiness budget could decide. They now carry the same derived budget as the sibling chaos and guardrail files, with four boot and stop budgets for the restart cell that boots twice --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
058c84476a
|
fix(mcp): leave out tool texts too long to embed instead of failing tool search (#43554)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
47e4ed88c2
|
fix(anthropic): keep refusal stop_reason and stop_details on the logged response (#46104)
* fix(anthropic): keep refusal stop_reason and stop_details on the logged response * fix(anthropic): keep the refusal stop-fields boundary tolerant of a non-dict upstream body * test(anthropic): use Anthropic's documented refusal example as the unit-test fixture * fix(streaming): keep provider fields on async stream cache hits and received_model_id on synthesized finish chunks * fix(caching): replay cached completion_tokens_details on stream cache hits and skip non-object cached provider fields * test(integration): cover Anthropic-family refusal stop_details on the wire, the cache and under chaos * fix(caching): skip a cached completion_tokens_details that fails validation on stream replay * fix(anthropic): keep refusal stop fields on a JSON-mode non-stream response Also adds the round 3 audit cells for JSON-mode refusals, the proxy response serialization, and wrong-typed cached usage details on both stream replays * test(anthropic): expect content_filter on streamed JSON tool refusals after main's stream change --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
83980c9d63
|
fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493)
* fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): keep only the raising call inside pytest.raises for the pre-content stream tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): yield the primed first chunk from a helper generator so the stream iterators keep their complexity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): expect the empty invoke stream to be reported from router.completion now that the first chunk is read there Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: give router stream mocks a first chunk or an ending so the eager first read neither hangs nor retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): name the first-chunk helpers after what they do (_read_first_chunk, _FirstChunk, _chunks_with_first) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): drop the string annotation on _FirstChunk.chunk, ModelResponseStream is imported at runtime --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
542b254ca2
|
fix(anthropic): forward service_tier on /v1/messages for anthropic, azure_ai and bedrock_mantle (#46370)
* test(integration): add known-bug translation cases for request fields LiteLLM drops service_tier on /v1/messages, conversation on /v1/responses and metadata on /v1/chat/completions never reach the provider. Each gets a TranslationTestCase with the correct provider body, marked xfail(strict=True) through the new TranslationTestCase.known_bug field, plus a passing service_tier case for /v1/messages on the openai route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): widen the known-bug cases to azure_ai, bedrock_mantle, openai/responses and azure deployments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): forward service_tier on /v1/messages for anthropic, azure_ai and bedrock_mantle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
2d4472efc1
|
fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733)
* fix(vllm): write spend log rows for /vllm passthrough calls to router models The vLLM passthrough config never built a logged response for relayed calls, so stream and non-stream calls through /vllm with a router model wrote no spend row and never charged the key. vLLM now shares the OpenAI-shaped relay logging that Azure already had, moved into an openai_like helper both configs call. * ci: run the vllm passthrough unit tests in the llms-providers shard * fix(proxy): log /vllm text completions and ask router models for stream usage A chat stream relayed to a router model without stream_options now gets include_usage added on the way out, and the usage chunk vLLM appends is removed before the client sees it, so the spend row carries vLLM's own token counts instead of a local estimate. /vllm/completions calls now write a row too, stream and non-stream. * fix(vllm): keep usage, fingerprints, and disconnect rows correct on /vllm streams Inject and strip stream usage on /vllm/completions too, hold every finish frame for the fingerprint when n > 1, pass a forced-usage server's stream through untouched, stop turning continuous_usage_stats alone into per-chunk usage, close the upstream when the client hangs up so the row is written, and assemble text choices with one sort instead of a scan per choice * fix(vllm): keep n>1 stream order when holding finish chunks for the fingerprint A held finish chunk now goes out as soon as any other chunk arrives, so a choice that finishes while another still streams keeps its place. Only the finish chunks right before the usage chunk get vLLM's system_fingerprint. * fix(vllm): keep caller stream_options and log both success handlers Forward a stream_options the proxy cannot read, or a non-boolean include_usage, to vLLM as the caller sent it. Only a false always_include_stream_usage changes the /vllm relay, so clients get the same stream bytes under every setting. Record the passthrough origin when logging starts so the sync and async success handlers both parse the relayed body, hand callbacks a parsed text completion, and read relayed messages and prompt separately so a stray field no longer drops the local count. Add the audit's integration tests for the relay, the settings, and the Azure spend row. * test(vllm): expect the caller's stream under always_include_stream_usage true * fix(vllm): relay long SSE frames in linear time * test(vllm): time the relay's usage and finish_reason prefilter on one large frame --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |