333 commits

Author	SHA1	Message	Date
Jorik	492272a051	fix(azure): reuse the cached Entra token provider on the Responses path (#46421) ... get_azure_ad_token() built a new DefaultAzureCredential for every request when enable_azure_ad_token_refresh is on, so each Azure Responses call fetched a fresh Entra token synchronously on the event loop. Use the per-scope cached provider the SDK-client path already uses, so the credential's own token cache does its job. Fixes #46419	2026-10-19 08:27:05 -07:00
Habon Laszlo	a2773bdcad	fix(azure): route WebRTC client_secrets and calls to the GA v1 realtime API (#47316)	2026-10-19 06:10:11 -07:00
moyai-berriai[bot]	f370235bb4	fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874) ... * fix(guardrails): preserve Bedrock Converse redactions and honor tool-only scans * Preserve main's per-block streaming fixes and regression coverage * fix(bedrock): apply Converse guardrail rewrites texts-first and keep main's write-backs Map tool results onto their text and json parts only, accept a structured row the flat texts already reproduce, let a flat-text mask win over a structured leaf, ignore texts that do not line up with the request when structured rows come back, keep tool call arguments out of the rows under skip_tool_message_in_guardrail and send no tool definitions to tool-only guardrails * fix(bedrock): accept blank content on tool-call-only Converse turns A vendor that returns "" or [] where an assistant row carried only tool calls no longer fails the request closed. Also type the leaf lookup without Any and restore the original layout of the existing unit tests. * fix(bedrock): scan additionalModelRequestFields in Converse guardrail rows and split the handler Guardrails that read structured_messages instead of texts (Akto, Prompt Security, CrowdStrike AIDR and others) stopped seeing strings in additionalModelRequestFields, which Bedrock can forward into the model prompt. Those strings now go to the guardrail as one user row placed before the conversation, and a row-only edit to them is written back. The Converse to OpenAI row conversion moves to litellm/llms/bedrock/chat/converse_to_openai.py and the returned-row diffing to structured_edits.py, so the handler keeps only the glue. Rows record the Converse path of every text part, scope flags travel as one value, and path lookups return a value instead of raising. Unit tests use a real CustomGuardrail subclass instead of mocks, and the integration vendor modes get descriptive names. * fix(bedrock): keep new Converse helpers inside the type discipline rules * test(bedrock): pin Converse row edits that change shape, role or placement * fix(bedrock): type the Converse body once so its edits keep their reference * fix(bedrock): list request-field texts where their Converse row sits * fix(bedrock): send a reshaped Converse tool result as one text block A vendor that compresses or summarises a tool result, such as Headroom collapsing a JSON list into a table string, returned a 500 because the row could not be split back into the original text and JSON parts. The tool result now carries the vendor's string as one text block where its parts were, keeping images and documents in place. It still fails closed when the flat texts masked that tool result or the row changed its tool call id. * fix(bedrock): apply guardrail tool edits on Converse and keep compression guardrails off it A guardrail that masks a Converse tool description or schema now has that mask written into toolConfig, a tool it removes is dropped, and a tool it adds, renames or reorders is refused before Bedrock. Headroom, Compresr and TypeSafe get only the flat texts on Converse again, as on main, because the pass-through route has no loop to answer their retrieve tool * test(bedrock): refuse a guardrail that changes a Converse tool's kind * fix(bedrock): keep partial-row and row-rebuilding guardrails on flat Converse texts OpenAI moderation, the LLM judge, CrowdStrike AIDR, Prompt Security, PANW Prisma AIRS and HiddenLayer scan only part of the rows or rebuild their returned texts from them, so on Converse they get the flat texts only, as on main. When a vendor returns changed rows, its texts are applied only if they keep the flat layout and change only rows the vendor also changed, so row-shaped texts of the same length no longer overwrite unrelated Converse fields. * test(bedrock): pin how flat texts sit next to rewritten Converse rows * fix(bedrock): keep the Bedrock guardrail on flat Converse texts With experimental_use_latest_role_message_only, the Bedrock guardrail places its mask by counting text parts per row, and Converse rows do not line up with the flat texts, so the mask could land in tool call arguments. Give it the flat texts only, as on main. * fix(bedrock): keep Akto on flat Converse texts Akto maps its masked payload back onto the flat texts by value, and a Converse row joins or serialises tool call arguments and JSON tool results, so a masked card there could not be placed and the request was blocked. Give Akto the flat texts only, as on main, where the mask is applied. * fix(bedrock): list request-field texts last again, as on main Flat texts listed additionalModelRequestFields strings right after the system prompt, so flat-text guardrails got a different order than on main. Akto's live verdict on a benign request depends on that order and started blocking it. Rows already keep their own order and texts that do not follow rows are dropped, so the flat texts go back to main's order. * fix(bedrock): list request-field texts next to their row for row readers Row readers that rebuild texts from rows could swap a masked request field with a masked user turn when the field strings came last. Row readers now get those strings right after the system prompt, where their row sits. Flat-text guardrails keep the end-of-list order. * fix(bedrock): list nested request-field texts in their row's order for row readers Row readers that rebuild texts from rows could write a masked request field onto a sibling key when the field held nested objects, because the texts walked the fields depth first and the row breadth first. Row readers now get those strings in the row's own order. * fix(bedrock): walk request fields breadth first instead of resolving each path Resolving every request-field string by its path revalidated the parent object per string, so a wide additionalModelRequestFields took seconds. The string walk now goes breadth first for row readers, the same order the row lists them, in one pass. A unit test pins that texts and rows map the same way when the vendor masks them differently. * fix(bedrock): fail closed when a vendor's rows drop a mask its texts carry When a vendor returns changed rows whose texts do not line up with them, the rows win. A text the vendor masked on a field its rows left untouched (a tool call argument, say) then reached Bedrock unmasked. Refuse that rewrite instead, unless the edited body holds the changed text elsewhere, as texts rebuilt from the rows do. * fix(bedrock): read same-length texts by position when rows drop a mask Matching a dropped text against any string of the edited body let a row mask on another field with the same token cancel it, so the raw tool call argument still went out. Texts of the length that was sent are now checked position by position, and only texts of another length, which cannot be placed, are matched by value. * test(bedrock): count each rebuilt text against the rows' edits * fix(bedrock): keep PromptGuard on flat texts and refuse texts that cannot be placed PromptGuard rebuilds its returned texts from the user rows, and a vendor that does so while masking only row content let a card in a tool call argument reach Bedrock, which main masked. PromptGuard now gets the flat texts as on main, and a reply whose rows changed next to texts of another length fails closed, since what those texts mask cannot be checked. * test(bedrock): let the structured checks decide the unsafe tool call and missing row cases The replies in these tests carried texts that no longer matched the sent texts, so the new refusal for texts that cannot be placed answered first and the tool call and row count checks went unexercised. * fix(bedrock): send request field strings to row readers after the conversation, where main puts them Row readers got the additionalModelRequestFields row before the conversation. Singulr judges only the latest user message, so an injection in a request field that main blocked reached Bedrock. The row and its texts now come last, as main sends those strings in texts * test(bedrock): pin nested request field order for flat readers and the texts view Both request field tests now use nested fields whose depth-first and breadth-first orders differ, so a flat reader getting the row order or a texts view written in main's order fails * refactor(bedrock): ask each guardrail whether Converse should hand it rows Converse picked the guardrails that get only the flat texts from a list of provider names. CustomGuardrail now has supports_structured_messages, True by default, and the ten classes on that list override it to return False, so the handler asks the guardrail instead of the list. Every guardrail class gets the same answer as before * fix(bedrock): keep Singulr on flat Converse texts as on main Singulr wraps each flat text as a user message and its backend judges only the latest user message, so on main a Converse tool result was the turn it judged. Handed rows, the tool result arrives with role tool and the latest user row is the original question, so an injection in a tool result reached Bedrock. Singulr now returns False from supports_structured_messages * fix(bedrock): keep Qualifire on flat Converse texts as on main Qualifire builds its evaluation input from the last user row only when that row's content is a string, so Converse rows (content-block lists) gave an evaluation_id policy an empty input. Opt it out of structured messages so it gets main's flat texts again. * fix(bedrock): keep Conduct on flat Converse texts as on main The conduct-litellm-guard plugin scans only user rows, so with Converse rows a system prompt, assistant text or tool result left its scan. Opt it out of structured messages so it gets main's flat texts again, all sent as user turns. * fix(bedrock): refuse tool removals that leave a toolConfig Bedrock rejects A guardrail that removed every tool, or the tool toolChoice names, produced an empty tools list or a dangling toolChoice that Bedrock answers with an opaque ValidationException. Refuse those rewrites before Bedrock, like other tool changes this route cannot carry. * fix(bedrock): list request-field strings in main's order for every guardrail The request-field row walked additionalModelRequestFields breadth first, and guardrails that get rows had their texts follow that order. With two or more strings the last text changed, so Ovalix, which blocks only when the last text is flagged, rewrote an injection main blocked. The row now walks the fields in the order main lists them in the texts, and the texts use main's order for every guardrail. * test(bedrock): pin a container's own request-field strings before its nested ones * test(bedrock): pin tool_policy and MCP end-user filtering on Converse tools * fix(bedrock): rebuild Converse tools the guardrail changed and drop blanked text like chat A guardrail that adds, renames, reorders or retypes a tool now gets its list sent to Bedrock, converted the way chat completions converts tools, with cache points kept behind their tool. When every tool is removed, toolConfig is dropped and earlier tool calls and results become text, as chat does. Text blocks a guardrail blanks are dropped, and a turn left empty fails closed. The body copies used to place edits are JSON copies, so deeply nested tool JSON no longer hits the recursion limit. * test(bedrock): keep the client tool schema on Converse masks and refuse deep JSON text drops cleanly * fix(guardrails): keep Airia on flat texts for Bedrock Converse so its redactions still apply * fix(guardrails): a pipeline step whose rewrite the endpoint cannot apply follows on_fail on_error: allow sent the original request on when the endpoint refused a guardrail rewrite it could not apply, so a Converse request the guardrail masked reached Bedrock raw. The refusal now counts as a fail on every endpoint. * fix(guardrails): a pipeline step blocks when its rewrite cannot be applied, whatever on_fail and on_error say A rewrite the endpoint cannot apply is neither a content verdict nor a technical error, so on_fail next or allow and on_error allow must not send the original request on * test(bedrock): trim the Converse guardrail tests Drops the chaos and per-vendor integration cells and the duplicate unit cases, and leaves the existing handler tests as they are on main * fix(bedrock): put Converse request-field strings in the latest user message for row readers A guardrail that judges only the latest user message, like Noma v2, read the additionalModelRequestFields row placed after the conversation, so a request with extended thinking or anthropic_beta sent the user's card to Bedrock and let an injection through. Those strings now follow the latest user message's text in its row, and get a user row of their own only when no user message has text. * fix(bedrock): create toolConfig when a guardrail adds the first Converse tool A guardrail that returned a tool for a Converse request with no toolConfig got a 500, since the tool write-back only rebuilt an existing list. It now creates toolConfig, as chat completions does. Guardrails that get only flat texts have their returned tools ignored, as on main. * test(guardrails): check Converse flat-text opt-outs without patching apply_guardrail Each built-in opt-out is checked through its own supports_structured_messages(), the handler's flat path through an injected scripted guardrail, and Headroom through the real class, which would call its unreachable service if it got rows * fix(guardrails): keep Converse tool matching and write-back linear in request width Matching each returned tool rescanned the whole tool list, and every write-back step revalidated the list or object it passed through, so a request with thousands of tools or request fields held the event loop for seconds. Returned tools are now looked up by name in one index, and path steps read the child directly. * fix(guardrails): let later Converse guardrails scan guardContent text When a guardrail removes every tool on a Converse request that carries guardrailConfig, tool results are rewritten as guardContent blocks. The flat text extractor read only top-level text, so a later masker in the same request never saw them and a card in a tool result reached Bedrock unmasked. Chat completions and /v1/messages keep tool results visible to later guardrails after the tools are removed. The extractor now reads guardContent text the same way the structured rows already do, so flat-text guardrails also scan guardContent text a caller sends. * fix(guardrails): keep caller guardContent out of Converse flat texts and send tool history as plain text Flat-text guardrails read only top-level text blocks again, as on main. A guardContent block after the caller's text no longer becomes the last text, so guardrails that judge the last text (HiddenLayer, Qualifire) check what the caller wrote When a guardrail removes every tool, earlier tool calls and results are now written as plain text blocks instead of guardContent. Later guardrails still scan and mask them, and Bedrock's own guardrail keeps checking every message, since a single guardContent block makes it skip all plain text * fix(guardrails): split a newline-joined Converse message back into its parts A vendor that returns a multi-part message as one newline-joined string now has its masks written back to each part, guardContent included, when no part sent already holds a newline. * fix(bedrock): keep Converse tool history as tool messages until send and match chat on emptied turns When a guardrail removes every tool, the tool calls and results stay tool messages for the guardrails after it, so a scan_only_tool_results guardrail still masks them. The toolConfig with no tool is dropped and the history turned into text only when the request is signed, as chat completions does at request build. A caller's toolConfig with no tool gets the same treatment, where Bedrock returned 400 Request-field strings join the latest user message that has text when a tool-result-only turn follows it, so a guardrail judging only that message reads the caller's text with them A turn the vendor empties entirely is sent as written, as on main and chat completions, instead of a 500 Messages go to the guardrail through a JSON round trip instead of a deep copy, and a reply that changes nothing returns early, so a long Converse conversation guards about as fast as on /v1/messages --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 20:38:14 -07:00
devin-ai-integration[bot]	4bebcdf757	fix(databricks): use the native Anthropic route for Claude /v1/messages (#46091) ... * fix(databricks): use the native Anthropic route for Claude /v1/messages Claude models on a Databricks workspace were bridged through chat completions on /v1/messages, which drops the safeguards field and never returns safeguard_results. They now call the workspace's /serving-endpoints/anthropic/v1/messages, the proxy forwards the client's anthropic-beta and anthropic-version headers to Databricks deployments, and the beta header map marks dangerous-tool-use-2026-09-03 as supported there. Any route whose platform does not take that beta drops safeguards together with it instead of answering 400. Non-Claude Databricks models keep the chat completions bridge * fix(databricks): derive the dangerous-tool-use beta from safeguards instead of forwarding client betas A Databricks Claude serving endpoint answers 400 to betas Claude Code sends on every turn, prompt-caching-scope-2026-01-05 among them, so the proxy no longer forwards the client's anthropic-beta header to Databricks deployments. The beta the classifier needs is derived from the safeguards body field, the same way the Azure AI route derives it. The Rust beta-header mirror stops rejecting the Databricks dangerous-tool-use cell so its config-cell test matches the JSON again * fix(databricks): keep the billing block, build the messages route once, and pass the partner user agent * fix(databricks): route Unity Catalog Claude names to the gateway messages route and keep env credentials Unity Catalog model-service names now reach the AI Gateway's native Anthropic route instead of a serving-endpoints path that answers 404 for them, and the native route resolves the key and base through the same order the chat route uses, so environment-only deployments keep working. The native route is limited to Databricks foundation-model Claude names, an unlisted provider forwards safeguards and the beta as sent, and the ten betas the workspace rejects are marked unsupported for Databricks in the beta map and its Rust mirror. * test(rust): expect Databricks to drop the advanced-tool-use beta * fix(databricks): keep passthrough safeguards, partner user agent, and classifier fallback Only the OpenAI-compatible JSON providers drop safeguards now, so a deployment opted into /v1/messages passthrough forwards the field and the beta as sent. When a route drops safeguards and the proxy has a classifier model, the proxy answers the verdict itself, as the chat completions bridge did. The Databricks Messages route also reads the databricks_user_agent param like the chat route. * fix(databricks): drop rejected server tools under drop_params, short-circuit web search, route UC names from a full Messages URL * fix(databricks): drop container on the native Messages route, as the chat completions bridge did * fix(databricks): keep every api_base on a route the workspace serves and gate the derived beta on the map Gate the dangerous-tool-use beta derivation on the beta map the way #43955 does, so GitHub Copilot, DeepSeek, and opt-in passthroughs keep the merge base wire. Recognize /serving-endpoints/anthropic api_bases, fall back to the merge base routing for an api_base path the workspace trim does not know, strip Bedrock invocation metrics from the streamed message_stop, forward client betas to Databricks filtered by the map, and keep tool_choice when a dropped server tool has no name * test(databricks): cover the native Messages route, derived beta and JSON provider safeguards drop on the wire * test(anthropic): expect the dangerous-tool-use beta derived from safeguards next to the client's betas * fix(databricks): send client anthropic headers only on the native Messages route and keep SDK auth Databricks chat completions, called directly or through the Messages bridge, no longer receive the client's anthropic-beta and anthropic-version, as at the merge base. The native Messages route still forwards them, filtered by the beta map, without counting them as caller credentials, so a deployment with no api_key keeps falling back to Databricks SDK auth. A pathless api_base goes to the native route only for Unity Catalog names, matching the chat route. * test(databricks): cover SDK auth with client anthropic headers and a Unity Catalog workspace root on the native route * fix(databricks): keep client anthropic headers off an opted-in passthrough api_base A databricks Claude deployment whose api_base is no workspace route and that opts into /v1/messages with supported_endpoints falls back to the OpenAI-like passthrough. That config does not filter betas through the beta map, yet it received the native-only Databricks header entry, so Claude Code's prompt-caching-scope beta reached the upstream raw and got a 400 where the merge base answered 200. Each Messages config now says whether it is the provider's native route, the OpenAI-like passthrough says no, and the handler passes that answer when it reads provider-specific headers, so native-only entries reach only native configs. * test(databricks): cover client anthropic betas on an opted-in passthrough api_base A databricks Claude deployment whose api_base is no workspace route and whose model_info.supported_endpoints opts into /v1/messages takes the OpenAI-like passthrough. The cell sends Claude Code's prompt-caching-scope and context-1m betas through the proxy and asserts the caller gets the upstream's message and the gateway receives no anthropic-beta, as on the merge base. The scripted gateway answers 400 on the prompt-caching-scope beta, as the real one does * fix(anthropic): keep beta map loads off shared threads and build the safeguards evaluator only when the proxy classifies Unfiltered configs such as first-party Anthropic never read the beta map to decide safeguards support. Filtered configs await the map on its dedicated loader before reading it, on both the native and the MCP paths. The native handler decides after the upstream call and builds the evaluator only when the proxy classifies, so malformed messages keep the upstream 400. The Databricks dispatch tests assert the dispatched route instead of the config class, and the sail and Eden AI wire rows expect safeguards dropped, as JSON providers do. * test(databricks): cover a loaded beta map that marks the safeguards beta unsupported A remote beta map that nulls dangerous-tool-use for databricks while the bundled map allows it must send neither safeguards nor the beta to the serving endpoint, with or without a client beta, and the caller gets a 200 with no verdict. * test(anthropic): annotate the MCP beta map recordings as Final * fix(anthropic): drop safeguards on routes without the classifier only under drop_params A route whose upstream does not answer safeguards (a JSON provider, or a map that marks the dangerous-tool-use beta unsupported) now forwards the field and the beta as the merge base did unless drop_params is set or the proxy runs its own classifier, matching how litellm treats any unsupported param. Databricks keeps dropping both, since its merge base bridge never forwarded the field. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-18 02:44:38 +00:00
ryan-crabbe-berri	52023601a8	test(baseten): annotate test locals with Final (#47205) ... Follow-up to #40152. Greptile flagged the two new reasoning_effort tests for assigning config and result without Final. Annotates every local in the file so it passes the LIT010 check cleanly	2026-10-17 17:47:15 -07:00
Yousef Joe	13335266f5	fix(baseten): forward reasoning_effort to Baseten Model API (#40152) ... * fix(baseten): forward reasoning_effort to Baseten Model API Baseten's Model API accepts a top-level reasoning_effort parameter and validates it server-side (https://docs.baseten.co/inference/model-apis/reasoning), but BasetenConfig.get_supported_openai_params did not list it, so LiteLLM silently dropped it when drop_params=True and raised UnsupportedParamsError otherwise. Add reasoning_effort to the supported params so map_openai_params forwards it verbatim, mirroring the hosted_vllm and deepseek providers. Baseten returns a clear 400 for values outside a model's supported set and ignores the parameter on non-reasoning models, so verbatim passthrough is safe for the whole catalog. [Claude_Code] * fix(baseten): forward reasoning_effort to Baseten Model API Baseten's Model API accepts a top-level reasoning_effort parameter and validates it server-side (https://docs.baseten.co/inference/model-apis/reasoning), but BasetenConfig.get_supported_openai_params did not list it, so LiteLLM silently dropped it when drop_params=True and raised UnsupportedParamsError otherwise. Add reasoning_effort to the supported params so map_openai_params forwards it verbatim, mirroring the hosted_vllm and deepseek providers. Baseten returns a clear 400 for values outside a model's supported set and ignores the parameter on non-reasoning models, so verbatim passthrough is safe for the whole catalog. --------- Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-10-17 17:16:03 -07:00
Mateo Wang	75faaeab64	feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) ... * feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response Count endpoints now say where each number came from in litellm_count_source (provider, fallback_model, local_estimate) and name the fallback model in litellm_count_fallback_model. When the provider answers 400 or 404 for a model it cannot count, the proxy counts the same request with token_count_fallback_model. token_count_framing adds a per-model constant to local estimates. Both settings ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info. * test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests * test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing * fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown Moves the cost map lookup off the event loop * ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty * fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too. * test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates * fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock calls and about 200 ms on every count, since Opus 5.5 cannot count there either. The Kimi K3 e2e also runs the large system prompt shape now. * test(count_tokens): use the shared integration port allocator in the fallback wire test * fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today * test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it * fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases. * fix(count_tokens): answer the provider's error whenever a framed estimate raises A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error. * fix(count_tokens): load the custom tokenizer inside the guarded framed estimate A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors. * fix(count_tokens): look up framing under the deployment's custom_llm_provider * fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error A framed deployment whose custom_tokenizer is not cached locally made every count wait out the Hugging Face hub's whole retry cycle when the hub could not be reached, and a burst of those loads held the default thread limiter that every count's cost map lookup also needs. Counts on a worker now share one download per minute, a failed download is reused for the rest of that minute, and after a provider error a count waits at most 5 s before answering the provider's error. The swallowed-estimate warning logs only the error type. * fix(count_tokens): keep one tokenizer download per key running and load the local route per request * test(count_tokens): type the scripted counter and wait on events instead of wall time where possible * fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500 * feat(proxy): count tokens with token_count_model in its pinned region token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable * fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys * fix(proxy): keep the upstream error envelope when adding the region hint * test(proxy): annotate the token count setting echo test * ci(cost map): require non-blank token_count_model strings and test the region rules * fix(count_tokens): answer the token_count_model's error even when framing is set * fix(count_tokens): keep the provider error for Google contents instead of a framed estimate * fix(count_tokens): estimate Google contents the route translated to messages * fix(count_tokens): keep the provider error for Google contents with non-text parts * test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock * fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags * feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer * test(bedrock): keep the Kimi K3 image check in-process * fix(count_tokens): join the region hint to an upstream message that already ends with a period Also pins that a blank token_count_model in config falls through to the cost map default * fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning * fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them A flat Responses API function or custom tool becomes the chat tool the Responses route sends to inference, and every other tool goes to the Converse body as is, so hosted tools Bedrock drops are left out of the count instead of being counted or turning it into an estimate. * test(count_tokens): annotate new token counter test locals with Final * fix(count_tokens): keep shipped count regions inside the deployment's geography A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added * fix(count_tokens): resolve count settings off the event loop	2026-10-17 17:11:27 -07:00
Mateo Wang	52f5a28679	fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it (#47100) ... * fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it * fix(bedrock): retry with the json tool when Bedrock refuses the native output schema Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller * fix(bedrock): make the native output retry send what an opted-out deployment sends The retry after a native output rejection now reruns the same optional param mapping completion() ran, with the deployment model_info opted out of supports_bedrock_native_structured_output, so tool_choice, effort and the json tool schema match an opted-out deployment instead of a hand-built tool. Each config keeps only the predicate that decides when native output came from the opt-in key. Bedrock batch records for Anthropic models map with the opt-in turned off, since a batch job has no retry. Models on supports_native_structured_output keep native output in batch * fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged * fix(bedrock): give sync converse stream errors the upstream body as text * fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising A Converse json_schema request whose caller outputConfig failed litellm's shape check raised a pydantic error before the request was sent, so the caller got a 500 where the merge base passed Bedrock's 400 through. The opt-in check now treats that outputConfig as carrying no native output. Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of native output, and extends the Bedrock native output integration tests. * fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported A caller can send its own outputConfig on Converse. When Bedrock refused that schema on an opt-in deployment, the retry still carried the caller's textFormat, so Bedrock got the same request twice and the warning told the operator to turn off an opt-in that was not the cause. The retry is now skipped when the opted-out request would still carry a native textFormat, so Bedrock's 400 reaches the caller after one call as on the merge base. The opt-out advice is logged only for the account-level "Extra inputs are not permitted" rejection. * fix(bedrock): keep the caller's continuation message on the native output retry The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt. * test(bedrock): pin additional_drop_params on the /v1/messages native output retry * fix(bedrock): give the Converse native output retry the fake stream timeout	2026-10-18 00:06:07 +00:00
Aayush Jain	e2295b596f	feat(openinfer): add OpenInfer as an OpenAI-compatible provider (#41034) ... * feat(openinfer): add OpenInfer as OpenAI-compatible provider Register OpenInfer in the JSON provider registry with catalog models, proxy create fields, and unit coverage so requests route via openinfer/ Co-authored-by: Cursor <cursoragent@cursor.com> * fix(openinfer): assert catalog invariants instead of vendor rates Pinning per-token prices made CI fail when the vendor list changed. Keep the backup agreement and positivity checks our code owns. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(openinfer): add openinfer to the Rust OpenAI-compatible provider mirror * fix(openinfer): align catalog with OpenInfer's live /v1/models Replace the four catalog rows with the two models OpenInfer serves and publishes a price for (@oi/GLM-5.3-Flash, @oi/Qwen3.5-27B), using its listed per-token rates and context windows, and price-check them in tests * fix(openinfer): use OpenInfer's native Responses and Messages APIs, allow tools on unpriced models, price Qwen3.5-9B Declare /v1/responses and /v1/messages for openinfer, since OpenInfer serves both natively. Let JSON providers declare provider-wide capabilities in providers.json, read through get_provider_info, so tool calls reach OpenInfer for models without a cost map row. Send max_completion_tokens as is, set output caps to each model's context window, mark the three rows as reasoning models, add Qwen3.5-9B at its listed rate, and wire openinfer into models_by_provider so the openinfer/* wildcard lists its models. * fix(openinfer): let a model's own capability row win over provider defaults in get_model_info Provider defaults now only fill capability fields the cost-map row leaves unset, so a row or deployment that turns function calling off stays off. Also marks openinfer_models as Final. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 23:22:19 +00:00
Mateo Wang	16645cc963	fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162) ... * fix(router): bound fake-streamed calls by the request timeout, not stream_timeout A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream. * test(timeout): cover stream_timeout on live streams only and the router's timeout split * fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management * fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface A faked stream now takes the longer of timeout and stream_timeout, so a caller who raised stream_timeout to cover a slow fake stream keeps that headroom. Azure o-series dispatch is mirrored when deciding whether a stream is faked. text_completion, /v1/messages and /v1/responses through the Router now get timeout and stream_timeout separately, like chat calls. * fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages * fix(router): report and log the timeout each stream was bounded by Native /v1/messages and /v1/responses now record the timeout they applied, so x-litellm-timeout and the cooldown guard read the real limit instead of the raw request value. A prompt-managed model also passes on only the caller's or its own timeout, so the target deployment's timeout still wins. * fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit * fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses * fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit * test(integration): cover faked, live and converted stream timeout limits per endpoint * test(router): regression tests for faked-stream timeouts	2026-10-17 22:08:02 +00:00
Mateo Wang	897dda7920	fix(router)!: stop a chat-mode Mantle deployment from rewriting the shared Responses routing row (#46263) ... * fix(router): stop a chat-mode Mantle deployment from rewriting the shared Responses routing row A deployment with an explicit custom_llm_provider and model_info.mode: chat used to overwrite the shared litellm.model_cost row, moving every other deployment and plain SDK calls onto Chat Completions until restart, because the downgrade guard read a double-prefixed key and never fired. The guard now reads the real key, and a deployment's own mode decides its routing in responses_api_bridge_check, except on chatgpt and github_copilot where Chat Completions is dead for responses-mode models. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): keep the bridge-check mode tests in the sharded bridge flag file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): let a deployment mode pin replace only the shared-row mode so the bridge capability rules still run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep Responses-only backends bridged under a deployment chat pin and satisfy the type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): give the bridge casts a reason and drop the bare noqa so the type discipline gate passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep a responses row with no supported_endpoints bridged under a deployment chat pin Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): read the downgrade guard's mode from the row registration writes A chat-mode openai/<name> deployment merged into the bare <name> row while the guard read openai/<name>, so the guard never fired and the shared responses row turned to chat. An explicit responses/ prefix now also outranks a deployment's chat pin. * fix(router): let an explicit responses/ prefix outrank a registered chat row The router now registers a chat-pinned openai/responses/<model> deployment under its own name, so the lookup found mode chat and the request went to Chat Completions as responses/<model>. * fix(router): keep the shared row's mode for every deployment pin and honor the pin on /v1/messages A deployment's mode pin now never rewrites the shared cost-map row in either direction, including the responses/-stripped twin key, so siblings and SDK calls keep routing by the registered mode. The backend key formula goes back to main's, which keeps capability fields off rows that registration never wrote at base. The /v1/messages adapter and handler read the deployment's model_info, so a chat-pinned deployment gets a plain reasoning effort on chat completions. * fix(router): keep copilot responses opt-in, make unmapped pins order-free, send Mantle auto summary as auto A responses pin on a github_copilot model or a model missing from the cost map still marks the shared row, since that is how those models opt into the Responses API. A chat pin never marks a row, so registration order no longer matters. Mantle's OpenAI path gets summary auto in place of the auto summary detailed. The Azure o3-pro and o3-deep-research rows drop Chat Completions, which Azure does not serve for them. * fix(router): skip a non-string model_info.mode when keeping the registered mode * fix(router): keep chat-pinned Mantle tool calls and summaries on the Responses API A Mantle row that also serves the Responses API now follows the GPT-5 tool and reasoning summary bridge rules, since Mantle Chat Completions rejects function tools on gpt-5.4+ unless reasoning is off and rejects the summary aliases. Mantle chat gets a dict reasoning_effort as its effort level. The opt-in auto summary asks each Responses config for its level, auto on Mantle's OpenAI path, so a client's own detailed summary follows drop_params as at base. No mode replaces a registered responses mode, and chatgpt and github_copilot rows take any pin until they are responses, which restores the Copilot chat opt-out. * fix(responses): ask the target for its auto summary level on the chat bridge and reasoning_effort * test(integration): run the Mantle region-prefixed v1 base case again This PR routes that deployment by its own chat pin, so the case passes and no longer needs the skip * fix(bedrock_mantle): keep web search and reasoning summaries on the Responses API under a chat pin A chat-pinned Mantle GPT-5 deployment now sends web_search_options and a dict reasoning_effort carrying a summary to the Responses API, which serves them. Mantle Chat Completions drops n above 1 and a reasoning summary under drop_params and answers 400 without it. The /v1/messages adapter probes the bridge with only the client's own summary, so an auto-injected one never moves the request * fix(bedrock_mantle): check n and reasoning summaries only on the Chat Completions request * fix(router): keep a non-string deployment mode off the shared routing row A deployment whose model_info.mode was an int, a list, or an object still reached the shared cost-map row, so an unpinned sibling on the same backend stopped routing to the Responses API. The shared registration now drops a mode that is not a string, and the deployment keeps its own raw value. Adds integration cells for per-deployment routing across the OpenAI and Anthropic SDKs, raw httpx, wildcard routes, malformed modes, an upstream outage, config changes under traffic, and a worker kill plus restart. * test(integration): find each routing cell's spend row by the response it logged A bridged call logs its row under the wrapped Responses id, which the caller never sees on a streamed Messages answer, so look the row up by the answer's own ids and the upstream ids for its marker. Int and list modes 500 their own model group on main too, so those cells check only the sibling, and the summary cell sends the auto level Mantle's Responses path accepts * fix(bedrock_mantle): run extra_body fields through the chat checks A chat-pinned Mantle request that nests n or reasoning_effort in extra_body now gets the same 400, drop, and effort-level handling as the top-level fields. The bridge test injects a MockTransport client instead of patching HTTPHandler.post. * test(router): cover nested reasoning_effort routing, messages thinking summaries and per-target auto summary levels * fix(router): probe a pinned deployment's own API on /health and keep a requested summary beside an extra_body effort * fix(router): keep a custom-priced deployment's mode pin off the shared backend row on each request * fix(router): run per-request custom pricing through the shared mode guard so a health probe keeps the row's mode * fix(router): drop the debug log that wrote a request's model name from keep_registered_mode * test(router): cover custom-priced chat pins, their health probes and summary aliases beside an extra_body effort * fix(proxy): pass only the mode pin to a /health probe so a failing probe never cools its deployment down * test(router): cover failing health probes leaving their deployment out of cooldown, with and without background checks * test(router): kill the worker that holds burst calls, so the worker kill cell never depends on how the kernel spread them * fix(router): route a fine-tune the same whichever deployment registers first Main's register_model now seeds an unmapped fine-tune's row from its base model's built-in row, mode included, on the first registration. The mode guard read only the live row, so a responses pin marked the shared row only when it registered before its siblings. The guard now reads the row the registration merges onto, the live row or else the row it would seed. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 21:39:14 +00:00
Mateo Wang	ef1d3959a0	fix(count_tokens)!: answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens (#47163) ... * fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens * fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests Vertex has no count-tokens route for Mistral or codestral models, so they now get the same 400 as other partner models without a count API instead of a 404 that points operators at IAM. * fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens * fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model * fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors * fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer * fix(count_tokens): pass through an OpenAI provider error that omits param or code * fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route * test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy * test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes	2026-10-17 21:15:38 +00:00
devin-ai-integration[bot]	46ae68021e	fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096) ... * fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events When the proxy renders a complete Anthropic Messages response as an SSE stream for a client that asked to stream, the message_delta event carried only stop_reason and stop_sequence. A real Anthropic stream puts safeguard_results and stop_details inside message_delta.delta, so a client such as Claude Code auto mode lost the server-side classifier verdict and a refusal lost its stop_details. Both stream builders now copy those keys from the source response through one shared helper when they are present. * fix(guardrails): keep safeguard_results and stop_details when re-emitting a rewritten Anthropic stream A post-call guardrail that rewrites an Anthropic stream (presidio, bedrock, tool permission, model armor) now carries the upstream message_delta safeguard_results and stop_details into the re-emitted stream. The router fallback message_start no longer leaks those final fields early * fix(guardrails): keep stop_details and safeguard_results in rewritten Anthropic streams without explanations A guardrail that rewrites a /v1/messages stream now carries stop_details (type and category) and the dangerous_tool_use verdicts for the tool calls it still delivers, with every explanation dropped so a masked stream never echoes the unmasked text through those fields. * fix(guardrails): copy only upstream stop_details and forward-compatible safeguard_results into rewritten Anthropic streams A rewritten stream now takes stop_details and safeguard_results from the upstream message_delta alone, so a refusal the upstream sent without stop_details is never given one by the OpenAI round trip. Unknown safeguard types, statuses, reasons and extra fields pass through, a verdict follows a tool call only when its input is unchanged, and tool_permission keeps the explanations since it rewrites no text. * fix(guardrails): apply the safeguard_results and stop_details rules to policy pipeline stream rewrites * fix(router): deliver a refused stream built from a finished agentic turn instead of falling back Websearch, MCP and advisor streams replay a complete reply whose tools already ran, so a refusal fallback would run every tool again. Also give the recovered response's message_start a null stop_details as the API sends it, drop safeguard_results a masking rewrite cannot read, and add the audit cells for synthesized streams. * test(router): annotate the finished-turn refusal stream test locals with Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 16:48:37 +00:00
devin-ai-integration[bot]	eb5295b20c	fix(anthropic): add the dangerous-tool-use beta when a /v1/messages request carries safeguards (#43955) ... * fix(anthropic): add the dangerous-tool-use beta when a /v1/messages request carries safeguards * test(azure_ai): pin the outgoing Foundry request keeps the dangerous-tool-use beta next to safeguards * test(azure_ai): inject the Foundry client and scope the beta cache reset to the test * test(azure_ai): own and close the mock transport client, drop the docstring entry * test(integration): audit the safeguards beta derivation across anthropic-compatible providers * fix(github_copilot): keep safeguards from deriving the dangerous-tool-use beta * test(integration): kill the chaos victim worker through the harness helper * fix(anthropic): derive the dangerous-tool-use beta only where the beta map supports it and honor additional_drop_params The safeguards beta is now added only for providers whose beta map column accepts it, so github_copilot, Eden AI and the OpenAI-compatible JSON providers keep their base wire. additional_drop_params now runs before the provider headers are built, so a dropped param no longer drives its derived beta. * fix(anthropic): load the beta map off the event loop and pin a dropped context_management The first /v1/messages request that needs the remote beta map now loads it in a worker thread, so a slow fetch no longer stalls every other request on the worker. An integration cell pins that a deployment dropping context_management sends neither the field nor its derived beta. * fix(anthropic): load the beta map only for safeguards requests, once per worker * fix(anthropic): gate safeguards on the bundled beta map and load the remote map on its own thread --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 16:46:25 +00:00
Mateo Wang	82ca10cc2d	feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes (#47040) ... * feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes * fix(bedrock): move Invoke body requestMetadata into the signed header and match Bedrock's ASCII whitespace rule * fix(bedrock): keep Invoke body requestMetadata in the body and answer bad metadata as AWS does Invoke requests whose body already carries requestMetadata keep the resolved pairs in the body instead of the 8,500-character signed header. Malformed caller metadata now answers 400 with the ValidationException error type the AWS SDK expects, and drop_params drops bad pairs on both the pass-through routes and /v1/chat/completions. The caller-pair dedupe is linear, and the chat validator treats \s as ASCII whitespace, as Bedrock does * fix(bedrock): match Bedrock's 400s and header limit on pass-through request metadata Invoke metadata too long for the 8,500-character header now goes in the body, Invoke requestMetadata null answers 400 like Bedrock, Converse type errors carry SerializationException, violation messages name the failing key, and the reserved user_api_key_ prefix is matched case-insensitively after trimming whitespace * fix(bedrock): trim the Invoke metadata header to Bedrock's limit and drop bad requestMetadata under drop_params without the setting * fix(bedrock): leave /bedrock pass-through bodies as sent when request metadata forwarding is off Global drop_params no longer rewrites requestMetadata on the pass-through routes unless bedrock_request_metadata_fields is set. Chat keeps dropping bad pairs under drop_params. Adds a tests/integration wire test for the four pass-through actions and removes an unused parameter from the Invoke header helper. * test(bedrock): drive the pass-through metadata wire test with boto3's bearer key env var * fix(bedrock): trim the pass-through invoke metadata header and stop Nova invoke from checking metadata it drops * test(bedrock): cover pass-through metadata header trimming, malformed caller metadata, and chat drop_params on the wire * fix(bedrock): match requestMetadata keys and values against the whole string so a trailing newline cannot pass the 256-character limit	2026-10-17 14:01:50 +00:00
yucheng-berri	9f115be88d	fix(bedrock)!: scan Claude invoke-with-response-stream passthrough output with post_call guardrails (#46524) ... * fix(bedrock): scan Claude invoke-with-response-stream passthrough frames with post_call guardrails (LIT-9618) Invoke streams of Claude models now go through the same frame-by-frame relay as converse-stream. Each chunk frame's Anthropic event is scanned by the /v1/messages stream translation, a block ends the stream with an AWS exception frame, and a held stream gets its masks written back into re-encoded chunk frames. Claude is told by its model id or, behind an application inference profile, by the anthropic_version its request body carries. Streams of other model families stay as they were * test(bedrock): cover invoke stream vendor outage, upstream exception and release on scan (LIT-9618) * fix(bedrock): hold Claude invoke streams for guardrails the relay cannot run and mark each scan as streaming (LIT-9618) A guardrail released on scan or run in parallel left a Claude invoke-with-response-stream unscanned, while converse-stream holds the whole stream and scans it once. Claude invoke streams now take the same buffered path, with the rewrites written back into the chunk frames that carried each text. Each invoke scan also sends stream true and the events relayed so far as responses, as /v1/messages does, so a vendor can tie the scans of one streamed reply together. * refactor(bedrock): mark the request dicts the buffered stream paths hand to post_call hooks (LIT-9618) * fix(bedrock): scan a Claude invoke stream the upstream breaks off as a finished stream (LIT-9618) An invoke stream that ended on an upstream exception, without a message_delta stop_reason, was scanned as still in flight, so its tool input never reached the guardrail and a held stream released it unscanned. The scan and its scan key now close such a stream the way converse-stream treats the end of its upstream. * test(bedrock): pass the converse-stream endpoint where the tool call without input test frames its stream (LIT-9618) * test(bedrock): expect the ValidationException main now returns when a guardrail blocks a Claude invoke stream before its first frame (LIT-9618) * test(bedrock): type the Claude invoke event helpers in the passthrough guardrail tests * test(bedrock): type the remaining parameters of the Claude invoke stream tests * refactor(bedrock): bind the buffered invoke rewrite in its condition --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 01:17:19 -07:00
yucheng-berri	ef1cb19351	fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering (#46307) ... * test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515) * fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515) A post_call guardrail that scans through apply_guardrail now runs on /bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the caller as the upstream sends them, every fifth frame and the end of the stream are scanned, and a block after the first frame ends the stream with an AWS exception frame. streaming_buffer_until_moderated still holds the stream and applies masks per content block. Guardrails that need the whole response (own async_post_call_success_hook, mask_response_content, native hooks, policy pipelines) keep the buffered path. * fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515) * fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515) * fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515) - The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the upstream ends with messageStop, an exception or error frame, or a dropped connection - A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no failure row - A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does - Closing the response before its body starts closes the guardrail layers and the upstream - Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never reaches frames a later layer still holds - Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser * fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515) * test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515) * fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515) Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them from the request the way chat completions streams do * fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515) * fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515) * fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515) A held post_call guardrail with streaming_buffer_release_on_scan releases each window as received once its scan passes, so the relay delivered the original text instead of the guardrail's masks. Such a stream keeps the buffered path, which applies masks per content block as on main. * fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515) The relay ran every guardrail in config order, so a run_in_parallel guardrail listed before a masking guardrail scanned the unmasked text. Main's buffered path runs parallel guardrails after the sequential ones and discards their rewrites, which the relay cannot reproduce, so these streams stay buffered * fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves - Noma v2 scans on the converse-stream relay carry stream, and the last one carries the finished Converse response with its stopReason, as chat sends them, so Noma keeps one record per reply. Other guardrails get the request unchanged. When the last sampled scan already covered the whole answer, the end-of-stream scan still runs for Noma so it gets the finished answer - A held relay stream whose client leaves before the first frame stops reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet, the budget reservation and the parallel-request slot are released - A cancelled wait for the first streamed chunk now stops the upstream read instead of leaving it running - A Noma v2 scan cancelled before Noma answers records not_run instead of success with an empty response * fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks * fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input * test(bedrock): expect the ValidationException main now returns for a held block before the first frame Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400. * fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the messageStop, skipped its write-back and released the held text without its masks. The Converse handler now reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text * fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame The relay skipped releasing the slot and budget reservation whenever a spend flush was scheduled, but a flush of a partial first frame builds no response and logs nothing, so the slot stayed held until it expired. The flush now says whether it logged the stream. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 00:53:53 -07:00
yuneng-jiang	d346d769f0	fix(caching): keep prompt cache usage on streamed cache hits (#47073) ... * fix(caching): keep prompt cache usage on streamed cache hits A stream=true request served from the response cache rebuilt its usage from the prompt, completion and total counts only, so prompt_tokens_details and the cache read and creation counts were dropped on the replayed stream. /v1/messages clients routed to Bedrock Converse saw input_tokens equal to the whole prompt and no cache tokens on a hit. Both replays now build the usage through one helper that also restores prompt_tokens_details and the integer cache read and creation counts, skipping cached values that do not validate. * fix(caching): coerce replayed token counts the way the live usage does The replay accepted only exact ints for the token counts, so an integral float or numeric string in a cached entry became 0 where the previous replay kept it. Counts now go through pydantic int coercion and fall back only when they do not validate.	2026-10-16 23:28:11 -07:00
Mateo Wang	2c5c9b8d84	fix(bedrock): use native structured output on Haiku 5.5 and Sonnet 5.5, and fix Fable 5 and Sonnet 4.6 schema 400s (#46870) ... * fix(bedrock): use native structured output on every Claude model whose Bedrock profiles accept it Bedrock accepts outputConfig structured output on every reachable profile of Claude Opus 4.7, 4.8, 5, 5.5, Sonnet 5, 5.5, Haiku 5.5, and Fable 5.1, so json_schema now goes out natively there instead of as a forced tool, which kept Bedrock from returning adaptive thinking. Fable 5 rejects outputConfig, so its keys go back to the tool path. GovCloud keys keep their flags, and the drift tests let GovCloud lag its commercial base but never lead it. * fix(bedrock): clean Claude schemas for native structured output and keep recursive schemas on the old path Bedrock native output rejects minimum/maximum, minItems, uniqueItems, oneOf, additionalProperties true, and recursive or unresolved $ref, all of which the forced tool accepted. Claude schemas on the Bedrock native paths now go through the cleaner the Anthropic native path already uses, and a schema with a recursive or unresolved $ref keeps the forced tool, inline, or drop path it had. * fix(bedrock): keep older installs on the json tool with a Bedrock-only native output key Older installs fetch the model map from main, so flipping supports_native_structured_output moved them onto native output with their old schema handling. The new supports_bedrock_native_structured_output key is ignored by older code. Schemas native output cannot hold (recursive refs, open objects such as dict fields) keep the json tool on the new key, models on the old key keep their current behavior for those shapes, and the Anthropic API, Vertex AI, and Azure keep their current behavior. * chore(cost-map): regenerate the model prices schema for the Bedrock native output key * fix(bedrock): keep native output to profiles every account accepts and route schemas Bedrock rejects to the json tool Only Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 keep the Bedrock native output key, since Opus 4.8 and Sonnet 5 answer 400 on some accounts and Opus 5 garbles escaped characters. Lookaround patterns and string formats outside Bedrock's list now keep the json tool, objects typed as a union with null are closed, and the cleaner keeps minItems of 0 or 1. * fix(bedrock): keep Opus 5.5 and Fable 5.1 on the json tool, since native output stops their adaptive thinking * fix(bedrock): keep shared $defs in native schemas and send refused schemas to the json tool on every native model Native output now keeps $defs instead of inlining each use, so a submodel reused across a schema counts once against Bedrock's optional and union limits. Schemas native output cannot hold use the json tool on Converse and Invoke chat for both native keys, and minItems is only kept when it is an integer 0 or 1 * fix(bedrock): keep the json tool unless thinking is requested and bound native output schemas The Bedrock native output key now applies only when the call asks for thinking, so calls without thinking keep the json tool as before. Schemas past Bedrock's documented union and optional counts, or past a grammar size estimate fitted to live probes, keep the json tool on both map keys. Open objects keep the json tool on the Anthropic API, Vertex AI and Azure AI, where native output returned them empty. The $ref check no longer expands $defs, so doubling definitions no longer stall the request. * fix(bedrock): resolve root-path $defs refs so nested Pydantic models keep native output on Invoke * test(bedrock): cover native structured output and json tool routing through the proxy Integration cells drive the proxy against a scripted Bedrock peer that enforces Bedrock's structured output schema rules, covering thinking with native output on Converse, Converse streaming, Invoke, /v1/responses and /v1/messages, the deployment opt-in, Fable 5, schema cleaning on every native entry, open objects, and malformed schemas. Vertex, Anthropic and Azure AI cells cover open objects on native-capable models and the cleaner's minItems and nullable object handling. The unit table now covers reasoning_effort none and disabled thinking * fix(bedrock): keep native output for numeric formats and wide old-key schemas, and keep thinking in json mode Check string formats only on nodes that can be strings, apply the 64-field cap only to the Bedrock-only key, carry thinking blocks into the json mode message, and send malformed json tool schemas as object schemas. * fix(anthropic): keep native output for dict fields unless the json tool can be forced * fix(anthropic): keep native output for dict fields when the caller sends tools * test(bedrock): close the dangerous-tool-use test schema, since Bedrock native output requires closed objects * test(anthropic): cover forced and unforced open-object routing, json mode thinking, numeric formats and wide schemas Add proxy cells for open-object schemas that keep native output when the json tool cannot be forced (a model that refuses forced tools on the Anthropic API and Vertex AI, thinking, caller tools), for the thinking block a json mode answer keeps on Vertex AI and Bedrock Invoke, and for integer and number formats and 65-field schemas on each Bedrock map key. Point the forced open-object cells at the Anthropic API and Azure AI only, since no Vertex AI model in the cost map takes both native output and a forced tool * fix(bedrock): keep the native output opt-in on the deployment that sets it, and route effort none open objects to native output * test(bedrock): cover deployment-scoped native output opt-in, register_model and effort none open objects	2026-10-16 20:57:23 -07:00
joshua-berri	6d0d0bb68b	feat: add Nace Drex decisions provider (#46523) ... * feat: add Nace Drex decisions provider * fix: expose Nace in dashboard provider metadata * ci: include Nace tests in the provider shard * fix: complete Nace provider catalog and preserve OpenRouter tests * test: mark Nace test directories as packages * fix: share Drex score parsing across decision providers --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-16 18:07:50 -07:00
Kerry Lu	e8550c1454	fix(openai): send chat completions through client.post so the SDK's per-message param transform stops holding the event loop (#46967) ... * fix(openai): send chat completions through client.post so the SDK's per-message param transform stops holding the event loop create() walks every message, content part and tool of the request through openai._utils._transform before posting, 91 ms at 300k tokens and ~300 ms at 1M tokens on the event loop, and returns the same dict for the plain dicts litellm builds. client.post is what create() calls after that walk, same request, headers, retries, raw response and stream objects, so the chat helpers now call it directly through the helper embeddings already used * fix(openai): type the chat client.post result as LegacyAPIResponse so the type gate sees .headers and .parse * fix(openai): assert the chat client.post result is a LegacyAPIResponse instead of casting it * test(openai): stub client.post with a LegacyAPIResponse spec in the empty-response and no-leak tests * test(core-utils): stub client.post with a LegacyAPIResponse spec in the agentic loop leak test * test: every client.post stub on the OpenAI chat path returns a LegacyAPIResponse-spec'd mock --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-16 18:06:01 -07:00
Mateo Wang	3ef0b35255	fix(bedrock)!: block realtime replay after an input-first start and reject malformed Nova Sonic session settings (#46936) ... * fix(bedrock): block realtime replay after input starts the session and reject non-string instructions and voice * test(bedrock): name the malformed realtime setting test for its string cases * fix(bedrock): replay the opening realtime message until Bedrock accepts the stream and refuse any fallback after commit Also rejects a malformed modalities, input_sample_rate_hertz or tools setting with invalid_type, dropped under drop_params. * fix(bedrock): replay a refused opening message on any fallback and reject OpenAI voices and non-string audio formats * fix(realtime): read the client scope only when the websocket has one and reject any-case OpenAI and empty Bedrock voices * test(bedrock): cover realtime replay, refusal and committed failure through a real proxy Fix the scripted Bedrock peer's port import and the nested tool call cell's stale expected events, so the Nova Sonic integration file collects again	2026-10-17 00:29:50 +00:00
jesus-berri	7f603da9f1	fix(openai_like)!: surface provider response headers on non-streaming chat completions (#46928) ... * fix(openai_like): surface provider response headers on non-streaming chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(openai_like): mark the header test locals Final * fix(proxy): leave out response header values HTTP cannot carry Non-streaming OpenAI-like chat now forwards every provider header, so on the httpx transport a provider header value outside latin-1 made the proxy answer 500 after the provider had served the call. get_custom_headers now leaves such a value out and keeps the other headers, which also ends the same 500 on streaming Adds integration cells for the provider request id on non-streaming chat and the Responses bridge, LiteLLM headers an upstream cannot overwrite, spend parity under an upstream cost header, the latin-1 case, and the logging callback payload * fix(cost): ignore an upstream response cost header that is negative or not a number * test(integration): cover invalid upstream cost headers and non-latin-1 headers on every endpoint * fix(prometheus): keep deployment metrics when a provider rate-limit header is not a number * test(prometheus): type the rate-limit header test and drop its clock reads * test(integration): cover deployment metrics under an unparseable provider rate-limit header --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 17:25:50 -07:00
ryan-crabbe-berri	9aab6ec47b	fix(azure_ai): count Grok reasoning tokens in completion_tokens (#46969) ... * fix(azure_ai): count Grok reasoning tokens in completion_tokens Grok on Azure AI Foundry reports reasoning_tokens outside completion_tokens, like xAI does, so azure_ai/grok-* calls billed only max(visible, reasoning) output tokens. Fold them with XAIChatConfig.fold_reasoning_tokens_into_completion in transform_response and in a new streaming iterator, matching the xai provider. Split out of #46377. * fix(azure_ai): fold reasoning tokens for every deployment name, streaming or not Non-streaming checked the deployment name for grok while streaming checked the chunk model field, so a Grok deployment named without grok was only folded when streaming. The fold already only fires when total_tokens equals prompt plus completion plus reasoning, so it now runs on every azure_ai response.	2026-10-16 16:54:38 -07:00
Mateo Wang	b39eadcb82	feat(openai)!: honor per-deployment workload identity on non-chat routes (#39966) ... * feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * feat(openai): honor per-deployment workload identity on non-chat routes Image generation, audio speech and transcription, moderation, text completions, files, batches, and fine-tuning built their OpenAI client from the env vars alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file failed with the api_key error on every route other than chat, Responses, and embeddings. Those handlers now build their client through the shared workload identity resolver, and the managed batch file-access credential extraction keeps the trio * fix(openai): cache workload identity clients and let deployment identity outrank process-wide keys Non-chat handlers built a fresh SDK client per call, so files, batches, fine-tuning, text completion and sync moderation exchanged the OIDC token on every request. The builders now reuse the cached workload identity client per identity, base, timeout, retries and organization; static-key clients are still built fresh. A process-wide OPENAI_API_KEY (or litellm.api_key / litellm.openai_key) no longer silently overrides a deployment's openai_identity_* params. An explicit distinct api_key on the deployment or call still wins. * fix(openai): keep shared workload identity clients on the process http session The workload identity client is cached and shared across requests, so it must never borrow the transport a single caller passed in. Static-key clients keep using the caller's http client; federated clients always ride the process-wide session. Also makes the http client helpers on BaseOpenAILLM public since the builder now needs them from outside the class. * test(openai): pin partial deployment identity beating the process key * test(openai): type the managed-object invoke callback * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * feat(openai): honor deployment workload identity on image edits, vector stores, containers, videos, and evals The httpx-based OpenAI handlers built their Authorization header from the deployment api_key or OPENAI_API_KEY alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file sent "Bearer None". They now resolve the bearer through resolve_openai_bearer_token, which exchanges the deployment's identity token the same way the SDK-based routes do. A deployment that carries only part of the identity trio now raises a 500 naming the missing field instead of silently falling back to the process key, and the SDK client builders take a factory for the static-key http client so the federated branch never touches litellm.client_session. * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * feat(openai): resolve per-deployment workload identity for the assistants routes OpenAIAssistantsAPI built its OpenAI and AsyncOpenAI clients from a locals() scrape, so a deployment carrying openai_identity_provider_id, openai_service_account_id and openai_identity_token_file never exchanged a token. Thread litellm_params through the class and build both clients with the shared workload identity builders. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * feat(openai): resolve per-deployment workload identity for image variations Build the image variations client through the shared workload identity builders instead of scraping locals(), and give the async path the async http session it was missing. Let a federation trio serve the model-less batch fallback, and hold the standard LRU size of exchanged tokens. * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * style(tests): keep two blank lines before the batch_cost_is_final block * test(batches): type the workload identity fallback test fixtures * fix(moderation): stop validating unrelated kwargs on the sync workload identity path * fix(auth): refuse server-owned workload identity keys in array form and inside a client router config * fix(openai): keep a deployment's own key ahead of workload identity on every route Thread the caller's own api_key into the identity resolver for embedding, transcription, moderation, image generation, and assistants so a deployment key equal to the process key still wins. A partial identity trio now falls back to a configured static key, unset max_retries keeps the SDK default, and /v1/decisions plus OpenAI count_tokens resolve the exchanged bearer for identity-only deployments * fix(containers): pass deployment params to container file auth headers Container file routes built their auth headers without the deployment's litellm_params, so a keyless deployment with an OpenAI workload identity sent Bearer None on upload, list, retrieve, delete and content calls * fix(openai): keep process-wide key precedence and move token exchange off the event loop Restore the merge base order where an explicit or process-wide OpenAI key wins over federation, so exporting OPENAI_API_KEY keeps working the way it did before. Run the header-building validate_environment calls (image edit, videos, containers, vector stores, vector store files) and the decisions and System One request preparation in a worker thread, so a slow token exchange no longer stalls the loop. Federate realtime websocket sessions, live sessions, realtime client secrets and transcription sessions, and the OpenAI moderation guardrail when no key is set. * fix(openai): answer 401 for a partial identity trio and federate realtime health checks A deployment with part of the workload identity trio and no static key now fails as an auth error naming the missing params, so the router does not retry it as a server error. The realtime health check sends the exchanged bearer for a keyless federated OpenAI deployment, and API token counting reads the same static keys as every other route before minting. * fix(openai): keep local token counting without an identity trio and federate the moderation guardrail from its own config * fix(openai): skip a partial-trio moderation guardrail and let vault keys win in token counting * test(integration): refuse caller-chosen identity params in user_config and bracketed form keys * fix(openai): send the honored secret manager key from SDK client builders A deployment whose OPENAI_API_KEY lives only in the secret manager now sends that key on image, assistants, files, batches, and other SDK-built routes instead of failing with no api_key, matching what the workload identity resolver already treats as a static key * test(openai): check the token exchange runs off the event loop instead of timing the loop --------- Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>	2026-10-16 22:25:15 +00:00
ryan-crabbe-berri	9cf09c18fe	fix(providers): route Azure Responses past dated api versions, count Grok reasoning tokens on azure_ai, refresh provider sets after register_model, fix gpt-6.1-sol Azure rows (#46377) ... * fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version * fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider * fix(utils): refresh provider model sets after register_model * fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row * test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version * fix(utils): keep router deployment ids out of provider model sets register_model now refreshes the provider sets, so the per-request and router registrations keyed by a deployment id put those opaque ids into wildcard model listings. Those callers now pass add_to_provider_model_sets=False. * fix(azure): carry responses_api_version through the chat to Responses bridge completion() dropped responses_api_version from the litellm_params it hands the Responses bridge, so azure/responses/<model> on a dated chat api_version still built the 404ing /openai/responses URL. Also cover the auto-router marker never forwarding the new param. * fix(azure): stop listing a wrong set of reasoning efforts when none is rejected The message said only low, medium and high are supported, which is wrong for models that also take xhigh such as gpt-6.1-sol. * test(azure): use the latest grok and gpt-5.4 rows in the new azure tests * fix(proxy): list each wildcard-expanded model once A provider set can hold a model both bare and provider-prefixed, which now happens after any priced request registers its openai/<model> key, so the openai/* listing showed that model twice. * fix(azure): route Responses past dated GA api_versions by default A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before * chore(azure_ai): mark the grok chunk_parser dict as an override contract * fix(azure): rewrite only the route's /openai segment when moving a call to v1 The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses. Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too, so the rewrite now only touches the last /openai path segment. The Responses version resolver no longer duplicates the builder's default for an unset api_version. Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite now has one test for the dated GA default and one where only responses_api_version can make the call work * test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params * fix(utils): list ai21 chat and vertex code models under their provider wildcard register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too. * fix(main): keep caller-chosen model names out of provider wildcard listings --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai>	2026-10-16 15:23:46 -07:00
yuneng-jiang	0403f8953e	fix(cost): bill bridged /v1/messages Responses replies like native /v1/responses for reasoning tokens and service tier (#46975) ... * fix(cost): price reasoning tokens of /v1/messages bridged onto the Responses API in the cost header and breakdown * fix(cost): record the bridged Responses API result only after the served body is built * fix(cost): keep the served service tier when a bridged Responses reply has no message item	2026-10-16 15:23:11 -07:00
AxelRay	388251a5b8	fix(bedrock): drop dangerous-tool-use beta without safeguards (#44471) ... * fix(bedrock): drop dangerous beta without safeguards * fix(bedrock): drop the dangerous-tool-use beta only next to structured outputs, keep it on Mantle Bedrock rejects the pair with 'invalid beta flag' on Claude models before Opus 4.7, while requests without structured outputs and Bedrock Mantle accept the beta, so the drop now only applies to Invoke requests with output_config.format or a strict tool and no safeguards. * fix(anthropic): keep the AnthropicThinkingParam import that main now uses * fix(bedrock): drop the client's dangerous-tool-use beta whenever safeguards is absent Bedrock rejects the beta with 'invalid beta flag' on Sonnet 4.5, Opus 4.5, and Opus 4.6 even without structured outputs, so dropping it only next to structured outputs left those beta-only requests failing. The beta does nothing without safeguards, so Invoke now drops it in every such request. Mantle keeps it. The thinking display helper goes back to main's version. * fix(bedrock): drop the client's dangerous-tool-use beta on Invoke chat without safeguards * test(bedrock): wire cells for the dangerous-tool-use beta on Invoke messages, chat and responses * test(bedrock): type the new dangerous-tool-use beta tests * test(bedrock_mantle): type the dangerous-tool-use beta Mantle test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 21:00:37 +00:00
Mateo Wang	f8217d2a03	fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count (#46843) ... * fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count /v1/messages runs clear_tool_uses context edits before it sends an Anthropic body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted the body unedited, so a request with older tool results cleared counted about 2.8x what inference bills. The count now runs the same history and context-edit step inference runs, honors additional_drop_params on the deployment, and skips compact edits so a count never makes a paid summary call. * fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs Run compact edits in the count without the summary call so trigger checks and history slicing match /v1/messages, honor a request-level additional_drop_params, keep the full body when any deployment in the model group drops context_management, and answer 400 naming a block Gemini cannot count when an edit trigger fails on it * fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check * fix(count_tokens): log a failed edit in the count at debug, since the send already logs it * fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does * fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits. * fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name * fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to * fix(messages): drop an invalid context_management spec under drop_params so the send matches the count * fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes. * test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code	2026-10-16 17:53:06 +00:00
Mateo Wang	7ddc9f1c96	fix(bedrock)!: keep reasoning, thinking blocks and the json_mode finish_reason on fake streams (#46863) ... * fix(bedrock): keep reasoning, thinking blocks and the json_mode finish_reason on fake streams A Bedrock Converse model without native structured output fakes the stream when response_format carries a json_schema. The fake stream transformed the response without json_mode and rebuilt a chunk from text, tool_use and token counts only, so it dropped reasoning_content, the signed thinking_blocks and the reasoning token count, and ended on finish_reason tool_calls. The three fake-stream sites now pass json_mode to _transform_response and stream the resulting ModelResponse through the shared convert_model_response_to_streaming converter, which now numbers each streamed thinking block with its index the way real streams do * fix(bedrock): parse text tool calls on Converse fake streams like the non-stream answer * fix(responses): stream the answer text when one chat chunk carries reasoning and the answer * fix(streaming): keep every field on the OpenAI and openai_like fake streams and split multi-block thinking Azure o-series and openai_like fake streams now use the shared MockResponseIterator, so extra tool calls, reasoning and cached and reasoning tokens survive; the Bedrock-local iterator is deleted. The /v1/responses bridge also splits a chunk whose thinking block is signed or redacted with no reasoning text, and the sync iterator closes the reasoning item before the answer opens, like the async one. The /v1/messages bridge gives each thinking block in a chunk its own content block, starts a new block when the thinking block index changes, and streams redacted thinking as redacted_thinking. Snowflake replays drop the stream index from thinking blocks. * fix(anthropic): stream each thinking block's text once on /v1/messages An unsigned thinking chunk now opens its content block with empty text, since its thinking_delta carries the text. A multi-block chunk keeps each piece's thinking block, so an unsigned block followed by a signed one keeps its own text and index. * docs(anthropic): describe which thinking block start carries its text * fix(streaming): close merged reasoning before the answer when one chunk carries both * test(streaming): cover fake-stream reasoning across the proxy bridges and type the Snowflake replay test	2026-10-16 17:27:57 +00:00
devin-ai-integration[bot]	859529ccbc	fix(realtime)!: keep turn_detection.type when a transcript guardrail disables auto-response (#44939) ... * fix(realtime): keep turn_detection.type when a transcript guardrail disables auto-response * test(realtime): type the GA session.update helper and mark its locals Final * test(realtime): expect typed turn_detection on the voice guardrail session updates * test(realtime): accept either gate path for the client's first voice session.update * fix(gemini): drop VAD tuning when the guardrail turns auto-response off * fix(gemini): check create_response without a redundant isinstance * fix(vertex_ai): keep the client's VAD tuning on Vertex when auto-response is off * fix(realtime): keep the transcript-guardrail gate typeless on xAI so it does not auto-answer * fix(realtime): replace any client turn_detection with the untyped gate on xAI xAI answers on its own under a typed server_vad even with create_response off, so a client's typed turn_detection must not reach xAI while the transcript guardrail gates the turn. * fix(realtime): keep a client's early turn_detection and hide the guardrail cancel's error A session.update sent before session.created reached OpenAI ahead of the proxy's own gate, so the gate replaced the client's turn_detection. The proxy now re-sends the client's gated turn_detection after the gate when the two differ. The guardrail's response.cancel now carries a litellm_guardrail_ event_id, and the error OpenAI returns for it (response_cancel_not_active when nothing is answering) no longer reaches the client. * fix(realtime): send the guardrail gate before a client's early session.update and let Muse and Chirp fall back to server_vad under drop_params * fix(realtime): hide only the cancel errors that echo the guardrail's own response.cancel ids Track the event ids of the guardrail cancels this session sent and drop a backend error only when its error.event_id is one of them, so a client's own event id with the same prefix still gets its error. Cover the scripted response.cancel reply in the integration upstream, update the stale list-mode test to the typed gate, and mark the xAI cancel error cell as an expected failure, since xAI replies with its own event id. * test(integration support): deliver the disconnect a real server sends after the client goes away The scripted upstream now keeps reading frames after a reply hits a closed socket, so the unit test's fake ASGI receive has to end with the websocket.disconnect message a real server delivers. * fix(realtime): delete the blocked transcript item before the guardrail's refusal turn OpenAI GA answered the blocked question in some refusal turns because the blocked user item stayed in the conversation. The raw OpenAI-protocol path now sends conversation.item.delete for it right after the cancel. Provider config backends have no such event, so that path no longer passes an item id. Also bound the guardrail integration file by the shared graceful-stop timeout and make the xAI cancel-error xfail require an AssertionError. * fix(realtime): keep the blocked item on xAI, which cannot delete an audio input item * test(realtime): assert the xAI voice block sends no conversation.item.delete upstream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 15:24:40 +00:00
Rick	f81e657903	feat(volcengine): add Doubao-Seedance video generation via the OpenAI /v1/videos surface (#46156) ... * feat(volcengine): add Doubao-Seedance video generation via the OpenAI /v1/videos surface New VolcEngineVideoConfig maps litellm's OpenAI Videos API onto Volcengine Ark's contents/generations task API: create hits /api/v3/contents/generations/tasks and returns a cgt-... task id, status and content poll /api/v3/contents/generations/tasks/{id}, list maps the same collection (limit -> page_size, Ark's filter.* through extra_query), delete maps DELETE on the task URL, and edit/extension reuse create with a reference_video content item since Ark models both as ordinary generation tasks. Ark's own edit and extension require ratio adaptive (edit also duration -1), so those are pinned unless the caller overrides them. Remix has no Ark equivalent and raises. The body is flat apart from a typed content array: the prompt is a text item and every image/video/audio input is a url item carrying a role, which is the only way to express reference-to-video, edit and extension jobs, so a caller-supplied content array wins outright. input_reference becomes a first_frame item, base64 encoded into a data URI for file inputs. OpenAI params translate per Ark's contract: seconds becomes duration and size yields both a ratio and a named resolution tier, with the ratio snapped onto the set Ark accepts (854x480 reduces to 427:240, which Ark rejects) and the tier taken from the shortest side. Emitting only the ratio leaves the request on the model default and bills at that default's rate. Billing reads the delivered duration and resolution rather than the requested ones, because duration -1 and ratio adaptive both let Ark deliver something else. Ark bills per output token (tokens = width * height * 24 * seconds / 1024), so the cost map carries per-second rates by resolution tier, derived from the console's CNY per million token list rates (2.5 70, 2.0 46, 2.0-fast 37, 2.0-mini 23 without a video input) at the repo's 7.0 CNY/USD convention. Requests that do carry a video input bill at the lower rate instead, which litellm's per-second tiers cannot express, so those are overestimated. usage.total_tokens rides along for invoice reconciliation. Ark echoes the delivered geometry only as a resolution tier plus a ratio, and the 2.0 family's rendered frame does not follow the tier name: its 16:9 "480p" canvas is 864x496, not the 854x480 the tier implies, and its other ratios differ from the tier's nominal shortest side too. A size reconstructed from the tier alone therefore reports pixels the model never produces and scales the 480p rate against the wrong pixel count, so both the reported size and the 2.0 480p rates come from the measured frame (a per-family table, sourced from the Seedance 2.0 Mini model card and re-verified against a live gateway). The create response reports the requested duration and resolution, since only the create, edit and remix calls are costed and Ark's create response carries no usage block, so without it every video logs zero spend. A caller that leaves seconds off is not free either: Ark runs its own default length, which is 5s for the 2.0 family, so that is what the create response bills, while the 2.5 family's default is -1 (Ark picks) and stays unbilled until the status response. The create response also echoes the requested seconds and the frame the job will be rendered at. Editing and extending need the source video as a URL: Ark's task query is a GET while litellm's edit prefetch hook only POSTs, so a task id cannot be resolved into one. The model comes from an explicit extra_body.model, else the encoded video id, else the model the router already resolved, with litellm's provider prefix dropped. The credential's api_base is the Anthropic-compat surface (/api/compatible), so the base is cut at the first /api/ marker rather than by a /api/v3 suffix before the video paths append their own version. The cost-map schema test's exception list gains the 2.0 4K tier, whose $1.28/second rate exceeds its $1 sanity bound, alongside the existing runwayml/seedance2 entry. * fix(volcengine): answer 4xx when a Seedance video is downloaded before it finishes Downloading a task that is still queued or running raised a bare ValueError, which the video content handler synthesizes into a 500. The proxy then retried the same poll twice and reported the caller's own early download as a gateway fault, and the router went looking for a model group fallback for a job that was simply unfinished The download path now raises VolcEngineVideoError carrying the status the OpenAI videos API documents for content that is not ready (404), 400 for a task the provider reported as failed, 404 for an expired task record, and 502 for a succeeded task with no artifact. None of them are 5xx, so the router stops retrying them Regression tests drive avideo_content and assert the caller sees NotFoundError or BadRequestError instead of an internal server error * fix(volcengine): clear the type-discipline and override findings on the Seedance video path The module annotated four mutable collections and carried five `# mutable-ok` markers on lines that suppressed nothing, so the type-discipline gate counted LIT001 x4 and LIT013 x5 against the PR. The content-item and task-dump helpers hand back read-only views of what they build, so they now say Mapping. VideoObject.error is declared as a dict by the type it feeds, so that one marker moves onto the annotation it actually suppresses transform_video_status_retrieve_response also dropped the client keyword the base class declares, which basedpyright reported as an incompatible override * fix(volcengine): bill every Seedance video at create and keep its polls on the deployment that made it Ark bills a task when it finishes while litellm bills it at create, so a request whose cost is not known then is rejected: Ark-picked length (duration -1), frames, a video input, and with it edit and extension. An omitted length is sent as 5s, since the 2.5 family otherwise lets Ark pick it and logged $0. Video ids now carry the requested and the served model group, read from litellm_metadata, so polls and downloads go back to the deployment that created the task. The routing and input_reference helpers move out of the DashScope config into a shared module both providers use. A download before the render finishes answers 400, unreadable or unmapped task bodies answer 502, malformed seconds, size, parameters and Ark fields raise unless drop_params is set, an unpriced resolution tier is refused, and the rendered frame table moves into the cost map. Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(volcengine): reject a text item in content so the guarded prompt is the only text sent Input guardrails scan data["prompt"], so a caller could pass a harmless prompt for scanning and a different, unscanned text item in content, which this branch kept and the guarded prompt was dropped. The prompt now always leads the content array and a second text item answers 400. Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(volcengine): route video to VOLCENGINE_API_BASE when the deployment sets no api_base * fix(volcengine): let a model with no tier prices keep its resolution instead of rejecting every tier * fix(volcengine): answer 400 for malformed or unknown create fields, send a 720p default tier, and stop the list cursor * fix(volcengine): map size to the closest tier by pixel count, reject draft, remix and list cursors, map user to safety_identifier, read ARK_API_BASE * fix(volcengine): drop unsupported create values under drop_params, take OpenAI's object input_reference, default to 4 seconds, price endpoint ids by base_model * fix(videos): unwrap nested file tuples without recursion * fix(volcengine): price Seedance tiers at Ark's own per-resolution token rates Ark prices 1080p and 4k per token above and below the 480p/720p rate, so the single 46 CNY per million tokens the rows were derived from billed 2.0 1080p and 2.5 1080p about 10% low and 2.0 4k 77% high. Ark's per-resolution table and its own 16:9 examples agree: 1080p 2.48 CNY/s, 4k 5.05 CNY/s for 2.0 and 3.74 CNY/s for 2.5 1080p, which is 51, 26 and 77 CNY per million tokens. Rewrites the tier test to check each rate against the rendered pixels at that tier's own price instead of assuming every tier scales from the base rate, and drops the 4k cost exception that only existed for the old rate. Co-Authored-By: Bytechoreographer <noreply@bytechoreographer.com> Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com> * fix(volcengine): unpack Ark parameters sent through extra_body * fix(volcengine): ignore the extra_body keys litellm maps itself The shared request builder merges the caller's raw extra_body back over the mapped params, so a key this adapter maps itself comes back a second time as a field of its own, and the unknown field check turned the supported request into a 400. 2aefc18b0a covered parameters; size, seconds and user still failed that way. All three are already mapped from the top-level copy the same builder writes, so the leftover is ignored rather than rejected. characters and image stay rejected, since Ark has no equivalent and accepting them would hide a parameter that never took effect. Co-Authored-By: Bytechoreographer <noreply@bytechoreographer.com> Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com> * fix(volcengine): reject litellm request fields inside parameters and accept litellm's own controls A seconds, size, user, prompt or nested parameters inside an Ark parameters object, and an extra_body copy of prompt, model or extra_headers, answer 400 again unless drop_params is on, instead of being ignored. A user that is not a string answers 400 the same way. drop_params, additional_drop_params and a forwarded organization are no longer rejected as unknown Ark fields, and additional_drop_params removes the fields it names. * fix(volcengine): apply additional_drop_params before mapping request fields A drop list naming user, seconds, size, parameters or an unsupported field such as image had no effect, because the fields were already mapped to Ark names before the list was applied. The list now removes them first, so the request sends what it would send if those fields were never set --------- Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: zijie.zhou <zijie.zhou@momenta.ai> Co-authored-by: Bytechoreographer <noreply@bytechoreographer.com>	2026-10-16 13:52:15 +00:00
Ayush Shekhar	7930c0e434	feat(router): mid-stream fallback continuation for chat completions (#41127) ... * feat(router): continue chat streams on fallback after a mid-stream break A chat-completions stream that breaks after content has been delivered was re-raised and the fallback deployment never ran (#40404), while the Responses-API path already continues via a prefilled assistant turn. Add the same for chat completions behind `enable_mid_stream_fallback_continuation` (opt-in, async-only), so a post-content break re-enters the fallback chain with the partial text as an assistant prefill instead of surfacing the error. Continuation is only attempted when it is safe: the stream emitted plain assistant text (no tool/function calls, thinking blocks, reasoning items, audio or images), the request is not constrained output (response_format or a forced tool_choice) and not merge-reasoning mode, and the fallback target's model supports assistant prefill. The target check runs as a deployment pre-call filter, so a chain with no prefill-capable deployment empties and the original error is surfaced rather than a duplicated or rejected request being sent. Plain reasoning_content is treated as out-of-band and does not block a continuation, matching the Responses-API path; Anthropic thinking is excluded because its signed thinking blocks cannot ride a text-only prefill. CustomStreamWrapper now latches whether a disqualifying delta was streamed and carries it on MidStreamFallbackError so the router can decide without re-scanning chunks. Default behavior is unchanged. * fix(router): harden mid-stream continuation eligibility and prefill folding Address review of the mid-stream continuation: - keep response_format={"type": "text"} eligible; only json_object / json_schema and other structured formats decline, since "text" is the unconstrained default - fold a new partial into an existing trailing assistant prefill instead of appending a second one, so a nested mid-stream break stays a single prefill turn even on providers that do not merge consecutive assistant messages - split the deployment prefill-capability lookup into two readable steps - cover the merge_reasoning decline branch, the text response_format pass, and the prefill-folding path with tests * fix(router): address review of mid-stream continuation - register the continuation deployment filter as a process-global singleton instead of tracking it per router, so discarding one router no longer removes the type-deduplicated filter that other live routers still depend on - disqualify reasoning_content: it reaches the caller as visible reasoning that a text-only prefill cannot carry, so a reasoning-capable fallback would re-derive it and produce an incoherent response - move MID_STREAM_CONTINUATION_KWARG to constants and lazy-import the filter class in the router, breaking the module-level import cycle CodeQL flagged - trim the added docstrings to the repository comment policy * fix(router): keep mid-stream continuation within the type-discipline budget The LIT* type-discipline gate counts mutable-collection use the ruff pass does not see. Annotate the reads with read-only Sequence/Mapping views, compare response_format against a frozen MappingProxyType sentinel, and mark the genuinely mutable litellm message and deployment-list boundaries with mutable-ok reasons, so LIT001 and LIT002 stay under their limits. * fix(router): clear basedpyright operator error and the import-cycle alert - narrow the accumulated stream delta content to str before appending, so the shared helper no longer trips reportOperatorIssue (str += object) - keep the continuation marker in the pre-call-check module and lazy-import it at the two router call sites instead of routing it through constants, so the router has no module-level dependency that closes an import cycle * fix(router): make the mid-stream continuation marker unforgeable The deployment filter is process-global and read the continuation marker as a truthy value from request_kwargs. Since the proxy can forward arbitrary request-body fields into the router, a client could send `"_mid_stream_continuation": true` on a normal request to filter a mixed model group down to prefill-capable deployments and steer their prompt onto that provider. Set a private sentinel object internally and check it by type instead of truthiness: a JSON request body cannot construct one, so a forged flag is ignored. Adds a regression test that forged client values (true, "true", 1, a dict) leave the deployment list untouched. * feat(router): honor per-deployment model_info for continuation prefill capability The deployment filter only read supports_assistant_prefill from the cost map, so a model configured with `model_info: {supports_assistant_prefill: true}` on the deployment (the usual way to declare a capability for a model that is not in the cost map, or to override it) was wrongly skipped. Check the deployment's model_info first (True or False both authoritative), then fall back to the cost map. Adds tests for the override in both directions. * fix(router): bound mid-stream continuation to the caller's output allowance A continuation handed the fallback the full max_tokens / max_completion_tokens even though part of the answer had already streamed, so the total could exceed the caller's limit and repeated interruptions could re-grant the allowance on every hop, amplifying spend. Reduce each configured ceiling by the tokens already emitted before re-entry, and decline the continuation (surface the original error) when the allowance is exhausted. The reductions compound across hops because the trimmed ceiling is what the next hop sees. Also add the missing explanatory comment on the model_info fall-through except. * refactor(router): extract mid-stream decline to stay within the complexity budget The output-ceiling branch pushed stream_with_fallbacks past the C901 limit. Both decline paths (ineligible after content, and exhausted allowance) re-raised the original provider error the same way, so pull that into _raise_original_mid_stream_error. It removes the duplicated branching, drops the function back under the complexity ceiling, and reads clearer. * fix(cost-map): mark Claude Sonnet 4.6 as not supporting assistant prefill The Anthropic API and Bedrock both reject an assistant-last message on claude-sonnet-4-6 with "This model does not support assistant message prefill", so the continuation filter must not route a mid-stream continuation to it. Flip the flag on every claude-sonnet-4-6 entry and on databricks' opus-4-6 entry, whose anthropic sibling already says false. The prefill check tests now register synthetic cost-map entries instead of pinning real models, which broke when claude-3-opus left the map. * refactor(router): type the streaming fallback kwargs and drop narration comments Removes an unused mutable-ok marker (LIT013) and the explanatory comments Greptile flagged, and types initial_kwargs on both streaming fallback iterators as dict[str, Any] like their Responses siblings so the new continuation helpers take a typed mapping and the basedpyright budget stays at or below main. * fix(router): run post_call_rules over the joined text of a continued stream Each provider stream wrapper validates only the fragment it produced, so a rule that trips on the primary's prefix plus the continuation's text never fired. The router now feeds litellm.post_call_rules the emitted text plus every continuation chunk before yielding it. Also drops a narrating test-helper docstring. * refactor(router): read a deployment's declared prefill support without an empty except * test(router): cover _raise_original_mid_stream_error directly for the router coverage gate * fix(router): fold the continuation prefill into a trailing assistant turn A request that already ends in an assistant message took the partial output as a second assistant turn, which prefill providers reject. The partial now extends that turn in place, with prefix set, so the continuation request never ends in two assistant messages. * fix(router): keep content blocks on a trailing assistant turn when extending it with the partial output A trailing assistant message whose content is a list of blocks was stringified before the partial output was appended, so the fallback received the Python repr of the blocks as its prefill. The partial now lands as one more text block, and a content shape that cannot be extended surfaces the original stream error instead of a garbled continuation. * fix(router): continue the trailing text block in place when extending a prefill with content blocks Anthropic strips trailing whitespace per content block, so appending the partial output as a new text block after an assistant turn that ends in "Sure, " would send the prefill as "Sure,here". The continuation now extends the last text block itself and only opens a new text block when the trailing block is not text (an image, say), so the prefill reads exactly as the text the caller already received * test(router): type the prefill check's deployment helper * fix(router): keep a continued stream on the primary's identity and widen what can continue Continuation chunks now keep the primary stream's id and created time and drop the fallback's second role delta. Requests with n > 1 no longer continue. Prefill support reads model_info.base_model and litellm_params.base_model before the deployment model. enable_mid_stream_fallback_continuation is runtime-updatable through update_settings and registers the prefill filter. The bridge rebuild of MidStreamFallbackError keeps the inner stream's disqualifier. A fallback that repeats the prefill is not doubled in the continuation input. * fix(router): decline mid-stream continuation after a content filter, merged reasoning, or a plain trailing assistant turn Also count the partial stream's completion tokens in the continued stream's usage, accept string values for enable_mid_stream_fallback_continuation, and list the new router helpers in the router coverage check * fix(router): drop re-emitted whitespace at the continuation join and sum both legs' usage consistently * fix(router): keep the full prefill so a nested break keeps the separator * fix(router): bill both calls of a continued stream and fix Bedrock Converse, Databricks, and Vertex DeepSeek prefill routes * test(router): integration cells for mid-stream fallback continuation Covers the continued stream the caller gets, the prefill turn and reduced max_tokens the fallback receives, the prefill-capability filter, the runtime toggle, repeated requests through the response cache, and the spend row of a continued stream * fix(router): log a continued stream under the caller's id, price each cut on its own prompt, and keep prefill flags per deployment * fix(router): price and log each cut leg before its continuation reuses the logging object Price the cut stream's partial and let its failure log finish (bounded) before the fallback overwrites a shared proxy logging object, and skip continuation when an output guardrail is registered or a request fallback carries its own messages. * test(router): integration cells for the output-guardrail and request-fallback declines and the cut call's failure log Cover a registered post_call guardrail and a request fallback carrying its own messages keeping the original error, an input-only guardrail leaving continuation on, and the cut call's failure event naming its own deployment and price. * fix(router): skip mid-stream continuation when a request fallback overrides request params A request-body fallback entry that sets anything besides model (stream, messages, max_tokens) changes the request the prefill continuation relies on, so a stream: false entry ended the answer with no error. * fix(router): skip mid-stream continuation for a fallback group target that overrides request params The override check read only top-level fallback entries, so a target inside a group's list, such as {"primary": [{"model": "backup", "stream": false}]} in the request, the config, or a key's router settings, still ended the answer with no error. * fix(router): check only the request's own fallback chain for request param overrides Scanning every group's chain turned continuation off for all requests when one unrelated group's target overrode a param. Resolve the chain the router will use for this request, the same way the fallback path does. * fix(router): keep the original error when request fallbacks are malformed The request-param override check resolved a standard-shape chain outside the router's own error handling, so an empty entry or a non-list chain raised StopIteration or TypeError and replaced the original mid-stream error. Skip empty entries and treat a non-list chain as no chain. * fix(router): read deployment-configured constraints, decline deployment-id fallbacks without prefill, strip only the last Bedrock text block * fix(router): read a deployment's tool_choice ahead of router defaults in the continuation check, and restore the renamed prefill test names * fix(router): filter continuation deployments on every routing strategy, read extra_body and fallback deployments' own params The deployment filter now also runs on the sync selection path that usage-based-routing v1 and custom strategies use, one predicate reads n, response_format, tool_choice and merge_reasoning_content_in_choices with extra_body laid over them, fallback deployments whose own config constrains the output are skipped, and continuation declines when the cut deployment's id is not in the router (per-request credential deployments) * fix(router): decline a cut mid-echo, reduce extra_body output limits, and check post_call_rules once on nested continuations --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 13:51:32 +00:00
Marcus	642a2ad934	fix(prompt-templates): treat initial_prompt_value and final_prompt_value as optional (#39762) ... * feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(model_hub): surface model_info.description in model group info and Model Hub UI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): aggregate model group description without in-place mutation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prompt-templates): treat initial_prompt_value and final_prompt_value as optional completion() only stores a custom prompt template key when its value is truthy, and custom_prompt() already defaults both prompt values to "". Seven call sites read them back with direct indexing, so a template that sets only roles raises KeyError instead of rendering, on ollama, petals, vllm, predibase, codestral, anthropic text completion, and response_schema_prompt. Five other providers already read defensively, so the same config works on sagemaker and 500s on ollama. Read both keys with .get(key, ""), matching those five providers and the defaults custom_prompt() already declares. Templates that set all three keys render exactly as before. * test(prompt-templates): drop the explanatory docstrings from the optional-keys regression test * fix(health): skip background health check DB writes when the latest-row read fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prompt-templates): read every custom prompt template field through one typed helper Templates that omit roles, initial_prompt_value, or final_prompt_value now render with empty defaults across every provider that reads them, instead of failing with KeyError or TypeError. A field with the wrong type fails with a validation error naming it. * fix(prompt-templates): keep templates that rendered on main working and fail bad fields with a 400 Only the roles a request uses are checked, null fields mean unset, and extra role keys are ignored, so every template main rendered renders the same. A wrong-type field now raises a BadRequestError naming it. initial_prompt_value is declared litellm-owned so it stays out of provider request bodies, like final_prompt_value. The template tests move into the mapped factory test file, and the predibase test asserts the real rendered prompt * fix(prompt-templates): keep bad-template failover and render non-mapping role entries like main A wrong-type template field now raises a 500 InternalServerError naming the field, so the router still retries and cools down the bad deployment. A roles value or role entry that is not a mapping renders unwrapped as it did on main, and role keys are only checked for roles a request uses. * fix(prompt-templates): warn when a custom prompt template roles value is not a mapping * fix(prompt-templates): keep deployments in rotation on a bad template and reject a template with nothing to render * test(integration): audit custom prompt templates across endpoints, providers, and chaos --------- Co-authored-by: milan <milan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 10:30:59 +00:00
Aryan Pardeshi	a844dadcd2	feat(gemini): allow disabling thought summaries while keeping thinking on (#36460) ... * feat(gemini): allow disabling thought summaries while keeping thinking on Both reasoning-effort mappers hardcoded includeThoughts to True whenever thinking was on, so thinking ON with summaries OFF was unreachable even though that is the google SDK's own default. Summaries are the narration rather than the reasoning, and for callers that discard them they are pure added latency. Adds litellm.gemini_include_thoughts_default, defaulting to True so nothing changes for anyone who does not set it. It gates the includeThoughts=True branches in the thinkingBudget mapper, the thinkingLevel mapper, and the anthropic-style thinking param mapper. The disable and none paths still force includeThoughts=False: with thinking off there is nothing to summarise. * test(gemini): assert thought-summary flag through the outbound Gemini request * feat(gemini): let a request's thought summary ask override the gateway default thinking.display (summarized, updates, omitted) and a reasoning_effort object carrying summary now decide includeThoughts for that request; gemini_include_thoughts_default applies only when the request says nothing * feat(gemini): let a thought summary ask on /v1/messages and /v1/responses override the gateway default * fix(gemini): read a text gateway default as a boolean and leave display omitted to the flag Normalize gemini_include_thoughts_default through normalize_drop_params so a YAML or env value written as "false" or "0" turns thought text off and a null or unknown value keeps the default on. thinking.display "omitted" no longer counts as a per-request ask, so the gateway flag decides, matching /v1/messages. Adds proxy integration cells for chat, /v1/messages and /v1/responses that prove the flag, every per-request ask, streaming, and a burst across workers. * test(gemini): match the streamed Responses usage, which omits null token details * feat(gemini): read the AI SDK reasoning summary alias and thinking.summary as chat asks * fix(gemini): keep the logged reasoning_effort unchanged and key the response cache on the summary alias A dict reasoning_effort next to the AI SDK reasoning_summary alias now counts as a summary ask, and the alias is part of the cache key so a plain request and an alias ask never share a reply * fix(caching): key the response cache on the summary alias wherever the request carries it The cache key now reads reasoning_summary and reasoningSummary through the same reader as the summary fold, so an alias inside extra_body gets its own key too * test: type the new summary-ask unit tests and mark their locals Final * test: annotate the remaining summary-ask unit tests the PR touches --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 09:01:33 +00:00
debanshd	1b1ddef1b2	fix: preserve non-null anyOf branches when stripping null types for Vertex (#41683) ... * fix: preserve non-null anyOf branches when stripping null types for Vertex * refactor: build anyOf branches immutably and drop explanatory comments * style: apply ruff format * fix: annotate anyOf list construction for the LIT002 budget * fix(vertex_ai): send a null-only anyOf as one nullable null instead of raising * test(vertex_ai): pin empty anyOf passthrough * fix(vertex_ai): forward a non-list anyOf unchanged and add wire cells for the null fold * fix(vertex_ai): forward a non-object anyOf branch for Google to judge instead of crashing * fix(vertex_ai): fold null out of type arrays and union branches before sending to Gemini --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 08:00:48 +00:00
devin-ai-integration[bot]	4e49394ade	fix(anthropic): keep trailing user messages with None or empty list content (#41397) ... * fix(anthropic): keep user messages with None or empty list content so requests do not end on an assistant turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): avoid mutable empty-list construction in sanitizer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): avoid rebinding sanitizer input Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): keep a trailing user turn whose content key was dropped litellm.completion strips None fields from every message before the provider transform, so a user message sent with content null reaches the Anthropic converter with no content key and raised KeyError. Read the content with get so that turn gets the same placeholder as an empty list. * fix(anthropic): only fill user turns with empty content and keep their cache_control * fix(anthropic): fill each empty user message in place like an empty string * fix(anthropic): add the placeholder only for an empty turn or a cache breakpoint, in linear time * test(anthropic): cover the tool-result guard with cache_control and widen the linear-time bound * test(anthropic): cover empty user turns through the proxy on chat, responses and messages * test(anthropic): type the empty user turn tests and move the linear-time check to a benchmark --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 07:41:05 +00:00
moyai-devin-berriai[bot]	a3251a3bfd	fix(responses): preserve provider request IDs in streaming failure logs (#45453) ... * fix(responses): preserve provider request IDs in streaming failure logs * fix(bedrock): log the provider request id for in-stream errors on bridged Bedrock streams Bedrock models that run through the chat-completions bridge (Claude, converse, invoke) raise in-stream failures from the event-stream decoder as a BedrockError without headers, so their failure logs kept error_provider_request_id null. The decoder now passes the stream's request-id headers to those errors. Both the native Responses iterator and the decoder pass only the PROVIDER_REQUEST_ID_HEADERS subset, so error responses and failure logs pick up the request id and none of the 200 response's other headers. MidStreamFallbackError gets its response at construction instead of mutating it afterwards. --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-16 00:29:11 -07:00
Mateo Wang	b21e659c24	fix(proxy)!: count endpoints return errors in their source API shape instead of silent local counts (#46815) ... * fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): reject unconvertible blocks nested inside a tool_result * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system * fix(proxy): count endpoints return missing-key and request errors in their source API shape * fix(proxy): drop imports the count error change left unused * fix(proxy): let Entra-authenticated Azure AI Claude count tokens and keep Google count errors in the Google envelope * fix(proxy): send Azure environment keys as x-api-key on Azure AI Claude counts and log unexpected Google count failures * fix(proxy): count with the global OpenAI and Anthropic keys chat already uses * fix(proxy): resolve the OpenAI count base with the key so a global key stays on its gateway * fix(proxy): keep unexpected Google count failures generic so internal details stay out of the response * fix(proxy): resolve the Azure AI Claude count base from the same settings chat reads * fix(proxy): count OpenAI tokens with the workload identity bearer chat already uses * fix(proxy): mint the OpenAI workload identity bearer inside the count error handling * fix(proxy): report a failed OpenAI workload identity exchange on the count endpoints as a 401 --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>	2026-10-16 00:11:39 -07:00
David Bochenski	f072498304	fix(bedrock): grant sts:GetWebIdentityToken and sts:TagGetWebIdentityToken in web identity session policy for claude_platform (#46670) ... * fix(bedrock): grant sts:GetWebIdentityToken in web identity session policy for claude_platform Claude Platform on AWS calls STS GetWebIdentityToken (outbound identity federation) with the caller's session before serving inference. The session policy passed to AssumeRoleWithWebIdentity is a permission ceiling and never granted it, so every bedrock/claude_platform request using OIDC web identity auth failed with 403 "no session policy allows the sts:GetWebIdentityToken action", even when the role's identity policy allowed it. Add a ClaudePlatformOutboundIdentityLiteLLM statement granting only sts:GetWebIdentityToken, plus tests that it is granted, scoped to its own statement, and that no sts:* wildcard is introduced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(bedrock): also grant sts:TagGetWebIdentityToken for claude_platform web identity Claude Platform on AWS tags the outbound web identity token it requests, so the caller also needs sts:TagGetWebIdentityToken. Found end-to-end on a deployment: with only GetWebIdentityToken granted, calls failed with "not authorized to perform: sts:TagGetWebIdentityToken on resource: arn:aws:sts::<account>:self"; granting both made /v1/chat/completions and /v1/messages (streamed and not) succeed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(bedrock): only let Claude Platform use the outbound identity STS grant Scope the ClaudePlatformOutboundIdentityLiteLLM session policy statement with aws:CalledViaLast = aws-external-anthropic.amazonaws.com, so a web identity session can no longer call sts:GetWebIdentityToken or sts:TagGetWebIdentityToken directly. Drop the code comment and fix the stale test messages. * test(bedrock): fail when the Claude Platform caller condition reaches other statements --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 06:37:49 +00:00
Dan Lemon	c8b73cde84	feat(soniox): decode proxy form fields for context, translation and flags (#40605) ... * feat(soniox): decode proxy form fields for context, translation and flags Multipart form fields reach the proxy as strings, so Soniox received context and translation as JSON strings and booleans as "true". Decode them at the handler entry point and add typed shapes for the context and translation request fields. * fix(soniox): keep bracket-prefixed free-form context and move types under llms Free-form context such as "[inaudible] consultation" is valid Soniox input, so a string that fails JSON parsing is now forwarded unchanged instead of returning a 400. The Soniox request types live next to the provider code. * test(soniox): exercise SDK-typed context and translation shapes through the decoder * refactor(soniox): derive handler options and provider params without mutating a local dict * chore(soniox): drop explanatory comment on the non-forwarded param set * refactor(soniox): return invalid form bool as a value and raise once in the handler * fix(soniox): validate decoded form fields and keep language with hints Validate structured context before sending it as an object, answer 400 for a bad polling interval or flag unless drop_params is on, nest SDK bracketed extra_body fields, keep the OpenAI language at the head of language_hints, and return the translated text for one-way translation without tags. * fix(soniox): keep the source transcript, rebuild SDK list fields, and reject malformed JSON values * fix(soniox): keep cleanup strings whole and drop the unused translation type * fix(soniox): let explicit language_hints win over language and test the proxy form path --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 06:37:34 +00:00
Mateo Wang	79ca820933	fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert (#46772) ... * fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): reject unconvertible blocks nested inside a tool_result * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>	2026-10-16 03:19:55 +00:00
Mateo Wang	dba86bec28	fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them (#46767) ... * fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>	2026-10-16 02:32:24 +00:00
Emmanuel Acheampong	de4d28653d	fix(crusoe): update base URL to current endpoint (#33121) ... Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 19:10:58 -07:00
Mateo Wang	d25b58f744	fix(count_tokens): count Anthropic-format requests for OpenAI the way /v1/messages inference converts them (#46759) ... * fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * test(count_tokens): expect the converted Responses input in the budgeted-key count integration test --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>	2026-10-16 01:58:46 +00:00
mrinal-berri	bd60cc8608	fix(bedrock): forward Nova Sonic realtime tool calls with their arguments (#46214)	2026-10-16 01:10:53 +00:00
Mateo Wang	d489c04554	fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking (#46741) ... * fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking * fix(bedrock): drop unknown thinking keys under drop_params on display updates rebuilds Bedrock Invoke copied every caller thinking key when adding display updates, so an unknown key still reached the provider with drop_params on. Filter that copy the same way, and cover each chat call site plus the global litellm.drop_params path in tests * fix(anthropic): drop budget_tokens from rebuilt adaptive thinking under drop_params * test(anthropic): cover drop_params sources for rebuilt adaptive thinking on /v1/messages and Bedrock Invoke	2026-10-15 23:09:30 +00:00
Mateo Wang	7090588bdf	feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240) ... * feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions * fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel * feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing early-access row, carrying the same limits and capabilities. The AI Studio row mirrors the early-access entry; the Vertex row differs only in provider and pricing source. Prices follow once Google publishes them. Extend the mapped metadata test to cover all three ids, that each routes to its provider, that its backup copy matches, and that the public rows carry the same limits as the early-access row. * feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops * chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap * fix(gemini): follow continuation hops in a loop instead of recursion The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls * fix(gemini): bill Google Search grounding from every continuation request Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain * fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch * chore(gemini): name the type errors the dual-mode stream iterator trips The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up * fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment * fix(gemini): keep one response id across a continued answer A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key * test(gemini): add integration cells for long decode continuation Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once * fix(vertex_ai): fetch the access token again before each continuation request A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged * fix(vertex_ai): start each continuation stream with an empty parser buffer When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async * fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way * fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes * fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not * fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0 A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer Adds the audit's wire tests for both, which fail 24 cells before this change * test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed * fix(vertex_ai): log a continued answer's raw response as one object A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch * test(integration): check a continued answer logs one merged raw response A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch * test(vertex_ai): cover a follow-up tool call and its thought signature in long decode * test(gemini): pin that a one-request answer logs Google's body unchanged	2026-10-15 22:54:23 +00:00
Mateo Wang	84ff9f84a9	fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses (#46720) ... * fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses * fix(ui): send detail on playground Responses image parts and code snippets * test(messages): cover image detail on the Responses bridge wire	2026-10-15 22:33:09 +00:00
Fede Kamelhar	68759a239f	fix(oci): drop non-function tools under drop_params and gate reasoning_effort on declared levels (#40732) ... * fix(oci): skip non-function tools and gate reasoning_effort on models that reject it Fixes #31449. Two OCI GENERIC request-building defects that made Grok-on-OCI unusable for tool-using coding-agent clients (Codex over the Responses API, Claude Code over the Anthropic Messages API): 1. `adapt_tool_definition_to_oci_standard` raised `OCIError("OCI only supports function tools")` for any tool whose type is not "function", aborting the whole request when a client mixed a built-in/server tool in with its function tools. Non-function tools are now skipped with a warning; if none remain, `tools`/`toolChoice` are omitted from the request instead of sending an empty array. 2. The GENERIC param map forwarded `reasoning_effort` to every non-Cohere model, but OCI rejects it for all served xAI Grok models (`Model grok-4.20 does not support parameter reasoningEffort`) and for OpenAI's non-reasoning models (`Unrecognized request argument supplied: reasoning_effort` on gpt-4o / gpt-4.1); gpt-5, the o-series, gpt-oss-* and Meta/Google models accept it. `_model_supports_reasoning_effort` encodes that, `get_supported_openai_params` stops advertising the param for those models, and `map_openai_params` follows the standard unsupported-param contract (dropped under drop_params, clear error otherwise) so `drop_params` / `additional_drop_params` now work as documented instead of the 400 surfacing from the service. Behaviour verified against OCI GenAI in us-chicago-1 (2026-09-11) per model. The two existing reasoning_effort normalisation tests move from a Grok model to openai.gpt-5, which is the case they exercise. 21 new tests; the OCI suite is 406 passed. * fix(oci): drive reasoning_effort gating from the model catalog and drop tool_choice for skipped tools Review follow-ups on #40732: - `_model_supports_reasoning_effort` now reads the `supports_reasoning` flag of the model's `oci/` catalog entry through `get_model_info` instead of hardcoding xAI / gpt-oss branches. Catalogued models without the flag (xAI Grok, Meta Llama) no longer receive `reasoningEffort`; models absent from the catalog keep the historical pass-through. - Catalog: flag `oci/google.gemini-2.5-{flash,pro,flash-lite}` as reasoning models and add `oci/openai.gpt-oss-120b` / `oci/openai.gpt-oss-20b` (pricing from Oracle's public price list, 128K context per OCI docs). - `_model_uses_max_completion_tokens` routes on the `openai.*` prefix only. OCI-hosted Gemini accepts `maxCompletionTokens` but ignores it (the output cap silently disappears) and Llama rejects it with 400, so the catalog's reasoning flag must not drive that switch. - After tool adaptation, a `FUNCTION` tool_choice naming a tool that was skipped (built-in / server tools) is dropped with a warning instead of being sent to OCI, which rejects the unmatched choice. Verified live against OCI GenAI (us-chicago-1): Grok with mixed tools plus a forced built-in now completes and still calls the function tool; Gemini and gpt-oss forward reasoningEffort with maxTokens honoured; Llama drops it under drop_params; gpt-5 keeps maxCompletionTokens. * style(oci): use an immutable set for the tool-name lookup Keeps the type-discipline gate (LIT002, mutable collections) within its budget; the lookup is read-only. * fix(oci): fail closed when a forced tool_choice names a tool OCI will not receive Review follow-ups on #40732: - When tool_choice forces a function that is not among the adapted tools (a skipped built-in or a misspelled name), send `toolChoice: NONE` instead of removing the choice. Falling back to the service default would let the model call any other supplied function even though the caller forced a specific one, which an agent processing untrusted content must not do. - Move the #31449 regression tests into the mapped test module (`test_oci_chat_transformation.py`) instead of a separate file. Verified live against OCI GenAI: Grok and Llama with a forced skipped built-in now complete with no tool call; a forced kept function is still honoured. * test(oci): point reasoning_effort comments at the in-module gating tests The referenced module was folded into this file. * fix(oci): drop non-function tools only under drop_params and gate reasoning_effort on declared levels Non-function tools now follow the unsupported-param contract: a 400 naming drop_params without it, and dropped with a warning under it, with a forced dropped tool turning tool calls off. reasoning_effort is gated on the model's reasoning_effort_levels catalog entry, so OCI Llama keeps forwarding it and the Grok 4.20 models OCI serves, now in the catalog with their Oracle prices, stop sending a param the service rejects * fix(oci): gate reasoning_effort on oci/xai.grok-4.20, which rejects it * fix(oci): disable tool calls when tool_choice forces a dropped custom tool by name * fix(oci): drop tool_choice that forces a dropped tool and keep a kept function's name * fix(oci): keep Gemini 2.5 Flash Lite unflagged for reasoning so released proxies keep its max_tokens cap * fix(oci): answer 400 when a request depends on a dropped tool and leave Gemini max_tokens routing to its own PR Under drop_params, non-function tools still drop when function tools remain and tool_choice does not force a dropped tool. A request whose every tool is dropped, or whose tool_choice forces one, now answers 400 instead of a tool-less reply the model can fill with invented results, or a swap to another tool. The maxTokens routing change for OCI Gemini goes back to base, since it is outside this fix and an open PR carries it. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 22:20:02 +00:00