131 commits

Author	SHA1	Message	Date
shrey	85fdea0fd1	fix(guardrails): apply guardrail masks to Bedrock InvokeModel Claude requests and replies (#49947)	2026-11-04 23:12:50 +00:00
norkelbyte	3b35ae340d	fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks (#49831) ... * fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks * fix(rate-limit): count embedding media as media only on models that embed it A data URI sent to a text-only embedding model is read as text, so it keeps the text estimate there. On models flagged supports_embedding_image_input, image and audio use the chat media estimates and PDF and video count at most the model's max_input_tokens. The same estimate now feeds project and key limits, the usage-based, latency and cost routers' deployment tpm, and the deployment itpm check. * fix(proxy): count embedding media only where the backend reads it as media Vertex AI, Gemini, Azure AI and Bedrock count a strict base64 data URI as media per item. Cohere counts it as media only when the whole batch is media, and Voyage, OCI and unknown models keep the text estimate. File blocks carrying a data URI count like the bare URI. The router context window pre-call check and the batch limiter now use the same estimate, resolving each deployment or JSONL row model to its backends. The cost map gains supports_embedding_image_input on voyage/voyage-multimodal-3 and oci/cohere.embed-multilingual-image-v3.0. * fix(router): hold each embedding media item to the context window on its own A PDF or video data URI alone already reserves at most the model's max_input_tokens, but the context window pre-call check then ran the blanked input through the Responses transform and added a few framing tokens, so a PDF that Vertex accepts got a 400. When every input item is media, the check now returns the largest item's media estimate, since an embedding request embeds each item separately * fix(proxy): leave embedding media out of the context check and narrow Bedrock media scope The router context window check now counts only the text of an embedding input, so Titan image lists no longer 400. Bedrock Marengo and Nova with a PDF keep the text estimate since they read those inputs as text or reject them, async_invoke Nova counts images as media, and the unused batch limiter change is dropped. * fix(bedrock): bill Cohere embeddings by Bedrock's input token count header * fix(embeddings): check data URI base64 in place and keep Bedrock Cohere text usage The strict base64 check now scans the payload in place instead of decoding and re-encoding a full copy. Bedrock's billed token header is used only for image requests, so text-only Bedrock Cohere usage matches main. The deployment itpm estimator change is reverted, since the router never passes embedding input to that check. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 22:06:59 +00:00
yuneng-jiang	883a2d1572	revert(bedrock): restore STS identity probe behavior before #49763 (#49939)	2026-11-03 21:34:59 +00:00
yuneng-jiang	e1c03c5423	fix(e2e): restore main qualification across providers and UI (#49763) ... * test: check an unsupported budget sort field * test: wait for dashboard navigation readiness * test: support clustered provider caches with live fallback * Revert "test: check an unsupported budget sort field" This reverts commit 9694787bb5. * test: check an unsupported budget sort field * test(e2e): refresh retired provider fixtures * test(e2e): read Langfuse observations through the supported API * test(e2e): identify trace roots by kind and request path * fix(e2e): repair UI readiness and regional AWS qualification * fix(e2e): scope LangSmith batch readback to its workspace * fix(e2e): use the configured global Bedrock GPT profile * test(e2e): defer unprovisioned GovCloud coverage * test(e2e): defer Bedrock Runtime Codex cases blocked by IAM * test(e2e): defer intermittent Azure translation coverage	2026-11-03 20:38:47 +00:00
zoveki	baddc67041	fix(anthropic)!: carry prompt_cache_breakpoint to cache_control on chat completions to Claude (#49783) ... * fix(anthropic): carry prompt_cache_breakpoint to cache_control on chat completions to Claude A content part carrying OpenAI's prompt_cache_breakpoint on a /v1/chat/completions request to Claude (Anthropic, Bedrock Invoke and Converse, Vertex AI, Azure AI) now reaches Claude as cache_control {"type": "ephemeral"} on the same block, so prompt caching works for OpenAI-shaped clients instead of the marker being stripped. Only parts OpenAI accepts the marker on are translated, a client cache_control on the same block wins, and Converse translates only for models that take cache points. A 30m ttl or a malformed marker answers UnsupportedParamsError without drop_params and becomes a default 5m mark or no mark with it. * fix(anthropic): hold translated prompt_cache_breakpoint marks to the 4-breakpoint cap Marks past Claude's 4 cache breakpoints now answer 400 without drop_params and keep the earliest under it, counting the client's own message, tool, and top-level cache_control marks. The ttl and malformed-marker errors name drop_params for proxy users too and no longer name a single provider. * fix(bedrock): count Converse tool cache points in the breakpoint cap and gate unmapped models A client tool cache_control or a tool_config injection point becomes a Converse tool cachePoint, so it now takes one of the 4 slots before a translated marker does. Converse only translates prompt_cache_breakpoint for a model the cost map says takes prompt caching. An unmapped model such as an application inference profile ARN opts in through model_info supports_prompt_cache_breakpoint, so a marker sent to every model of a mixed fleet no longer turns a non-caching profile's 200 into Bedrock's 403. Router prompt-caching affinity now treats a marked block like a cache_control one, and requests without a marker skip the translation after one pass over their messages. * fix(bedrock): count only the tool cache points Converse sends against the breakpoint cap * fix(anthropic,bedrock): cap only the cache breakpoints each provider sends, and drop marks Nova cannot take on tool results * perf(anthropic,bedrock): find the breakpoint cut-off by binary search so trimming many marks stays fast * fix(anthropic): drop the duplicate drop_params left by the main merge * fix(bedrock): let an inference profile's cache breakpoint flag win over its base_model * fix(anthropic): keep the first and last prompt_cache_breakpoint marks past the cap Under drop_params the translated marks that fit the cap are now split between the start and the end of the request, so the stable prefix and the newest turns both stay cached. The cap count also includes a native tool search tool that keeps its cache_control on the wire. Bedrock Converse now translates the marker only for the Claude and Nova families, which Bedrock documents as taking cache checkpoints, unless the deployment sets supports_prompt_cache_breakpoint. A guarded user turn keeps its cache mark when the request fits the cap. * refactor(bedrock): decide guarded cache marks with a typed bool helper * fix(bedrock): validate guarded blocks before reading their cache mark * fix(anthropic): let injection points replace client marks and keep non-Claude routes untranslated When a deployment sets cache_control_injection_points, the client's prompt_cache_breakpoint marks are dropped on targets that do not read them, so the operator's injection points decide every Claude breakpoint. OpenCode Go models served over the Anthropic messages endpoint translate the marker only for Claude, and guarded Converse text keeps a cache mark only on a model known to take cache points. * fix(anthropic): keep client marks unless a message injection point replaces them on a translating route A tool_config-only injection point no longer strips client prompt_cache_breakpoint marks, a flagged Bedrock profile no longer counts as reading the marker natively, and guarded Converse text skips the tool count when no message carries a cache mark. * fix(anthropic): keep Bedrock and Claude deployments flagged for the marker on cache_control injection A supports_prompt_cache_breakpoint flag on a Bedrock profile or a Claude deployment made the injection hook write OpenAI markers and prompt_cache_options, which Bedrock answers 400 for. Routes that translate the marker now never take the OpenAI dialect. * fix(bedrock): reserve only the tool cache points Converse sends when trimming Claude cache marks under drop_params * fix(anthropic): name the marker's accepted ttl values in the malformed prompt_cache_breakpoint 400 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-11-03 12:56:22 +00:00
norkelbyte	864b98acec	fix(bedrock)!: fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 (#49170) ... * fix(bedrock): fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 The Bedrock SDK ends the output stream the same way for a dropped connection as for a session that sent sessionEnd, so the handler treated a dropped stream as a clean finish. The client got a 1000 close with no error event and the router logged 200 OK with no retry An output end now counts as clean only after the client left and the handler began closing the Bedrock input. An earlier end raises a 503 BedrockError, so the client gets an error event and a 1011 close, the failure handlers run, partial spend is still logged, and a retry of a committed session is still refused * fix(bedrock): fail a dropped Nova Sonic session right away and log it as failed When Bedrock's output side ends mid-session, by a drop or a modeled stream error, the client side no longer spends the close timeout sending close messages into the dead input stream, so the error reaches the app right after the drop. A session the app ends itself still sends Bedrock its close messages and closes the input The replay refusal now leads with the committed cause, so the error event and the cut 1011 close reason both start with it. It stays a 400 and is tagged as a failure outside the selected deployment, so router retries and fallbacks of a dropped session no longer add extra failure counts A clean realtime session now counts once toward its deployment's cooldown success counter, so cooldown reads the real failure rate instead of 100 percent. Realtime tpm and rpm counting is unchanged The spend row of a session the provider ended now reads failure with its error information, while the events already forwarded are still billed. A session the app ended keeps logging success * fix(bedrock): agree on a dropped realtime session's failure across loggers and the spend row A realtime session's success event now reports status failure, with the same error_information the spend row stores, when the session recorded a provider failure in its metadata. Usage already streamed stays billed and the router still counts one cooldown failure The spend row builder now runs every row's error_information through sanitize_error_information_for_spend_logs (DB size cap plus prompt redaction), so the realtime failure rows get the same treatment as the proxy failure hook, which no longer sanitizes on its own to avoid truncating twice Drop the selection-failure tag on the committed-session refusal, since the shared logging object already keeps a retried refusal out of cooldown, and pin the router's realtime success counting with an OpenAI realtime deployment * fix(bedrock): count realtime router successes only for sessions that logged one, and log first-turn drops The router counts a realtime success only when the session's success handlers ran, so a refused handshake or an upstream close before any frame no longer offsets its failure. The success event's payload status stays success again, so each logger gets one failure record per dropped session, while the spend row still reads failure from the metadata the Bedrock handler records. A committed Bedrock session that fails before any logged event now gets one zero-cost Logs row. After an output failure the input stream is closed in the background, so the SDK body writer finishes without delaying the client's error. Every input send failure reports the same 503 stream-ended cause as an output drop * fix(realtime): count a router success only for a session that ended cleanly A realtime session the upstream closes with 1011 or 1006 after its first frame now marks itself as ended abnormally where that close is seen, in the shared relay and in the OpenAI Live handler, so the router no longer counts it as a cooldown success. The success callbacks still run, so the proxy keeps the budget and parallel-slot settlement it had A Bedrock session that fails before Bedrock sends any usage now logs no success event, as on base. The spend-row failure fields go into a new metadata mapping instead of the request's own dict, validated with a pydantic TypeAdapter instead of an unchecked TypeIs guard. The handler tests drop a wall-clock bound and type their helpers * fix(bedrock): log a dropped realtime session's usage from its own logging state The failure event replaced the shared standard_logging_object while the queued success dispatch was still running its callbacks, so loggers received $0 and 0 tokens for a billed dropped session. * fix(proxy): settle a rate-limit reservation once when a request logs a failure and a success A dropped realtime session fires a failure event and then a success event for the same call. The v3 limiter refunded the TPM reservation on the failure and released it again on the success, so the key's TPM counter went negative. The success path now skips a reservation the failure already released and marks it settled for a later failure. The realtime handler test callbacks are typed * fix(rate-limit): settle each token reservation once when a request logs both a failure and a success A dropped realtime session logs a success for its usage and then a failure. After a success the limiter now clears every reserved amount (TPM, PTU, ITPM, OTPM), so a later failure refunds nothing twice. A new reservation written in the same context after a released one now starts clean, so reserve, fail, reserve, succeed charges the second request's usage once instead of 78 TPM or 28 ITPM for 27 and 20 used * fix(rate-limit): take a token hold before awaiting so concurrent success and failure callbacks settle it once * fix(rate-limit): track the ITPM/OTPM hold's release apart from TPM so a nested pre-call keeps the request's project IO charge * test(rate-limit): pin the split hold-release flags across nested MCP pre-calls * fix(realtime): count a relay that fails on its own as an abnormal end * test(realtime): pin the relay failure exit with a client frame the provider rejects --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:56:23 +00:00
tessellark	5b546259d5	fix(cache_control)!: keep one deployment's prompt-cache opt-in from reaching siblings on the same model (#49685) ... * fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry Two deployments that share one model string share one key in the live litellm.model_cost map, and router registration writes each deployment's supports_prompt_cache_breakpoint into that key. The cache control hook read the live map, so a flagless deployment picked up a sibling's opt-in and sent prompt_cache_options and prompt_cache_breakpoint upstream, which the provider rejects with a 400 When a call carries the routed deployment's model_info, the prompt cache checks now use its declared supports_prompt_cache_breakpoint, and fall back to the loaded catalog instead of the live map. This covers the hook on completion, acompletion, Responses and the /v1/messages route, the azure_ai chat transform, the OpenAI Responses transform under drop_params, and the /v1/messages to Responses adapter * fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a provider other than openai, such as azure_ai OpenAI models and bedrock_mantle. Bedrock on every route and providers that build Anthropic messages keep cache_control, which Bedrock Converse turns into cachePoint The openai chat reader now passes the routed model_info and checks base_model. The Bedrock cache point reader takes the deployment's flag first and, for a routed call with no flag, reads the catalog as loaded instead of the live map that sibling deployments write into. model_info is threaded from the converse transform, the factory message and tool builders, and the responses bridge * fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations. * fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways * fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases * test(anthropic): cover auto caching on an unmapped Claude deployment * fix(cache_control): keep the merge base's reads where the per-deployment flag changed them A register_model entry under a provider-prefixed name reaches routed calls again, since the overlay now resolves the bare and prefixed names the way runtime registrations are stored. The hook and the openai chat reader no longer judge base_model, as at the merge base, so a client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint and get a 400. A Bedrock deployment flag that is not a real true reads as false again, as at the merge base. An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again, as at the merge base. Other wildcards still leave both cache flags to the catalog. The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps its own flag and that a flagless unmapped ARN keeps its cache points after the delete. * fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name Routed calls read register_model entries through an overlay that merged the bare name after the provider-prefixed one whatever order they were registered in, so a later opt-out under one name lost to an earlier opt-in under the other. Each registration now moves its key to the end of the kept registrations, and the overlay merges both names oldest first, which matches what SDK calls read from the live model map. Also lists the wildcard cache flag helper in the router coverage script, since the hook's wildcard tests drive it rather than calling it by name. * fix(cache_control): keep register_model entries visible while a cost map reload replays them The previous commit moved a re-registered name to the end of the saved registrations by removing and re-adding it, and the overlay walked that dict to order a bare and a provider-prefixed name. A reload replays every saved entry, so a request on another thread could briefly miss an entry or fail on a dict that changed size mid-walk. Entries are written in place again, a per-name sequence records which registration came last, and both the overlay and the reload replay order by it, so the later registration still wins after a reload. * fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:55:51 +00:00
tessellark	f4a2e2b3af	fix(anthropic): send an https PDF image_url to Claude as a document (#49708) ... * fix(anthropic): send an https PDF image_url to Claude as a document A PDF link sent as image_url to Claude on Anthropic went out as an image block and Anthropic answered 400. A link whose path ends in .pdf is now downloaded like Vertex links are, and the served Content-Type picks the document or image block, so CDN links that serve a PNG from a .pdf path keep working. An explicit format of application/pdf sends a url document with no download. PDF media types match regardless of case or parameters, and count_tokens picks the same block as the request. * fix(anthropic): download https PDF links off the event loop The proxy downloaded an https link ending in .pdf synchronously while building the Anthropic request, so on an async call every other request on that worker waited for the download. Those links are now inlined before the request is built, the same way http links already are, and the /v1/messages guardrail write-back does the same. A non-string format on an image_url is ignored the way it already was inside the image_url mapping, instead of raising an AttributeError. * fix(anthropic): send a PDF link the proxy cannot download as a link and keep guardrail url blocks as sent * test(anthropic): expect the PNG media type the served bytes carry at a .jpg path * fix(anthropic): bound the PDF link download and keep query-string PDF names Give up on a .pdf link after NON_LLM_CONNECTION_TIMEOUT instead of the client's 6000 s default, fall back to main's whole-URL extension check so download?file=report.pdf still counts as a PDF, treat application/x-pdf as a PDF, and set the url document's application/pdf format only for the guardrail write-back so other /v1/messages providers send what main sends * fix(anthropic): probe a .pdf link and send it to Claude as a url document Instead of downloading a .pdf link and inlining it as base64, ask the host for its first kilobyte and send the link as a url document when it serves a PDF. Every other link goes out as the url image main sends, so robots.txt handling, caching and the wire format stay as they are on main. Also restore the guardrail adapter and the content type inference to main, keep the client's url document blocks (title, context, citations, cache_control) through the guardrail write-back, treat a PDF filename in the query string as a PDF link, probe links concurrently on the sync path, cap the async probe's read at the download limit, and catch only transport, URL and SSRF failures in the probe * fix(anthropic): read only the first kilobyte when probing a PDF link, and restore each url document from its own place The probe now streams the response and keeps the first 1 KB, so a host that ignores Range no longer fills memory with up to 20 parallel bodies. The guardrail write-back now matches url documents inside a tool_result and outside one separately, since the write-back moves tool results ahead of other content * fix(anthropic): remember each PDF link probe's verdict and cap its total time, and skip the probe on Bedrock invoke * fix(anthropic): keep a PDF probe's error status for a minute, not ten --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:05:15 +00:00
tavren27	94b5b6386a	fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool (#49703) ... * fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows * fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool Native output without thinking now applies to every Bedrock Claude model that cannot force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5 needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose. The invoke chat path keeps the native output_format it built instead of inlining it, and GovCloud region-path ids read their own false rows. * fix(bedrock): clean caller output_format schemas before sending them natively on invoke * fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke * fix(bedrock): keep the faked Converse stream for native output that replaces the json tool A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture. * test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 01:56:52 +00:00
simonp9	61da6fcb4d	fix(mcp)!: answer 400 for a hosted MCP tool a translated route cannot run, drop it under drop_params (#48931) ... * fix(bedrock, gemini, deepseek, databricks, gigachat, anthropic): answer 400 for a hosted MCP tool a route cannot run, drop it under drop_params A `type: "mcp"` tool sent to a route that rebuilds tools for its provider (Bedrock Converse and Nova, Bedrock Invoke Claude, Vertex AI Claude, Vertex AI and Google AI Studio Gemini, DeepSeek, Databricks Claude, GigaChat) was dropped silently and the request still answered 200. These routes now answer UnsupportedParamsError (400) naming the model, route and server, and pointing to the LiteLLM MCP gateway. Under drop_params the tool is dropped with a warning, other tools are kept, and no empty tools field is sent. Anthropic, Azure AI Claude and Claude Platform on AWS keep mapping the tool to the MCP connector. Native Responses providers and raw OpenAI-compatible forwarders are unchanged. * fix(mcp): check hosted MCP tools once per request and keep base tool_choice handling Each route now runs one request-level step before its param mapping. When the drop leaves no tools, the request sends no tools, tool_choice or parallel_tool_calls. Converse and GigaChat go back to their base tool_choice and empty-tools handling. * refactor(deepseek): hold the sent tools in a tuple * fix(mcp): keep tool_choice for legacy functions when every hosted MCP tool drops * fix(mcp): answer 400 for MCP servers a route cannot run on Messages, Ollama and the Responses playground /v1/messages requests with mcp_servers on a chat-adapter or Responses-bridge route now answer 400 naming each server, or drop it with a warning under drop_params, instead of being forwarded or silently ignored. Ollama generate no longer pastes a hosted MCP tool (headers included) into the prompt, and ollama_chat filters hosted MCP tools the same way. The Admin UI Responses playground now sends litellm_proxy/mcp references like the other endpoints. * fix(ollama): leave JSON mode off when drop_params drops every hosted MCP tool Ollama generate set format json for any request with tools, even after the hosted MCP check dropped all of them, so a plain answer came back as JSON or as a call to a tool the model never had. The check now runs before the JSON mode switch, which stays on only while a tool is left for the prompt. * fix(messages): honor additional_drop_params for mcp_servers and read the beta map only for hosted MCP tools * fix(mcp_security): accept gateway server names and toolsets the gateway resolves The guardrail compared litellm_proxy/mcp/<name> refs against registry keys, which are server ids, so a saved server referenced by name, alias, or toolset name was blocked although the gateway routes it. It now resolves each ref the way the gateway does and still blocks unknown names. * fix(tool_fields): keep MCP drop warnings on one log line Server labels, model names, and routes come from the caller, so a line break in one could forge extra log lines. The drop warnings now strip CR and LF from each value. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:13:03 -08:00
zoveki	6be98d8c75	fix(bedrock)!: give Invoke and Mantle Claude the model's own default max_tokens (#49692) ... * fix(bedrock): resolve Invoke Claude default max_tokens from the base model id Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): build Invoke Claude request params without mutating Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): type tool-search normalizer return to clear unknown-type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): resolve Invoke and Mantle Claude default max_tokens from the deployment id, ARN base_model, and Mantle row * fix(bedrock): default Invoke Claude max_tokens from the deployment's own cap and honor AnthropicConfig Read the called deployment's model_info.max_output_tokens before the base model's cost row, skip the shared regional key the router fills from every deployment, and leave the default to litellm.AnthropicConfig(max_tokens) when it is set. * fix(bedrock): send the mapped default max_tokens on bedrock_mantle Claude chat * fix(bedrock): bound a global AnthropicConfig max_tokens by the deployment's own cap * test(bedrock): annotate the new Mantle default max_tokens tests with Final * refactor(bedrock): inject the AnthropicConfig max_tokens reader into the Invoke Claude config --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 11:42:45 -08:00
tessellark	a2ab27373f	fix(proxy): count the context_management edits /v1/messages applies on every bridged provider (#49234) ... * fix(proxy): count the context_management edits /v1/messages applies on every bridged provider * fix: apply context_management edits in count_tokens on Responses bridges, compaction blocks and Bedrock Claude The Responses bridge now applies the same edits inference applies for every provider, the chat bridge cuts the history at a client compaction block, and Bedrock Claude forwards the edits AWS accepts with their beta to the mantle and runtime count APIs instead of dropping them. * refactor(count_tokens): return count-view failures as values and raise them once in the caller * fix(bedrock): keep the runtime's 400 when mantle does not serve the model it retries * fix(count_tokens): forward context edits on Vertex and Azure Claude counts and keep Bedrock's mantle answer without edits * fix(proxy): count edited bodies on native routes without a count API, keep counts on editor failures * fix(proxy): keep the Bedrock pass-through count shape and provider 400s on empty bridged bodies The pass-through count picks Converse or InvokeModel from the content again, and the edited InvokeModel count wraps string turns in text blocks. A bridged view with no messages sends the body as before. Azure and Vertex load the beta config off the event loop before a count with edits, and a model with no deployment reads drop_params from the request and router defaults. * fix(bedrock): send an empty-string turn as is when a count forwards context edits Wrapping an empty string into a text block made AWS reject the count with 'text content blocks must be non-empty', while base and /v1/messages accept the empty string. Only non-empty string turns are wrapped now. * test(azure_ai): compare the count's betas with the betas Azure inference sends * test(azure_ai): record the Azure inference request with respx instead of appending to a list --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 10:38:35 -08:00
simonp9	f7b2cd0000	fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500 (#49237) ... * fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500 A list role or an object or list tool_call_id crashed the chat conversion with "unhashable type". Depending on the step that hit it, callers got a 500 or a 400 that never named the field. A failed conversion now answers 400 naming the first non-string role or tool call id, and the prompt caching hook reads a non-string answer id as matching no tool call. * fix(anthropic): name the field the conversion failed on, and answer 400 for a non-string Responses call_id The chat field check now runs only when the conversion raised a type error, and scans roles first, then assistant tool call ids, then tool answer ids, so an orphan tool answer that modify_params drops is no longer blamed. A tool output call_id that is not a string on /v1/responses answers 400 naming input[N].call_id, and the clear_tool_uses polyfill leaves a non-string tool_use_id for the provider * fix(anthropic): name a non-string role whatever error the conversion raised An int or bool role fails the conversion with an invalid message error, not a type error, so it lost the named 400. Roles are now named for any conversion failure, while tool call ids stay limited to type errors * fix(responses): name a non-string message item role by its input index * fix(responses): name a malformed input field only where litellm reads it, by the caller's index A tool output call_id is named only when it is an object or a list and the conversion fails, so an orphan numeric id still converts and another field's failure is never blamed on it. On the bridge to Claude a non-string role or function_call call_id is named by its input item, counting items before a compaction item. A numeric assistant tool call id on chat is named too. * fix(anthropic): name a tool id only for the error it causes and count Codex tool items in input indexes A numeric tool answer id that modify_params drops was blamed for any later type or attribute error, so the 400 hid the real cause. Tool ids are now named only for the error their value raises, and a hashable answer id never. The /v1/responses bridge and its input token count now read the input the caller sent, with only additional_tools hoisted into tools, so input[N] counts those items too. * fix(anthropic): name a tool id only for the error its own value causes and keep Codex tool item text out of the request * fix(responses): name a tool output call_id only when the items before it convert * fix(responses): name a non-string input field only on inference so token counting and batch records keep their answer * fix(responses): let a client field named caller_input or names_non_string_fields reach the bridge * test(anthropic): build the non-string id histories without editing them in place * fix(anthropic): never name an empty role for another field's failure * fix(anthropic): name an empty list or object role when the conversion fails on it * fix(anthropic): name a list role under the drop_params cache cap and keep the Bedrock batch line prefix --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 21:13:24 +00:00
yucheng-berri	efc2304b7e	fix(streaming)!: fail a stream that ends without a finish reason (#47665) ... * fix(streaming): fail on clean EOF without finish reason * fix(streaming): cover shared OpenAI-compatible EOF paths * test(router): complete mocked streaming responses * fix(streaming): keep the EOF error message and cover Anthropic clean EOF The request-less RemoteProtocolError tripped exception mapping into 'The .request property has not been set.' for non-OpenAI providers. Adds Anthropic and generic-chunk unit tests, an integration test across chat, the /v1/messages bridge and the /v1/responses bridge, and moves main's token-estimate test onto a finished stream with a sibling that checks the cut stream still bills the estimate on its failure log. * fix(streaming): count a finish reason any chunk sent, keep LangGraph async streams whole Gemini marks every chunk unfinished and a later usage chunk reset intermittent_finish_reason, so a finished Gemini stream failed at EOF. LangGraph's __aiter__ opened a new aiter_lines on every wrapper call, which hit StreamConsumed after the first chunk and now surfaced as a false EOF error instead of a silent truncation. * fix(streaming): keep the synthetic stop for a stream that sent no chunks Gemini's OpenAI-compatible endpoint answers with only [DONE] when reasoning uses up max_tokens. Failing every openai/azure stream that sent nothing turned that 200 into a 500, while every other provider already kept the stop for an empty stream. Adds integration cells for Gemini's trailing usage chunk, multi-event LangGraph runs, a mocked deployment and the [DONE]-only answer. * test(streaming): keep pytest.raises to one statement, drop the spend read for an empty stream An empty stream writes no spend row on main either, so the [DONE]-only cell checks only what the client sees. * fix(streaming): read the chunk's finish reason for the sticky flag * fix(agentcore): end a stream on messageStop when the runtime sends no metadata or final message * test(streaming): pin the logged-chunks guard, inject the LangGraph client, use current model names * fix(agentcore): say why the end-of-stream flags rebind * test(streaming): cover an AgentCore runtime that ends on messageStop through the proxy * test(streaming): cover SDK clients, request fallbacks and a concurrent burst for a cut stream * test(agentcore): inject the async transport instead of patching the aiohttp flag * test(guardrails): end the decision model chat fixture with a finish reason, as a real stream does * test(streaming): shape the AgentCore and LangGraph fixtures like captures from a real runtime and server * test(streaming): keep each stream's ending and land every spend row once through a spend log outage * test(streaming): end each stream by its provider's finish after a proxy worker is killed mid burst * test(streaming): run the outage test as one burst before and one during the outage instead of a capped wave loop * test(streaming): connect the outage test's proxy to the relay without TLS so its trigger sees the spend-log insert on an ssl=on Postgres * test(streaming): type the new stream EOF unit tests * test(streaming): a merged reasoning stream cut mid reasoning now fails Main's think tag test expected a stream that ends mid reasoning without a finish reason to finish with a closed think block. With this PR that stream fails, so the case moves to its own test that expects MidStreamFallbackError with the open think block as the partial content --------- Co-authored-by: King Star <mcxin.y@gmail.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-31 11:29:26 -07:00
mika-lt	c4b01b2ac8	fix(decisions)!: honor a deployment's own supports_audio_input and supports_video_input flags (#49207) ... * fix(decisions): honor a deployment's own supports_audio_input and supports_video_input flags A Decisions deployment's model_info flags for audio and video now win over the cost map, per media kind, the same way supports_box_questions already does. An unlisted model flagged true gets its clips sent upstream, and a false flag refuses them (or drops them under drop_params). OpenAI-format providers still refuse clips since their request shape has no field for them, and the SystemOne request builder no longer re-reads the cost map. * fix(decisions): ignore media flags on a deployment whose own mode is not decisions A chat-mode deployment's supports_audio_input describes chat input, so it no longer turns on audio or video clips for Decisions. The cost map decides for it, as before. Adds a proxy integration test for flagged, partly flagged, unflagged, and drop_params deployments on both /v1/decisions and /v1/systemone, and for caller-sent model_info. * fix(decisions): follow the cost map's mode for a flagged deployment without its own mode A chat deployment flagged for audio or video usually leaves mode unset, so its flags described chat input yet still let clips through on /v1/decisions. When the deployment sets no mode, the cost map's mode for the model now decides whether its media flags apply, and an unlisted model still takes them. The audio and video lookups also run only when the request carries a clip. * fix(decisions): read the mode a flagged deployment falls back to from the shipped cost map The router copies each deployment's model_info into the model's shared cost map row, so another deployment's mode there could switch off a flagged deployment's media flags. The fallback now reads the cost map as loaded, untouched by router registrations * fix(decisions): route clips in a load-balanced group to a deployment that takes them A Decisions or SystemOne request carrying audio or video now skips group deployments whose own flags or the cost map refuse that media, and keeps the whole group when none takes it, so the 400 still names the media. * fix(decisions): keep deployments that drop or cannot resolve clips in the media filter * fix(router): skip clip-refusing Decisions deployments after cooldown and tag filtering Running the media filter before cooldown turned the 400 a group answered when its only clip-taking deployment was cooling down into a 429. It now runs next to the image filter, just before order filtering, in both the async and sync selection paths. * fix(router): honor router default drop_params when skipping clip-refusing Decisions deployments * test(router): list the router default params helper in the router coverage check * test(bedrock): stop reloading the cost map module in the GovCloud fixture --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 01:09:46 -07:00
Vineeth Sai Varikuntla	a7b66dd8b9	fix(openai): stop stripping cache_control from the caller's own messages and tools (#35602) ... * fix(openai): stop stripping cache_control from the caller's messages Copy before removing the field, so the caller's own message objects are left alone. * fix(fireworks_ai): strip cache_control on a copy so a fallback keeps the caller's hints * fix: keep the caller's request intact on more providers and trim extra Claude cache hints under drop_params * fix: drop only the oldest extra cache hints inside one message when trimming to the Claude limit * fix: keep replayed session turns yielding their hints as whole turns while a request's own message keeps its newest hints * fix(cache_control): count only hints that reach Claude and cap hints on OpenAI-compatible Claude routes under drop_params * fix(cache_control): keep the capped Claude messages a list in OpenAI-compatible request bodies * fix(caching): cap Claude hints for string drop_params, a Claude base_model, and Converse tool messages The 4-hint cap now reads drop_params through drops_unsupported_params on chat, Converse, and Anthropic routes, names Claude by base_model on OpenAI-compatible routes, and counts a Converse tool message's hints as the one cache point it sends. * fix(caching): trim Claude cache hints in one wire-order pass and reserve a top-level cache_control on OpenAI-compatible routes Dropping the oldest hint one at a time re-walked the whole request per hint, which took seconds on long tool histories. The cap now drops each message's oldest hints in the order the provider counts them, so a string assistant turn loses its text hint before its tool call hints. OpenAI-compatible Claude routes also keep room for a request-level or extra_body cache_control under drop_params. * test: drop narrating comments and stacked generator clauses from cache_control tests * fix(vertex_ai): put every system message in the Gemini context cache so generateContent never sends it beside cachedContent --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 00:29:10 -07:00
tessellark	5d5e3685c1	fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths (#48933) ... * fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths * fix(logging): keep guardrail and token-count reads of a request from logging drop warnings Only the real Anthropic-messages adapter request reports a dropped thinking budget. Guardrail, shadow-eval and token-count translations of the same request now drop it silently, so one request logs one warning. The 422 retry also no longer hashes each loc entry, so a list inside loc cannot raise TypeError. * fix(bedrock): keep the DeepSeek R1 Converse reasoning_effort drop at debug level Only allowed_openai_params reaches that branch, and it puts reasoning_effort back after the provider mapping, so a warning there claimed a drop that never happened * fix(logging): warn on Gemini non-integer budget drops and keep the auto-router baseline silent Gemini 2.x now logs the drop of a non-integer thinking budget. The auto-router savings baseline builds a body it never sends, so it no longer warns. Logged values are capped with reprlib, Azure AI marks only its drop_params retry as gated, and Bedrock Chat Completions names the model without its route prefix. * fix(logging): warn on the Gemini 3, Magistral, Ollama, Fireworks and DeepSeek ARN reasoning drops Gemini 3 and later now log a dropped thinking budget, Magistral logs the level it swaps for a reasoning prompt, Ollama logs a level it maps to think off, Fireworks logs chat_template_kwargs effort keys dropped for a model the map does not list as reasoning, and DeepSeek on Bedrock logs a drop when the model is addressed by an ARN that advertises the param. * fix(logging): warn on the remaining Ollama, DeepSeek and Tencent reasoning drops The ollama/ completion route now shares the chat route's think mapping, so a level it turns off logs the same warning. DeepSeek warns when it drops thinking.budget_tokens, and Tencent warns when an adaptive-only model gets a non-integer budget. Magistral no longer warns on low, medium, high or enabled thinking, the values its reasoning prompt stands in for. The Gemini 3 budget warning no longer claims the model uses its default level. * fix(reasoning): warn when Claude drops a non-string reasoning_effort Claude chat and native /v1/messages dropped a reasoning_effort like 5 or {"effort": 5} with no log line. Both now log one warn-only WARNING. A Magistral thinking.type sent as a list no longer raises TypeError. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 08:48:43 -07:00
Siddhant Badola	b71491c6c9	fix(bedrock): route a Bedrock model id with no Invoke provider to Converse (#43290) ... * fix(bedrock): route unmapped converse-only family model ids to converse #42713 stopped Bedrock-native OpenAI ids that are not in the cost map yet from falling to the invoke route. The same gap exists for the other families that have no invoke transformation at all: minimax, nvidia, writer, zai, google and xai. An id from one of them that is newer than the cost map hits get_bedrock_route -> "invoke" -> get_bedrock_invoke_provider() -> None -> BedrockError, when Converse would have served it Add a second family regex for those six and default them to converse, the same way the openai family is handled. _OPENAI_FAMILY_MODEL_RE is left alone because the prompt-caching rule is openai-specific Every shipped id of these families already resolves to a converse entry via its base model (checked against the cost map), so this only changes behaviour for ids that are not in the map yet Tests: six unmapped ids route to converse, and an unmapped meta.* id still routes to invoke since meta has an invoke handler * fix(bedrock): route a model id with no Invoke provider to Converse A bedrock/<model> id missing from the cost map fell to the Invoke route, where an id that names no Invoke provider (a new vendor's model, a bare provisioned-model, imported-model, or custom-model-deployment ARN) fails locally with "Unknown provider=None" before any request is sent. Such ids now go to Converse, which replaces the per-family regex with the general rule. Ids that name an Invoke provider and explicit route prefixes are unchanged. * test(bedrock): expect Converse for ids with no Invoke provider in the nova ARN and not-found tests * fix(bedrock): accept tools on Converse for a model with no cost-map row or Invoke provider, and keep a 404 for an invalid id * fix(bedrock): answer 404 for an invalid model id instead of AWS's 400 AWS answers a mistyped Bedrock model id with a 400 ValidationException. The mapping already raised NotFoundError, but passing AWS's 400 response through made the openai base class report 400, so the proxy answered 400 and the router skipped its cooldown * fix(bedrock): drop lookaround regexes from tool schemas for a model the cost map does not list * fix(bedrock): apply the unknown-model defaults only to bare ids LiteLLM knows nothing about A bare Bedrock id with no cost-map row and no Invoke vendor gets Converse's open params, its reasoning_effort as sent, lookaround-free tool schemas, and a 404 for AWS's invalid-identifier reply, which keeps AWS's response headers. An explicit route like converse/, an ARN, or an id LiteLLM recognizes keeps main's params, regexes and error status, so drop_params still drops tools there * fix(bedrock): judge lookaround support by model id on every route, ignore price-only rows The regex check now strips route and region prefixes from the model and its base_model before asking whether LiteLLM knows the model, so bedrock/converse/<id> gets the same answer in a direct transform call as on the completion path. A cost-map row counts only when it names a provider, so a price-only register_model row leaves an unlisted id unknown, as on the proxy. AmazonInvokeConfig now inherits the Invoke provider lookup instead of keeping an identical copy. * fix(bedrock): treat a region path like the bare id when the cost map does not list it A deployment like bedrock/us-east-1/<unlisted id> now gets the same tools, reasoning_effort and typo 404 as bedrock/<unlisted id>, so Claude Code works on it and a mistyped one still answers 404. * fix(bedrock): keep today's 404 for ids that reach Converse only by default, and keep an empty id local * fix(bedrock): answer 404 for an invalid id only when Converse rejected it, and keep a region path with no id local * fix(bedrock): keep a path-shaped id local, and map /v1/completions errors with the model's provider An id with a slash that is not an ARN, like bedrock/../../converse or an unknown region path, keeps Invoke and its local 404 instead of reaching AWS Converse. atext_completion mapped every error as OpenAI, so a Bedrock typo on /v1/completions answered 400 while chat answered 404. It now resolves the provider from the request's model, as the other async entry points do. * fix(text completion): keep /v1/completions error mapping as it is on main Mapping async /v1/completions errors with each provider's rules is wider than this Bedrock routing change. It moves classes, statuses, cooldown and fallbacks for most providers, and some lose context-window fallbacks. So this restores main's mapping here and leaves that change to its own PR. A Bedrock typo on /v1/completions answers AWS's 400, while chat keeps the 404. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 06:27:59 -07:00
zoveki	68b5a737a1	fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params (#49102) ... * fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params The native /v1/messages path sent tools with `strict` straight through, so when Claude's strict-tool limits rejected them with a 400 the caller got that 400 even with drop_params set. Under drop_params, a 400 that matches Claude's strict-limit phrases on a request with a strict tool is now sent once more with `strict` removed from every tool. Without drop_params nothing changes. The Claude strict-limit matcher moves from bedrock/common_utils.py to anthropic/common_utils.py so Bedrock converse and the base messages config share it, and the Rust bridge routes strict tools under drop_params to Python. * fix(anthropic): retry without strict only when a tool sends strict true A tool with only strict false no longer triggers a resend of an identical body, the warning no longer blames tool schemas when the rejected schema may be the output format, and requests with only additional_drop_params stay on the Rust messages route since the Python retry ignores them * fix(anthropic): keep sync messages calls with strict tools on Rust Python has no sync /v1/messages handler, so routing a sync call with a strict tool to Python raised ValueError where Rust served it before. The strict-tools detector now applies to async calls only. The retry warning also names the strict tool count without a plural. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 17:18:59 -07:00
tavren27	b25ea48c4b	fix(anthropic)!: drop beta-gated body fields under drop_params (#48793) ... * fix(anthropic): strip beta-gated body fields when a leg drops the anthropic-beta When a provider leg drops an anthropic-beta value (the chat handler's allowlist filter, /v1/messages' filtered headers, Bedrock Converse's additionalModelRequestFields), also drop the body fields that beta gates (thinking.block_binding, thinking.display=updates, mid-conversation system clear_at, output_config, tool_addition/tool_removal blocks) so the request degrades on that leg instead of failing with a provider 400. Copy-on-write so a fallback leg still sends the full request, and each strip is logged at WARNING. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove a leg that drops a beta sends neither the beta nor clear_at Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): strip beta-gated fields against the final body anthropic_beta list Converse counts the body anthropic_beta list as active, Invoke strips after its auto-injected betas are known and injects thinking-binding-controls for thinking.block_binding, test helpers are fully typed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated system fields before an older model's system turn becomes a user turn Models without supports_mid_conversation_system send no mid-conversation beta, so tool_addition and tool_removal blocks are dropped (and logged) before the turn is converted instead of riding along as user content. The Invoke output_config regression test reads the real cost map instead of patching _bedrock_model_supports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): pin the integration proxy to the bundled beta allowlist and type the helper locals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): annotate every new test local with Final and type the fixtures and returns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated body fields only under drop_params Without drop_params the request now goes out as sent and the provider answers its own 400, like any other unsupported param. inline-tools-2026-09-15 also unlocks tool_addition and tool_removal blocks, so a leg that carries it keeps them. * fix(anthropic): strip beta-gated fields on Foundry chat and Bedrock chat Invoke under drop_params Foundry chat and Bedrock chat Invoke now drop beta-gated body fields the same way the other legs do when drop_params is on and the request does not send the gating beta. The mid-conversation system normalizer validates only on the conversion path, and the strip WARNING names the beta the caller can send to keep the field. * fix(anthropic): send the thinking betas for block_binding and display updates, strip Databricks chat thinking Every Claude leg now adds the thinking-binding-controls beta when thinking carries block_binding and the display-updates beta for display updates, so those fields go through instead of a 400. Databricks chat rejects both fields with or without the beta, so it drops them under drop_params. A system turn the strip empties stays when it still carries output_config, and the drop warning names both causes. * fix(anthropic): send the thinking betas on every leg and keep fields where the beta is unknown Vertex messages and Claude Platform chat now add the thinking betas when thinking carries block_binding or display updates. Converse sends only the caller's header betas when it has them. A provider with no row in the beta config forwards beta-gated fields instead of dropping them, an emptied trailing system turn that would leave an assistant prefill is sent as is, and the prompt cache baseline body matches the body that goes out. The drop WARNING now names the beta that unlocks each field * fix(anthropic): filter Converse body betas, keep Foundry auto betas, read the bundled beta config for row checks * fix(anthropic): validate messages only when a beta-gated drop runs A non-object messages entry on a Claude model without mid-conversation system support raised a raw pydantic ValidationError before the beta-config and drop_params checks. The strip now passes the messages through untouched unless it drops something, so the provider's 400 reaches SDK callers as BadRequestError again * fix(anthropic): keep system blocks with a non-string type instead of raising --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 06:25:31 -07:00
tessellark	1a05aab72a	fix(bedrock)!: route region-prefixed embedding models to that region and bill its price (#49072) ... * fix(bedrock): send region-prefixed embedding model ids to that region without the prefix * chore(bedrock): hold the embedding fix back until the risk stamp * fix(bedrock): send region-prefixed embedding model ids to that region without the prefix * fix(bedrock): keep a model_id's region and price embeddings at the region the call goes to * fix(bedrock): read a model_id ARN's region for embeddings and keep unlisted regions mappable * fix(bedrock): route async_invoke embeddings whether the region comes before or after it * fix(bedrock): price embeddings by region_name and keep the path region with a plain model_id * fix(bedrock): price a model_id ARN at its region's row * fix(bedrock): keep chat's region rules for embeddings and leave ARN pricing as on main A plain model_id skips the region in the model, as chat does, so a deployment that worked on main keeps calling the same region. An empty aws_region_name fails region validation again instead of falling back. This reverts the ARN regional pricing, since many regional rows lack the bare row's priority, batch, and long-context rates --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 06:24:04 -07:00
Mateo Wang	0cf2bb7296	fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params (#47761) ... * fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params A GPT-5 request whose reasoning_effort the model map does not enable for that model lost the effort with no log line under drop_params. An Azure deployment name missing from the cost map lost reasoning_effort none this way and ran at the model's default effort. The gate now logs a warning naming the model, the level, the map entry it checked, and the model_info.base_model and model_info.reasoning_effort_levels fixes. * fix(openai): warn when the Responses route or a deployment's declared levels drop a reasoning effort The /v1/responses gate dropped a row-disabled GPT-5 effort, and reasoning for an OpenAI model without supports_reasoning, with no log line, and the router logged a declared-level drop only at INFO. Both Responses drops now log a WARNING and the router logs its drop at WARNING. The warning now leads with model_info.reasoning_effort_levels, since base_model only helps a deployment named differently from its underlying model. * fix: warn only when a reasoning level is actually dropped, and on generic reasoning_effort drops * fix: warn when thinking or a Bedrock Converse reasoning level is dropped, and name the router drop a reasoning level * fix: skip the drop warning when a non-reasoning model is asked for no reasoning, and honor declared levels on Bedrock Responses * fix: stay silent when reasoning is turned off on /v1/responses and warn on the Bedrock Responses fallback drop * fix(reasoning): warn when any reasoning-off request is dropped and give each route only the advice it honors A request to turn reasoning off on a model name the catalog lacks now warns on every route, since it may be a reasoning model. Chat suggests model_info.reasoning_effort_levels and, for a name with no catalog entry, model_info.base_model. OpenAI Responses suggests declared levels only where the config honors them, and Bedrock suggests neither. The Bedrock and Mantle Responses forwarding of declared levels is reverted until it can be proven live. * fix(reasoning): suggest reasoning_effort_levels and base_model only where the route reads them The chat gate suggested model_info.reasoning_effort_levels on routes that never pass declared levels (github_copilot, chatgpt and other OpenAIConfig delegates) and model_info.base_model on OpenAI and Azure AI chat, whose gate looks up the deployment name rather than base_model. /v1/responses/compact suggested levels it never reads either. Declared levels are now None when a caller passes none, so the advice appears only when a caller fed them, and the base_model advice is limited to the Azure chat gate, the one that looks up base_model. The openai row of the chat warning test now expects no base_model advice, the Responses warning test goes through get_optional_params_responses_api the way /v1/responses and compact call it, and the Azure direct-config test passes declared levels the way get_optional_params does. Long test lines are wrapped. * fix: read only the built-in catalog when choosing the reasoning drop warning	2026-10-29 04:46:29 -07:00
zoveki	310c96162b	fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893) ... * fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema * fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema. * fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them. * fix(bedrock): drop tool custom object on the Mantle count_tokens fallback Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way. * fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages * test(vertex): annotate count_tokens tool test locals with Final * fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names * fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:36:20 -07:00
rj-cairn	d06b889d41	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) ... * fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream The Gemini and Vertex AI streaming path read the SSE body with httpx iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085, U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds one of them was cut in two. The tail is not a data: line and the head never parses, and the accumulate-and-retry fallback then carries the unparsed prefix forward, so every later event is appended to it and the stream ends with no more content and no usage. SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines split decoded text on exactly those, treating a CR at the end of one chunk and an LF at the start of the next as one break. The first request and every long-decode hop now read through them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks. * fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator * fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed Switch the remaining httpx stream readers (Anthropic chat, Google GenAI, Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a raw U+2028 or U+2029 inside a JSON string no longer splits an event. Keep the chunk that closes a think block in the logged and cached response when a stream ends without a finish_reason, close the block before a legacy function_call, and make sync Codestral FIM streaming send stream=true and use the text-completion-codestral stream parser. Test fakes that only mocked iter_lines now stream real httpx bytes and assert the content they feed in. * fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use split_sse_lines from sse_lines.py, which also gains a str.split fast path. Adds tests/unit/llms/codestral to the unit workflow shard. * fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines * test: type the new stream tests, drop a comment, and benchmark the SSE line splitter * test: type the iterator class parameters in the line-separator stream tests * fix(responses): stream tool call arguments opened in a text chunk * fix(responses): hold a tool call that shares a chunk with text until the text ends * test: type the a2a SSE handler helper and the sandbox fake responses --------- Co-authored-by: Tim Vink <vinktim@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>	2026-10-28 17:00:29 -07:00
rj-cairn	f89b9c6150	refactor(types): replace Any with proven types in 28 files (#48903) ... * refactor(types): replace Any with proven types in 52 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): use typing_extensions.TypedDict for validated seams Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): revert strict Anthropic response validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): accept a null finish reason from Bedrock Cohere and TwelveLabs Pegasus * test(bedrock): annotate the Pegasus null finish reason test --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 16:12:10 -07:00
tessellark	43ce3e22aa	fix(responses)!: answer 400 before counting a lone Responses built-in tool on Claude (#48798) ... * test(responses): audit #47363 Adds integration cells for the input token count paths #47363 changed: the provider id of a managed file and the 403 for another team's file on OpenAI, file_search and additional_tools parity plus cache-mark stripping and untranslatable tools on Claude, and the Bedrock runtime count leaving out the options it refuses. * fix(count-tokens): reject a lone tool with no chat equivalent before the Claude count call A Claude input token count whose only tool was mcp or computer_use_preview skipped the pre-call check and went to the provider, so the caller got the provider error instead of the 400 naming the tool. The check now runs whenever a tool would get a placeholder name. * fix(responses): count a lone Anthropic toolset and name the uncountable tool The lone-tool 400 now fires only for OpenAI Responses built-in tool types, so a nameless Anthropic toolset still counts. The error says LiteLLM cannot count that tool type for the model. Adds a Bedrock unit cell for the lone-tool 400 and pins the runtime wire test's tool shape per request type. * fix(responses): take the uncountable tool types from the OpenAI SDK and keep the region hint off LiteLLM's 400 The Responses built-in tool types now come from openai's ToolParam union, so a type the SDK adds is refused before the count call too. The AWS region hint no longer lands on an error LiteLLM raised before calling the provider. Tests cover two more lone types and the hint. * fix(responses): keep LiteLLM's own list of uncountable tool types and pass a non-string tool type through * fix(responses): count tool_search and namespace tools the way Claude chat inference sends them The count refused a lone tool_search with LiteLLM's 400 and sent a lone namespace raw or squashed it into an empty tool, while Claude chat inference maps tool_search to Anthropic's tool search tool and unpacks a namespace into its function tools. Both now go through the deployment's own map_openai_params, so the count sends what inference sends, and tool_search leaves the list of uncountable types. * fix(bedrock): count a Claude request with its deployment's own chat translation Bedrock invoke and Mantle inference turn a chat tool_search tool into Bedrock's tool search tool, but the Bedrock counter always translated with the Anthropic config, so the count answered 400 for a request inference serves. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 12:12:45 -07:00
m6atlas	a0c490ea54	fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params (#48733) ... * fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params * test(bedrock): cover the strict tools 400 retry under drop_params * fix(bedrock): anchor the strict tools rejection match and strip toolSpec strict * test(bedrock): return captured requests from the strict tools test helper * fix(bedrock): retry without strict only when a tool sent strict to Bedrock A strict-limit 400 on a request whose tools carry strict: false, or on a model litellm never sends strict to, was sent again with an identical body. The fallback now takes the model and returns nothing unless a tool puts strict on the wire. Also types the fallback test parameter and records the mock transport's requests on the recorder instead of a mutated list * fix(bedrock): decide the strict retry on the profile's base model and retry grammar timeouts An application inference profile with base_model sends strict on the wire, so the retry check now resolves the capability model the way the request transformation does. Bedrock's "Grammar compilation timed out" 400 is a strict-grammar rejection too, so it takes the same single retry under drop_params. * test(bedrock): annotate the strict-tools retry test helper's values as Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 04:04:39 -07:00
Kerry Lu	0b66a27528	feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost (#48278) ... * feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bill native Bedrock guardrail trace usage on chat, responses and pass-through Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): record the Bedrock guardrail trace of a streamed metadata chunk on the logging object Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(bedrock): apply guardrailConfig on /v1/messages and price its trace, record the stream trace on both builder paths, redact the native guardrail_response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): validate the guardrail trace with one pydantic model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): record the native guardrail entry when litellm_params metadata is None Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): import the renamed messages invoke transformation module Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): skip native guardrail billing on cache hits and redact the trace with message logging off Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): walk the guardrail trace iteratively when redacting it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): record the native guardrail trace where the Bedrock response is parsed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): type the native guardrail trace fixtures and locals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 12:10:45 -07:00
Mateo Wang	14193435b8	refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510) ... * refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages Move the unified /v1/messages code out of a directory called pass_through, since it is not the passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner model experimental_pass_through package becomes messages. The old packages stay as import shims that warn once and alias every submodule in sys.modules, so old deep imports still resolve Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): keep Final variables out of loops in the rename shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): match the warning text in the rename shim test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): resolve old pass_through imports lazily through a meta path finder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): mark the alias loader spec restore for the type discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): format the renamed package shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): keep the rebind-ok marker on the spec restore line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): carry the new package search path on alias specs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): declare the shim test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 10:08:25 -07:00
Mateo Wang	7e14638001	refactor(llms): rename model-version transformation files to family names (#48509) ... gpt_5_transformation.py becomes gpt_reasoning_transformation.py (OpenAI and Azure), the Bedrock anthropic_claude3_transformation.py files become anthropic_claude_transformation.py, anthropic_claude2_transformation.py becomes anthropic_legacy_text_completions_transformation.py, and vertex_ai_partner_models/llama3 becomes vertex_ai_partner_models/llama. Class and symbol names are unchanged. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-26 18:53:16 -07:00
Kerry Lu	560372456e	fix(bedrock): price claude_platform calls from their own cost map rows (#48150) ... * fix(bedrock): price claude_platform calls from the anthropic cost map rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: avoid recursive Claude Platform lookups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): price claude_platform calls from their own cost map rows Replace the claude_platform -> anthropic lookup redirect with bedrock/claude_platform/* rows, and apply the inference_geo multiplier in the bedrock cost path so US-only inference is priced at 1.1x Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(bedrock): exempt claude_platform rows from the Converse routing guard The claude_platform/ route is explicit in the row key, so these rows are not accidental non-Converse Bedrock chat rows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-24 13:54:18 -07:00
yucheng-berri	f7e6ddec66	feat(proxy): add LITELLM_OFFLINE_MODE, one switch for the fetches LiteLLM makes on its own (#48096) ... * feat(proxy): add LITELLM_OFFLINE_MODE to stop every request the proxy makes on its own * fix(ui): name LITELLM_OFFLINE_MODE where the cost map panel says local mode is forced * test: isolate the offline-mode tokenizer tests from the local Hugging Face cache * test: cover the offline policy templates and the cached Kimi K3 tokenizer * test(integration): count Kimi K3 tokens in the offline-mode egress test * refactor(token_counter): skip non-bundled tokenizers in offline mode instead of listing downloaded ones --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-23 16:54:19 -07:00
Mateo Wang	66c8f5b1a9	fix(llms): join api_base and route with one shared helper (#47535) ... * fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * fix(llms): join api_base and route with one shared helper A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base * refactor(llms): move per-provider api_base strips onto join_api_base Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join * ci: run the trailing-slash URL builder test in the llms shard * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell * fix(llms): send every provider route through join_api_base Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash * fix(llms): keep the route skip only where a builder already had it join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path already ends in it, and it is used only where the merge base or the trailing-slash test PR head already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the path so a host named anthropic-* builds /anthropic/v1/messages * fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL * test(llms): run the Gemini RAG trailing-slash case through aingest The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears * test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base * test(integration): cover api_base joins across providers, query strings and named hosts * feat(url_utils): add join_api_base and ensure_api_route helpers * fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host join_api_base and ensure_api_route now keep the // of a base left with only a scheme, so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it * fix(azure_ai): build the count_tokens URL from the messages URL * fix(azure_ai): match anthropic as a whole path segment in the messages URL * fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints Azure AI chat and messages go back to cutting the base at the first /anthropic, as on main, and count_tokens keeps its path check, so a gateway path like /anthropic-proxy builds the same URLs as main. A base that already ends in /v1/messages still gets count_tokens next to it. A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is used as its text again in AgentCore, Bedrock responses and the RAGFlow base, instead of falling back to the default host. get_volcengine_base_url and GigaChat's get_api_base strip a trailing slash again for outside callers. The cache-hit wire test is skipped as a known proxy bug, since one cache hit writes two cache_hit spend rows. * fix(llms): drop a slash before the api_base query and tighten the route test checks * test(llms): annotate new test locals as Final and flatten the builder generator * fix(url_utils): keep a scheme-only api_base hostless so it fails before sending --------- Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com>	2026-10-21 06:59:32 +00:00
yucheng-berri	e6065b9f4f	fix(bedrock)!: apply the deployment guardrailConfig on /bedrock pass-through routes (#47429) ... * fix(bedrock): apply the deployment guardrailConfig on the /bedrock pass-through routes The router-model pass-through dropped a Bedrock deployment's guardrailConfig, so converse, converse-stream, invoke and invoke-with-response-stream reached Bedrock unguarded while /v1/chat/completions on the same deployment was blocked. The signer now sends it the way Bedrock reads it, in the body for Converse and as the X-Amzn-Bedrock-Guardrail* headers for Invoke, replacing any guardrail the caller sent * test(bedrock): reject version-less guardrails in the scripted Bedrock and cover the 400 The mantle and fallback fixtures named a guardrail with no guardrailVersion, which real Bedrock refuses, yet the scripted Bedrock answered them as guarded successes. The fake now answers a version-less guardrail with a 400 ValidationException, the fixtures carry versions, and a new case checks a version-less deployment guardrail reaches Bedrock and the caller gets its rejection on all four actions --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-20 14:48:19 -07:00
Priyansh Nandwana	c6fc5b01bb	fix(bedrock): send batch jobs to a configured control-plane endpoint (#47260) ... * fix(bedrock): send batch jobs to a configured control-plane endpoint Bedrock batch create, retrieve and cancel always built the regional bedrock.{region} host, so a VPC endpoint or compatible gateway could not be used. Add aws_bedrock_control_plane_endpoint as a deployment setting read by the signed HTTP path and the boto3 path, carry it through the proxy credential filter, and refuse it in request bodies like the other endpoint overrides. api_base stays out of it because on a Bedrock deployment it points at the runtime host. Fixes #45056 * fix(bedrock): read the control-plane endpoint from typed kwargs in job status The named parameter added one unknown-argument error in litellm/batches/main.py, which unpacks untyped kwargs into this function * docs(bedrock): explain batch control-plane endpoint configuration * fix(bedrock): keep the batch control-plane endpoint out of chat request bodies A deployment that sets aws_bedrock_control_plane_endpoint also serves chat, and the unregistered key was swept into the Bedrock request body, so every chat call on it failed with "Extra inputs are not permitted". Register it with the other Bedrock batch connection fields so it stays a LiteLLM-owned kwarg on every route while batches still read it from litellm_params. * test(bedrock): verify endpoint signatures without internal mocks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-20 12:30:16 -07:00
Abhishek Acharya	7655c2c839	fix(bedrock): return a 400 for malformed batch JSONL lines (#45992) ... * fix(bedrock): return a 400 for malformed batch JSONL lines A malformed line in a Bedrock batch input file escaped as a JSONDecodeError from the upload and file content conversions, which the proxy reported as a 500. Parse those files line by line and raise a 400 that names the bad line. Object name inference keeps its fallback for files that are not JSONL. Fixes #45060 * test(bedrock): cover blank lines in malformed batch JSONL * fix(bedrock): reject a malformed batch line whatever the upload's MIME type The upload only converts, and so re-checks, batch files whose MIME type is on the allowlist. Falling back to the generic object name let a malformed file sent as application/x-jsonl skip both checks and land in S3 unchanged, so the object-name step now runs the same strict per-line parse and raises the 400. * fix(bedrock): reject every malformed batch input line with a 400 naming the line Non-object lines, a missing or non-object body, a non-string body.model, non-UTF-8 bytes, nesting too deep to parse, and rows the Bedrock mapping cannot convert escaped as AttributeError, TypeError, UnicodeDecodeError or RecursionError and surfaced as 500s. They now raise a 400 BedrockError in the same "Batch input file line N ..." wording the proxy pre-check and the Vertex adapter use. Object naming reads only the first line and never raises, and a UTF-8 byte order mark is accepted. * docs(bedrock): point the batch file name comment at the helper that builds it * test(bedrock): mark new batch test locals Final and give the line builders explicit params * test(bedrock): use single quotes inside the batch-line f-strings so the file parses on Python 3.10 and 3.11 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-10-20 11:58:43 -07:00
Mateo Wang	4b1928387f	test(bedrock): detect Vault reads on the event-loop thread (#47597)	2026-10-20 10:45:04 -07:00
Mateo Wang	f3513bca9e	fix(bedrock): resolve AWS region and secret lookups off the event loop (#47331) ... * fix(bedrock): resolve AWS region and secret lookups off the event loop Bedrock request paths that resolve their region, endpoint or bearer token inside the async handler now do it in the AWS signing executor, so a slow secret manager read no longer stalls every other request on the worker * fix(bedrock): type the STS client kwargs and claude platform environment override Keeps the basedpyright counts at or below the merge base after the off-loop wraps, by typing the STS client kwargs as a TypedDict, the claude platform environment override with concrete dicts, and the realtime region call through the typed getter * test(bedrock): detect event loop stalls without wall-clock timing The Vault fake now answers a read only after the test loop runs a callback, so a read made on the loop thread is caught deterministically instead of by a sleep and a heartbeat threshold	2026-10-20 17:04:41 +00:00
Mateo Wang	91724bb0a7	fix(responses)!: count input tokens the way inference sends the request (#47363) ... * fix(count-tokens): translate chat tool use before Claude token counts Responses and chat tool use (function tools, function_call items, tool messages) is now converted the way Claude chat inference converts it before the Anthropic, Azure AI and Vertex partner counters call count_tokens. Translation errors answer 400 before any provider call. Anthropic-shaped requests and /v1/messages/count_tokens keep their body as sent. * test(count-tokens): compare Claude count bodies with inference for Responses tool use * fix(responses): forward the caller's Responses body to OpenAI token counts OpenAI deployments now get the caller's /v1/responses/input_tokens body as sent, minus LiteLLM-only fields, so OpenAI's own count and errors come back. previous_response_id is checked for ownership like inference and routes to the deployment that served it * fix(responses)!: count Claude Responses bodies the way inference sends them Non-native Responses counts now run the body through the inference bridge, so text.format, tool_choice, reasoning, built-in tools, and file_search count the way inference sends them on Anthropic, Azure AI, Vertex, and Bedrock. The OpenAI native count body uses the drop_params-resolved tool fields. * fix(responses): honor drop_params and replay previous turns in Claude counts Claude input_tokens counts now read the same drop_params setting inference merges from the deployment, router defaults, and request, so a forced tool_choice downgrades to auto under drop_params. previous_response_id replays the stored turns through inference's session loader, and chat messages carrying fields beyond role and content go through the provider's chat translation * fix(responses): keep proxy-only request keys out of OpenAI counts and map chat tools in Bedrock counts * fix(count-tokens): import on Python 3.11 and keep stored prompts in OpenAI counts under drop_params * fix(responses): leave tool_choice out of counts when drop_params drops every tool * fix(count-tokens): count Mantle options, drop cache marks, and reject input inference cannot send The Bedrock Mantle fallback now counts thinking, tool_choice and output_config like the runtime count. Claude Responses counts leave out cache_control marks again, so more than four marks count instead of failing. Counts answer 400 before any provider call for user blocks chat inference refuses and for Responses tools with no Chat Completions equivalent, instead of dropping the block or counting a placeholder tool. Adds the wire tests for these paths * fix(count-tokens): count Responses tools the way the inference bridge sends them A Claude count now hoists additional_tools input items into tools, strips a malformed tool flag under drop_params and rejects it without, and leaves tools out when the deployment lists tools in additional_drop_params, so the count matches the body Claude inference sends for the same request * fix(count-tokens): keep Bedrock runtime counts option-free and send OpenAI the provider file id * test(count-tokens): capture the Anthropic count request at the HTTP layer instead of patching the handler * fix(count-tokens): ignore a caller-sent file id mapping and keep non-string file ids as sent	2026-10-20 15:13:59 +00:00
devin-ai-integration[bot]	dd20bb0ae9	fix(bedrock): apply the 16 maxTokens floor to Kimi K3 on Converse (#43875) ... * fix(bedrock): apply the 16 maxTokens floor to Kimi K3 on Converse Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): use lane AWS credentials for the Kimi K3 messages test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): bypass the proxy cache and declare test metadata for the Kimi K3 probe --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-20 13:01:45 +00:00
Mateo Wang	270d108ace	fix(anthropic): json_schema envelope edge cases on the Anthropic json tool path (#47478) ... * test(anthropic): cover json_schema envelope edge cases on the Anthropic json tool path * fix(anthropic): unwrap required-only keys, walk past tool-named envelopes, keep non-ASCII text, flush cut streams, check Vertex output_format * fix(anthropic): send held json tool text on cut Bedrock Invoke streams and drop a non-object output_format under drop_params A Bedrock Invoke stream that ends before the json tool block closes now sends the held text and closing brace, as the direct Anthropic stream already did. A non-object output_format on the Anthropic messages route and its providers is dropped under drop_params and otherwise forwarded, so the provider answers its own 400. Vertex Claude names the proxy setting in its 400 and reads drop_params through the shared helper. Adds integration cases for the json_schema envelope and the cut stream. * fix(anthropic): drop a non-object output_format on chat, unwrap a whole envelope on a cut stream, escape lone surrogates Chat completions to Anthropic, Azure AI, Bedrock Invoke and Vertex now drop a non-object output_format under drop_params, matching /v1/messages. A stream cut after a complete held envelope is unwrapped like a normal end. A json_schema answer holding a lone surrogate keeps its \u escape so the response can still be encoded * fix(anthropic): join a split answer when the stream ends in its last block A cut during the second json tool block now runs the held block through the same flush as a block stop, so it joins the streamed keys or the earlier split blocks instead of sending two objects back to back. The Vertex output_format fold drops its unreachable drop_params branch, since both parent configs drop a non-object value first. * test(vertex): expect the LiteLLM 400 for a non-object output_format on /v1/messages	2026-10-20 08:38:40 +00:00
joshua-berri	3124797399	fix(bedrock): resolve profile pricing before applying custom rates (#47419) ... * fix(bedrock): price Converse calls using the invoked model profile * fix(pricing): retain registered rates across metadata updates * fix(pricing): honor temporary registered prices for Bedrock profiles * fix(bedrock): record invocation model ID for each completion attempt * test(bedrock): recalculate fallback cost after metadata updates * fix(pricing): preserve cache-only registered price overrides * fix(bedrock): preserve priced overrides without widening shared pricing * fix(bedrock): retain explicit pricing through token cost lookup * fix(bedrock): layer cache overrides over invoked profile rates * fix(bedrock): keep effective prices consistent across billing * fix(bedrock): share profile pricing and custom rate resolution * fix(router): remove unused pricing suppression * fix(cost): preserve rates across mixed token thresholds * fix(router): share deployment pricing during startup * fix(bedrock): preserve Router pricing inheritance for profiles * fix(bedrock): retain configured pricing for unmapped profiles --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-20 00:10:49 -07:00
Mateo Wang	7b82b2a3c3	feat(bedrock): route token counts from model metadata (#47441) ... * feat(bedrock): route token counts from model metadata * fix(bedrock): keep counting metadata compatible with older proxies	2026-10-19 21:36:13 -07:00
Mateo Wang	8b9cd12a89	test: catch regressions in the validation lines #44798 added (#47420) ... * test: pin the validation lines #44798 kept with failing cases * fix(containers): accept endpoint table entries that omit their param lists * test: keep the endpoint table and xAI adapter as on main The endpoint table ships inside the package, so optional param lists fixed no reachable state and turned a loud failure into silently dropped params. Drops the Gemini case that main's tests already cover. * test: cover the cost map backup and Gemini image config validation lines The cost map backup adapter's hide flag, and the validation itself, could be removed with every test still green. The Gemini imageConfig adapter could be narrowed with only the integration suite noticing, which does not run on PRs. The endpoint table cases now read a real file through the module's location instead of replacing open, so reading the file another way keeps them green. * test: pin the container handler's loud table failure and loosen two pinned drops A container endpoint table entry without path_params or query_params fails the call on main, and no test held that. The new case trims each key from the packaged table and expects the validation error. The Gemini imageConfig case now accepts a drop or UnsupportedParamsError for a non-object value, and the Bedrock beta case checks only that numbers come back as strings, so neither pins behavior the drop_params convention may change. * test: check a numeric beta array is never split as text and use the latest Gemini image model The Bedrock case now also fails when a JSON array of numbers falls through to the comma split, which returns bracketed fragments, while a drop or a string coercion both stay allowed.	2026-10-20 01:06:37 +00:00
moyai-berriai[bot]	f370235bb4	fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874) ... * fix(guardrails): preserve Bedrock Converse redactions and honor tool-only scans * Preserve main's per-block streaming fixes and regression coverage * fix(bedrock): apply Converse guardrail rewrites texts-first and keep main's write-backs Map tool results onto their text and json parts only, accept a structured row the flat texts already reproduce, let a flat-text mask win over a structured leaf, ignore texts that do not line up with the request when structured rows come back, keep tool call arguments out of the rows under skip_tool_message_in_guardrail and send no tool definitions to tool-only guardrails * fix(bedrock): accept blank content on tool-call-only Converse turns A vendor that returns "" or [] where an assistant row carried only tool calls no longer fails the request closed. Also type the leaf lookup without Any and restore the original layout of the existing unit tests. * fix(bedrock): scan additionalModelRequestFields in Converse guardrail rows and split the handler Guardrails that read structured_messages instead of texts (Akto, Prompt Security, CrowdStrike AIDR and others) stopped seeing strings in additionalModelRequestFields, which Bedrock can forward into the model prompt. Those strings now go to the guardrail as one user row placed before the conversation, and a row-only edit to them is written back. The Converse to OpenAI row conversion moves to litellm/llms/bedrock/chat/converse_to_openai.py and the returned-row diffing to structured_edits.py, so the handler keeps only the glue. Rows record the Converse path of every text part, scope flags travel as one value, and path lookups return a value instead of raising. Unit tests use a real CustomGuardrail subclass instead of mocks, and the integration vendor modes get descriptive names. * fix(bedrock): keep new Converse helpers inside the type discipline rules * test(bedrock): pin Converse row edits that change shape, role or placement * fix(bedrock): type the Converse body once so its edits keep their reference * fix(bedrock): list request-field texts where their Converse row sits * fix(bedrock): send a reshaped Converse tool result as one text block A vendor that compresses or summarises a tool result, such as Headroom collapsing a JSON list into a table string, returned a 500 because the row could not be split back into the original text and JSON parts. The tool result now carries the vendor's string as one text block where its parts were, keeping images and documents in place. It still fails closed when the flat texts masked that tool result or the row changed its tool call id. * fix(bedrock): apply guardrail tool edits on Converse and keep compression guardrails off it A guardrail that masks a Converse tool description or schema now has that mask written into toolConfig, a tool it removes is dropped, and a tool it adds, renames or reorders is refused before Bedrock. Headroom, Compresr and TypeSafe get only the flat texts on Converse again, as on main, because the pass-through route has no loop to answer their retrieve tool * test(bedrock): refuse a guardrail that changes a Converse tool's kind * fix(bedrock): keep partial-row and row-rebuilding guardrails on flat Converse texts OpenAI moderation, the LLM judge, CrowdStrike AIDR, Prompt Security, PANW Prisma AIRS and HiddenLayer scan only part of the rows or rebuild their returned texts from them, so on Converse they get the flat texts only, as on main. When a vendor returns changed rows, its texts are applied only if they keep the flat layout and change only rows the vendor also changed, so row-shaped texts of the same length no longer overwrite unrelated Converse fields. * test(bedrock): pin how flat texts sit next to rewritten Converse rows * fix(bedrock): keep the Bedrock guardrail on flat Converse texts With experimental_use_latest_role_message_only, the Bedrock guardrail places its mask by counting text parts per row, and Converse rows do not line up with the flat texts, so the mask could land in tool call arguments. Give it the flat texts only, as on main. * fix(bedrock): keep Akto on flat Converse texts Akto maps its masked payload back onto the flat texts by value, and a Converse row joins or serialises tool call arguments and JSON tool results, so a masked card there could not be placed and the request was blocked. Give Akto the flat texts only, as on main, where the mask is applied. * fix(bedrock): list request-field texts last again, as on main Flat texts listed additionalModelRequestFields strings right after the system prompt, so flat-text guardrails got a different order than on main. Akto's live verdict on a benign request depends on that order and started blocking it. Rows already keep their own order and texts that do not follow rows are dropped, so the flat texts go back to main's order. * fix(bedrock): list request-field texts next to their row for row readers Row readers that rebuild texts from rows could swap a masked request field with a masked user turn when the field strings came last. Row readers now get those strings right after the system prompt, where their row sits. Flat-text guardrails keep the end-of-list order. * fix(bedrock): list nested request-field texts in their row's order for row readers Row readers that rebuild texts from rows could write a masked request field onto a sibling key when the field held nested objects, because the texts walked the fields depth first and the row breadth first. Row readers now get those strings in the row's own order. * fix(bedrock): walk request fields breadth first instead of resolving each path Resolving every request-field string by its path revalidated the parent object per string, so a wide additionalModelRequestFields took seconds. The string walk now goes breadth first for row readers, the same order the row lists them, in one pass. A unit test pins that texts and rows map the same way when the vendor masks them differently. * fix(bedrock): fail closed when a vendor's rows drop a mask its texts carry When a vendor returns changed rows whose texts do not line up with them, the rows win. A text the vendor masked on a field its rows left untouched (a tool call argument, say) then reached Bedrock unmasked. Refuse that rewrite instead, unless the edited body holds the changed text elsewhere, as texts rebuilt from the rows do. * fix(bedrock): read same-length texts by position when rows drop a mask Matching a dropped text against any string of the edited body let a row mask on another field with the same token cancel it, so the raw tool call argument still went out. Texts of the length that was sent are now checked position by position, and only texts of another length, which cannot be placed, are matched by value. * test(bedrock): count each rebuilt text against the rows' edits * fix(bedrock): keep PromptGuard on flat texts and refuse texts that cannot be placed PromptGuard rebuilds its returned texts from the user rows, and a vendor that does so while masking only row content let a card in a tool call argument reach Bedrock, which main masked. PromptGuard now gets the flat texts as on main, and a reply whose rows changed next to texts of another length fails closed, since what those texts mask cannot be checked. * test(bedrock): let the structured checks decide the unsafe tool call and missing row cases The replies in these tests carried texts that no longer matched the sent texts, so the new refusal for texts that cannot be placed answered first and the tool call and row count checks went unexercised. * fix(bedrock): send request field strings to row readers after the conversation, where main puts them Row readers got the additionalModelRequestFields row before the conversation. Singulr judges only the latest user message, so an injection in a request field that main blocked reached Bedrock. The row and its texts now come last, as main sends those strings in texts * test(bedrock): pin nested request field order for flat readers and the texts view Both request field tests now use nested fields whose depth-first and breadth-first orders differ, so a flat reader getting the row order or a texts view written in main's order fails * refactor(bedrock): ask each guardrail whether Converse should hand it rows Converse picked the guardrails that get only the flat texts from a list of provider names. CustomGuardrail now has supports_structured_messages, True by default, and the ten classes on that list override it to return False, so the handler asks the guardrail instead of the list. Every guardrail class gets the same answer as before * fix(bedrock): keep Singulr on flat Converse texts as on main Singulr wraps each flat text as a user message and its backend judges only the latest user message, so on main a Converse tool result was the turn it judged. Handed rows, the tool result arrives with role tool and the latest user row is the original question, so an injection in a tool result reached Bedrock. Singulr now returns False from supports_structured_messages * fix(bedrock): keep Qualifire on flat Converse texts as on main Qualifire builds its evaluation input from the last user row only when that row's content is a string, so Converse rows (content-block lists) gave an evaluation_id policy an empty input. Opt it out of structured messages so it gets main's flat texts again. * fix(bedrock): keep Conduct on flat Converse texts as on main The conduct-litellm-guard plugin scans only user rows, so with Converse rows a system prompt, assistant text or tool result left its scan. Opt it out of structured messages so it gets main's flat texts again, all sent as user turns. * fix(bedrock): refuse tool removals that leave a toolConfig Bedrock rejects A guardrail that removed every tool, or the tool toolChoice names, produced an empty tools list or a dangling toolChoice that Bedrock answers with an opaque ValidationException. Refuse those rewrites before Bedrock, like other tool changes this route cannot carry. * fix(bedrock): list request-field strings in main's order for every guardrail The request-field row walked additionalModelRequestFields breadth first, and guardrails that get rows had their texts follow that order. With two or more strings the last text changed, so Ovalix, which blocks only when the last text is flagged, rewrote an injection main blocked. The row now walks the fields in the order main lists them in the texts, and the texts use main's order for every guardrail. * test(bedrock): pin a container's own request-field strings before its nested ones * test(bedrock): pin tool_policy and MCP end-user filtering on Converse tools * fix(bedrock): rebuild Converse tools the guardrail changed and drop blanked text like chat A guardrail that adds, renames, reorders or retypes a tool now gets its list sent to Bedrock, converted the way chat completions converts tools, with cache points kept behind their tool. When every tool is removed, toolConfig is dropped and earlier tool calls and results become text, as chat does. Text blocks a guardrail blanks are dropped, and a turn left empty fails closed. The body copies used to place edits are JSON copies, so deeply nested tool JSON no longer hits the recursion limit. * test(bedrock): keep the client tool schema on Converse masks and refuse deep JSON text drops cleanly * fix(guardrails): keep Airia on flat texts for Bedrock Converse so its redactions still apply * fix(guardrails): a pipeline step whose rewrite the endpoint cannot apply follows on_fail on_error: allow sent the original request on when the endpoint refused a guardrail rewrite it could not apply, so a Converse request the guardrail masked reached Bedrock raw. The refusal now counts as a fail on every endpoint. * fix(guardrails): a pipeline step blocks when its rewrite cannot be applied, whatever on_fail and on_error say A rewrite the endpoint cannot apply is neither a content verdict nor a technical error, so on_fail next or allow and on_error allow must not send the original request on * test(bedrock): trim the Converse guardrail tests Drops the chaos and per-vendor integration cells and the duplicate unit cases, and leaves the existing handler tests as they are on main * fix(bedrock): put Converse request-field strings in the latest user message for row readers A guardrail that judges only the latest user message, like Noma v2, read the additionalModelRequestFields row placed after the conversation, so a request with extended thinking or anthropic_beta sent the user's card to Bedrock and let an injection through. Those strings now follow the latest user message's text in its row, and get a user row of their own only when no user message has text. * fix(bedrock): create toolConfig when a guardrail adds the first Converse tool A guardrail that returned a tool for a Converse request with no toolConfig got a 500, since the tool write-back only rebuilt an existing list. It now creates toolConfig, as chat completions does. Guardrails that get only flat texts have their returned tools ignored, as on main. * test(guardrails): check Converse flat-text opt-outs without patching apply_guardrail Each built-in opt-out is checked through its own supports_structured_messages(), the handler's flat path through an injected scripted guardrail, and Headroom through the real class, which would call its unreachable service if it got rows * fix(guardrails): keep Converse tool matching and write-back linear in request width Matching each returned tool rescanned the whole tool list, and every write-back step revalidated the list or object it passed through, so a request with thousands of tools or request fields held the event loop for seconds. Returned tools are now looked up by name in one index, and path steps read the child directly. * fix(guardrails): let later Converse guardrails scan guardContent text When a guardrail removes every tool on a Converse request that carries guardrailConfig, tool results are rewritten as guardContent blocks. The flat text extractor read only top-level text, so a later masker in the same request never saw them and a card in a tool result reached Bedrock unmasked. Chat completions and /v1/messages keep tool results visible to later guardrails after the tools are removed. The extractor now reads guardContent text the same way the structured rows already do, so flat-text guardrails also scan guardContent text a caller sends. * fix(guardrails): keep caller guardContent out of Converse flat texts and send tool history as plain text Flat-text guardrails read only top-level text blocks again, as on main. A guardContent block after the caller's text no longer becomes the last text, so guardrails that judge the last text (HiddenLayer, Qualifire) check what the caller wrote When a guardrail removes every tool, earlier tool calls and results are now written as plain text blocks instead of guardContent. Later guardrails still scan and mask them, and Bedrock's own guardrail keeps checking every message, since a single guardContent block makes it skip all plain text * fix(guardrails): split a newline-joined Converse message back into its parts A vendor that returns a multi-part message as one newline-joined string now has its masks written back to each part, guardContent included, when no part sent already holds a newline. * fix(bedrock): keep Converse tool history as tool messages until send and match chat on emptied turns When a guardrail removes every tool, the tool calls and results stay tool messages for the guardrails after it, so a scan_only_tool_results guardrail still masks them. The toolConfig with no tool is dropped and the history turned into text only when the request is signed, as chat completions does at request build. A caller's toolConfig with no tool gets the same treatment, where Bedrock returned 400 Request-field strings join the latest user message that has text when a tool-result-only turn follows it, so a guardrail judging only that message reads the caller's text with them A turn the vendor empties entirely is sent as written, as on main and chat completions, instead of a 500 Messages go to the guardrail through a JSON round trip instead of a deep copy, and a reply that changes nothing returns early, so a long Converse conversation guards about as fast as on /v1/messages --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 20:38:14 -07:00
Mateo Wang	75faaeab64	feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) ... * feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response Count endpoints now say where each number came from in litellm_count_source (provider, fallback_model, local_estimate) and name the fallback model in litellm_count_fallback_model. When the provider answers 400 or 404 for a model it cannot count, the proxy counts the same request with token_count_fallback_model. token_count_framing adds a per-model constant to local estimates. Both settings ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info. * test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests * test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing * fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown Moves the cost map lookup off the event loop * ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty * fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too. * test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates * fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock calls and about 200 ms on every count, since Opus 5.5 cannot count there either. The Kimi K3 e2e also runs the large system prompt shape now. * test(count_tokens): use the shared integration port allocator in the fallback wire test * fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today * test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it * fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases. * fix(count_tokens): answer the provider's error whenever a framed estimate raises A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error. * fix(count_tokens): load the custom tokenizer inside the guarded framed estimate A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors. * fix(count_tokens): look up framing under the deployment's custom_llm_provider * fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error A framed deployment whose custom_tokenizer is not cached locally made every count wait out the Hugging Face hub's whole retry cycle when the hub could not be reached, and a burst of those loads held the default thread limiter that every count's cost map lookup also needs. Counts on a worker now share one download per minute, a failed download is reused for the rest of that minute, and after a provider error a count waits at most 5 s before answering the provider's error. The swallowed-estimate warning logs only the error type. * fix(count_tokens): keep one tokenizer download per key running and load the local route per request * test(count_tokens): type the scripted counter and wait on events instead of wall time where possible * fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500 * feat(proxy): count tokens with token_count_model in its pinned region token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable * fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys * fix(proxy): keep the upstream error envelope when adding the region hint * test(proxy): annotate the token count setting echo test * ci(cost map): require non-blank token_count_model strings and test the region rules * fix(count_tokens): answer the token_count_model's error even when framing is set * fix(count_tokens): keep the provider error for Google contents instead of a framed estimate * fix(count_tokens): estimate Google contents the route translated to messages * fix(count_tokens): keep the provider error for Google contents with non-text parts * test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock * fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags * feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer * test(bedrock): keep the Kimi K3 image check in-process * fix(count_tokens): join the region hint to an upstream message that already ends with a period Also pins that a blank token_count_model in config falls through to the cost map default * fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning * fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them A flat Responses API function or custom tool becomes the chat tool the Responses route sends to inference, and every other tool goes to the Converse body as is, so hosted tools Bedrock drops are left out of the count instead of being counted or turning it into an estimate. * test(count_tokens): annotate new token counter test locals with Final * fix(count_tokens): keep shipped count regions inside the deployment's geography A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added * fix(count_tokens): resolve count settings off the event loop	2026-10-17 17:11:27 -07:00
Mateo Wang	52f5a28679	fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it (#47100) ... * fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it * fix(bedrock): retry with the json tool when Bedrock refuses the native output schema Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller * fix(bedrock): make the native output retry send what an opted-out deployment sends The retry after a native output rejection now reruns the same optional param mapping completion() ran, with the deployment model_info opted out of supports_bedrock_native_structured_output, so tool_choice, effort and the json tool schema match an opted-out deployment instead of a hand-built tool. Each config keeps only the predicate that decides when native output came from the opt-in key. Bedrock batch records for Anthropic models map with the opt-in turned off, since a batch job has no retry. Models on supports_native_structured_output keep native output in batch * fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged * fix(bedrock): give sync converse stream errors the upstream body as text * fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising A Converse json_schema request whose caller outputConfig failed litellm's shape check raised a pydantic error before the request was sent, so the caller got a 500 where the merge base passed Bedrock's 400 through. The opt-in check now treats that outputConfig as carrying no native output. Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of native output, and extends the Bedrock native output integration tests. * fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported A caller can send its own outputConfig on Converse. When Bedrock refused that schema on an opt-in deployment, the retry still carried the caller's textFormat, so Bedrock got the same request twice and the warning told the operator to turn off an opt-in that was not the cause. The retry is now skipped when the opted-out request would still carry a native textFormat, so Bedrock's 400 reaches the caller after one call as on the merge base. The opt-out advice is logged only for the account-level "Extra inputs are not permitted" rejection. * fix(bedrock): keep the caller's continuation message on the native output retry The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt. * test(bedrock): pin additional_drop_params on the /v1/messages native output retry * fix(bedrock): give the Converse native output retry the fake stream timeout	2026-10-18 00:06:07 +00:00
Mateo Wang	16645cc963	fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162) ... * fix(router): bound fake-streamed calls by the request timeout, not stream_timeout A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream. * test(timeout): cover stream_timeout on live streams only and the router's timeout split * fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management * fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface A faked stream now takes the longer of timeout and stream_timeout, so a caller who raised stream_timeout to cover a slow fake stream keeps that headroom. Azure o-series dispatch is mirrored when deciding whether a stream is faked. text_completion, /v1/messages and /v1/responses through the Router now get timeout and stream_timeout separately, like chat calls. * fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages * fix(router): report and log the timeout each stream was bounded by Native /v1/messages and /v1/responses now record the timeout they applied, so x-litellm-timeout and the cooldown guard read the real limit instead of the raw request value. A prompt-managed model also passes on only the caller's or its own timeout, so the target deployment's timeout still wins. * fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit * fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses * fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit * test(integration): cover faked, live and converted stream timeout limits per endpoint * test(router): regression tests for faked-stream timeouts	2026-10-17 22:08:02 +00:00
Mateo Wang	82ca10cc2d	feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes (#47040) ... * feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes * fix(bedrock): move Invoke body requestMetadata into the signed header and match Bedrock's ASCII whitespace rule * fix(bedrock): keep Invoke body requestMetadata in the body and answer bad metadata as AWS does Invoke requests whose body already carries requestMetadata keep the resolved pairs in the body instead of the 8,500-character signed header. Malformed caller metadata now answers 400 with the ValidationException error type the AWS SDK expects, and drop_params drops bad pairs on both the pass-through routes and /v1/chat/completions. The caller-pair dedupe is linear, and the chat validator treats \s as ASCII whitespace, as Bedrock does * fix(bedrock): match Bedrock's 400s and header limit on pass-through request metadata Invoke metadata too long for the 8,500-character header now goes in the body, Invoke requestMetadata null answers 400 like Bedrock, Converse type errors carry SerializationException, violation messages name the failing key, and the reserved user_api_key_ prefix is matched case-insensitively after trimming whitespace * fix(bedrock): trim the Invoke metadata header to Bedrock's limit and drop bad requestMetadata under drop_params without the setting * fix(bedrock): leave /bedrock pass-through bodies as sent when request metadata forwarding is off Global drop_params no longer rewrites requestMetadata on the pass-through routes unless bedrock_request_metadata_fields is set. Chat keeps dropping bad pairs under drop_params. Adds a tests/integration wire test for the four pass-through actions and removes an unused parameter from the Invoke header helper. * test(bedrock): drive the pass-through metadata wire test with boto3's bearer key env var * fix(bedrock): trim the pass-through invoke metadata header and stop Nova invoke from checking metadata it drops * test(bedrock): cover pass-through metadata header trimming, malformed caller metadata, and chat drop_params on the wire * fix(bedrock): match requestMetadata keys and values against the whole string so a trailing newline cannot pass the 256-character limit	2026-10-17 14:01:50 +00:00
yucheng-berri	9f115be88d	fix(bedrock)!: scan Claude invoke-with-response-stream passthrough output with post_call guardrails (#46524) ... * fix(bedrock): scan Claude invoke-with-response-stream passthrough frames with post_call guardrails (LIT-9618) Invoke streams of Claude models now go through the same frame-by-frame relay as converse-stream. Each chunk frame's Anthropic event is scanned by the /v1/messages stream translation, a block ends the stream with an AWS exception frame, and a held stream gets its masks written back into re-encoded chunk frames. Claude is told by its model id or, behind an application inference profile, by the anthropic_version its request body carries. Streams of other model families stay as they were * test(bedrock): cover invoke stream vendor outage, upstream exception and release on scan (LIT-9618) * fix(bedrock): hold Claude invoke streams for guardrails the relay cannot run and mark each scan as streaming (LIT-9618) A guardrail released on scan or run in parallel left a Claude invoke-with-response-stream unscanned, while converse-stream holds the whole stream and scans it once. Claude invoke streams now take the same buffered path, with the rewrites written back into the chunk frames that carried each text. Each invoke scan also sends stream true and the events relayed so far as responses, as /v1/messages does, so a vendor can tie the scans of one streamed reply together. * refactor(bedrock): mark the request dicts the buffered stream paths hand to post_call hooks (LIT-9618) * fix(bedrock): scan a Claude invoke stream the upstream breaks off as a finished stream (LIT-9618) An invoke stream that ended on an upstream exception, without a message_delta stop_reason, was scanned as still in flight, so its tool input never reached the guardrail and a held stream released it unscanned. The scan and its scan key now close such a stream the way converse-stream treats the end of its upstream. * test(bedrock): pass the converse-stream endpoint where the tool call without input test frames its stream (LIT-9618) * test(bedrock): expect the ValidationException main now returns when a guardrail blocks a Claude invoke stream before its first frame (LIT-9618) * test(bedrock): type the Claude invoke event helpers in the passthrough guardrail tests * test(bedrock): type the remaining parameters of the Claude invoke stream tests * refactor(bedrock): bind the buffered invoke rewrite in its condition --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 01:17:19 -07:00