mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
66 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yucheng-berri
|
efc2304b7e
|
fix(streaming)!: fail a stream that ends without a finish reason (#47665)
* fix(streaming): fail on clean EOF without finish reason * fix(streaming): cover shared OpenAI-compatible EOF paths * test(router): complete mocked streaming responses * fix(streaming): keep the EOF error message and cover Anthropic clean EOF The request-less RemoteProtocolError tripped exception mapping into 'The .request property has not been set.' for non-OpenAI providers. Adds Anthropic and generic-chunk unit tests, an integration test across chat, the /v1/messages bridge and the /v1/responses bridge, and moves main's token-estimate test onto a finished stream with a sibling that checks the cut stream still bills the estimate on its failure log. * fix(streaming): count a finish reason any chunk sent, keep LangGraph async streams whole Gemini marks every chunk unfinished and a later usage chunk reset intermittent_finish_reason, so a finished Gemini stream failed at EOF. LangGraph's __aiter__ opened a new aiter_lines on every wrapper call, which hit StreamConsumed after the first chunk and now surfaced as a false EOF error instead of a silent truncation. * fix(streaming): keep the synthetic stop for a stream that sent no chunks Gemini's OpenAI-compatible endpoint answers with only [DONE] when reasoning uses up max_tokens. Failing every openai/azure stream that sent nothing turned that 200 into a 500, while every other provider already kept the stop for an empty stream. Adds integration cells for Gemini's trailing usage chunk, multi-event LangGraph runs, a mocked deployment and the [DONE]-only answer. * test(streaming): keep pytest.raises to one statement, drop the spend read for an empty stream An empty stream writes no spend row on main either, so the [DONE]-only cell checks only what the client sees. * fix(streaming): read the chunk's finish reason for the sticky flag * fix(agentcore): end a stream on messageStop when the runtime sends no metadata or final message * test(streaming): pin the logged-chunks guard, inject the LangGraph client, use current model names * fix(agentcore): say why the end-of-stream flags rebind * test(streaming): cover an AgentCore runtime that ends on messageStop through the proxy * test(streaming): cover SDK clients, request fallbacks and a concurrent burst for a cut stream * test(agentcore): inject the async transport instead of patching the aiohttp flag * test(guardrails): end the decision model chat fixture with a finish reason, as a real stream does * test(streaming): shape the AgentCore and LangGraph fixtures like captures from a real runtime and server * test(streaming): keep each stream's ending and land every spend row once through a spend log outage * test(streaming): end each stream by its provider's finish after a proxy worker is killed mid burst * test(streaming): run the outage test as one burst before and one during the outage instead of a capped wave loop * test(streaming): connect the outage test's proxy to the relay without TLS so its trigger sees the spend-log insert on an ssl=on Postgres * test(streaming): type the new stream EOF unit tests * test(streaming): a merged reasoning stream cut mid reasoning now fails Main's think tag test expected a stream that ends mid reasoning without a finish reason to finish with a closed think block. With this PR that stream fails, so the case moves to its own test that expects MidStreamFallbackError with the open think block as the partial content --------- Co-authored-by: King Star <mcxin.y@gmail.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Vineeth Sai Varikuntla
|
a7b66dd8b9
|
fix(openai): stop stripping cache_control from the caller's own messages and tools (#35602)
* fix(openai): stop stripping cache_control from the caller's messages Copy before removing the field, so the caller's own message objects are left alone. * fix(fireworks_ai): strip cache_control on a copy so a fallback keeps the caller's hints * fix: keep the caller's request intact on more providers and trim extra Claude cache hints under drop_params * fix: drop only the oldest extra cache hints inside one message when trimming to the Claude limit * fix: keep replayed session turns yielding their hints as whole turns while a request's own message keeps its newest hints * fix(cache_control): count only hints that reach Claude and cap hints on OpenAI-compatible Claude routes under drop_params * fix(cache_control): keep the capped Claude messages a list in OpenAI-compatible request bodies * fix(caching): cap Claude hints for string drop_params, a Claude base_model, and Converse tool messages The 4-hint cap now reads drop_params through drops_unsupported_params on chat, Converse, and Anthropic routes, names Claude by base_model on OpenAI-compatible routes, and counts a Converse tool message's hints as the one cache point it sends. * fix(caching): trim Claude cache hints in one wire-order pass and reserve a top-level cache_control on OpenAI-compatible routes Dropping the oldest hint one at a time re-walked the whole request per hint, which took seconds on long tool histories. The cap now drops each message's oldest hints in the order the provider counts them, so a string assistant turn loses its text hint before its tool call hints. OpenAI-compatible Claude routes also keep room for a request-level or extra_body cache_control under drop_params. * test: drop narrating comments and stacked generator clauses from cache_control tests * fix(vertex_ai): put every system message in the Gemini context cache so generateContent never sends it beside cachedContent --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
5d5e3685c1
|
fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths (#48933)
* fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths
* fix(logging): keep guardrail and token-count reads of a request from logging drop warnings
Only the real Anthropic-messages adapter request reports a dropped thinking budget. Guardrail,
shadow-eval and token-count translations of the same request now drop it silently, so one request
logs one warning. The 422 retry also no longer hashes each loc entry, so a list inside loc cannot
raise TypeError.
* fix(bedrock): keep the DeepSeek R1 Converse reasoning_effort drop at debug level
Only allowed_openai_params reaches that branch, and it puts reasoning_effort back after the provider mapping, so a warning there claimed a drop that never happened
* fix(logging): warn on Gemini non-integer budget drops and keep the auto-router baseline silent
Gemini 2.x now logs the drop of a non-integer thinking budget. The auto-router
savings baseline builds a body it never sends, so it no longer warns. Logged
values are capped with reprlib, Azure AI marks only its drop_params retry as
gated, and Bedrock Chat Completions names the model without its route prefix.
* fix(logging): warn on the Gemini 3, Magistral, Ollama, Fireworks and DeepSeek ARN reasoning drops
Gemini 3 and later now log a dropped thinking budget, Magistral logs the
level it swaps for a reasoning prompt, Ollama logs a level it maps to
think off, Fireworks logs chat_template_kwargs effort keys dropped for a
model the map does not list as reasoning, and DeepSeek on Bedrock logs a
drop when the model is addressed by an ARN that advertises the param.
* fix(logging): warn on the remaining Ollama, DeepSeek and Tencent reasoning drops
The ollama/ completion route now shares the chat route's think mapping, so
a level it turns off logs the same warning. DeepSeek warns when it drops
thinking.budget_tokens, and Tencent warns when an adaptive-only model gets a
non-integer budget. Magistral no longer warns on low, medium, high or
enabled thinking, the values its reasoning prompt stands in for. The Gemini 3
budget warning no longer claims the model uses its default level.
* fix(reasoning): warn when Claude drops a non-string reasoning_effort
Claude chat and native /v1/messages dropped a reasoning_effort like 5 or
{"effort": 5} with no log line. Both now log one warn-only WARNING. A
Magistral thinking.type sent as a list no longer raises TypeError.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Siddhant Badola
|
b71491c6c9
|
fix(bedrock): route a Bedrock model id with no Invoke provider to Converse (#43290)
* fix(bedrock): route unmapped converse-only family model ids to converse #42713 stopped Bedrock-native OpenAI ids that are not in the cost map yet from falling to the invoke route. The same gap exists for the other families that have no invoke transformation at all: minimax, nvidia, writer, zai, google and xai. An id from one of them that is newer than the cost map hits get_bedrock_route -> "invoke" -> get_bedrock_invoke_provider() -> None -> BedrockError, when Converse would have served it Add a second family regex for those six and default them to converse, the same way the openai family is handled. _OPENAI_FAMILY_MODEL_RE is left alone because the prompt-caching rule is openai-specific Every shipped id of these families already resolves to a converse entry via its base model (checked against the cost map), so this only changes behaviour for ids that are not in the map yet Tests: six unmapped ids route to converse, and an unmapped meta.* id still routes to invoke since meta has an invoke handler * fix(bedrock): route a model id with no Invoke provider to Converse A bedrock/<model> id missing from the cost map fell to the Invoke route, where an id that names no Invoke provider (a new vendor's model, a bare provisioned-model, imported-model, or custom-model-deployment ARN) fails locally with "Unknown provider=None" before any request is sent. Such ids now go to Converse, which replaces the per-family regex with the general rule. Ids that name an Invoke provider and explicit route prefixes are unchanged. * test(bedrock): expect Converse for ids with no Invoke provider in the nova ARN and not-found tests * fix(bedrock): accept tools on Converse for a model with no cost-map row or Invoke provider, and keep a 404 for an invalid id * fix(bedrock): answer 404 for an invalid model id instead of AWS's 400 AWS answers a mistyped Bedrock model id with a 400 ValidationException. The mapping already raised NotFoundError, but passing AWS's 400 response through made the openai base class report 400, so the proxy answered 400 and the router skipped its cooldown * fix(bedrock): drop lookaround regexes from tool schemas for a model the cost map does not list * fix(bedrock): apply the unknown-model defaults only to bare ids LiteLLM knows nothing about A bare Bedrock id with no cost-map row and no Invoke vendor gets Converse's open params, its reasoning_effort as sent, lookaround-free tool schemas, and a 404 for AWS's invalid-identifier reply, which keeps AWS's response headers. An explicit route like converse/, an ARN, or an id LiteLLM recognizes keeps main's params, regexes and error status, so drop_params still drops tools there * fix(bedrock): judge lookaround support by model id on every route, ignore price-only rows The regex check now strips route and region prefixes from the model and its base_model before asking whether LiteLLM knows the model, so bedrock/converse/<id> gets the same answer in a direct transform call as on the completion path. A cost-map row counts only when it names a provider, so a price-only register_model row leaves an unlisted id unknown, as on the proxy. AmazonInvokeConfig now inherits the Invoke provider lookup instead of keeping an identical copy. * fix(bedrock): treat a region path like the bare id when the cost map does not list it A deployment like bedrock/us-east-1/<unlisted id> now gets the same tools, reasoning_effort and typo 404 as bedrock/<unlisted id>, so Claude Code works on it and a mistyped one still answers 404. * fix(bedrock): keep today's 404 for ids that reach Converse only by default, and keep an empty id local * fix(bedrock): answer 404 for an invalid id only when Converse rejected it, and keep a region path with no id local * fix(bedrock): keep a path-shaped id local, and map /v1/completions errors with the model's provider An id with a slash that is not an ARN, like bedrock/../../converse or an unknown region path, keeps Invoke and its local 404 instead of reaching AWS Converse. atext_completion mapped every error as OpenAI, so a Bedrock typo on /v1/completions answered 400 while chat answered 404. It now resolves the provider from the request's model, as the other async entry points do. * fix(text completion): keep /v1/completions error mapping as it is on main Mapping async /v1/completions errors with each provider's rules is wider than this Bedrock routing change. It moves classes, statuses, cooldown and fallbacks for most providers, and some lose context-window fallbacks. So this restores main's mapping here and leaves that change to its own PR. A Bedrock typo on /v1/completions answers AWS's 400, while chat keeps the 404. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tavren27
|
b25ea48c4b
|
fix(anthropic)!: drop beta-gated body fields under drop_params (#48793)
* fix(anthropic): strip beta-gated body fields when a leg drops the anthropic-beta When a provider leg drops an anthropic-beta value (the chat handler's allowlist filter, /v1/messages' filtered headers, Bedrock Converse's additionalModelRequestFields), also drop the body fields that beta gates (thinking.block_binding, thinking.display=updates, mid-conversation system clear_at, output_config, tool_addition/tool_removal blocks) so the request degrades on that leg instead of failing with a provider 400. Copy-on-write so a fallback leg still sends the full request, and each strip is logged at WARNING. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove a leg that drops a beta sends neither the beta nor clear_at Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): strip beta-gated fields against the final body anthropic_beta list Converse counts the body anthropic_beta list as active, Invoke strips after its auto-injected betas are known and injects thinking-binding-controls for thinking.block_binding, test helpers are fully typed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated system fields before an older model's system turn becomes a user turn Models without supports_mid_conversation_system send no mid-conversation beta, so tool_addition and tool_removal blocks are dropped (and logged) before the turn is converted instead of riding along as user content. The Invoke output_config regression test reads the real cost map instead of patching _bedrock_model_supports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): pin the integration proxy to the bundled beta allowlist and type the helper locals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): annotate every new test local with Final and type the fixtures and returns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated body fields only under drop_params Without drop_params the request now goes out as sent and the provider answers its own 400, like any other unsupported param. inline-tools-2026-09-15 also unlocks tool_addition and tool_removal blocks, so a leg that carries it keeps them. * fix(anthropic): strip beta-gated fields on Foundry chat and Bedrock chat Invoke under drop_params Foundry chat and Bedrock chat Invoke now drop beta-gated body fields the same way the other legs do when drop_params is on and the request does not send the gating beta. The mid-conversation system normalizer validates only on the conversion path, and the strip WARNING names the beta the caller can send to keep the field. * fix(anthropic): send the thinking betas for block_binding and display updates, strip Databricks chat thinking Every Claude leg now adds the thinking-binding-controls beta when thinking carries block_binding and the display-updates beta for display updates, so those fields go through instead of a 400. Databricks chat rejects both fields with or without the beta, so it drops them under drop_params. A system turn the strip empties stays when it still carries output_config, and the drop warning names both causes. * fix(anthropic): send the thinking betas on every leg and keep fields where the beta is unknown Vertex messages and Claude Platform chat now add the thinking betas when thinking carries block_binding or display updates. Converse sends only the caller's header betas when it has them. A provider with no row in the beta config forwards beta-gated fields instead of dropping them, an emptied trailing system turn that would leave an assistant prefill is sent as is, and the prompt cache baseline body matches the body that goes out. The drop WARNING now names the beta that unlocks each field * fix(anthropic): filter Converse body betas, keep Foundry auto betas, read the bundled beta config for row checks * fix(anthropic): validate messages only when a beta-gated drop runs A non-object messages entry on a Claude model without mid-conversation system support raised a raw pydantic ValidationError before the beta-config and drop_params checks. The strip now passes the messages through untouched unless it drops something, so the provider's 400 reaches SDK callers as BadRequestError again * fix(anthropic): keep system blocks with a non-string type instead of raising --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
0cf2bb7296
|
fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params (#47761)
* fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params A GPT-5 request whose reasoning_effort the model map does not enable for that model lost the effort with no log line under drop_params. An Azure deployment name missing from the cost map lost reasoning_effort none this way and ran at the model's default effort. The gate now logs a warning naming the model, the level, the map entry it checked, and the model_info.base_model and model_info.reasoning_effort_levels fixes. * fix(openai): warn when the Responses route or a deployment's declared levels drop a reasoning effort The /v1/responses gate dropped a row-disabled GPT-5 effort, and reasoning for an OpenAI model without supports_reasoning, with no log line, and the router logged a declared-level drop only at INFO. Both Responses drops now log a WARNING and the router logs its drop at WARNING. The warning now leads with model_info.reasoning_effort_levels, since base_model only helps a deployment named differently from its underlying model. * fix: warn only when a reasoning level is actually dropped, and on generic reasoning_effort drops * fix: warn when thinking or a Bedrock Converse reasoning level is dropped, and name the router drop a reasoning level * fix: skip the drop warning when a non-reasoning model is asked for no reasoning, and honor declared levels on Bedrock Responses * fix: stay silent when reasoning is turned off on /v1/responses and warn on the Bedrock Responses fallback drop * fix(reasoning): warn when any reasoning-off request is dropped and give each route only the advice it honors A request to turn reasoning off on a model name the catalog lacks now warns on every route, since it may be a reasoning model. Chat suggests model_info.reasoning_effort_levels and, for a name with no catalog entry, model_info.base_model. OpenAI Responses suggests declared levels only where the config honors them, and Bedrock suggests neither. The Bedrock and Mantle Responses forwarding of declared levels is reverted until it can be proven live. * fix(reasoning): suggest reasoning_effort_levels and base_model only where the route reads them The chat gate suggested model_info.reasoning_effort_levels on routes that never pass declared levels (github_copilot, chatgpt and other OpenAIConfig delegates) and model_info.base_model on OpenAI and Azure AI chat, whose gate looks up the deployment name rather than base_model. /v1/responses/compact suggested levels it never reads either. Declared levels are now None when a caller passes none, so the advice appears only when a caller fed them, and the base_model advice is limited to the Azure chat gate, the one that looks up base_model. The openai row of the chat warning test now expects no base_model advice, the Responses warning test goes through get_optional_params_responses_api the way /v1/responses and compact call it, and the Azure direct-config test passes declared levels the way get_optional_params does. Long test lines are wrapped. * fix: read only the built-in catalog when choosing the reasoning drop warning |
||
|
rj-cairn
|
f89b9c6150
|
refactor(types): replace Any with proven types in 28 files (#48903)
* refactor(types): replace Any with proven types in 52 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): use typing_extensions.TypedDict for validated seams Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): revert strict Anthropic response validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): accept a null finish reason from Bedrock Cohere and TwelveLabs Pegasus * test(bedrock): annotate the Pegasus null finish reason test --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
m6atlas
|
a0c490ea54
|
fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params (#48733)
* fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params * test(bedrock): cover the strict tools 400 retry under drop_params * fix(bedrock): anchor the strict tools rejection match and strip toolSpec strict * test(bedrock): return captured requests from the strict tools test helper * fix(bedrock): retry without strict only when a tool sent strict to Bedrock A strict-limit 400 on a request whose tools carry strict: false, or on a model litellm never sends strict to, was sent again with an identical body. The fallback now takes the model and returns nothing unless a tool puts strict on the wire. Also types the fallback test parameter and records the mock transport's requests on the recorder instead of a mutated list * fix(bedrock): decide the strict retry on the profile's base model and retry grammar timeouts An application inference profile with base_model sends strict on the wire, so the retry check now resolves the capability model the way the request transformation does. Bedrock's "Grammar compilation timed out" 400 is a strict-grammar rejection too, so it takes the same single retry under drop_params. * test(bedrock): annotate the strict-tools retry test helper's values as Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
0b66a27528
|
feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost (#48278)
* feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bill native Bedrock guardrail trace usage on chat, responses and pass-through Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): record the Bedrock guardrail trace of a streamed metadata chunk on the logging object Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(bedrock): apply guardrailConfig on /v1/messages and price its trace, record the stream trace on both builder paths, redact the native guardrail_response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): validate the guardrail trace with one pydantic model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): record the native guardrail entry when litellm_params metadata is None Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): import the renamed messages invoke transformation module Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): skip native guardrail billing on cache hits and redact the trace with message logging off Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): walk the guardrail trace iteratively when redacting it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): record the native guardrail trace where the Bedrock response is parsed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): type the native guardrail trace fixtures and locals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
7e14638001
|
refactor(llms): rename model-version transformation files to family names (#48509)
gpt_5_transformation.py becomes gpt_reasoning_transformation.py (OpenAI and Azure), the Bedrock anthropic_claude3_transformation.py files become anthropic_claude_transformation.py, anthropic_claude2_transformation.py becomes anthropic_legacy_text_completions_transformation.py, and vertex_ai_partner_models/llama3 becomes vertex_ai_partner_models/llama. Class and symbol names are unchanged. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
66c8f5b1a9
|
fix(llms): join api_base and route with one shared helper (#47535)
* fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * fix(llms): join api_base and route with one shared helper A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base * refactor(llms): move per-provider api_base strips onto join_api_base Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join * ci: run the trailing-slash URL builder test in the llms shard * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell * fix(llms): send every provider route through join_api_base Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash * fix(llms): keep the route skip only where a builder already had it join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path already ends in it, and it is used only where the merge base or the trailing-slash test PR head already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the path so a host named anthropic-* builds /anthropic/v1/messages * fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL * test(llms): run the Gemini RAG trailing-slash case through aingest The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears * test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base * test(integration): cover api_base joins across providers, query strings and named hosts * feat(url_utils): add join_api_base and ensure_api_route helpers * fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host join_api_base and ensure_api_route now keep the // of a base left with only a scheme, so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it * fix(azure_ai): build the count_tokens URL from the messages URL * fix(azure_ai): match anthropic as a whole path segment in the messages URL * fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints Azure AI chat and messages go back to cutting the base at the first /anthropic, as on main, and count_tokens keeps its path check, so a gateway path like /anthropic-proxy builds the same URLs as main. A base that already ends in /v1/messages still gets count_tokens next to it. A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is used as its text again in AgentCore, Bedrock responses and the RAGFlow base, instead of falling back to the default host. get_volcengine_base_url and GigaChat's get_api_base strip a trailing slash again for outside callers. The cache-hit wire test is skipped as a known proxy bug, since one cache hit writes two cache_hit spend rows. * fix(llms): drop a slash before the api_base query and tighten the route test checks * test(llms): annotate new test locals as Final and flatten the builder generator * fix(url_utils): keep a scheme-only api_base hostless so it fails before sending --------- Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com> |
||
|
Priyansh Nandwana
|
c6fc5b01bb
|
fix(bedrock): send batch jobs to a configured control-plane endpoint (#47260)
* fix(bedrock): send batch jobs to a configured control-plane endpoint
Bedrock batch create, retrieve and cancel always built the regional
bedrock.{region} host, so a VPC endpoint or compatible gateway could not
be used. Add aws_bedrock_control_plane_endpoint as a deployment setting
read by the signed HTTP path and the boto3 path, carry it through the
proxy credential filter, and refuse it in request bodies like the other
endpoint overrides. api_base stays out of it because on a Bedrock
deployment it points at the runtime host.
Fixes #45056
* fix(bedrock): read the control-plane endpoint from typed kwargs in job status
The named parameter added one unknown-argument error in litellm/batches/main.py,
which unpacks untyped kwargs into this function
* docs(bedrock): explain batch control-plane endpoint configuration
* fix(bedrock): keep the batch control-plane endpoint out of chat request bodies
A deployment that sets aws_bedrock_control_plane_endpoint also serves chat,
and the unregistered key was swept into the Bedrock request body, so every
chat call on it failed with "Extra inputs are not permitted". Register it
with the other Bedrock batch connection fields so it stays a LiteLLM-owned
kwarg on every route while batches still read it from litellm_params.
* test(bedrock): verify endpoint signatures without internal mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
dd20bb0ae9
|
fix(bedrock): apply the 16 maxTokens floor to Kimi K3 on Converse (#43875)
* fix(bedrock): apply the 16 maxTokens floor to Kimi K3 on Converse Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): use lane AWS credentials for the Kimi K3 messages test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): bypass the proxy cache and declare test metadata for the Kimi K3 probe --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
270d108ace
|
fix(anthropic): json_schema envelope edge cases on the Anthropic json tool path (#47478)
* test(anthropic): cover json_schema envelope edge cases on the Anthropic json tool path * fix(anthropic): unwrap required-only keys, walk past tool-named envelopes, keep non-ASCII text, flush cut streams, check Vertex output_format * fix(anthropic): send held json tool text on cut Bedrock Invoke streams and drop a non-object output_format under drop_params A Bedrock Invoke stream that ends before the json tool block closes now sends the held text and closing brace, as the direct Anthropic stream already did. A non-object output_format on the Anthropic messages route and its providers is dropped under drop_params and otherwise forwarded, so the provider answers its own 400. Vertex Claude names the proxy setting in its 400 and reads drop_params through the shared helper. Adds integration cases for the json_schema envelope and the cut stream. * fix(anthropic): drop a non-object output_format on chat, unwrap a whole envelope on a cut stream, escape lone surrogates Chat completions to Anthropic, Azure AI, Bedrock Invoke and Vertex now drop a non-object output_format under drop_params, matching /v1/messages. A stream cut after a complete held envelope is unwrapped like a normal end. A json_schema answer holding a lone surrogate keeps its \u escape so the response can still be encoded * fix(anthropic): join a split answer when the stream ends in its last block A cut during the second json tool block now runs the held block through the same flush as a block stop, so it joins the streamed keys or the earlier split blocks instead of sending two objects back to back. The Vertex output_format fold drops its unreachable drop_params branch, since both parent configs drop a non-object value first. * test(vertex): expect the LiteLLM 400 for a non-object output_format on /v1/messages |
||
|
joshua-berri
|
3124797399
|
fix(bedrock): resolve profile pricing before applying custom rates (#47419)
* fix(bedrock): price Converse calls using the invoked model profile * fix(pricing): retain registered rates across metadata updates * fix(pricing): honor temporary registered prices for Bedrock profiles * fix(bedrock): record invocation model ID for each completion attempt * test(bedrock): recalculate fallback cost after metadata updates * fix(pricing): preserve cache-only registered price overrides * fix(bedrock): preserve priced overrides without widening shared pricing * fix(bedrock): retain explicit pricing through token cost lookup * fix(bedrock): layer cache overrides over invoked profile rates * fix(bedrock): keep effective prices consistent across billing * fix(bedrock): share profile pricing and custom rate resolution * fix(router): remove unused pricing suppression * fix(cost): preserve rates across mixed token thresholds * fix(router): share deployment pricing during startup * fix(bedrock): preserve Router pricing inheritance for profiles * fix(bedrock): retain configured pricing for unmapped profiles --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
52f5a28679
|
fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it (#47100)
* fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it
* fix(bedrock): retry with the json tool when Bedrock refuses the native output schema
Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller
* fix(bedrock): make the native output retry send what an opted-out deployment sends
The retry after a native output rejection now reruns the same optional param
mapping completion() ran, with the deployment model_info opted out of
supports_bedrock_native_structured_output, so tool_choice, effort and the json
tool schema match an opted-out deployment instead of a hand-built tool. Each
config keeps only the predicate that decides when native output came from the
opt-in key.
Bedrock batch records for Anthropic models map with the opt-in turned off,
since a batch job has no retry. Models on supports_native_structured_output
keep native output in batch
* fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged
* fix(bedrock): give sync converse stream errors the upstream body as text
* fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising
A Converse json_schema request whose caller outputConfig failed litellm's
shape check raised a pydantic error before the request was sent, so the
caller got a 500 where the merge base passed Bedrock's 400 through. The
opt-in check now treats that outputConfig as carrying no native output.
Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of
native output, and extends the Bedrock native output integration tests.
* fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported
A caller can send its own outputConfig on Converse. When Bedrock refused that
schema on an opt-in deployment, the retry still carried the caller's textFormat,
so Bedrock got the same request twice and the warning told the operator to turn
off an opt-in that was not the cause. The retry is now skipped when the opted-out
request would still carry a native textFormat, so Bedrock's 400 reaches the
caller after one call as on the merge base. The opt-out advice is logged only for
the account-level "Extra inputs are not permitted" rejection.
* fix(bedrock): keep the caller's continuation message on the native output retry
The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt.
* test(bedrock): pin additional_drop_params on the /v1/messages native output retry
* fix(bedrock): give the Converse native output retry the fake stream timeout
|
||
|
Mateo Wang
|
16645cc963
|
fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162)
* fix(router): bound fake-streamed calls by the request timeout, not stream_timeout A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream. * test(timeout): cover stream_timeout on live streams only and the router's timeout split * fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management * fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface A faked stream now takes the longer of timeout and stream_timeout, so a caller who raised stream_timeout to cover a slow fake stream keeps that headroom. Azure o-series dispatch is mirrored when deciding whether a stream is faked. text_completion, /v1/messages and /v1/responses through the Router now get timeout and stream_timeout separately, like chat calls. * fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages * fix(router): report and log the timeout each stream was bounded by Native /v1/messages and /v1/responses now record the timeout they applied, so x-litellm-timeout and the cooldown guard read the real limit instead of the raw request value. A prompt-managed model also passes on only the caller's or its own timeout, so the target deployment's timeout still wins. * fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit * fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses * fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit * test(integration): cover faked, live and converted stream timeout limits per endpoint * test(router): regression tests for faked-stream timeouts |
||
|
Mateo Wang
|
82ca10cc2d
|
feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes (#47040)
* feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes * fix(bedrock): move Invoke body requestMetadata into the signed header and match Bedrock's ASCII whitespace rule * fix(bedrock): keep Invoke body requestMetadata in the body and answer bad metadata as AWS does Invoke requests whose body already carries requestMetadata keep the resolved pairs in the body instead of the 8,500-character signed header. Malformed caller metadata now answers 400 with the ValidationException error type the AWS SDK expects, and drop_params drops bad pairs on both the pass-through routes and /v1/chat/completions. The caller-pair dedupe is linear, and the chat validator treats \s as ASCII whitespace, as Bedrock does * fix(bedrock): match Bedrock's 400s and header limit on pass-through request metadata Invoke metadata too long for the 8,500-character header now goes in the body, Invoke requestMetadata null answers 400 like Bedrock, Converse type errors carry SerializationException, violation messages name the failing key, and the reserved user_api_key_ prefix is matched case-insensitively after trimming whitespace * fix(bedrock): trim the Invoke metadata header to Bedrock's limit and drop bad requestMetadata under drop_params without the setting * fix(bedrock): leave /bedrock pass-through bodies as sent when request metadata forwarding is off Global drop_params no longer rewrites requestMetadata on the pass-through routes unless bedrock_request_metadata_fields is set. Chat keeps dropping bad pairs under drop_params. Adds a tests/integration wire test for the four pass-through actions and removes an unused parameter from the Invoke header helper. * test(bedrock): drive the pass-through metadata wire test with boto3's bearer key env var * fix(bedrock): trim the pass-through invoke metadata header and stop Nova invoke from checking metadata it drops * test(bedrock): cover pass-through metadata header trimming, malformed caller metadata, and chat drop_params on the wire * fix(bedrock): match requestMetadata keys and values against the whole string so a trailing newline cannot pass the 256-character limit |
||
|
Mateo Wang
|
2c5c9b8d84
|
fix(bedrock): use native structured output on Haiku 5.5 and Sonnet 5.5, and fix Fable 5 and Sonnet 4.6 schema 400s (#46870)
* fix(bedrock): use native structured output on every Claude model whose Bedrock profiles accept it Bedrock accepts outputConfig structured output on every reachable profile of Claude Opus 4.7, 4.8, 5, 5.5, Sonnet 5, 5.5, Haiku 5.5, and Fable 5.1, so json_schema now goes out natively there instead of as a forced tool, which kept Bedrock from returning adaptive thinking. Fable 5 rejects outputConfig, so its keys go back to the tool path. GovCloud keys keep their flags, and the drift tests let GovCloud lag its commercial base but never lead it. * fix(bedrock): clean Claude schemas for native structured output and keep recursive schemas on the old path Bedrock native output rejects minimum/maximum, minItems, uniqueItems, oneOf, additionalProperties true, and recursive or unresolved $ref, all of which the forced tool accepted. Claude schemas on the Bedrock native paths now go through the cleaner the Anthropic native path already uses, and a schema with a recursive or unresolved $ref keeps the forced tool, inline, or drop path it had. * fix(bedrock): keep older installs on the json tool with a Bedrock-only native output key Older installs fetch the model map from main, so flipping supports_native_structured_output moved them onto native output with their old schema handling. The new supports_bedrock_native_structured_output key is ignored by older code. Schemas native output cannot hold (recursive refs, open objects such as dict fields) keep the json tool on the new key, models on the old key keep their current behavior for those shapes, and the Anthropic API, Vertex AI, and Azure keep their current behavior. * chore(cost-map): regenerate the model prices schema for the Bedrock native output key * fix(bedrock): keep native output to profiles every account accepts and route schemas Bedrock rejects to the json tool Only Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 keep the Bedrock native output key, since Opus 4.8 and Sonnet 5 answer 400 on some accounts and Opus 5 garbles escaped characters. Lookaround patterns and string formats outside Bedrock's list now keep the json tool, objects typed as a union with null are closed, and the cleaner keeps minItems of 0 or 1. * fix(bedrock): keep Opus 5.5 and Fable 5.1 on the json tool, since native output stops their adaptive thinking * fix(bedrock): keep shared $defs in native schemas and send refused schemas to the json tool on every native model Native output now keeps $defs instead of inlining each use, so a submodel reused across a schema counts once against Bedrock's optional and union limits. Schemas native output cannot hold use the json tool on Converse and Invoke chat for both native keys, and minItems is only kept when it is an integer 0 or 1 * fix(bedrock): keep the json tool unless thinking is requested and bound native output schemas The Bedrock native output key now applies only when the call asks for thinking, so calls without thinking keep the json tool as before. Schemas past Bedrock's documented union and optional counts, or past a grammar size estimate fitted to live probes, keep the json tool on both map keys. Open objects keep the json tool on the Anthropic API, Vertex AI and Azure AI, where native output returned them empty. The $ref check no longer expands $defs, so doubling definitions no longer stall the request. * fix(bedrock): resolve root-path $defs refs so nested Pydantic models keep native output on Invoke * test(bedrock): cover native structured output and json tool routing through the proxy Integration cells drive the proxy against a scripted Bedrock peer that enforces Bedrock's structured output schema rules, covering thinking with native output on Converse, Converse streaming, Invoke, /v1/responses and /v1/messages, the deployment opt-in, Fable 5, schema cleaning on every native entry, open objects, and malformed schemas. Vertex, Anthropic and Azure AI cells cover open objects on native-capable models and the cleaner's minItems and nullable object handling. The unit table now covers reasoning_effort none and disabled thinking * fix(bedrock): keep native output for numeric formats and wide old-key schemas, and keep thinking in json mode Check string formats only on nodes that can be strings, apply the 64-field cap only to the Bedrock-only key, carry thinking blocks into the json mode message, and send malformed json tool schemas as object schemas. * fix(anthropic): keep native output for dict fields unless the json tool can be forced * fix(anthropic): keep native output for dict fields when the caller sends tools * test(bedrock): close the dangerous-tool-use test schema, since Bedrock native output requires closed objects * test(anthropic): cover forced and unforced open-object routing, json mode thinking, numeric formats and wide schemas Add proxy cells for open-object schemas that keep native output when the json tool cannot be forced (a model that refuses forced tools on the Anthropic API and Vertex AI, thinking, caller tools), for the thinking block a json mode answer keeps on Vertex AI and Bedrock Invoke, and for integer and number formats and 65-field schemas on each Bedrock map key. Point the forced open-object cells at the Anthropic API and Azure AI only, since no Vertex AI model in the cost map takes both native output and a forced tool * fix(bedrock): keep the native output opt-in on the deployment that sets it, and route effort none open objects to native output * test(bedrock): cover deployment-scoped native output opt-in, register_model and effort none open objects |
||
|
AxelRay
|
388251a5b8
|
fix(bedrock): drop dangerous-tool-use beta without safeguards (#44471)
* fix(bedrock): drop dangerous beta without safeguards * fix(bedrock): drop the dangerous-tool-use beta only next to structured outputs, keep it on Mantle Bedrock rejects the pair with 'invalid beta flag' on Claude models before Opus 4.7, while requests without structured outputs and Bedrock Mantle accept the beta, so the drop now only applies to Invoke requests with output_config.format or a strict tool and no safeguards. * fix(anthropic): keep the AnthropicThinkingParam import that main now uses * fix(bedrock): drop the client's dangerous-tool-use beta whenever safeguards is absent Bedrock rejects the beta with 'invalid beta flag' on Sonnet 4.5, Opus 4.5, and Opus 4.6 even without structured outputs, so dropping it only next to structured outputs left those beta-only requests failing. The beta does nothing without safeguards, so Invoke now drops it in every such request. Mantle keeps it. The thinking display helper goes back to main's version. * fix(bedrock): drop the client's dangerous-tool-use beta on Invoke chat without safeguards * test(bedrock): wire cells for the dangerous-tool-use beta on Invoke messages, chat and responses * test(bedrock): type the new dangerous-tool-use beta tests * test(bedrock_mantle): type the dangerous-tool-use beta Mantle test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
7ddc9f1c96
|
fix(bedrock)!: keep reasoning, thinking blocks and the json_mode finish_reason on fake streams (#46863)
* fix(bedrock): keep reasoning, thinking blocks and the json_mode finish_reason on fake streams A Bedrock Converse model without native structured output fakes the stream when response_format carries a json_schema. The fake stream transformed the response without json_mode and rebuilt a chunk from text, tool_use and token counts only, so it dropped reasoning_content, the signed thinking_blocks and the reasoning token count, and ended on finish_reason tool_calls. The three fake-stream sites now pass json_mode to _transform_response and stream the resulting ModelResponse through the shared convert_model_response_to_streaming converter, which now numbers each streamed thinking block with its index the way real streams do * fix(bedrock): parse text tool calls on Converse fake streams like the non-stream answer * fix(responses): stream the answer text when one chat chunk carries reasoning and the answer * fix(streaming): keep every field on the OpenAI and openai_like fake streams and split multi-block thinking Azure o-series and openai_like fake streams now use the shared MockResponseIterator, so extra tool calls, reasoning and cached and reasoning tokens survive; the Bedrock-local iterator is deleted. The /v1/responses bridge also splits a chunk whose thinking block is signed or redacted with no reasoning text, and the sync iterator closes the reasoning item before the answer opens, like the async one. The /v1/messages bridge gives each thinking block in a chunk its own content block, starts a new block when the thinking block index changes, and streams redacted thinking as redacted_thinking. Snowflake replays drop the stream index from thinking blocks. * fix(anthropic): stream each thinking block's text once on /v1/messages An unsigned thinking chunk now opens its content block with empty text, since its thinking_delta carries the text. A multi-block chunk keeps each piece's thinking block, so an unsigned block followed by a signed one keeps its own text and index. * docs(anthropic): describe which thinking block start carries its text * fix(streaming): close merged reasoning before the answer when one chunk carries both * test(streaming): cover fake-stream reasoning across the proxy bridges and type the Snowflake replay test |
||
|
moyai-devin-berriai[bot]
|
a3251a3bfd
|
fix(responses): preserve provider request IDs in streaming failure logs (#45453)
* fix(responses): preserve provider request IDs in streaming failure logs * fix(bedrock): log the provider request id for in-stream errors on bridged Bedrock streams Bedrock models that run through the chat-completions bridge (Claude, converse, invoke) raise in-stream failures from the event-stream decoder as a BedrockError without headers, so their failure logs kept error_provider_request_id null. The decoder now passes the stream's request-id headers to those errors. Both the native Responses iterator and the decoder pass only the PROVIDER_REQUEST_ID_HEADERS subset, so error responses and failure logs pick up the request id and none of the 200 response's other headers. MidStreamFallbackError gets its response at construction instead of mutating it afterwards. --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
f470e418ea
|
fix(anthropic): keep signed empty thinking blocks and honor thinking.block_binding (#46342)
* fix(anthropic): treat adaptive thinking as enabled when mapping response_format
is_thinking_enabled only recognised thinking.type=enabled, so a request that
paired thinking.type=adaptive with response_format took the synthetic JSON
tool path with a forced tool_choice, which Anthropic rejects alongside
thinking. The same check also gates the Bedrock Converse and Databricks
forced tool_choice downgrades, so they now see adaptive thinking too.
* fix(anthropic): keep signed thinking blocks with empty text on chat completions
Models that default to display: omitted return signed thinking blocks whose
text is empty, and Anthropic rejects the next turn when they are not replayed
unchanged. is_unsignable_thinking_block dropped them along with unsigned
blocks, and the inline content path required non-empty text, so a tool-use
turn on Opus 4.7+ or the 5 families lost its thinking blocks and 400ed on the
tool-result turn. Only a missing signature now drops a block, and the
mid-conversation system placement mirrors that rule. The /v1/messages strip
helper is left to #41362.
* fix(anthropic): skip the thinking-strip retry when block_binding asks for an error
The /v1/messages retry loop strips thinking blocks and resends on any
invalid-thinking 400, including the binding mismatch a caller opted into
with thinking.block_binding.prefix_mismatch_behavior: error. The retry
decision now sees the request body and surfaces that 400 instead of
overriding the caller's choice. #42888 covers the litellm_params side.
* fix(bedrock): carry thinking.block_binding through the Invoke thinking rebuilds
Bedrock Invoke rebuilt the thinking param from scratch when display: updates
was requested, on both the /v1/messages and chat completions paths, and the
legacy enabled-to-adaptive translation plus the clear_thinking injection did
the same, so thinking.block_binding never reached Bedrock. Every rebuild now
spreads the caller's thinking and only overrides the keys it owns, and
AnthropicThinkingParam types block_binding. The thinking-binding-controls
beta was already on the Bedrock allowlist.
* fix(anthropic): keep forcing the response_format tool under adaptive thinking
This reverts
|
||
|
Kerry Lu
|
0d741a9816
|
fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them (#46611)
* fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them * fix: satisfy the type discipline and basedpyright gates for the thinking block index * chore: sync the OpenAPI snapshot and schema.d.ts with the thinking block index field --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
c0663d5202
|
fix(anthropic)!: translate the OpenAI tool_search tool on /v1/chat/completions for Claude models (#46389)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models
* fix(responses): define the tool_search_call output item locally for older openai SDKs
* fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields
Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500
* fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out
A replayed Claude tool search call that the caller answered with a tool message kept its
tool_result in the user turn while its tool_use was dropped from the assistant turn, so
Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when
no tool message answers it.
The Responses bridge now strips defer_loading from every tool whenever no tool search tool is
sent, whether the caller sent none or the route cannot run it, since a deferred tool with no
search tool can never be loaded. Namespace members had started carrying the flag to every
bridge provider, which providers that reject unknown tool keys answered 422.
The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit.
* fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact
A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried
* fix(anthropic): translate the OpenAI tool_search tool on /v1/chat/completions for Claude models
* fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups
With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does
A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result
The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator
* fix(bedrock): answer 400 for the OpenAI tool_search tool on Converse unless drop_params is set
* fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic
A function tool whose allowed_callers is not a list of strings now answers 400
instead of 500. The guardrail scan reads tools without malformed flags, so the
route decides between a 400 and a drop under drop_params. When the caller sends
a native Anthropic search tool beside tool_search, only the native one goes out.
Databricks Claude no longer counts as an Anthropic request route.
A previous_response_id follow-up off Anthropic also drops a search call the
client replayed as a function_call with a srvtoolu_ id, and its output. The
answered tool id scan in the Anthropic message factory runs once per assistant
run instead of once per message. A null created_by on tool_search_call and
tool_search_output output items is left out of the serialized response, so a
cache hit returns the same keys as the first call.
* fix(anthropic): send one search tool and judge tool search on the real model for chat tool_search
Several OpenAI tool_search items now collapse into one tool_search_tool_regex
tool, since Anthropic answers 400 when two tools share a name. A native
Anthropic search tool the caller already sent wins over the OpenAI item.
Bedrock invoke and Vertex AI now decide tool search support on the real model
before tool-based response_format swaps in a stub model name, so a Claude 5.5
request with response_format no longer gets a 400 on a route that runs tool
search.
* fix(anthropic): name the proxy drop_params setting in the tool_search 400
* fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan
The scan translated tool_search into Anthropic's regex search tool on every
route, so a guardrail returning only function tools removed tool_search from
native OpenAI requests and OpenAI answered 400 for the deferred tools
* fix(anthropic): run tool search on azure_ai, gate it on Databricks Claude, and strip deferral without recursion
Anthropic's Foundry docs list no tool search exclusion for either hosting
option, so the claude-tool-search rule now fills azure_ai. Databricks Claude
models route their tools through the same unsupported-tool-search gate, so a
tool_search tool answers 400 or drops under drop_params instead of a 500.
_without_deferral strips two fixed levels instead of walking recursively.
* fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai
* fix(anthropic): keep answered tool search replays paired with their answers
* fix(anthropic): drop Claude tool search calls from history sent to other routes, name the fix in the 400
A second turn that carries Claude's srvtoolu_ search call and its result now has both removed
before completion() sends it to a route that does not build an Anthropic request, including the
responses bridge. The tool search 400 now names model_info supports_tool_search for a model the
cost map does not know, and bedrock/invoke for a Converse model that runs tool search there.
* fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails
* fix(chat): keep answered Claude tool search calls in history sent to other providers
Only a srvtoolu_ search call with no tool result is dropped before a
non-Anthropic route, so an answered pair keeps the caller's shape where
OpenAI and Gemini take it. The history check no longer copies the list
through a TypeAdapter.
* fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object
* fix(chat): drop unanswered Claude tool search calls only on routes that reject them
OpenAI, Azure OpenAI and Bedrock Converse answer 400 for a srvtoolu_ call
with no tool result, while Gemini answers 200. Keep the history as sent
everywhere else so routes that already work see what the caller sent.
* fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent
* test(integration): cover chat tool_search routing, history drop and load on the wire
Audit round 1 of 3 at
|
||
|
ryan-crabbe-berri
|
6c78089d34
|
fix(bedrock): use base_model for Claude capability checks on application inference profile ARNs (#46569) | ||
|
ryan-crabbe-berri
|
e889d9d653
|
fix(bedrock)!: replay redacted_thinking on Converse and keep its data as Claude's own string (#46532)
* fix(bedrock): send redacted_thinking back to Converse as reasoningContent.redactedContent Replaying an assistant turn that carried a redacted_thinking block to Bedrock Converse turned it into an empty unsigned reasoningText, which the assistant content builder then dropped as blank text. Bedrock returned 200 so callers never noticed. Map redacted_thinking to redactedContent in both the thinking_blocks and inline content paths of the sync and async builders * fix(bedrock): drop blank redacted_thinking data and keep non-str signatures signed on Converse replay * fix(bedrock): replay redacted_thinking to Converse only when the target is Claude Nova, Llama, DeepSeek and gpt-oss reject reasoningContent.redactedContent with a 400, so a Claude history replayed to them through a Router fallback or model switch keeps dropping the redacted block the way main does. Claude is resolved from the Bedrock base model of the target, and an application inference profile ARN counts as Claude when its deployment's base_model names a Claude model. Adds integration contracts for the replayed Converse request shape, stream and non-stream, with a tool call, and for non-Claude and ARN targets. * test(e2e): cover Bedrock Converse replaying redacted_thinking live through the proxy Turn 1 triggers redacted thinking on Claude Sonnet 4.6, turn 2 replays it, and a live edge between the proxy and Bedrock checks the replayed Converse body carries the same reasoningContent.redactedContent. Covers nonstream, stream and a tool call round trip * fix(bedrock): validate litellm_params before reading the deployment base_model litellm_params arrives as an untyped dict, so passing it straight into the typed helper grew reportUnknownArgumentType. Validating it into Mapping[str, object] first keeps the read typed without growing the type-check gate * fix(bedrock): wrap non-Converse redacted_thinking data and let a named model id win over base_model * test(bedrock): cover redacted_thinking minted outside Converse, model_id and the sync Router base_model path * fix(bedrock): match bare claude- base_model names without reading the untyped cost map * fix(bedrock): keep redacted_thinking data as Claude's own string on every route Converse hands redactedContent back base64-wrapped, and that wrapper used to leak out as data, so a Converse turn replayed to Invoke or the Anthropic API was rejected. Converse replies now decode it, Converse requests wrap it exactly once, and the Anthropic-shaped builders unwrap histories saved with the old wrapping. Responses API encrypted reasoning is no longer sent to Claude on Converse, which rejected it with a 400 |
||
|
devin-ai-integration[bot]
|
47e4ed88c2
|
fix(anthropic): keep refusal stop_reason and stop_details on the logged response (#46104)
* fix(anthropic): keep refusal stop_reason and stop_details on the logged response * fix(anthropic): keep the refusal stop-fields boundary tolerant of a non-dict upstream body * test(anthropic): use Anthropic's documented refusal example as the unit-test fixture * fix(streaming): keep provider fields on async stream cache hits and received_model_id on synthesized finish chunks * fix(caching): replay cached completion_tokens_details on stream cache hits and skip non-object cached provider fields * test(integration): cover Anthropic-family refusal stop_details on the wire, the cache and under chaos * fix(caching): skip a cached completion_tokens_details that fails validation on stream replay * fix(anthropic): keep refusal stop fields on a JSON-mode non-stream response Also adds the round 3 audit cells for JSON-mode refusals, the proxy response serialization, and wrong-typed cached usage details on both stream replays * test(anthropic): expect content_filter on streamed JSON tool refusals after main's stream change --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
83980c9d63
|
fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493)
* fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): keep only the raising call inside pytest.raises for the pre-content stream tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): yield the primed first chunk from a helper generator so the stream iterators keep their complexity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): expect the empty invoke stream to be reported from router.completion now that the first chunk is read there Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: give router stream mocks a first chunk or an ending so the eager first read neither hangs nor retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): name the first-chunk helpers after what they do (_read_first_chunk, _FirstChunk, _chunks_with_first) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): drop the string annotation on _FirstChunk.chunk, ModelResponseStream is imported at runtime --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
457b5d26c7
|
fix(anthropic): stream json_schema answers as content on Bedrock Mantle and Claude Platform, and join split answers (#46331)
* fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude
Vertex AI Claude deployments that emulate response_format through a forced
tool sometimes nest the schema object under an extra key such as
`parameters` or `input`, and that envelope reached message.content with a
200. The converter now unwraps exactly one nested object that matches the
schema, leaving a payload that is itself schema-shaped alone.
Native structured-output rows sent the deprecated top-level `output_format`,
which Vertex rejects with a 400. The Vertex request now carries it as
`output_config.format`, with an explicit caller value winning.
* fix(anthropic): unwrap tool-emulated json_schema envelopes on streams and tool-name wrappers
* test(anthropic): expect the json tool description through a shared constant
* fix(anthropic): drop the json tool description that makes sonnet-5 omit required keys
* fix(anthropic): keep schema-accepted answers whole and decide held streams at the second key
An answer the json_schema accepts is returned as is before any envelope rule runs, the tool-name
wrapper is unwrapped only when the schema rejects it, a malformed schema turns the unwrap off, a
streamed answer whose first key is a schema property named name or tool_name waits for its second
key before streaming, the held fragments go through a StringIO, and a prefix that stays undecided
past 4 KB is held to the end
* test(anthropic): keep the stream family cells to providers whose stream iterator gets the schema
* fix(vertex_ai): unwrap claude json_schema answers nested under up to three wrapper keys
A tool-emulated json_schema answer was unwrapped only when the real object sat one key down. In live QA, 1 of 58 answers (/v1/responses, claude-sonnet-5) came back as {"parameter": {"value": {...}}} and reached the client still wrapped
The unwrap now follows single-key wrappers up to three levels down and stops at the first object that has a key the schema declares. That object is returned only when it has every required key. A payload wrapped deeper than three levels, a level with more than one key, and a level whose key is one of the schema's own property names all come back exactly as they arrived
The streamed path needs no change because the buffered answer goes through the same unwrap when the tool block closes, and a test now pins that both paths agree
* fix(bedrock): unwrap streamed claude json_schema answers on claude platform and mantle
A streamed json_schema answer on bedrock/claude_platform and Bedrock Mantle Claude reached the client as tool_calls deltas holding the json_tool_call wrapper, with finish_reason tool_calls and no content. The shared HTTP handler built the async stream iterator without json_mode, and neither stream path passed the response_format schema
The handler now asks the provider config for the iterator through get_model_response_iterator_for_request, which carries json_mode and the request's optional params on both the sync and async paths. Claude Platform and Mantle override it to hand the json tool schema to the Anthropic stream iterator, so the streamed answer is the bare schema object with finish_reason stop, the same as the non-streamed one
Databricks is the only other provider on this handler whose iterator reads json_mode, so its async Claude json mode stream now returns the answer as content like its sync stream already did
The stream family integration test covers claude_platform_unmapped and bedrock_mantle again
* fix(anthropic): unwrap json tool envelopes by structure instead of schema validation
The unwrap now reads only the schema's top-level property names. A lone
undeclared key holding an object, alone or beside a name equal to the json
tool's own name, is the envelope, and the unwrap repeats while the inner
object shares no key with the schema. jsonschema and referencing are no
longer imported on the request path. The stream holds only an answer that
opens with the tool's name or with an undeclared key holding an object, and a
schema without top-level properties streams as it arrives.
* fix(llms): keep other providers' async streams as they were
The request-aware iterator hook passed json_mode on async streams to every provider on the shared HTTP handler, which changed Databricks and any other provider whose iterator reads it. The default hook now makes the same two calls the handler made before, so only Claude Platform and Bedrock Mantle change, and the Databricks test that pinned the wider change is removed
* fix(anthropic): walk json tool envelopes iteratively
* test(llms): pin which stream iterators get json_mode from the shared handler
A provider config that does not override the request-aware iterator hook still gets json_mode on a sync stream and still does not get it on an async stream, the same as before the hook existed
* fix(anthropic): keep json tool arguments too deep to decode instead of raising
* fix(anthropic): unwrap json tool envelopes beside undeclared string labels
* fix(anthropic): keep json answers whole on schemas that allow extra keys and stream held text as sent
A schema that allows undeclared keys and requires none now keeps an undeclared wrapper key on both the stream and non-stream paths. When no envelope is found, the held stream text is now sent exactly as the model wrote it. On Vertex /v1/messages, output_format now wins over output_config.format when both are set, which matches the Anthropic adapter.
* fix(anthropic): join a json_schema answer the model splits into one tool block per field
With tool emulation the model sometimes answers with one json_tool_call block per schema field, each naming the field and holding its value. The non-stream path returned only the first block and the stream sent every block's text back to back, so the client got one field or invalid JSON. Both paths now build one schema object from those blocks, and any other set of blocks keeps today's output
* refactor(anthropic): set the streamed finish reason in one expression
* test(anthropic): cover label envelopes, dynamic-key schemas, as-written held streams and output_format precedence
* fix(anthropic): unwrap a streamed values wrapper like the non-stream answer
* fix(anthropic): join streamed split answers on schemas without required keys, keep client tool calls and length on json_schema streams
* fix(anthropic): join streamed split answers that carry text labels on schemas without required keys
* test(bedrock): script the wrapped json_schema stream with the wrapper Bedrock Mantle sends
* test(bedrock): script the wrapped json_schema stream with the wrapper Bedrock Mantle sent raw
* fix(anthropic): join a split json_schema answer whose fields all hold the same string
* fix(anthropic): join a split json_schema answer sent as one partial object per field
* fix(anthropic): join field blocks after an empty streamed answer without a leading comma
|
||
|
devin-ai-integration[bot]
|
2f3ccbe7c7
|
feat(decisions)!: send images to Cloudflare Clef through its images field (#45734)
* feat(decisions): send images to Cloudflare Clef OpenAI-format input_image parts and the System One images field now reach Cloudflare clef and clef-flash in its images field, with remote URLs downloaded and inlined first. Providers that cannot take images drop them under drop_params and answer 400 otherwise. * fix(decisions): follow drop_params for images and use the call timeout for image downloads * fix(decisions): drop a malformed images entry on its own under drop_params * fix(decisions): refuse images next to input whatever drop_params says * fix(decisions): bound clef image fetches, honor drop_params for images, and log caller URLs Refuse more than 4 images for Cloudflare clef before any download, keep the caller's timeout as one deadline across image fetch retries, accept images next to input under drop_params, validate image strings and objects, and pass the caller's image URLs (not inlined bytes) to logging callbacks. * fix(decisions): check input_image URLs and clef image sizes * fix(decisions): accept data URL media type parameters and check clef image types * fix(decisions): count clef images after dropping unsupported types and flag clef vision --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c581c85559
|
fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude (#45871)
* fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude Vertex AI Claude deployments that emulate response_format through a forced tool sometimes nest the schema object under an extra key such as `parameters` or `input`, and that envelope reached message.content with a 200. The converter now unwraps exactly one nested object that matches the schema, leaving a payload that is itself schema-shaped alone. Native structured-output rows sent the deprecated top-level `output_format`, which Vertex rejects with a 400. The Vertex request now carries it as `output_config.format`, with an explicit caller value winning. * fix(anthropic): unwrap tool-emulated json_schema envelopes on streams and tool-name wrappers * test(anthropic): expect the json tool description through a shared constant * fix(anthropic): drop the json tool description that makes sonnet-5 omit required keys * fix(anthropic): keep schema-accepted answers whole and decide held streams at the second key An answer the json_schema accepts is returned as is before any envelope rule runs, the tool-name wrapper is unwrapped only when the schema rejects it, a malformed schema turns the unwrap off, a streamed answer whose first key is a schema property named name or tool_name waits for its second key before streaming, the held fragments go through a StringIO, and a prefix that stays undecided past 4 KB is held to the end * test(anthropic): keep the stream family cells to providers whose stream iterator gets the schema * fix(anthropic): unwrap json tool envelopes by structure instead of schema validation The unwrap now reads only the schema's top-level property names. A lone undeclared key holding an object, alone or beside a name equal to the json tool's own name, is the envelope, and the unwrap repeats while the inner object shares no key with the schema. jsonschema and referencing are no longer imported on the request path. The stream holds only an answer that opens with the tool's name or with an undeclared key holding an object, and a schema without top-level properties streams as it arrives. * fix(anthropic): walk json tool envelopes iteratively * fix(anthropic): keep json tool arguments too deep to decode instead of raising * fix(anthropic): unwrap json tool envelopes beside undeclared string labels * fix(anthropic): keep json answers whole on schemas that allow extra keys and stream held text as sent A schema that allows undeclared keys and requires none now keeps an undeclared wrapper key on both the stream and non-stream paths. When no envelope is found, the held stream text is now sent exactly as the model wrote it. On Vertex /v1/messages, output_format now wins over output_config.format when both are set, which matches the Anthropic adapter. * test(anthropic): cover label envelopes, dynamic-key schemas, as-written held streams and output_format precedence --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
0ebec123fd
|
chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)
* chore(typing): validate provider payloads at the boundary to clear 464 basedpyright Any errors Replaces untyped reads of decoded JSON in 48 provider, guardrail and proxy modules with one pydantic validation at the point the payload enters the module, so every value read afterwards carries a real type. reportAny 6111 -> 5663 and reportExplicitAny 1416 -> 1400. No basedpyright, ruff-strict or type-discipline rule is higher repo-wide or in any single file, and no cast, ignore or suppression marker was added. * chore(typing): replace 47 Any annotations that every caller already satisfies with object Each annotation here is a parameter or return type whose value is only stored, compared, logged or forwarded, so object describes it exactly and no function body changes. A site was kept only if no basedpyright, ruff-strict or type-discipline rule rose in any file that references it. reportExplicitAny 1400 -> 1353 and reportAny 5663 -> 5655. * chore(typing): type upstream responses and loose annotations across providers, proxy and integrations Second pass of the Any reduction. Decoded upstream JSON is validated once at the provider boundary and read through typed views afterwards, annotations that every caller already satisfies move from Any to a concrete type, and the nine search transformations validate each result item on its own so error text no longer carries the item index. No cast, no type: ignore, no suppression marker and no budget file edit * ci(unit): run the brave, google_pse, nlp_cloud and opensandbox tests in the providers shard |
||
|
devin-ai-integration[bot]
|
249add6f0d
|
refactor(bedrock): rename is_claude_4_5_on_bedrock to supports_bedrock_prompt_cache_ttl (#46058)
Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e0398455d0
|
test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759)
* test: migrate selected llm translation contracts * test: migrate anthropic completion translation tests * test: migrate bedrock and cohere translation tests * test: tighten bedrock translation migration * test: migrate azure and openai model translation tests * test: migrate openai and small-provider translation tests * test: migrate gemini and vertex translation tests * test: migrate optional params and rerank translation tests * test: migrate huggingface, json and openrouter translation tests * test: fix openrouter integration expectations * test: migrate databricks, xai, cometapi and fireworks translation tests * test: migrate ovhcloud, prompt and small provider translation tests * test: split reasoning effort grid into unit and e2e * test: migrate router and realtime guardrail translation tests * test: migrate google interactions translation tests * test: migrate a2a, evals, skills and hosted vllm translation tests * test: remove remaining llm translation suite * test: tighten migrated llm translation unit tests and land missing rows Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover * test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths * test: close lost-intent gaps found in review of the llm translation migration Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised * test: assert provider acceptance in the reasoning effort e2e The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid * test: cover async and stream-end paths the legacy translation suite exercised Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker * test: type the new llm translation e2e calls and mark moved fixture dirs as packages * test: name the reasoning effort e2e models as module constants * test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name * test: block the prompt the shared guardrail denies and drop the reasoning presence check * test: drop the duplicate litellm import left by the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
1dbde0845e
|
test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758)
* test: move guardrails, image, audio, batches, search legacy tests to unit * test: restore the custom guardrail and remaining chunk 1 unit rows * test: move the ocr matrix to e2e * test: drop comments from moved tests and fix vertex ocr credentials * test: move litellm utils and unified google legacy tests to their tiers * test: move logging callback and pass-through unit legacy tests to their tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: move store_model_in_db legacy tests and start the admin ui migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: finish the proxy job rows and drop their CircleCI jobs Moves the remaining multi-instance, openai endpoint, otel, pass-through and top-level proxy tests into tests/integration with scripted upstreams, rewrites the weak replacements to assert the outbound wire and exact responses, and removes the build_and_test job group this PR owns from CircleCI * test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names * test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist * test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in * test: type the e2e beta header, prompt cache and secret manager values * test: assert the missing model error message on the OpenAI routes * test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks * test: make the langfuse fixture directory a package for the layout check * test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane * test: keep the ocr e2e pdf fixture next to the test * test: put meta below covers on the streaming tool search e2e * test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures * test: count only OCR calls in the OCR matrix recorder * test: pin OCR parse bodies and billed cost, group new e2e cases into classes * ci: restore the python image anchor dropped in the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a2e334febe
|
test: migrate local_testing SDK and provider tests out of build_and_test (#45760)
* test: migrate local_testing SDK and provider tests out of build_and_test * test: cover sagemaker region, hf template and nova paths end to end Region rows now go through litellm.completion on the sagemaker/ handler that the legacy tests exercised, the prompt template test mocks the Hugging Face tokenizer config instead of switching to custom_prompt_dict, streaming tests assert role, finish reason and body, and Nova single turn and streaming run through litellm.completion with a recorded request body * test: tighten migrated integration tests for local OpenAI routes, timeouts and air-gapped boot Split the catch-all providers migration file by behavior. The fake endpoint route tests now hit the owned upstream directly, as the legacy harness tests did. LM Studio asserts the outbound key, path and stripped model. The embedding timeout test asserts the proxy maps a deployment timeout to a 408 and forwards the read timeout to the provider client, instead of timing out the test client The Azure streaming tool-call test also asserts the second identical stream is served from the Redis cache with the same reassembled tool call The Docker network test always skipped in CI because the security job has no litellm/litellm image and no egress to pull one, and it tested a published image rather than the checkout. It now boots the checkout proxy with every outbound HTTP connection refused and asserts readiness, the bundled cost map fallback and that the cost map host is the only egress attempted * test: tighten the migrated e2e rows for local_testing SDK and provider tests Drop the duplicated Anthropic covers marker and the Anthropic cell wrongly claimed by the Bedrock and Vertex cache tests. Keep the file-wide Anthropic model as it is on main. Send the 2023-06-01 version header to Anthropic through the deployment's extra_headers so the header actually reaches the provider, and mark the creation-token case 1h then 5m like the legacy test did Reuse the Bedrock file's own registration for the pydantic response test, swap the hand-built PNGs for the shared cat.jpg fixture, assert the SageMaker Nova image answer names the cat and that top_logprobs=3 is honored, and register the SageMaker Nova opt-in markers the same way as the other opt-in markers * test: restore lost assertions in migrated test_main tests Restore the role alternation check in test_completion_anthropic_hanging, the embedding and streaming response-header checks, the logit_bias drop check and the gemini system and safety settings checks. Add an azure 429 retry-after test for test_exception_with_headers, trigger the router fallback with mock_timeout like the legacy test, and stop the mock timeout tests sleeping 3s * test: restore lost legacy assertions in migrated vertex ai unit tests * test: tighten migrated cost, streaming and embedding tests and scope the hosted mock guard * test: strengthen migrated anthropic, azure, bedrock and cohere provider tests Assert full request bodies and usage instead of presence checks, restore the dummy-tool and custom_llm_provider assertions the legacy tests made, add the async and per-model variants the legacy parametrization covered, split streamed tool arguments across deltas, and move the Azure chat streaming tests to the azure.py mirror file * test: tighten migrated llms provider tests and cover legacy intents lost in the move Strengthen the migrated fireworks, gemini, mistral, together, openai text completion, openrouter, perplexity and openai error tests so they assert what the legacy tests proved (cost, provider tag, usage-only-on-last-chunk, function_call rebuild, reasoning param, sync/async/stream error fields). Replace the huggingface and jina tests that doubled litellm code with respx end-to-end tests. Add the tests the pre-existing mapped tests never covered: cohere 400 mapping, token-array text completion, hosted_vllm/llamafile/lm_studio env api base embeddings, watsonx embedding auth, ollama ssl_verify client and ollama vision prompt * test: cover residual legacy paths for anthropic tool streams, watsonx async stream, ollama http images and caller async embedding clients * test: restore single request assertions in the deep_infra and azure_ai embedding tests * test: drop a no-op streaming cost flag from the openai streaming cost test * test: cover residual legacy-test lines in caching, main, utils, replicate, vertex and bedrock invoke * test: cover streamed audio assembly, choice membership, untranslated responses events and the stream log guard * test: pin cost rates in unit tests, use sdk for cohere e2e, drop vertex llama e2e * test: send the vertex maps e2e request through the openai sdk * test: drop the local_testing jobs that select no tests after the migrations * test: wait for the streamed cache entry and keep custom Mistral rates test-local * test: keep untouched Mistral test lines as they were --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
Shubham Jain
|
985574f3dc
|
fix(bedrock): keep GPT tool-result images beside the tool result (#45579)
* fix(bedrock): keep GPT tool-result images beside the tool result Bedrock gpt-6.1-sol rejects an image nested in toolResult.content. Put it on the same user turn instead. * fix(bedrock): read tool-result image support from the price map OpenAI GPT rows on Bedrock Converse reject an image inside toolResult.content, so those images sit beside the tool result. * fix: sync price-map test schema and import order CI validates model_prices_and_context_window.json against an inline schema in test_utils, and ruff checks import order in converse_transformation. * fix(bedrock): satisfy basedpyright in tool-result image helper Drop Final assignments inside loops and use TypedDict key checks so the lint type-check gate stays within its merge-base ceiling. * test(bedrock): cover tool-result image placement helper Exercise Claude no-op paths, text-only GPT tool results, status preservation, and split text/image tool content for codecov. * refactor(bedrock): reuse the shared tool-result image placeholder text --------- Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> |
||
|
devin-ai-integration[bot]
|
9f5de05b9d
|
fix(bedrock): drop stop sequences for GPT 5.6 and newer instead of forwarding them to a 400 (#45664)
Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
8080601a95
|
feat(bedrock): default Grok on Bedrock to native Chat Completions, mint unique tool call ids and drop stop (#45473)
* fix(bedrock): route Grok 4.7 tools with reasoning to native Chat Completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): mint unique tool call ids on native Chat Completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): send a blank text block for Converse tool results with no supported content Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(bedrock): default Grok on Bedrock to native Chat Completions and leave Converse unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): gate tool call id minting to Grok on native Chat Completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): check the Grok tool call id gate at both call sites Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): remint positional tool call ids on native Chat Completions for every model AWS returned call_0, call_1 for GPT 5.6 as well as Grok on 2026-10-09 (and unique ids for GPT 5.6 earlier the same day), so the id format is not fixed per model. Drop the Grok-only gate and remint any call_<digits> id on the route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): drop stop sequences for Grok instead of forwarding them to a 400 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
3250ccc802
|
test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554)
Moves 58 legacy nodes (cyberark, logging callback manager, langfuse handler and helpers, SQS, app crypto, bedrock nova and invoke, cloudflare, litellm_proxy, nvidia nim, replicate, triton, convert_dict_to_response, get_model_info, vcr live-call probe) into the tests/unit files that mirror the modules they exercise. Calls that reached huggingface.co, langfuse or 0.0.0.0 are refused at the HTTP boundary with respx. Tests that need credentials, a license, a spawned server, sleeps, a subprocess, or assert nothing stay in their legacy files |
||
|
joshua-berri
|
911aff2752
|
refactor(sdk): separate core AWS and tokenizer dependencies (#44447)
* refactor(sdk): separate core AWS and tokenizer dependencies * fix(sdk): preserve runtime tokenizer alias compatibility * test(sdk): compare tokenizer installs through existing entry point * fix(sdk): preserve bearer headers and optional dependency interfaces * fix(sdk): retain safe tokenizer fallback diagnostics * test(sdk): inject tokenizer dependency for fallback diagnostics * fix(core): preserve AWS dependency errors with retries * fix(core): centralize optional AWS dependency handling * refactor(core): reuse optional import helper for Invoke streams * test(core): compare complete shared dependency requirements * fix(core): keep tokenizer logging import compatible with main * fix(core): preserve native token decoding and Responses dependency errors * test(core): isolate optional dependency import failures * fix(sdk): retain typed exception message access * test(sdk): scope HTTP verification environment changes * fix(sdk): preserve bearer request typing across Bedrock callers --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
6a2d3edce3
|
fix(bedrock): split the <reasoning> tag for gpt-oss only on native Chat Completions (#44432)
* fix(bedrock): split the <reasoning> tag for gpt-oss only on native Chat Completions The native Chat Completions route moved a leading <reasoning>...</reasoning> block into reasoning_content for every model, while only gpt-oss writes its reasoning inline. A GPT 5.6 or Grok answer that starts with a literal <reasoning> tag lost that text, streaming and non-streaming alike. Both paths now split only when the model id is gpt-oss. * fix(bedrock): drive the inline reasoning split from a cost-map flag --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4f62bbfd8b
|
fix(bedrock): remove the bare xai.grok-4.7 cost-map row that AWS cannot invoke (#45460)
Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
fa2c8984ba
|
test: move the unit half of 126 mixed legacy files into tests/unit (#45090)
* test: move the unit half of 126 mixed legacy files into tests/unit * test: restore litellm globals that moved tests set * test: finalize migration test cleanup * test: restore original bodies of moved legacy tests The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated * test: drop module imports shadowed by restored local imports * test: assert on LiteLLM output in no-assertion moved tests and isolate leaks Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags * test: drain queued logging tasks before the Phoenix span test The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
e2971e0af4
|
refactor(llms): expose public names for private provider helpers (#45037)
* refactor(litellm): migrate private usage in llms Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve Bedrock batch signature marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): migrate llms private usage symbols Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve UUID Watsonx project IDs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): retarget llms mocks to public names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): limit llms changes to renames and forwarders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): keep GCS mock client patching private Vertex auth methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): assert forwarder arguments and type forwarder signatures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
dfe4df8c8a
|
test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)
* test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
c3a23fe499
|
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
f81a3f5243
|
fix(responses): report truncated bridged output as incomplete and echo request params (#44460)
* fix Bedrock json mode bug * fix finish reason when max tokens hit * fix hardcoded completion for streaming responses * fix: echo Responses API request params onto bridged response * refactor(responses): drop explanatory comments and build null finish_reason test without mutation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): type the terminal event and Bedrock mock helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep truncated WebSocket turns in history and drop invalid echoed params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
1b98748528
|
fix(bedrock): honor the per-request timeout on Converse and Invoke streaming (internal copy of #38210) (#44134)
* fix(bedrock): propagate timeout to streaming requests * test(bedrock): prove streaming fails at the request timeout against a slow upstream * test(bedrock): simulate the slow upstream in process instead of over a local socket * test(bedrock): audit the Converse and Invoke stream timeout on the proxy Two integration files drive the per-request timeout on Bedrock streams through the real proxy against an owned wire peer: the wire file covers every surface (chat, messages, responses, invoke, pass-through), the sad, edge and precedence rows, and the chaos file covers bursts, a dropping upstream, a killed worker and a proxy stopped mid-burst. The wire peer gains Reply.drop_connection so a cell can close the socket before any response, and the harness's graceful stop grace is now INTEGRATION_PROXY_STOP_SECONDS (default unchanged at 30), since a two-worker supervisor's interpreter finalization takes longer than that on a loaded box. * test(bedrock): pin the fallback audit cell to one proxy worker The fallback cell created both deployments through /model/new on one worker and sent the chat request to the other, whose registry read-through loads only the requested model, so the fallback target was unknown there until the periodic DB poll. The cell now warms the fallback model and sends the request over one keep-alive client, so one TCP connection stays with one uvicorn worker, and it expects the fallback upstream to see both requests. --------- Co-authored-by: Sainyam Kapoor <hello@sainyam.me> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |