mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-13 08:52:02 +00:00
126 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Mukesh Dewangan
|
4ed339b2fc
|
fix(vertex_ai): map missing image on Gemini image edit to BadRequestError (400) (#50244) | ||
|
Mateo Wang
|
98acc01e0d
|
feat(videos): add first-class OpenRouter video request fields (#50319)
* feat(videos): add first-class OpenRouter video request fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): remove recursive image parameter fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): preserve EdenAI native parameter passthrough Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): deduplicate frame roles in linear time Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): preserve image selection and dropped parameter semantics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): preserve provider scalar parameters and raw extra body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): retain accepted scalar inputs for Vertex and DashScope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
3f38730d2c
|
refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986)
* refactor(typing): clear basedpyright Any errors (wave 1) * refactor(typing): clear basedpyright Any errors (wave 2) * refactor(typing): clear basedpyright Any errors (wave 3) * refactor(typing): clear basedpyright Any errors (wave 4) * refactor(typing): clear basedpyright Any errors (wave 5) * refactor(typing): clear basedpyright Any errors (wave 6) * fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves * fix(logging): keep callbacks running when litellm_params is a params model should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request Adds behavior tests for the changed lines codecov listed as not run * fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call. * fix: bill regional uplift multipliers float() accepts, bytes included * fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type |
||
|
Julien Parenti
|
6fc9555f83
|
fix(vertex_ai): honor vertex_location for the Cloud Text-to-Speech endpoint (#50255)
Chirp and other Cloud TTS models now call us-texttospeech or eu-texttospeech.googleapis.com when the deployment's vertex_location is a US or EU location, keep the global host otherwise, and an explicit api_base still wins. Gemini TTS accepts more US and EU regions. |
||
|
ryan-crabbe-berri
|
48ff826cbc
|
feat(proxy)!: only let clients pick S3 buckets the admin allowed (#47496)
* feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way. s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls * refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map * fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request |
||
|
qtim0
|
5085022580
|
fix(audio-transcription): keep transcripts out of transcription error messages (#50137)
* fix(gemini): omit raw response body from audio transcription error messages * fix(gemini): raise incomplete transcription interactions as non-retryable 400 and drop response bodies from Vertex transcription errors * fix(audio_transcription): keep response bodies out of transcription errors Keep the upstream status on a Gemini interaction that did not complete, so the router still retries it as it does on main. Gemini stops the same long clip on some tries and not on others, so a retry can recover it. Stop quoting the reply in the Deepgram, ElevenLabs, Watsonx, Mistral, OVHcloud and Scaleway parse errors, and hide pydantic input values on the Gemini and Vertex response models. * fix(audio-transcription): keep reply bodies out of watsonx, whisper, xai and validation errors Watsonx and whisper no longer print the parsed reply after Got: when it has no text field. TranscriptionResponse and the xAI response model hide input in validation errors, so a malformed field cannot quote transcript fragments. Deepgram and ElevenLabs keep the original error text, which is now safe. * fix(audio-transcription): keep transcripts out of the shared converter's parse errors * test(audio-transcription): compare wrapped lookup errors instead of pinning upstream wording --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d947ec1f7d
|
refactor(prompt_templates): move JSON schema helpers out of common_utils into json_schema_utils (#50241)
* refactor(schema): extract JSON schema helpers from prompt utilities Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Extract cohesive prompt utility modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Preserve legacy prompt utility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
f069da8e0e
|
fix(anthropic): report temperature/top_p as supported only when the model keeps them (#50144)
* fix(anthropic): report temperature/top_p as supported only when the model keeps them Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vertex_ai): accept the generic unsupported-param message for Claude batch rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Meryem Sakin
|
4a7fb30b0a
|
fix(guardrails)!: check Anthropic and Responses request tools in tool_permission pre_call (#41011)
* fix(guardrails): check Anthropic-format request tools in tool_permission pre_call
The pre_call hook only collected tools shaped as {"type": "function", "function": {...}},
so flat tools from /v1/messages ({"name", "input_schema"}) and /v1/responses
({"type": "function", "name"}) were never checked and deny rules silently passed.
Reuse the shared anthropic_tool_names helper so every request tool name is checked,
and handle Anthropic tool_choice {"type": "tool"} in both block and rewrite modes.
Fixes #40583
* fix(guardrails): detect Responses API named tool_choice in tool_permission
Responses API named choices put the name at the top level ({"type": "function", "name": ...}),
so rewrite mode removed the denied tool but left a tool_choice pointing at it.
* test(guardrails): run tool_permission pre_call tests through real default_on gating
Build the guardrail as an always-on pre_call guardrail instead of patching should_run_guardrail.
* test(guardrails): type the new tool_permission pre_call test parameters
* fix(guardrails): keep the Responses API custom tool type distinct
Anthropic's user-defined tools carry type "custom", so pre_call mapped
that to "function" for tool_type rules. The Responses API has its own
custom tool type, and mapping it let a custom tool slip past a
tool_type deny rule when the default action is allow.
Normalize only for the formats where custom means a plain function, and
read tool names directly in the rewrite path, which never needed a type.
* test(guardrails): cover flat tools through Messages and Responses
Add blocked and permitted flat-tool cases through the running proxy, with
endpoint coverage and metadata for the current e2e harness
Require a guardrail policy verdict when checking denied tools so an
unrelated provider error cannot satisfy the regression test
Co-authored-by: Christopher Krah <christopher.krah@plexito.de>
* fix(guardrails): check namespace and nameless tools and relax tool_choice in tool permission rewrite
* fix(guardrails): check forced, grouped, loaded, and hosted tools by their real type
* fix(guardrails): drop references to removed tools and count loaded tools in tool permission rewrite
* fix(guardrails): check tools sent outside top-level tools in tool permission
* fix(guardrails): judge namespace members by their bare name as a fallback and keep rewrites valid for Gemini, Bedrock and hosted tools
A namespace member is checked by namespace__name first, and by its bare name when no rule names the joined one. Rewrite mode now prunes Gemini allowedFunctionNames and turns off forced calling when no function is left, drops a removed hosted tool from allowed_tools, reads Bedrock toolSpec and systemTool blocks by name, skips cachePoint markers, and leaves a tools field that is not a list for the provider to reject instead of failing with a 500
* fix(guardrails): let only name rules decide a namespace member's bare-name fallback
* fix(guardrails): judge malformed tool types, empty Gemini name lists and pass-through Anthropic tools like the endpoints do
A tool or tool_choice whose type is not a string now reads as untyped instead of crashing the guardrail with a 500, and a null type key is never read as a Gemini tool part. An empty allowedFunctionNames list no longer turns off the functions a rewrite kept. An Anthropic custom tool reads as a function by its input_schema on every route, so a pass-through route judges it the same as /v1/messages. A Responses WebSocket frame now forwards the extra_body tools a rewrite stripped
* fix(guardrails): judge a named hosted tool by its type and keep a nested chat custom tool custom
* fix(guardrails): keep judging Anthropic client tools and type-named server tools by their name
* fix(guardrails): judge a named tool by its name alone only when the name is its type's own
* fix(guardrails): judge the function a tool names beside a declaration list or a top-level name
* fix(guardrails): judge Gemini hosted keys whatever the tool type and a namespace's function
* fix(guardrails): judge legacy functions entries like tools and send Gemini a namespace's members
* fix(guardrails): judge a namespace member by every name a provider reads and send Gemini its schema converted
* fix(guardrails): judge a legacy functions entry as a function and relax a tool_choice in extra_body
* fix(gemini): keep a namespace whole when inlining its members would repeat a function name
* fix(guardrails): judge a namespace member by every bare name that shares its qualified name
* fix(vertex_ai): declare a repeated namespace member name once instead of sending namespace names to Gemini
* fix(guardrails): judge a tool_choice sent in extra_body like a top-level one
A forced name only extra_body.tool_choice carried was never checked, so block mode let a denied name through to the provider
* fix(guardrails): judge a Gemini namespace member returned by its bare name and force a named namespace's members
* test: keep formatter churn out of the tool_permission and Gemini test files
* fix(guardrails): judge extra_body.input tools, argument-aware alias rules, and exact forced Gemini functions
Tools loaded in extra_body.input are now checked and stripped like top-level input. A returned
namespace member is judged by the alias whose first name rule its arguments satisfy. A forced
Gemini name that matches a top-level function keeps that function. Alias grouping and Gemini
declaration dedupe now run in one pass instead of quadratic scans.
* fix(guardrails): judge and rewrite every tool field sent inside extra_body
extra_body is merged into the upstream body, so functions, function_call,
mcp_servers and toolConfig nested there reached the model unjudged. Treat
extra_body as a second request layer that is collected and rewritten the
same way as the top-level request.
* fix(guardrails): judge Gemini hosted keys by key and count only the tools a provider receives after extra_body
* fix(guardrails): relax a tool_choice that names a dropped namespace member by its bare name
Gemini and Anthropic chat send namespace members under their bare names, so rewrite mode now treats a dropped member's bare name, and an emptied namespace's own name, as removed when no kept tool still carries it
* fix(gemini,guardrails): keep a top-level function over a same-named namespace member and leave malformed tool values to the provider
Gemini and Vertex chat now declare a top-level function over a namespace member that shares its name, so a forced call gets the top-level schema. post_call judges a returned name a top-level tool carries as that tool, and pre_call no longer raises on an unhashable or non-string tool name, input item type, or function_call name.
* fix(guardrails): judge a returned namespace member named like a hosted MCP server as that member
* fix(gemini,guardrails): send a namespace as main did when another tool has a member's name
Gemini chat inlines a namespace's members only when no other tool, namespace
member, or legacy functions entry declares the same name, and otherwise sends the
namespace by its own name as main did. A tool_choice naming such a namespace
forces the namespace itself. The tool permission post_call reads top-level names
the way Gemini declares them and judges a call the Responses bridge tags with a
namespace as that member.
* fix(gemini): send colliding namespace members as namespace__name functions and judge them in post_call
When a namespace member shares a name with another tool, Gemini chat now gets that namespace's members as namespace__name functions instead of one function named after the namespace, so every function it can call is a name the rules judged. Declared names are counted once, and a forced member maps to each holder's qualified name.
post_call judges a returned namespace__name as that member, by its qualified or bare name the way pre_call does, so a bare-name allowlist or denylist applies to it.
* fix(gemini): keep generated member names unique and let a top-level function own its exact name
A namespace that listed a member twice, or a generated namespace__name that
repeated a top-level function, sent Gemini two declarations with one name, and
Gemini answered 400 where main answered 200. A name a namespace lists twice is
now one member, and a top-level function keeps its exact name.
post_call now judges a returned name a top-level function declares as that
function, never as the namespace member the name could also mean.
* fix(gemini): qualify namespace members by the namespace's top-level name
The guardrail judges a member as `name__member` from the namespace's top-level
name. Gemini built the name from `function.name` when the tool also carried one,
so a request could pass pre-call and reach Gemini under a name no rule judged.
* fix(vertex_ai): force only the namespace members Gemini is sent
A forced namespace or member on Gemini chat could still allow a generated name that a top-level function or an earlier namespace owns, so tool_choice forced an unrelated function. The forced names now come from the same collision checks the declarations use
* fix(vertex_ai): reject a forced namespace Gemini gets no member of, and resolve forced names in one pass
* fix(guardrails): judge a namespace member by a rule on its namespace's name
---------
Co-authored-by: Christopher Krah <christopher.krah@plexito.de>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
tavren27
|
640a87f70d
|
fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature (#49697)
* fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): gemini tool-call signature rejected by the backend is resent once with the placeholder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): keep the placeholder resend inside the cache rebuild so either rejection order ends in an accepted body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): resend a cached-prefix signature inline and cover native generateContent When the rejected signature sits inside a cachedContents prefix, the placeholder resend now carries the history inline, since replacing the signature in the request body cannot reach the cache. The native generateContent handlers get the same one-time resend. * fix(gemini): send the history inline when Vertex rejects a signature while creating the cache * fix(gemini): keep the context cache on a signature resend and go inline only when that is rejected too --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
simonp9
|
61da6fcb4d
|
fix(mcp)!: answer 400 for a hosted MCP tool a translated route cannot run, drop it under drop_params (#48931)
* fix(bedrock, gemini, deepseek, databricks, gigachat, anthropic): answer 400 for a hosted MCP tool a route cannot run, drop it under drop_params A `type: "mcp"` tool sent to a route that rebuilds tools for its provider (Bedrock Converse and Nova, Bedrock Invoke Claude, Vertex AI Claude, Vertex AI and Google AI Studio Gemini, DeepSeek, Databricks Claude, GigaChat) was dropped silently and the request still answered 200. These routes now answer UnsupportedParamsError (400) naming the model, route and server, and pointing to the LiteLLM MCP gateway. Under drop_params the tool is dropped with a warning, other tools are kept, and no empty tools field is sent. Anthropic, Azure AI Claude and Claude Platform on AWS keep mapping the tool to the MCP connector. Native Responses providers and raw OpenAI-compatible forwarders are unchanged. * fix(mcp): check hosted MCP tools once per request and keep base tool_choice handling Each route now runs one request-level step before its param mapping. When the drop leaves no tools, the request sends no tools, tool_choice or parallel_tool_calls. Converse and GigaChat go back to their base tool_choice and empty-tools handling. * refactor(deepseek): hold the sent tools in a tuple * fix(mcp): keep tool_choice for legacy functions when every hosted MCP tool drops * fix(mcp): answer 400 for MCP servers a route cannot run on Messages, Ollama and the Responses playground /v1/messages requests with mcp_servers on a chat-adapter or Responses-bridge route now answer 400 naming each server, or drop it with a warning under drop_params, instead of being forwarded or silently ignored. Ollama generate no longer pastes a hosted MCP tool (headers included) into the prompt, and ollama_chat filters hosted MCP tools the same way. The Admin UI Responses playground now sends litellm_proxy/mcp references like the other endpoints. * fix(ollama): leave JSON mode off when drop_params drops every hosted MCP tool Ollama generate set format json for any request with tools, even after the hosted MCP check dropped all of them, so a plain answer came back as JSON or as a call to a tool the model never had. The check now runs before the JSON mode switch, which stays on only while a tool is left for the prompt. * fix(messages): honor additional_drop_params for mcp_servers and read the beta map only for hosted MCP tools * fix(mcp_security): accept gateway server names and toolsets the gateway resolves The guardrail compared litellm_proxy/mcp/<name> refs against registry keys, which are server ids, so a saved server referenced by name, alias, or toolset name was blocked although the gateway routes it. It now resolves each ref the way the gateway does and still blocks unknown names. * fix(tool_fields): keep MCP drop warnings on one log line Server labels, model names, and routes come from the caller, so a line break in one could forge extra log lines. The drop warnings now strip CR and LF from each value. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
a2ab27373f
|
fix(proxy): count the context_management edits /v1/messages applies on every bridged provider (#49234)
* fix(proxy): count the context_management edits /v1/messages applies on every bridged provider * fix: apply context_management edits in count_tokens on Responses bridges, compaction blocks and Bedrock Claude The Responses bridge now applies the same edits inference applies for every provider, the chat bridge cuts the history at a client compaction block, and Bedrock Claude forwards the edits AWS accepts with their beta to the mantle and runtime count APIs instead of dropping them. * refactor(count_tokens): return count-view failures as values and raise them once in the caller * fix(bedrock): keep the runtime's 400 when mantle does not serve the model it retries * fix(count_tokens): forward context edits on Vertex and Azure Claude counts and keep Bedrock's mantle answer without edits * fix(proxy): count edited bodies on native routes without a count API, keep counts on editor failures * fix(proxy): keep the Bedrock pass-through count shape and provider 400s on empty bridged bodies The pass-through count picks Converse or InvokeModel from the content again, and the edited InvokeModel count wraps string turns in text blocks. A bridged view with no messages sends the body as before. Azure and Vertex load the beta config off the event loop before a count with edits, and a model with no deployment reads drop_params from the request and router defaults. * fix(bedrock): send an empty-string turn as is when a count forwards context edits Wrapping an empty string into a text block made AWS reject the count with 'text content blocks must be non-empty', while base and /v1/messages accept the empty string. Only non-empty string turns are wrapped now. * test(azure_ai): compare the count's betas with the betas Azure inference sends * test(azure_ai): record the Azure inference request with respx instead of appending to a list --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tavren27
|
1046716b6d
|
fix(anthropic)!: sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept (#49409)
* fix(anthropic): sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept * test(e2e): send the azure_ai beta check without the code execution tool * test(anthropic): require a vertex_ai and azure_ai decision for every anthropic beta * fix(anthropic): forward the betas Claude Code sends that anthropic, vertex_ai, and azure_ai accept * test(e2e): send Vertex's header-only betas in the beta acceptance case * fix(anthropic): keep forwarding Vertex computer-use betas and point beta tests at platforms that still lack them A live probe found each Claude 4.5 and 4.6 model on Vertex needs one of the two computer-use betas for the computer tool, so dropping both broke it. The e2e acceptance test leaves those two out of its all-at-once Vertex call, and the integration and unit tests that relied on a beta being dropped now use one the platform still lacks. * test(anthropic): point unsupported-beta tests at betas the platforms still drop The Azure AI handler test used claude-code-20250219 as the beta Azure drops, and the Rust parse test used it as an unknown value. This PR forwards it on Azure AI and maps it, so both now use values that stay dropped or unknown. * fix(anthropic): add computer-use-2025-11-24 for the computer_20251124 tool The computer_20251124 tool fell through to the default computer-use-2024-10-22 beta, which the provider rejects for that tool version. Map it to computer-use-2025-11-24, which the beta map now forwards on anthropic and azure_ai * fix(anthropic): send no computer use beta for an unknown computer tool version An unknown computer tool version fell back to computer-use-2024-10-22, which the anthropic and azure_ai columns now forward. It now gets no beta, as every platform effectively sent before. Every computer tool version also skips defer_loading and allowed_callers, computer_20251124 included * test(azure_ai): read the branch's beta map in the auto-injected beta wire test * fix(anthropic): keep defer_loading and allowed_callers on the computer_20251124 chat tool * test(anthropic): read the branch's beta map in the azure_ai safeguards beta wire test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
572832dea1
|
fix(gemini)!: answer 400 for an unsupported chat audio format, drop it under drop_params (#49423)
* fix(gemini): answer 400 for an unsupported chat audio format, drop it under drop_params * fix(gemini): return the audio format error as a value and keep direct mapper callers refused The format helper returns its UnsupportedParamsError for the caller to raise. The shared voice mapper refuses an unsupported format again for direct SDK callers, the error caps the echoed format at 100 characters, and its advice no longer claims the model only emits pcm16 * fix(gemini): refuse an unsupported audio format nested in speech_config A format inside audio.speech_config skipped the check and was silently dropped with a 200. It now gets the same 400 unless drop_params is set. The 400 test also pins the model name in the chat-path message. * fix(gemini): show a non-string audio format as sent in the 400 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
a713ed6bab
|
fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them (#47692)
* test(e2e): add gemini tool call id length repro for LIT-8912 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them * fix(gemini): pair a tool result with its call when the client re-adds the thought signature to the call id Strands, which the AgentCore harness runs, puts provider_specific_fields.thought_signature back on the assistant tool call id as <id>__thought__<signature> while its tool result keeps the bare id, so the Gemini conversion found no matching call and answered 500. Ids now match with the signature suffix stripped. Also adds thought_signature_in_tool_call_id to the litellm_params inventory test and gives the pass-through tests' mocked logging objects the litellm_params a real Logging object always has. * fix(gemini): keep the thought signature on streamed Messages and Responses tool calls and accept a string setting * fix(gemini): read the thought signature back from Responses input and coerce the per-request setting A Responses function_call input item now keeps its provider_specific_fields, so a client that sends the item back gets its real thought signature replayed to Gemini. A per-request thought_signature_in_tool_call_id of "false" or 0 now keeps ids short, and a value that is not a boolean answers 400, or is dropped under drop_params, instead of a 500. The e2e test now uses the sdk fixture and @meta subjects * fix(caching): key the response cache on thought_signature_in_tool_call_id * fix(responses): send thought signature fields only with short ids and drop malformed routing kwargs whole * fix(main): validate routing kwargs after the locals snapshot so fallbacks and model_list calls keep working The e2e test now uses gemini-3.8-flash with enough max_tokens to finish thinking before the tool call. * fix(gemini): keep non-chat routes tolerant of a malformed thought_signature_in_tool_call_id, match tool results by exact id first, and keep the deployment's value when drop_params drops a malformed override * fix(router): validate strict litellm params through a helper so star imports cannot re-bind a public Final * perf(router): skip strict validation when a request overrides no deployment param * fix(responses): keep a dict tool call's thought signature on the streamed function_call done item * refactor(main): return invalid supplemental params as a value and test the streaming tool call with a real response --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Vineeth Sai Varikuntla
|
a7b66dd8b9
|
fix(openai): stop stripping cache_control from the caller's own messages and tools (#35602)
* fix(openai): stop stripping cache_control from the caller's messages Copy before removing the field, so the caller's own message objects are left alone. * fix(fireworks_ai): strip cache_control on a copy so a fallback keeps the caller's hints * fix: keep the caller's request intact on more providers and trim extra Claude cache hints under drop_params * fix: drop only the oldest extra cache hints inside one message when trimming to the Claude limit * fix: keep replayed session turns yielding their hints as whole turns while a request's own message keeps its newest hints * fix(cache_control): count only hints that reach Claude and cap hints on OpenAI-compatible Claude routes under drop_params * fix(cache_control): keep the capped Claude messages a list in OpenAI-compatible request bodies * fix(caching): cap Claude hints for string drop_params, a Claude base_model, and Converse tool messages The 4-hint cap now reads drop_params through drops_unsupported_params on chat, Converse, and Anthropic routes, names Claude by base_model on OpenAI-compatible routes, and counts a Converse tool message's hints as the one cache point it sends. * fix(caching): trim Claude cache hints in one wire-order pass and reserve a top-level cache_control on OpenAI-compatible routes Dropping the oldest hint one at a time re-walked the whole request per hint, which took seconds on long tool histories. The cap now drops each message's oldest hints in the order the provider counts them, so a string assistant turn loses its text hint before its tool call hints. OpenAI-compatible Claude routes also keep room for a request-level or extra_body cache_control under drop_params. * test: drop narrating comments and stacked generator clauses from cache_control tests * fix(vertex_ai): put every system message in the Gemini context cache so generateContent never sends it beside cachedContent --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
a084d13635
|
fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures (#49075)
* fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures Pass-through failures on Gemini and Vertex routes carry the model only in the URL, so the failure spend row and the Prometheus failure counter logged an empty model. The failure payload now falls back to the model parsed from the URL on those routes, and the Prometheus provider label honors the provider the request declares before inferring one from the model name * fix(passthrough): keep model and provider on mid-stream and Vertex failure logs Build the failure payload from the URL at both streamed relay sites so a 200 stream that dies mid-body still logs the model, derive vertex_ai for Vertex routes that never declare a provider, and have the Vertex logging handler reuse get_vertex_model_id_from_url instead of its own regex * fix(passthrough): bound the Prometheus provider label and name Vertex failures like the success handler Declared providers only reach the api_provider label when they name a registered provider, otherwise the label falls back to inference. Vertex failure rows now use the success handler's model naming (vertex_ai/ prefix on rawPredict and streamRawPredict, vertex_ai/search_api on :search) and its host-based provider detection, and the detected-stream failure sites are covered by tests. * fix(passthrough): name Google failures from the URL on untracked methods too A Gemini or Vertex pass-through failure on a method the success handler does not track (countTokens, for one) still logged model "" and no provider, although the model sits in the URL. The failure payload now falls back to the Google AI host for the provider and reads /models/<name> from the URL, so countTokens 400s name their model and provider like generateContent ones do. Non-Google hosts keep the tracked-route rule and still log nothing extra. * fix(passthrough): trust the declared Google provider and Vertex-shaped paths on custom bases The failure path only named the Google provider from the URL when the method was one the success handler tracks or the host was a Google API host, so a Gemini countTokens failure behind a custom GEMINI_API_BASE (or a Vertex one behind a custom base) still logged an empty model. A route that declares gemini or vertex_ai now wins outright, and a URL whose path carries /projects/<p>/locations/<l>/ counts as Vertex on any host, while a tracked method on a non-Google host no longer passes as Vertex. * fix(pass_through): stop the URL model parser at the query string so a configured key never lands in a spend row's model * test(pass_through): run the URL-model failure tests through the real ProxyLogging with a recording callback * fix(passthrough): scope declared provider to pass-through failures and match Google hosts only Prometheus trusts the resolved provider as before and reads the request-declared provider only on pass-through failures. Vertex attribution matches Google hosts, strips operation poll suffixes, prefixes vertex_ai/ only for rawPredict, reads the URL path only, covers Vertex Live WebSocket failures, and labels a router alias failure with the alias the client requested * fix(passthrough): attribute failures from the URL only on built-in Google routes and keep other routes' metric labels A user-defined pass-through route to a Google host keeps its empty model and provider, since its URL names an upstream model rather than a LiteLLM-managed one. Prometheus failure labels read the route's provider and model_group only for gemini and vertex_ai, so Azure, OpenRouter, Bedrock and other pass-through routes keep their earlier labels. The Gemini URL parser drops the query string, Google host matching needs a dot or dash boundary, and the Vertex Claude prefix follows alt=sse like the success handler. * fix(passthrough): log the URL model on guardrail blocks and the named model on Vertex Live failures A pre-call guardrail block on a pass-through route now builds its failure payload the same way upstream failures do, so the model and provider read from the URL reach spend logs and metrics. Vertex Live websocket failures (connect errors, upstream rejections, and the credential failure) now carry the model from the ?model= query parameter under the vertex_ai provider * fix(passthrough): mark guardrail-block and Vertex Live credential failures as pass-through calls Prometheus reads a failure's declared provider only on pass-through calls, so without call_type it fell back to guessing from the model name. That labeled a guardrail block on the Gemini route as vertex_ai and the Live credential failure as None * fix(passthrough): let the route name a failure's provider and give guardrail blocks their own spend row * fix(pass-through): keep URL model attribution off routes that name another provider A built-in route that names a non-Google provider, like /openai pointed at Gemini's OpenAI-compatible host, no longer takes its failure model from the Google URL. A guardrail block response now returns the x-litellm-call-id of its spend row. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b195d15b93
|
fix(vertex_ai,azure_ai)!: auto-add the compaction, thinking display, and tool change betas on Claude routes (#44747)
* fix(vertex_ai): forward and auto-add the compact-2026-09-04 beta on Claude routes * test(vertex_ai): prove the compact-2026-09-04 beta reaches the wire through both Claude handlers * fix(vertex_ai): move a client-sent body compact-2026-09-04 beta to the request header * refactor(vertex_ai): build the Claude beta set without mutation and inject the client in the messages wire test * fix(vertex_ai): keep client body betas in the body and answer 400 on a malformed anthropic_beta * fix(rust): allow the compact-2026-09-04 beta on vertex_ai in the Rust beta table The beta header config now maps compact-2026-09-04 for vertex_ai, so the Rust table that mirrors it must stop rejecting that beta there, and the Rust Vertex messages route adds it for a compaction request like the Python route does. * fix(vertex_ai): keep client body betas out of the Vertex body next to litellm betas, honor drop_params, and send the compaction beta on count_tokens Litellm's own betas replace a client's body anthropic_beta again, as on main, so a body beta Vertex rejects no longer turns a 200 into a 400. A malformed beta drops under drop_params and names its field otherwise. inline-tools-2026-09-15 joins compact-2026-09-04 as header-only, since Vertex rejects both in the body. Vertex count_tokens sends compact-2026-09-04 when the request carries compaction. * fix(logging): log and bill Anthropic messages compaction responses on the Rust route The Rust route hands the served Anthropic body to the logger, which validates it against AnthropicResponse. That model had no compaction content block and no compaction stop reason, so a compaction reply raised and wrote no spend row. * fix(vertex_ai,azure_ai): forward compact-2026-09-04 to Azure AI Foundry, merge Vertex body betas, and redact compaction summaries Azure AI Foundry needs the compact-2026-09-04 header for compaction, a replayed compaction block, and count tokens, so the beta map, the native compaction providers, and the Rust beta table now allow it there. Vertex keeps a client's body betas next to the ones litellm adds, as the Anthropic route does, and turn_off_message_logging now redacts compaction summaries too * fix(vertex_ai,azure_ai): auto-add thinking display and tool change betas, redact streamed compaction summaries Folds the header changes from #44103 into this PR. Vertex AI and Azure AI Claude routes now add thinking-display-updates-2026-08-18 when thinking.display is "updates" and mid-conversation-tool-changes-2026-07-01 when a tool_addition block is present, matching the Anthropic and Bedrock routes. With turn_off_message_logging on, compaction summaries from streamed messages and streamed chat completions are now redacted in spend logs and callback payloads, like the non-streamed response already was. * fix(vertex_ai,azure_ai,rust): add tool change betas on count tokens and Rust messages, bill Rust server tool replies Vertex and Azure count tokens now add mid-conversation-tool-changes-2026-07-01 when the history carries a tool_addition or tool_removal block, through one shared helper with the compaction beta. The Rust /v1/messages route adds thinking-display-updates-2026-08-18 and mid-conversation-tool-changes-2026-07-01 when a request uses them, on the providers that accept each beta. Rust-route logging parses the served body with the Anthropic response parser, so web_search, pause_turn, mcp_tool_use and container_upload replies get a spend row. Legacy enabled thinking keeps display updates on anthropic too, and the Vertex transform reads drop_params through drops_unsupported_params. * style(rust): rewrap beta map test line with cargo fmt * test(rust): expect the display beta on Databricks, which main's beta list now accepts --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
5d5e3685c1
|
fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths (#48933)
* fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths
* fix(logging): keep guardrail and token-count reads of a request from logging drop warnings
Only the real Anthropic-messages adapter request reports a dropped thinking budget. Guardrail,
shadow-eval and token-count translations of the same request now drop it silently, so one request
logs one warning. The 422 retry also no longer hashes each loc entry, so a list inside loc cannot
raise TypeError.
* fix(bedrock): keep the DeepSeek R1 Converse reasoning_effort drop at debug level
Only allowed_openai_params reaches that branch, and it puts reasoning_effort back after the provider mapping, so a warning there claimed a drop that never happened
* fix(logging): warn on Gemini non-integer budget drops and keep the auto-router baseline silent
Gemini 2.x now logs the drop of a non-integer thinking budget. The auto-router
savings baseline builds a body it never sends, so it no longer warns. Logged
values are capped with reprlib, Azure AI marks only its drop_params retry as
gated, and Bedrock Chat Completions names the model without its route prefix.
* fix(logging): warn on the Gemini 3, Magistral, Ollama, Fireworks and DeepSeek ARN reasoning drops
Gemini 3 and later now log a dropped thinking budget, Magistral logs the
level it swaps for a reasoning prompt, Ollama logs a level it maps to
think off, Fireworks logs chat_template_kwargs effort keys dropped for a
model the map does not list as reasoning, and DeepSeek on Bedrock logs a
drop when the model is addressed by an ARN that advertises the param.
* fix(logging): warn on the remaining Ollama, DeepSeek and Tencent reasoning drops
The ollama/ completion route now shares the chat route's think mapping, so
a level it turns off logs the same warning. DeepSeek warns when it drops
thinking.budget_tokens, and Tencent warns when an adaptive-only model gets a
non-integer budget. Magistral no longer warns on low, medium, high or
enabled thinking, the values its reasoning prompt stands in for. The Gemini 3
budget warning no longer claims the model uses its default level.
* fix(reasoning): warn when Claude drops a non-string reasoning_effort
Claude chat and native /v1/messages dropped a reasoning_effort like 5 or
{"effort": 5} with no log line. Both now log one warn-only WARNING. A
Magistral thinking.type sent as a list no longer raises TypeError.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
4c8b90945f
|
fix(realtime)!: send the Gemini Live setup at the first content so session.update settings apply (#39864)
* fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its default audio setup at connect, so a client's session.update (GA output_modalities or beta modalities) was dropped and every answer came back as audio. The setup now goes out on the client's first message: a session.update becomes that setup, anything else gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a provider hook. Text setups label deltas response.output_text.delta, and downgrading TEXT to AUDIO on an audio-only model logs a warning * fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup * fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup Gemini and Vertex Live accept one setup per connection, so the bridge used to turn only the first session.update into that setup and drop the rest without telling the client. Now every session.update that arrives before the first content frame is held, deep-merged (later values win per field), acknowledged with a session.updated built from the setup it would produce, and the merged session becomes the one setup sent with the first conversation item, audio frame, or tool result. The session events also surface the systemInstruction text as instructions, which the old string check never matched because the setup carries it as a Content object. * fix(realtime): cap the session.update merge depth so the recursive detector accepts it _merge_session_fields recurses once per nested mapping level, so CI's recursive_detector flagged it. Bound the descent at eight levels (the OpenAI session object nests four deep at most), replace anything deeper wholesale with the later session.update's value, and list the helper with the other depth-capped functions the detector allows. * fix(gemini-live): accept a string output modality and drop values Live cannot produce A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once. * fix(gemini realtime): answer an error for session updates Gemini Live cannot apply An output modality the model cannot produce, or a session.update sent after the first content, now gets an invalid_request_error event instead of a silent swap or drop, unless drop_params is set. The synthetic session.created reports the model's default modality, setupComplete no longer adds an extra session.updated, and merging nested session updates is linear in their size * fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns A pre_call guardrail blocking the first typed message now goes out after the setup, so Gemini and Vertex no longer close the session with 1007. Push-to-talk sessions (turn_detection null or create_response false) wrap each audio turn in empty activityStart and activityEnd messages, which Gemini needs when automatic activity detection is off. A session with no content after 60 seconds sends its setup, since Vertex refuses a setup 300 seconds after connect. Held session.update frames past 10 MB answer session_update_too_large, and deferred mode now errors on a second session.update unless drop_params is set. * fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection An update that leaves the applied setup unchanged now gets session.updated instead of session_update_not_applied, so clients that resend their session with the first response keep working. Deferred setup mode merges held updates like the default mode, turn_detection null turns Gemini's activity detection off, drop_params drops a voice outside the documented Live voices, and the lazy setup deadline moves to 240 seconds * fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content * fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name. * fix(realtime): hold first content so a following session.update joins the Gemini setup Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too * fix(realtime): check AI Studio Live voices like Vertex * fix(realtime): forward any voice to AI Studio Live models that accept it * fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup. * fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock The deadline and grace tests waited on real 50 ms timers, and the merge cost check compared wall-clock runs. The session now reads an injected clock, time.monotonic by default, the two tests step it, and the merge cost check moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes the backend through wrap_backend, which main added after the test was written |
||
|
devin-ai-integration[bot]
|
56388fd13d
|
fix(count_tokens): count a history that carries a compaction block (#44784)
* fix(count_tokens): count a compaction block instead of answering 500 A message history that replays Anthropic's signed compaction block made /v1/messages/count_tokens answer HTTP 500: neither the Anthropic nor the Vertex AI partner count-tokens request carried the compact-2026-09-04 beta, the provider rejected the block with a 400, the proxy fell back to the local tokenizer, and that raised on the unknown block type. Both provider count paths now add the compaction beta whenever the history carries a compaction block, and the local tokenizer counts the block's summary text like a text block, skipping the signature. * fix(token_counter): skip the tokenizer for empty thinking and compaction text * fix(count_tokens): forward context_management on Anthropic counts and count chat compaction blocks locally Vertex AI compaction support for inference and counting moves to its own PR, so the Vertex count handler is back to main here * fix(anthropic): map list-form context_management to edits for count_tokens and inference betas Count requests forwarded the OpenAI list form of context_management unchanged, while inference maps it to Anthropic edits, and the inference beta helper read the list as no edits. Both now map it once. A replayed compaction block with an empty summary no longer reaches the tokenizer. * fix(anthropic): forward count context_management only with edits, map Vertex list-form betas, and return provider context_management * fix(anthropic): drop an unmappable context_management list under drop_params --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Gopal Kamineni
|
a7030a5a2c
|
fix(vertex_ai): batch Claude models in Anthropic's native row shape (#45715)
* fix(vertex_ai): batch Claude models in Anthropic's native row shape
Claude deployments on Vertex were filed under publishers/google/models/<claude>
when the batch input was uploaded, so create_batch asked Vertex for a Google
model that doesn't exist and got a bare 404 on batchPredictionJobs. The rows
were also written as Gemini GenerateContentRequests, which Vertex rejects for
Claude even with the right model path
Claude uploads now go under publishers/anthropic/models/<model> and each line
becomes {"custom_id", "request": <Anthropic Messages body with
anthropic_version vertex-2023-10-16>} using the existing Vertex Claude
transform. Output rows in the Anthropic message shape are converted back to
OpenAI batch rows (custom_id from the row, errors as failed rows), native rows
are costed from Anthropic usage, and create_batch fails fast with a 400 when a
Claude job targets the global location, which Vertex doesn't serve for
partner model batches
Fixes #45671
* test(vertex_ai): cover malformed Claude output rows and rows without custom_id
* fix(vertex_ai): build Claude batch rows with chat's param mapping and surface Vertex batch create errors
Claude batch rows now go through get_optional_params like Vertex Claude chat, so unsupported params, top_k, response_format and extra_body follow drop_params. Tool names Claude would rename get a 400 at upload. A refused batch create returns Vertex's own status and message.
* fix(vertex_ai): price Claude batch output rows without a request echo, keep @ version ids, refuse a json_tool_call user tool, keep Vertex error headers
* fix(vertex_ai): keep httpx.HTTPStatusError on Vertex batch create errors and apply deployment chat params to Claude batch rows
* fix(vertex_ai): keep Vertex's status code on batch create errors through the proxy
* fix(vertex_ai): keep Vertex's error body on batch create errors and require custom_id on Claude lines
* fix(vertex_ai): keep the old batch create error text and Google object names
A failed Vertex batch create keeps httpx's error line first and adds Vertex's body after it, and logs the status and body again. Only Claude object names keep the @ version, so Google models are stored under the same names as before
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Alon Sokolovski
|
2b4cb92c8b
|
fix(context_management)!: run clear_tool_uses in the gateway on the Responses API path (#47861)
* fix(context_management): run clear_tool_uses in the gateway on the Responses API path /v1/messages to openai/ models goes through the Responses API, which translated only compact_20260112 (to its native compaction) and dropped every other edit without a warning, so clear_tool_uses_20250919 never cleared anything. Edits without a Responses API form now run in the same polyfill as the chat completions path, before the call, and their applied_edits come back on the response and the final message_delta. compact stays native. * fix(context_management): check the sync polyfill result instead of casting it The strict ruff gate (TID251) bans typing.cast. * test(context_management): cover the Responses path's polyfill error handling A polyfill crash is a 500 before the call, a rejected edit under drop_params is skipped, and the sync stream reports its edits. normalize_spec never raises for a dict spec, so its guard is gone. * fix(context_management): count unreadable blocks and check tool_result text before clearing clear_tool_uses now maps blocks the token counter cannot read (server tool, compaction) to JSON text, so an input_tokens trigger no longer 500s on that history. The Responses API path checks malformed tool_result text on the original messages before clearing, as the chat completions path does. Also removes comments that repeated the code and makes the test fixtures immutable. * fix(context_management): count only blocks the model receives and keep list specs unsent clear_tool_uses now leaves blocks token_counter cannot read, like server tool results and compaction, out of its count instead of counting them as JSON text. No adapter sends them to the model, so an input_tokens trigger no longer fires on tokens the model never sees. On the Responses API path, an OpenAI-style list context_management stays unsent, as it was before, instead of reaching OpenAI's native compaction, where a threshold under 1000 is a 400. On the chat completions path, a clear edit listed before compact_20260112 now runs on the turns after the latest compaction block, as Anthropic does, instead of counting and clearing turns the compaction block drops. The crash test that replaced apply_context_management with a mock is removed. * fix(context_management): apply clear edits in OpenAI count_tokens and skip nested off-spec blocks when counting * fix(context_management): run edits on the history drop_params sends, so count_tokens and /v1/messages clear alike * fix(context_management): count clear_at_least freed tokens over sent blocks only * fix(context_management): count tools with JSON Schema shorthands and reject a non-object trigger or keep A tool whose schema has an array without items, a boolean subschema, or a search_result block without a title made the token counter raise, so a clear edit answered 500. A string trigger or keep also answered 500, even under drop_params. Both now count or get the API's 400, and drop_params drops the malformed field. * fix(token_counter): count a malformed tool schema keyword as absent and move shorthand framed-estimate cases * fix(count_tokens): pass the system prompt into clearing so count_tokens matches inference * fix(context_management): keep a malformed system prompt from failing clear_tool_uses on the Responses API A number, an object, or a list holding non-object items as system made the polyfill answer 500. The polyfill now keeps a list's object items and treats any other shape as no system, on both /v1/messages and count_tokens, so the two paths still count the same text --------- Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
be6d082bdc
|
fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166)
* fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts * refactor(anthropic): hold the chat completions document change for a later commit * fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai * fix(snowflake): turn file parts into document blocks for Cortex Claude * fix(prompt_security): sanitize chat file parts that carry file_data * fix: answer 400 or drop PDFs for models that cannot read them Models whose model_info or cost map entry sets supports_pdf_input to false, and Ollama, now get the PDF dropped under drop_params and a 400 from litellm without it. GigaChat uploads file parts, SAP keeps sending them as image_url data URLs, azure_ai passes dict content through, and annotations other than url_citation move to provider_specific_fields instead of failing the response. * fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider annotations to provider_specific_fields, and split them the same way when a stream is rebuilt. * fix(vertex_ai): send PDF file parts to Vertex partner models as image_url Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL. * fix(cost-map): leave native DeepSeek PDF support unset /v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there and would only change chat completions, which this PR does not otherwise touch. * fix(main): drop a user turn that held only a PDF under drop_params Dropping the PDF from a PDF-only user turn left an empty content list, which Foundry and OpenAI-shaped providers reject with a 400, so drop_params still failed for that request. The emptied user turn is now removed. * fix: read OCI PDFs, gate deployment base models, and stop empty drops OCI generic chat sends file parts as image data URLs again, so OCI Gemini keeps reading PDFs from /v1/messages and chat file parts. The PDF gate now reads model_info.base_model, drops PDFs under drop_params for azure_ai deployments that never declared PDF support (the old flattening behavior), and answers 400 when dropping the PDF leaves no user message. The cost map marks the Vertex partner text models that a live probe showed answering without reading the PDF. * test(integration): cover the PDF input gate, messages PDF file parts and provider annotations Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed * test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells * fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload * fix(main): PDF check reads the deployment's top-level base_model * fix(main): keep roles alternating after drop_params removes a PDF-only turn When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other * test: type the PDF and annotation tests the PR added * test: type the GigaChat, SAP and Snowflake PDF tests |
||
|
tavren27
|
b25ea48c4b
|
fix(anthropic)!: drop beta-gated body fields under drop_params (#48793)
* fix(anthropic): strip beta-gated body fields when a leg drops the anthropic-beta When a provider leg drops an anthropic-beta value (the chat handler's allowlist filter, /v1/messages' filtered headers, Bedrock Converse's additionalModelRequestFields), also drop the body fields that beta gates (thinking.block_binding, thinking.display=updates, mid-conversation system clear_at, output_config, tool_addition/tool_removal blocks) so the request degrades on that leg instead of failing with a provider 400. Copy-on-write so a fallback leg still sends the full request, and each strip is logged at WARNING. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove a leg that drops a beta sends neither the beta nor clear_at Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): strip beta-gated fields against the final body anthropic_beta list Converse counts the body anthropic_beta list as active, Invoke strips after its auto-injected betas are known and injects thinking-binding-controls for thinking.block_binding, test helpers are fully typed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated system fields before an older model's system turn becomes a user turn Models without supports_mid_conversation_system send no mid-conversation beta, so tool_addition and tool_removal blocks are dropped (and logged) before the turn is converted instead of riding along as user content. The Invoke output_config regression test reads the real cost map instead of patching _bedrock_model_supports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): pin the integration proxy to the bundled beta allowlist and type the helper locals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): annotate every new test local with Final and type the fixtures and returns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated body fields only under drop_params Without drop_params the request now goes out as sent and the provider answers its own 400, like any other unsupported param. inline-tools-2026-09-15 also unlocks tool_addition and tool_removal blocks, so a leg that carries it keeps them. * fix(anthropic): strip beta-gated fields on Foundry chat and Bedrock chat Invoke under drop_params Foundry chat and Bedrock chat Invoke now drop beta-gated body fields the same way the other legs do when drop_params is on and the request does not send the gating beta. The mid-conversation system normalizer validates only on the conversion path, and the strip WARNING names the beta the caller can send to keep the field. * fix(anthropic): send the thinking betas for block_binding and display updates, strip Databricks chat thinking Every Claude leg now adds the thinking-binding-controls beta when thinking carries block_binding and the display-updates beta for display updates, so those fields go through instead of a 400. Databricks chat rejects both fields with or without the beta, so it drops them under drop_params. A system turn the strip empties stays when it still carries output_config, and the drop warning names both causes. * fix(anthropic): send the thinking betas on every leg and keep fields where the beta is unknown Vertex messages and Claude Platform chat now add the thinking betas when thinking carries block_binding or display updates. Converse sends only the caller's header betas when it has them. A provider with no row in the beta config forwards beta-gated fields instead of dropping them, an emptied trailing system turn that would leave an assistant prefill is sent as is, and the prompt cache baseline body matches the body that goes out. The drop WARNING now names the beta that unlocks each field * fix(anthropic): filter Converse body betas, keep Foundry auto betas, read the bundled beta config for row checks * fix(anthropic): validate messages only when a beta-gated drop runs A non-object messages entry on a Claude model without mid-conversation system support raised a raw pydantic ValidationError before the beta-config and drop_params checks. The strip now passes the messages through untouched unless it drops something, so the provider's 400 reaches SDK callers as BadRequestError again * fix(anthropic): keep system blocks with a non-string type instead of raising --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
184ec7b894
|
fix(vertex_ai): build the cachedContents collection URL under a custom api_base (#44783)
* fix(gemini): use collection endpoint for custom context cache * fix(vertex_ai): build the cachedContents collection URL under a custom api_base * fix(vertex_ai): keep a gateway prefix that contains /publishers/ * test(integration): expect the cachedContents collection URL on a custom Gemini api_base * fix(vertex_ai): cut cache collection URLs at the base's location, graft a mount prefix * fix(vertex_ai): cache under a publisher model api_base in that model's location A Vertex AI api_base that names an endpoint or no model keeps its old cache URL, so no cache is created that the chat call cannot use. The cache body takes its project and location from the api_base model, matching the collection URL. * fix(vertex_ai): cache for the api_base's model and skip caching on a Gemini v1 base A Vertex api_base naming a publisher model now gives the cache its model as well as its project and location, so the cache matches the model generateContent calls. A Gemini api_base ending in /v1 skips context caching, since the v1 API has no cachedContents and rejects the cachedContent field. * fix(vertex_ai): check the cache token minimum against the api_base's model The cache is created for the model a publisher-model api_base names, so the minimum prompt size now comes from that model too. A prompt below its minimum is sent uncached instead of failing at cache create. * fix(vertex_ai): use the stricter cache minimum and send a Gemini base's 404 uncached --------- Co-authored-by: paulo037 <paulo.h.carneiro.18@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
310c96162b
|
fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893)
* fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema * fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema. * fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them. * fix(bedrock): drop tool custom object on the Mantle count_tokens fallback Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way. * fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages * test(vertex): annotate count_tokens tool test locals with Final * fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names * fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
3b9a59348f
|
fix(vertex_ai)!: send the caller's anthropic-version in the Vertex Claude /v1/messages body, not the header (#48814)
* fix(vertex_ai): drop the caller's anthropic-version header on Claude /v1/messages Vertex reads the API version from the body's anthropic_version and answers 400 when a forwarded anthropic-version header names a different one, which anthropic SDK 2.x always does. Strip the header on the Python and LITELLM_RUST Vertex messages routes. * fix(vertex_ai): carry the caller's anthropic-version into the Claude /v1/messages body Vertex reads the API version from the body alone, so the caller's anthropic-version header now becomes the body's anthropic_version instead of a pinned vertex-2023-10-16. The 2023-01-01 and 2023-06-01 first-party versions keep vertex-2023-10-16, and the header itself is still never sent. Same change on the Python proxy and the Rust gateway. * fix(vertex_ai): follow drop_params for an anthropic-version Vertex does not accept Only versions Vertex accepts in the body (vertex-2023-10-16, 2026-10-21) are carried into anthropic_version. 2023-06-01, 2023-01-01, a blank header, or no header keep vertex-2023-10-16. Any other value drops to vertex-2023-10-16 under drop_params and answers a 400 naming drop_params without it, on the Python proxy and the Rust gateway. * refactor(vertex_ai): return the unsupported anthropic-version error as a value and keep Rust helpers and tests in place * fix(rust gateway): resolve os.environ drop_params and accept the same flag words as Python The gateway read drop_params: os.environ/NAME and words like yes as off, so the drop_params opt-out from the new anthropic-version 400 did not work there * docs(rust router): name Router::from_config in the crate README --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
rj-cairn
|
d06b889d41
|
fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream The Gemini and Vertex AI streaming path read the SSE body with httpx iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085, U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds one of them was cut in two. The tail is not a data: line and the head never parses, and the accumulate-and-retry fallback then carries the unparsed prefix forward, so every later event is appended to it and the stream ends with no more content and no usage. SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines split decoded text on exactly those, treating a CR at the end of one chunk and an LF at the start of the next as one break. The first request and every long-decode hop now read through them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks. * fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator * fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed Switch the remaining httpx stream readers (Anthropic chat, Google GenAI, Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a raw U+2028 or U+2029 inside a JSON string no longer splits an event. Keep the chunk that closes a think block in the logged and cached response when a stream ends without a finish_reason, close the block before a legacy function_call, and make sync Codestral FIM streaming send stream=true and use the text-completion-codestral stream parser. Test fakes that only mocked iter_lines now stream real httpx bytes and assert the content they feed in. * fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use split_sse_lines from sse_lines.py, which also gains a str.split fast path. Adds tests/unit/llms/codestral to the unit workflow shard. * fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines * test: type the new stream tests, drop a comment, and benchmark the SSE line splitter * test: type the iterator class parameters in the line-separator stream tests * fix(responses): stream tool call arguments opened in a text chunk * fix(responses): hold a tool call that shares a chunk with text until the text ends * test: type the a2a SSE handler helper and the sandbox fake responses --------- Co-authored-by: Tim Vink <vinktim@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> |
||
|
moe-berri
|
9e878cfae2
|
test(gemini): cover every web source for a grounded passage (#48861)
* fix(gemini): preserve every grounding web source * test(gemini): cover all grounding sources per passage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Zabir Raihan <zabirraihan@icloud.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Zabir Raihan
|
7d79824391
|
fix(gemini): preserve every grounding web source (#48745) | ||
|
shrey
|
4fa860b64c
|
fix(vertex-ai): preserve image size aspect ratios (#48463) | ||
|
ryan-crabbe-berri
|
db5b940143
|
fix(vertex_ai): send Vertex Claude labels as X-Vertex-AI-Labels header instead of the request body (#48753)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
b9e3587721
|
fix(embeddings): forward Gemini and Vertex request timeouts (#48690)
Co-authored-by: DDT <141296842+Hugo-DDT@users.noreply.github.com> |
||
|
tin-berri
|
b9e4298506
|
fix: preserve media across Responses and Gemini conversions (#48484) | ||
|
yucheng-berri
|
788868a52f
|
fix(security)!: build proxy-only failure logs from an allowlist and drop caller guardrail info (#48599)
* fix(proxy): build proxy-only failure logs from an allowlist and ignore request-time GCS credentials * fix(proxy): drop caller metadata from proxy-only failure logs metadata and litellm_metadata leave the proxy-only failure-log allowlist, so a rejected request can no longer point its failure log at its own Langsmith or Langfuse host through nested metadata. * fix(proxy): keep allowlisted metadata in proxy-only failure logs * test(proxy): isolate proxy-only failure log metadata tests * fix(logging): ignore non-list request tags when building the logging payload A guardrail-rejected request with metadata tags set to a non-list value crashed the standard logging payload, so its spend row lost model attribution. * fix(proxy): keep router attribution and drop unauthenticated tags in proxy-only failure logs A request rejected before routing now logs the configured deployment and model group even when its metadata or litellm_metadata carries its own values. Requests rejected at auth no longer write caller tags into failure logs and tag spend. * fix(proxy): always strip client guardrail info and mirror failure attribution into litellm_metadata allow_client_pricing_override no longer keeps a caller-supplied standard_logging_guardrail_information on the chat or pass-through paths, since its guardrail_cost is billed. Proxy-only failure logs now also carry key and team attribution in litellm_metadata. * refactor(proxy): keep the proxy-only failure-log fixes inside the type gates Types the picked helpers so basedpyright and the type-discipline gate stay at main's counts. No runtime change. * fix(proxy): keep the WS frame guardrail-info strip and string interaction tags as sent The guardrail-info key left the pricing strip list, so the Responses WS frame guard now drops it by name. get_request_tags no longer raises on string tags, so background interaction polling checks for them before keeping them as sent. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
14193435b8
|
refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages Move the unified /v1/messages code out of a directory called pass_through, since it is not the passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner model experimental_pass_through package becomes messages. The old packages stay as import shims that warn once and alias every submodule in sys.modules, so old deep imports still resolve Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): keep Final variables out of loops in the rename shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): match the warning text in the rename shim test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): resolve old pass_through imports lazily through a meta path finder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): mark the alias loader spec restore for the type discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): format the renamed package shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): keep the rebind-ok marker on the spec restore line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): carry the new package search path on alias specs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): declare the shim test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
f2b59be6ea
|
fix(vertex): keep Messages authentication off the event loop (#48523) | ||
|
Mateo Wang
|
7e14638001
|
refactor(llms): rename model-version transformation files to family names (#48509)
gpt_5_transformation.py becomes gpt_reasoning_transformation.py (OpenAI and Azure), the Bedrock anthropic_claude3_transformation.py files become anthropic_claude_transformation.py, anthropic_claude2_transformation.py becomes anthropic_legacy_text_completions_transformation.py, and vertex_ai_partner_models/llama3 becomes vertex_ai_partner_models/llama. Class and symbol names are unchanged. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
9a38ff4cb5
|
fix(gemini): preserve tool media for older models (#48428) | ||
|
tin-berri
|
f96200bcea
|
fix(vertex_ai): bound shared credential cache retention (#48442) | ||
|
tin-berri
|
1ecc782986
|
fix(vertex_ai): reuse rerank credential cache (#48430) | ||
|
Kerry Lu
|
dc0237ccd7
|
fix(cost): bill gemini audio input and cached audio at flex, priority and batch rates (#48086)
* fix(gemini): price tiered audio token rates * fix: correct Gemini audio pricing catalog * fix(gemini): preserve cached modality pricing --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
66c8f5b1a9
|
fix(llms): join api_base and route with one shared helper (#47535)
* fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * fix(llms): join api_base and route with one shared helper A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base * refactor(llms): move per-provider api_base strips onto join_api_base Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join * ci: run the trailing-slash URL builder test in the llms shard * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell * fix(llms): send every provider route through join_api_base Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash * fix(llms): keep the route skip only where a builder already had it join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path already ends in it, and it is used only where the merge base or the trailing-slash test PR head already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the path so a host named anthropic-* builds /anthropic/v1/messages * fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL * test(llms): run the Gemini RAG trailing-slash case through aingest The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears * test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base * test(integration): cover api_base joins across providers, query strings and named hosts * feat(url_utils): add join_api_base and ensure_api_route helpers * fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host join_api_base and ensure_api_route now keep the // of a base left with only a scheme, so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it * fix(azure_ai): build the count_tokens URL from the messages URL * fix(azure_ai): match anthropic as a whole path segment in the messages URL * fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints Azure AI chat and messages go back to cutting the base at the first /anthropic, as on main, and count_tokens keeps its path check, so a gateway path like /anthropic-proxy builds the same URLs as main. A base that already ends in /v1/messages still gets count_tokens next to it. A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is used as its text again in AgentCore, Bedrock responses and the RAGFlow base, instead of falling back to the default host. get_volcengine_base_url and GigaChat's get_api_base strip a trailing slash again for outside callers. The cache-hit wire test is skipped as a known proxy bug, since one cache hit writes two cache_hit spend rows. * fix(llms): drop a slash before the api_base query and tighten the route test checks * test(llms): annotate new test locals as Final and flatten the builder generator * fix(url_utils): keep a scheme-only api_base hostless so it fails before sending --------- Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com> |
||
|
Mateo Wang
|
820725661e
|
fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage (#45194)
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first. * fix(proxy): log a provider error frame through an attempt-scoped logging copy The error-frame failure stashed the superseded attempt's usage on the logging object the retry shares, so the retry's success log carried the failed attempt's token counts. The failure now logs through a copy of the logging object with its own model_call_details, and the helper moves into the Anthropic passthrough logging handler so the proxy streaming handler only routes by endpoint type. * fix(proxy): scope the error-frame failure log to streams closed at the frame A stream that ran to its end at the provider's error frame kept logging through the success route before this PR, and the attempt-scoped failure log is now taken only for a stream the router closed at that frame, with the copy made at the close so the retry's deployment never reaches it * fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage A /v1/messages stream dropped after message_start and retried in its model group shares one Logging object across attempts, so the retry's success log inherited the dropped attempt's stashed usage and exception, and a second drop never reached the failure handlers because the first had already marked the object as logged. The dropped attempt now dispatches its failure handlers through an attempt-scoped copy, and the router discards the stashed usage right before it opens the next attempt, on same-group retries and on every fallback hop, so a request that runs out of attempts still bills the last attempt's partial usage. * fix(router): dispatch chat stream failures through the attempt-scoped logging copy The router now clears a dropped attempt's stashed usage before every fallback hop, so the chat stream wrapper's queued failure log must read that usage from its own copy of the logging object, like the /v1/messages path already does. The stash test asserts through a real success log, the cooldown test reads a real Router's cooldown list, and the integration helpers carry full types. * test(vertex): give the stream wrapper's logging stub an attempt_scoped_copy * test: route the pass-through logging mock's attempt copy to itself and read the callback sink through a reader The pass-through failure test stubbed dispatch_failure_handlers on a spec'd MagicMock, which the attempt-scoped copy no longer reached, so the logging worker never finished. The integration tests kept earlier sink batches in a caller-owned list the helper mutated; a small reader keeps them instead. * test(router): use real Logging in fallback cooldown test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): preserve superseded retry usage for billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): bill retry input after prior stream usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): track retry dispatch for terminal billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): use deterministic retry timestamp Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): avoid real clock in fallback tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): remove duplicate streaming logger import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): retain error-frame usage for exhausted retries * test(proxy): fix retry billing expectations and helper annotations * fix(proxy): preserve failure estimate without carried usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): audit retry helpers and fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): fix retry fixture timestamps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): observe retry stash at attempt boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): keep terminal failure attribution when dropped attempts log on copies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish a dropped attempt's payload only while it is still the current attempt Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover spend-log encoding error propagation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): skip adapter failure dispatch when the wrapped stream already logged it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): cool down the final fallback attempt when its failure callbacks were blocked Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): import Final in adapter regression Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop spend-log encoding expectation superseded by main's poison isolation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): finish the wrapped stream's failure log before surfacing the error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): wait for the wrapped stream's failure log to finish like the inline dispatch did Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): wait for the executor-run sync failure callback separately from the async caller boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): pin pass-through stream fixture start times Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload before awaiting failure callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload from the failure handler after baseline invalidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(logging): drop unused discard_partial_usage_for_failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Emerson Gomes
|
045fb7ba84
|
feat(vertex-ai)!: support gemini 3.1 flash tts (#31915)
* feat(vertex-ai): support gemini 3.1 flash tts
* fix(vertex-ai): keep TTS metadata compatible with the cost map guard
* fix(vertex-ai): preserve existing Gemini TTS models and voice mappings
* fix(vertex-ai): recognize provider-stripped TTS model names
* test(vertex-ai): match current speech HTTP error response
* fix(vertex-ai): preserve nested speech locale and voice
* fix(vertex-ai): use provider speech media type in proxy
* fix(gemini): preserve TTS dispatch and nested speech settings
* test(gemini): run TTS regressions in CI shards
* fix(vertex-ai): account for Gemini Cloud TTS audio cost
* fix(vertex-ai): bill Gemini TTS without optional audio decoder
* test(gemini): cover TTS mapping and billing edge cases
* test(vertex-ai): cover Gemini voice normalization in provider shard
* fix(vertex-ai): keep Gemini TTS routing with generic provider metadata
* chore(vertex-ai): remove obsolete lint suppressions
* fix(gemini): strip audio format from speech configuration
* chore(vertex-ai): remove obsolete mutation suppressions
* fix(vertex-ai): preserve FLAC billing and wrapped audio headers
* fix(vertex-ai): reject unmeasurable FLAC before billing
* fix(vertex-ai): reconcile speech support with current provider mappings
* fix(vertex-ai): validate speech mappings and usage types
* fix(vertex-ai): preserve speech mapper override forwarding
* fix(gemini): map voice aliases across speech request forms
* fix(vertex-ai): send only formats the speech bridge cannot serve to Cloud TTS
Gemini TTS detection now goes by model name, so newer Vertex Gemini TTS
models keep working, and only mp3, opus, alaw, mulaw and other container
formats move to Cloud Text-to-Speech. wav and pcm requests stay on
generateContent as on main.
A Cloud TTS response whose duration cannot be measured now returns the
audio unbilled with a warning instead of raising after a paid call. FLAC
handling is removed because Cloud TTS has no FLAC encoding.
The cost map, the google_genai mappers and the proxy error path go back
to main, since those changes were outside this feature.
* fix(vertex_ai): keep Gemini TTS formats, region and instructions on routes that serve them
Only mp3, opus, alaw and mulaw go to Cloud Text-to-Speech. Every other
format gets a local 400 that lists the supported ones, before Google is
called.
Cloud TTS uses the deployment's us or eu regional host. Other regions
and a custom api_base get a 400 that says to request wav or pcm.
instructions is sent as the Cloud TTS style prompt, and a request with
no voice keeps the Gemini model.
get_usage_object is back to main's code, so dict usages still go
through Usage.__init__, and the proxy no longer labels pcm16 or
linear16 output as WAV.
* fix(vertex_ai): send Gemini TTS to every regional Cloud TTS host and keep speak tags as text
Cloud Text-to-Speech serves Gemini TTS on a host named after each Vertex
location, so a non-global vertex_location now uses
{location}-texttospeech.googleapis.com instead of only the us and eu hosts.
Google answers for regions it does not serve.
Gemini TTS voices reject SSML, so input that contains <speak> is sent as
text for Gemini TTS models. Chirp keeps the SSML auto-detect.
Removes the proxy alaw and mulaw media-type entries. Cloud TTS returns
those encodings with a WAV header, so the response is labeled audio/wav
and the entries never applied.
Adds a regression test that custom per-token pricing bills the usage a
speech response reports.
* fix(vertex_ai): map Gemini TTS regions to Google's Cloud TTS locations and keep a bad voice from cooling down the deployment
US and EU Vertex regions go to the us and eu Cloud TTS hosts, northamerica-northeast1 to its own, global or unset to global, and any other region gets a local 400. A Cloud TTS 404 on the Gemini path, such as a misspelled voice, comes back as a 400 so the router keeps the deployment serving wav. Usage reads the sent request from the response instead of the logging object.
* fix(vertex_ai): default Gemini TTS Cloud formats to a Gemini voice and validate speakers
Parse only usage fields from the sent Cloud TTS request, so a numeric speakingRate no longer fails after Google bills it.
Send the Gemini voice Kore when no voice is named, and reject Cloud TTS voice names unless drop_params is set.
Reject malformed speaker entries unless drop_params is set, and translate Cloud TTS speaker fields on wav and chat audio.
Use the Vertex default region when no location is set, so mp3 and wav share a region.
* test(vertex_ai): cover two-speaker Gemini TTS mp3 through aspeech with fake credentials
Replaces two tests that patched the Vertex TTS config's own token and URL methods.
* fix(vertex_ai): reject malformed Gemini TTS voices and keep beta Chirp and Lyria params unmapped
* fix(vertex_ai): drop unknown Gemini TTS voice keys on wav and pcm instead of rejecting them
* fix(vertex_ai): reject unknown Gemini TTS voice keys on wav, pcm, and chat audio unless drop_params is set
* fix(vertex_ai): honor every drop_params setting in the Gemini TTS voice key check
Global drop_params, batch row and deployment drop_params on Vertex batch
file uploads now drop unknown voice keys. A voice object with both
speechConfig and speech_config gets a 400 for the ignored speech_config.
* fix(gemini): drop unknown nested voice fields under drop_params and detect TTS by served model
* fix(gemini): filter unknown speech config fields with fixed-depth helpers and pass audio params through unchanged
* fix(gemini): camel-case snake replicated voice keys so drop_params keeps them
* fix(vertex-ai): reject voice fields Cloud TTS cannot send unless drop_params is set
* fix(vertex-ai): reject wrong-typed Cloud TTS voice values and check speakers as sent
Gemini TTS on mp3, opus, alaw, or mulaw now answers 400 for a voice field whose value Cloud TTS cannot send, such as a list languageCode or a string voiceConfig, and for nested voice fields under a speaker that also names speakerId. drop_params still drops them.
* fix(vertex-ai): reject wrong-typed speech config wrappers and speakerId overriding a prebuilt voice on Cloud TTS
* fix(vertex-ai): keep speakers and voice_config beside a speech_config wrapper on Cloud TTS
* fix(vertex-ai): pick Cloud TTS voices from the same speech config the WAV path sends to Gemini
* fix(files): carry the deployment's drop_params on model= file uploads so Vertex batch rows drop unknown voice keys
* fix(gemini): keep a structured speech voice when the model is a model_alias_map alias for a Gemini TTS model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
11b5d1a817
|
feat(gemini): set the long decode budget for requests with no output limit from LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS (#47542)
* feat(gemini): LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS sets the long decode budget for requests that send no limit * test(gemini): cover a default long decode budget set by LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS * fix(gemini): apply the default output budget only to requests asking for text output * refactor(gemini): apply the default output budget once in the async request transform * fix(gemini): leave the default output budget off non-chat models and a snake_case generation_config * fix(gemini): skip the default output budget for audio-only models and continuation resumes Lyria and other Gemini chat models whose cost map row lists no text output answer 400 when sent a maxOutputTokens, and a request that resumes from a continuation token answers 400 when a limit is added beside it. Both now go out without the default budget. * fix(vertex_ai): skip the Gemini default budget for a model a deployment's model_info overrides, and pass odd generationConfig shapes through * fix(vertex_ai): cap the Gemini default budget per deployment at the loaded cost map and its own model_info limit * test(vertex_ai): wrap a long test signature under the 120 character limit * fix(vertex_ai): keep a direct call's default output budget under the loaded cost map limit A litellm.Router deployment's model_info writes onto the shared cost map row, so a later direct call with no model_info read that raised value and Vertex answered 400. The direct call now caps at the lower of the loaded row and the live row, so litellm.register_model can still lower the cap. * fix(vertex_ai): ignore a deployment output limit that is not positive A model_info.max_output_tokens of 0 or -1 capped the default budget at that value, and Google answers 400 to a limit that is not positive. It now counts as unreadable, so the cost map limit applies. * fix(gemini): never continue a request asking for more than one candidate in camelCase * fix(gemini): keep continuing a request whose candidate count is null or 0 |
||
|
alberto
|
f768fb32ef
|
fix(streaming)!: send the include_usage chunk with an empty choices array (#47109)
* fix: Send the include_usage chunk with an empty choices array Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Skip chunks with no choices when reading streams Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Strip usage from plain-iterator async stream chunks Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Expect empty choices on the usage chunk in the gemma tracing test Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage The usage chunk now has choices=[] only when the caller set stream_options.include_usage itself. When the OpenAI handler, the proxy's always_include_stream_usage setting, or the harness added include_usage on the caller's behalf, the chunk keeps its legacy single empty-delta choice, so clients that read choices[0] on every chunk keep working. The proxy and harness mark their injection with the private _litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy. Plain-iterator streams keep usage on their chunk when usage was not requested, and the LLM Shield trailing flush ignores chunks with no choices. * fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them. * fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route * refactor(router): move the stream usage injection check into router_utils * fix(router): read stream_options from pydantic deployment params when marking injected usage Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices. * fix(proxy): keep the stream usage injection marker out of stored request bodies * test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths * test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
c29a4f56a3
|
fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later (#47645)
* fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later Google ignores temperature, top_p and top_k on Gemini 3.6 Flash, Gemini 3.5 Flash-Lite and every later Gemini release, and says a later generation will return 400 for them. The model map now carries supports_sampling_params: false on those rows, and the Gemini and Vertex transformation reads the flag the way the Anthropic and Bedrock paths already do, dropping the params under drop_params and otherwise raising the client-side 400 that names drop_params BREAKING CHANGE: a request that sends temperature, top_p or top_k to a flagged Gemini model without drop_params now gets a 400 instead of a 200 whose sampling params Google silently ignored Fixes #47015 * fix(gemini): gate the passed-through top_k where the request body is built and keep allowed_openai_params working top_k is a provider-specific kwarg that never reaches the OpenAI param mapping, so the flag has to be applied while the Gemini request body is built, the same boundary Anthropic uses. The mapping now skips a flagged sampling param instead of raising, because the shared unsupported-param check already returns the 400 that names drop_params, and a param the caller lists in allowed_openai_params is forwarded verbatim as that 400 promises * fix(gemini): filter top_k out of the generation config instead of mutating optional_params * test(gemini): type the sampling-flag twin test and keep its locals out of the loop body * fix(vertex_ai): reject or drop sampling params on batch rows for flagged Gemini models The batch JSONL wrapper calls map_openai_params directly and skips get_optional_params, so a flagged model's temperature, top_p and top_k vanished from the uploaded row with no 400. Batch rows now get the same contract as a real-time call: a 400 naming drop_params, or the knobs dropped when litellm.drop_params is on * fix(vertex_ai): ignore null sampling knobs on batch rows for flagged Gemini models A batch row carrying temperature: null, top_p: null or top_k: null means no override, the same as on a real-time call, so the row is no longer refused with a 400 and the null knobs are simply not forwarded |
||
|
devin-ai-integration[bot]
|
f3e1307ed9
|
fix(vertex_ai): log Gemini 3 sampling deprecation once per model per worker (#40868)
* fix(vertex_ai): log Gemini 3 sampling deprecation once per process Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): bound the Gemini 3 sampling warn-once cache and test it by behavior * fix(vertex_ai): log the Gemini 3 sampling deprecation once even when first calls race Pin the warned-model cap to its own constant so the shared LRU size knob cannot turn the dedupe off, and serialize the cached call so concurrent first requests for one model log it once * fix(vertex_ai): stop logging the Gemini 3 sampling deprecation for new models once 64 are remembered * fix(vertex_ai): give each mapped Gemini 3 model its own sampling notice and share one for unmapped names * fix(vertex_ai): key the Gemini 3 sampling notice on the loaded catalog so runtime-priced names share one notice --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |