mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-04 07:27:51 +00:00
35 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Mateo Wang
|
a713ed6bab
|
fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them (#47692)
* test(e2e): add gemini tool call id length repro for LIT-8912 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them * fix(gemini): pair a tool result with its call when the client re-adds the thought signature to the call id Strands, which the AgentCore harness runs, puts provider_specific_fields.thought_signature back on the assistant tool call id as <id>__thought__<signature> while its tool result keeps the bare id, so the Gemini conversion found no matching call and answered 500. Ids now match with the signature suffix stripped. Also adds thought_signature_in_tool_call_id to the litellm_params inventory test and gives the pass-through tests' mocked logging objects the litellm_params a real Logging object always has. * fix(gemini): keep the thought signature on streamed Messages and Responses tool calls and accept a string setting * fix(gemini): read the thought signature back from Responses input and coerce the per-request setting A Responses function_call input item now keeps its provider_specific_fields, so a client that sends the item back gets its real thought signature replayed to Gemini. A per-request thought_signature_in_tool_call_id of "false" or 0 now keeps ids short, and a value that is not a boolean answers 400, or is dropped under drop_params, instead of a 500. The e2e test now uses the sdk fixture and @meta subjects * fix(caching): key the response cache on thought_signature_in_tool_call_id * fix(responses): send thought signature fields only with short ids and drop malformed routing kwargs whole * fix(main): validate routing kwargs after the locals snapshot so fallbacks and model_list calls keep working The e2e test now uses gemini-3.8-flash with enough max_tokens to finish thinking before the tool call. * fix(gemini): keep non-chat routes tolerant of a malformed thought_signature_in_tool_call_id, match tool results by exact id first, and keep the deployment's value when drop_params drops a malformed override * fix(router): validate strict litellm params through a helper so star imports cannot re-bind a public Final * perf(router): skip strict validation when a request overrides no deployment param * fix(responses): keep a dict tool call's thought signature on the streamed function_call done item * refactor(main): return invalid supplemental params as a value and test the streaming tool call with a real response --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tessellark
|
5d5e3685c1
|
fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths (#48933)
* fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths
* fix(logging): keep guardrail and token-count reads of a request from logging drop warnings
Only the real Anthropic-messages adapter request reports a dropped thinking budget. Guardrail,
shadow-eval and token-count translations of the same request now drop it silently, so one request
logs one warning. The 422 retry also no longer hashes each loc entry, so a list inside loc cannot
raise TypeError.
* fix(bedrock): keep the DeepSeek R1 Converse reasoning_effort drop at debug level
Only allowed_openai_params reaches that branch, and it puts reasoning_effort back after the provider mapping, so a warning there claimed a drop that never happened
* fix(logging): warn on Gemini non-integer budget drops and keep the auto-router baseline silent
Gemini 2.x now logs the drop of a non-integer thinking budget. The auto-router
savings baseline builds a body it never sends, so it no longer warns. Logged
values are capped with reprlib, Azure AI marks only its drop_params retry as
gated, and Bedrock Chat Completions names the model without its route prefix.
* fix(logging): warn on the Gemini 3, Magistral, Ollama, Fireworks and DeepSeek ARN reasoning drops
Gemini 3 and later now log a dropped thinking budget, Magistral logs the
level it swaps for a reasoning prompt, Ollama logs a level it maps to
think off, Fireworks logs chat_template_kwargs effort keys dropped for a
model the map does not list as reasoning, and DeepSeek on Bedrock logs a
drop when the model is addressed by an ARN that advertises the param.
* fix(logging): warn on the remaining Ollama, DeepSeek and Tencent reasoning drops
The ollama/ completion route now shares the chat route's think mapping, so
a level it turns off logs the same warning. DeepSeek warns when it drops
thinking.budget_tokens, and Tencent warns when an adaptive-only model gets a
non-integer budget. Magistral no longer warns on low, medium, high or
enabled thinking, the values its reasoning prompt stands in for. The Gemini 3
budget warning no longer claims the model uses its default level.
* fix(reasoning): warn when Claude drops a non-string reasoning_effort
Claude chat and native /v1/messages dropped a reasoning_effort like 5 or
{"effort": 5} with no log line. Both now log one warn-only WARNING. A
Magistral thinking.type sent as a list no longer raises TypeError.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
rj-cairn
|
d06b889d41
|
fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream The Gemini and Vertex AI streaming path read the SSE body with httpx iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085, U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds one of them was cut in two. The tail is not a data: line and the head never parses, and the accumulate-and-retry fallback then carries the unparsed prefix forward, so every later event is appended to it and the stream ends with no more content and no usage. SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines split decoded text on exactly those, treating a CR at the end of one chunk and an LF at the start of the next as one break. The first request and every long-decode hop now read through them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks. * fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator * fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed Switch the remaining httpx stream readers (Anthropic chat, Google GenAI, Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a raw U+2028 or U+2029 inside a JSON string no longer splits an event. Keep the chunk that closes a think block in the logged and cached response when a stream ends without a finish_reason, close the block before a legacy function_call, and make sync Codestral FIM streaming send stream=true and use the text-completion-codestral stream parser. Test fakes that only mocked iter_lines now stream real httpx bytes and assert the content they feed in. * fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use split_sse_lines from sse_lines.py, which also gains a str.split fast path. Adds tests/unit/llms/codestral to the unit workflow shard. * fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines * test: type the new stream tests, drop a comment, and benchmark the SSE line splitter * test: type the iterator class parameters in the line-separator stream tests * fix(responses): stream tool call arguments opened in a text chunk * fix(responses): hold a tool call that shares a chunk with text until the text ends * test: type the a2a SSE handler helper and the sandbox fake responses --------- Co-authored-by: Tim Vink <vinktim@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> |
||
|
moe-berri
|
9e878cfae2
|
test(gemini): cover every web source for a grounded passage (#48861)
* fix(gemini): preserve every grounding web source * test(gemini): cover all grounding sources per passage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Zabir Raihan <zabirraihan@icloud.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Zabir Raihan
|
7d79824391
|
fix(gemini): preserve every grounding web source (#48745) | ||
|
shrey
|
4fa860b64c
|
fix(vertex-ai): preserve image size aspect ratios (#48463) | ||
|
tin-berri
|
b9e4298506
|
fix: preserve media across Responses and Gemini conversions (#48484) | ||
|
Mateo Wang
|
14193435b8
|
refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages Move the unified /v1/messages code out of a directory called pass_through, since it is not the passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner model experimental_pass_through package becomes messages. The old packages stay as import shims that warn once and alias every submodule in sys.modules, so old deep imports still resolve Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): keep Final variables out of loops in the rename shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): match the warning text in the rename shim test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): resolve old pass_through imports lazily through a meta path finder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): mark the alias loader spec restore for the type discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): format the renamed package shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): keep the rebind-ok marker on the spec restore line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): carry the new package search path on alias specs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): declare the shim test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
9a38ff4cb5
|
fix(gemini): preserve tool media for older models (#48428) | ||
|
Kerry Lu
|
dc0237ccd7
|
fix(cost): bill gemini audio input and cached audio at flex, priority and batch rates (#48086)
* fix(gemini): price tiered audio token rates * fix: correct Gemini audio pricing catalog * fix(gemini): preserve cached modality pricing --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
820725661e
|
fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage (#45194)
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first. * fix(proxy): log a provider error frame through an attempt-scoped logging copy The error-frame failure stashed the superseded attempt's usage on the logging object the retry shares, so the retry's success log carried the failed attempt's token counts. The failure now logs through a copy of the logging object with its own model_call_details, and the helper moves into the Anthropic passthrough logging handler so the proxy streaming handler only routes by endpoint type. * fix(proxy): scope the error-frame failure log to streams closed at the frame A stream that ran to its end at the provider's error frame kept logging through the success route before this PR, and the attempt-scoped failure log is now taken only for a stream the router closed at that frame, with the copy made at the close so the retry's deployment never reaches it * fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage A /v1/messages stream dropped after message_start and retried in its model group shares one Logging object across attempts, so the retry's success log inherited the dropped attempt's stashed usage and exception, and a second drop never reached the failure handlers because the first had already marked the object as logged. The dropped attempt now dispatches its failure handlers through an attempt-scoped copy, and the router discards the stashed usage right before it opens the next attempt, on same-group retries and on every fallback hop, so a request that runs out of attempts still bills the last attempt's partial usage. * fix(router): dispatch chat stream failures through the attempt-scoped logging copy The router now clears a dropped attempt's stashed usage before every fallback hop, so the chat stream wrapper's queued failure log must read that usage from its own copy of the logging object, like the /v1/messages path already does. The stash test asserts through a real success log, the cooldown test reads a real Router's cooldown list, and the integration helpers carry full types. * test(vertex): give the stream wrapper's logging stub an attempt_scoped_copy * test: route the pass-through logging mock's attempt copy to itself and read the callback sink through a reader The pass-through failure test stubbed dispatch_failure_handlers on a spec'd MagicMock, which the attempt-scoped copy no longer reached, so the logging worker never finished. The integration tests kept earlier sink batches in a caller-owned list the helper mutated; a small reader keeps them instead. * test(router): use real Logging in fallback cooldown test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): preserve superseded retry usage for billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): bill retry input after prior stream usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): track retry dispatch for terminal billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): use deterministic retry timestamp Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): avoid real clock in fallback tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): remove duplicate streaming logger import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): retain error-frame usage for exhausted retries * test(proxy): fix retry billing expectations and helper annotations * fix(proxy): preserve failure estimate without carried usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): audit retry helpers and fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): fix retry fixture timestamps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): observe retry stash at attempt boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): keep terminal failure attribution when dropped attempts log on copies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish a dropped attempt's payload only while it is still the current attempt Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover spend-log encoding error propagation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): skip adapter failure dispatch when the wrapped stream already logged it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): cool down the final fallback attempt when its failure callbacks were blocked Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): import Final in adapter regression Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop spend-log encoding expectation superseded by main's poison isolation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): finish the wrapped stream's failure log before surfacing the error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): wait for the wrapped stream's failure log to finish like the inline dispatch did Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): wait for the executor-run sync failure callback separately from the async caller boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): pin pass-through stream fixture start times Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload before awaiting failure callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload from the failure handler after baseline invalidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(logging): drop unused discard_partial_usage_for_failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Emerson Gomes
|
045fb7ba84
|
feat(vertex-ai)!: support gemini 3.1 flash tts (#31915)
* feat(vertex-ai): support gemini 3.1 flash tts
* fix(vertex-ai): keep TTS metadata compatible with the cost map guard
* fix(vertex-ai): preserve existing Gemini TTS models and voice mappings
* fix(vertex-ai): recognize provider-stripped TTS model names
* test(vertex-ai): match current speech HTTP error response
* fix(vertex-ai): preserve nested speech locale and voice
* fix(vertex-ai): use provider speech media type in proxy
* fix(gemini): preserve TTS dispatch and nested speech settings
* test(gemini): run TTS regressions in CI shards
* fix(vertex-ai): account for Gemini Cloud TTS audio cost
* fix(vertex-ai): bill Gemini TTS without optional audio decoder
* test(gemini): cover TTS mapping and billing edge cases
* test(vertex-ai): cover Gemini voice normalization in provider shard
* fix(vertex-ai): keep Gemini TTS routing with generic provider metadata
* chore(vertex-ai): remove obsolete lint suppressions
* fix(gemini): strip audio format from speech configuration
* chore(vertex-ai): remove obsolete mutation suppressions
* fix(vertex-ai): preserve FLAC billing and wrapped audio headers
* fix(vertex-ai): reject unmeasurable FLAC before billing
* fix(vertex-ai): reconcile speech support with current provider mappings
* fix(vertex-ai): validate speech mappings and usage types
* fix(vertex-ai): preserve speech mapper override forwarding
* fix(gemini): map voice aliases across speech request forms
* fix(vertex-ai): send only formats the speech bridge cannot serve to Cloud TTS
Gemini TTS detection now goes by model name, so newer Vertex Gemini TTS
models keep working, and only mp3, opus, alaw, mulaw and other container
formats move to Cloud Text-to-Speech. wav and pcm requests stay on
generateContent as on main.
A Cloud TTS response whose duration cannot be measured now returns the
audio unbilled with a warning instead of raising after a paid call. FLAC
handling is removed because Cloud TTS has no FLAC encoding.
The cost map, the google_genai mappers and the proxy error path go back
to main, since those changes were outside this feature.
* fix(vertex_ai): keep Gemini TTS formats, region and instructions on routes that serve them
Only mp3, opus, alaw and mulaw go to Cloud Text-to-Speech. Every other
format gets a local 400 that lists the supported ones, before Google is
called.
Cloud TTS uses the deployment's us or eu regional host. Other regions
and a custom api_base get a 400 that says to request wav or pcm.
instructions is sent as the Cloud TTS style prompt, and a request with
no voice keeps the Gemini model.
get_usage_object is back to main's code, so dict usages still go
through Usage.__init__, and the proxy no longer labels pcm16 or
linear16 output as WAV.
* fix(vertex_ai): send Gemini TTS to every regional Cloud TTS host and keep speak tags as text
Cloud Text-to-Speech serves Gemini TTS on a host named after each Vertex
location, so a non-global vertex_location now uses
{location}-texttospeech.googleapis.com instead of only the us and eu hosts.
Google answers for regions it does not serve.
Gemini TTS voices reject SSML, so input that contains <speak> is sent as
text for Gemini TTS models. Chirp keeps the SSML auto-detect.
Removes the proxy alaw and mulaw media-type entries. Cloud TTS returns
those encodings with a WAV header, so the response is labeled audio/wav
and the entries never applied.
Adds a regression test that custom per-token pricing bills the usage a
speech response reports.
* fix(vertex_ai): map Gemini TTS regions to Google's Cloud TTS locations and keep a bad voice from cooling down the deployment
US and EU Vertex regions go to the us and eu Cloud TTS hosts, northamerica-northeast1 to its own, global or unset to global, and any other region gets a local 400. A Cloud TTS 404 on the Gemini path, such as a misspelled voice, comes back as a 400 so the router keeps the deployment serving wav. Usage reads the sent request from the response instead of the logging object.
* fix(vertex_ai): default Gemini TTS Cloud formats to a Gemini voice and validate speakers
Parse only usage fields from the sent Cloud TTS request, so a numeric speakingRate no longer fails after Google bills it.
Send the Gemini voice Kore when no voice is named, and reject Cloud TTS voice names unless drop_params is set.
Reject malformed speaker entries unless drop_params is set, and translate Cloud TTS speaker fields on wav and chat audio.
Use the Vertex default region when no location is set, so mp3 and wav share a region.
* test(vertex_ai): cover two-speaker Gemini TTS mp3 through aspeech with fake credentials
Replaces two tests that patched the Vertex TTS config's own token and URL methods.
* fix(vertex_ai): reject malformed Gemini TTS voices and keep beta Chirp and Lyria params unmapped
* fix(vertex_ai): drop unknown Gemini TTS voice keys on wav and pcm instead of rejecting them
* fix(vertex_ai): reject unknown Gemini TTS voice keys on wav, pcm, and chat audio unless drop_params is set
* fix(vertex_ai): honor every drop_params setting in the Gemini TTS voice key check
Global drop_params, batch row and deployment drop_params on Vertex batch
file uploads now drop unknown voice keys. A voice object with both
speechConfig and speech_config gets a 400 for the ignored speech_config.
* fix(gemini): drop unknown nested voice fields under drop_params and detect TTS by served model
* fix(gemini): filter unknown speech config fields with fixed-depth helpers and pass audio params through unchanged
* fix(gemini): camel-case snake replicated voice keys so drop_params keeps them
* fix(vertex-ai): reject voice fields Cloud TTS cannot send unless drop_params is set
* fix(vertex-ai): reject wrong-typed Cloud TTS voice values and check speakers as sent
Gemini TTS on mp3, opus, alaw, or mulaw now answers 400 for a voice field whose value Cloud TTS cannot send, such as a list languageCode or a string voiceConfig, and for nested voice fields under a speaker that also names speakerId. drop_params still drops them.
* fix(vertex-ai): reject wrong-typed speech config wrappers and speakerId overriding a prebuilt voice on Cloud TTS
* fix(vertex-ai): keep speakers and voice_config beside a speech_config wrapper on Cloud TTS
* fix(vertex-ai): pick Cloud TTS voices from the same speech config the WAV path sends to Gemini
* fix(files): carry the deployment's drop_params on model= file uploads so Vertex batch rows drop unknown voice keys
* fix(gemini): keep a structured speech voice when the model is a model_alias_map alias for a Gemini TTS model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
11b5d1a817
|
feat(gemini): set the long decode budget for requests with no output limit from LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS (#47542)
* feat(gemini): LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS sets the long decode budget for requests that send no limit * test(gemini): cover a default long decode budget set by LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS * fix(gemini): apply the default output budget only to requests asking for text output * refactor(gemini): apply the default output budget once in the async request transform * fix(gemini): leave the default output budget off non-chat models and a snake_case generation_config * fix(gemini): skip the default output budget for audio-only models and continuation resumes Lyria and other Gemini chat models whose cost map row lists no text output answer 400 when sent a maxOutputTokens, and a request that resumes from a continuation token answers 400 when a limit is added beside it. Both now go out without the default budget. * fix(vertex_ai): skip the Gemini default budget for a model a deployment's model_info overrides, and pass odd generationConfig shapes through * fix(vertex_ai): cap the Gemini default budget per deployment at the loaded cost map and its own model_info limit * test(vertex_ai): wrap a long test signature under the 120 character limit * fix(vertex_ai): keep a direct call's default output budget under the loaded cost map limit A litellm.Router deployment's model_info writes onto the shared cost map row, so a later direct call with no model_info read that raised value and Vertex answered 400. The direct call now caps at the lower of the loaded row and the live row, so litellm.register_model can still lower the cap. * fix(vertex_ai): ignore a deployment output limit that is not positive A model_info.max_output_tokens of 0 or -1 capped the default budget at that value, and Google answers 400 to a limit that is not positive. It now counts as unreadable, so the cost map limit applies. * fix(gemini): never continue a request asking for more than one candidate in camelCase * fix(gemini): keep continuing a request whose candidate count is null or 0 |
||
|
alberto
|
f768fb32ef
|
fix(streaming)!: send the include_usage chunk with an empty choices array (#47109)
* fix: Send the include_usage chunk with an empty choices array Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Skip chunks with no choices when reading streams Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Strip usage from plain-iterator async stream chunks Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Expect empty choices on the usage chunk in the gemma tracing test Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage The usage chunk now has choices=[] only when the caller set stream_options.include_usage itself. When the OpenAI handler, the proxy's always_include_stream_usage setting, or the harness added include_usage on the caller's behalf, the chunk keeps its legacy single empty-delta choice, so clients that read choices[0] on every chunk keep working. The proxy and harness mark their injection with the private _litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy. Plain-iterator streams keep usage on their chunk when usage was not requested, and the LLM Shield trailing flush ignores chunks with no choices. * fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them. * fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route * refactor(router): move the stream usage injection check into router_utils * fix(router): read stream_options from pydantic deployment params when marking injected usage Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices. * fix(proxy): keep the stream usage injection marker out of stored request bodies * test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths * test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
c29a4f56a3
|
fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later (#47645)
* fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later Google ignores temperature, top_p and top_k on Gemini 3.6 Flash, Gemini 3.5 Flash-Lite and every later Gemini release, and says a later generation will return 400 for them. The model map now carries supports_sampling_params: false on those rows, and the Gemini and Vertex transformation reads the flag the way the Anthropic and Bedrock paths already do, dropping the params under drop_params and otherwise raising the client-side 400 that names drop_params BREAKING CHANGE: a request that sends temperature, top_p or top_k to a flagged Gemini model without drop_params now gets a 400 instead of a 200 whose sampling params Google silently ignored Fixes #47015 * fix(gemini): gate the passed-through top_k where the request body is built and keep allowed_openai_params working top_k is a provider-specific kwarg that never reaches the OpenAI param mapping, so the flag has to be applied while the Gemini request body is built, the same boundary Anthropic uses. The mapping now skips a flagged sampling param instead of raising, because the shared unsupported-param check already returns the 400 that names drop_params, and a param the caller lists in allowed_openai_params is forwarded verbatim as that 400 promises * fix(gemini): filter top_k out of the generation config instead of mutating optional_params * test(gemini): type the sampling-flag twin test and keep its locals out of the loop body * fix(vertex_ai): reject or drop sampling params on batch rows for flagged Gemini models The batch JSONL wrapper calls map_openai_params directly and skips get_optional_params, so a flagged model's temperature, top_p and top_k vanished from the uploaded row with no 400. Batch rows now get the same contract as a real-time call: a 400 naming drop_params, or the knobs dropped when litellm.drop_params is on * fix(vertex_ai): ignore null sampling knobs on batch rows for flagged Gemini models A batch row carrying temperature: null, top_p: null or top_k: null means no override, the same as on a real-time call, so the row is no longer refused with a 400 and the null knobs are simply not forwarded |
||
|
devin-ai-integration[bot]
|
f3e1307ed9
|
fix(vertex_ai): log Gemini 3 sampling deprecation once per model per worker (#40868)
* fix(vertex_ai): log Gemini 3 sampling deprecation once per process Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): bound the Gemini 3 sampling warn-once cache and test it by behavior * fix(vertex_ai): log the Gemini 3 sampling deprecation once even when first calls race Pin the warned-model cap to its own constant so the shared LRU size knob cannot turn the dedupe off, and serialize the cached call so concurrent first requests for one model log it once * fix(vertex_ai): stop logging the Gemini 3 sampling deprecation for new models once 64 are remembered * fix(vertex_ai): give each mapped Gemini 3 model its own sampling notice and share one for unmapped names * fix(vertex_ai): key the Gemini 3 sampling notice on the loaded catalog so runtime-priced names share one notice --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Max Azatian
|
208cac49f0
|
fix(vertex_ai)!: send integer enums to Gemini instead of dropping them (#44908)
* fix(vertex_ai): send integer enums to Gemini instead of dropping them
Gemini rejects enum values that are not strings, so _fix_enum_types removed
every non-string enum. Integer enums are now sent as type INTEGER with
format "enum" and string values, which Gemini accepts and enforces.
* fix(vertex_ai): keep converted integer enums when a tool is reused
The schema is edited in place, so a tool definition passed again already
carries the converted enum. Keep an integer enum that is already in the
format "enum" form instead of dropping it, and type the new locals.
* fix(vertex_ai): use Sequence at runtime in the enum cast
* fix(vertex_ai): treat integer-string enum values as an integer enum
A reused schema already holds its integer enum as strings. Instead of
reading `format` to recognise that, accept integer strings as well as
integers on an integer-typed enum and send both in Gemini's form. This
drops the untyped `schema.get("format")` read.
* fix(vertex_ai): map Gemini tools from a copy so a fallback keeps the caller's schema
Gemini tool mapping rewrote the caller's tools in place, so a router fallback to another provider received the Gemini-converted integer enum (string values with format enum) and returned the integer as a string
* fix(vertex_ai): send integer enums only to Gemini 3 and newer, and only ones Gemini accepts
gemini-2.5-flash stalls for minutes on a larger integer enum, so Gemini 2.x
keeps main's behavior and drops it. An enum outside int64 or longer than 1000
values is dropped too, since Gemini answers 400 for those. A nullable integer
enum written as a type list now keeps its values like the anyOf form does
* fix(vertex_ai): drop integer enum strings longer than int64 instead of failing the request
* fix(vertex_ai): drop integer enums when a request's enums pass Gemini's size limit, and send int enum members as numbers
Gemini limits the total size of all enums in one request, so the per-enum 1000-value cap still let several large integer enums through to a 400. Integer enums are now sent only while every enum value in the request adds up to 2000 characters or fewer, and an (int, Enum) member goes out as its number instead of its name
* fix(vertex_ai): send only integer enums Gemini returns exactly, and count them in Gemini token counts
* fix(vertex_ai): send Gemini integer enums without format enum, which made gemini-3.8-flash think much longer
Also skip the response cache in the repeated-request integration test, since copying tools makes the second identical request a cache hit
* fix(vertex_ai): keep dropping integer enums in Vertex batch rows, which Vertex batch cannot parse
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Aryan Pardeshi
|
a844dadcd2
|
feat(gemini): allow disabling thought summaries while keeping thinking on (#36460)
* feat(gemini): allow disabling thought summaries while keeping thinking on Both reasoning-effort mappers hardcoded includeThoughts to True whenever thinking was on, so thinking ON with summaries OFF was unreachable even though that is the google SDK's own default. Summaries are the narration rather than the reasoning, and for callers that discard them they are pure added latency. Adds litellm.gemini_include_thoughts_default, defaulting to True so nothing changes for anyone who does not set it. It gates the includeThoughts=True branches in the thinkingBudget mapper, the thinkingLevel mapper, and the anthropic-style thinking param mapper. The disable and none paths still force includeThoughts=False: with thinking off there is nothing to summarise. * test(gemini): assert thought-summary flag through the outbound Gemini request * feat(gemini): let a request's thought summary ask override the gateway default thinking.display (summarized, updates, omitted) and a reasoning_effort object carrying summary now decide includeThoughts for that request; gemini_include_thoughts_default applies only when the request says nothing * feat(gemini): let a thought summary ask on /v1/messages and /v1/responses override the gateway default * fix(gemini): read a text gateway default as a boolean and leave display omitted to the flag Normalize gemini_include_thoughts_default through normalize_drop_params so a YAML or env value written as "false" or "0" turns thought text off and a null or unknown value keeps the default on. thinking.display "omitted" no longer counts as a per-request ask, so the gateway flag decides, matching /v1/messages. Adds proxy integration cells for chat, /v1/messages and /v1/responses that prove the flag, every per-request ask, streaming, and a burst across workers. * test(gemini): match the streamed Responses usage, which omits null token details * feat(gemini): read the AI SDK reasoning summary alias and thinking.summary as chat asks * fix(gemini): keep the logged reasoning_effort unchanged and key the response cache on the summary alias A dict reasoning_effort next to the AI SDK reasoning_summary alias now counts as a summary ask, and the alias is part of the cache key so a plain request and an alias ask never share a reply * fix(caching): key the response cache on the summary alias wherever the request carries it The cache key now reads reasoning_summary and reasoningSummary through the same reader as the summary fold, so an alias inside extra_body gets its own key too * test: type the new summary-ask unit tests and mark their locals Final * test: annotate the remaining summary-ask unit tests the PR touches --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
7090588bdf
|
feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240)
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions * fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel * feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing early-access row, carrying the same limits and capabilities. The AI Studio row mirrors the early-access entry; the Vertex row differs only in provider and pricing source. Prices follow once Google publishes them. Extend the mapped metadata test to cover all three ids, that each routes to its provider, that its backup copy matches, and that the public rows carry the same limits as the early-access row. * feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops * chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap * fix(gemini): follow continuation hops in a loop instead of recursion The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls * fix(gemini): bill Google Search grounding from every continuation request Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain * fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch * chore(gemini): name the type errors the dual-mode stream iterator trips The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up * fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment * fix(gemini): keep one response id across a continued answer A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key * test(gemini): add integration cells for long decode continuation Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once * fix(vertex_ai): fetch the access token again before each continuation request A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged * fix(vertex_ai): start each continuation stream with an empty parser buffer When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async * fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way * fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes * fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not * fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0 A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer Adds the audit's wire tests for both, which fail 24 cells before this change * test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed * fix(vertex_ai): log a continued answer's raw response as one object A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch * test(integration): check a continued answer logs one merged raw response A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch * test(vertex_ai): cover a follow-up tool call and its thought signature in long decode * test(gemini): pin that a one-request answer logs Google's body unchanged |
||
|
Sushit Lal Pakrashy
|
958cba7016
|
fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results (#38284)
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
fdab308646
|
fix(vertex_ai): keep Gemini cache creation token count out of request kwargs (#46152) | ||
|
Kerry Lu
|
0eff31ae4e
|
fix(gemini): bill context cache creation tokens from cachedContents responses (#46144)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Delweng
|
b24b68e576
|
fix(gemini): price tool-use prompt tokens as input (#46021)
* test(gemini): tool-use prompt tokens must be priced as input Signed-off-by: Delweng <delweng@gmail.com> * fix(gemini): price tool-use prompt tokens as input toolUsePromptTokenCount was folded into usage.prompt_tokens but left out of prompt_tokens_details.text_tokens, and generic_cost_per_token bills input from text_tokens, so URL context / code execution / File Search tool tokens were counted but never charged. Search-grounded responses keep adding zero via the grounding-aware billable count, and promptTokensDetails absent still leaves text_tokens None so billing falls back to the full prompt_tokens Signed-off-by: Delweng <delweng@gmail.com> --------- Signed-off-by: Delweng <delweng@gmail.com> |
||
|
Sukhdeep
|
87478c7b6d
|
fix(gemini): do not inject temperature for Gemini 3+ models when omitted (#45107)
* fix(gemini): do not inject temperature for Gemini 3+ models when omitted map_openai_params added temperature=1.0 to every Gemini 3+ request that did not set one, on both the gemini and vertex_ai routes. Google is deprecating temperature, top_p and top_k for Gemini 3+ and says upcoming models will reject them, so this default turns a request that never set temperature into one that will fail An omitted temperature now falls back to Google's own default, which their docs give as 1.0 for all Gemini 3 models. Explicit temperature, top_p and top_k are still forwarded Fixes #38663 * test(gemini): annotate sampling regression tests * test(gemini): drop the injected temperature from Gemini translation cases The nine Gemini base cases for chat completions, responses and messages send no temperature, so the provider request no longer carries the removed 1.0 default. Each new body was sent to the real Gemini API for gemini-3.5-flash, gemini-3.8-flash and gemini-3.1-pro-preview and returned 200, and the stored provider responses are unchanged * test(gemini): drop the default-temperature wording from the thinking level test docstring * test(gemini): cover omitted and explicit sampling on both Gemini routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e0398455d0
|
test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759)
* test: migrate selected llm translation contracts * test: migrate anthropic completion translation tests * test: migrate bedrock and cohere translation tests * test: tighten bedrock translation migration * test: migrate azure and openai model translation tests * test: migrate openai and small-provider translation tests * test: migrate gemini and vertex translation tests * test: migrate optional params and rerank translation tests * test: migrate huggingface, json and openrouter translation tests * test: fix openrouter integration expectations * test: migrate databricks, xai, cometapi and fireworks translation tests * test: migrate ovhcloud, prompt and small provider translation tests * test: split reasoning effort grid into unit and e2e * test: migrate router and realtime guardrail translation tests * test: migrate google interactions translation tests * test: migrate a2a, evals, skills and hosted vllm translation tests * test: remove remaining llm translation suite * test: tighten migrated llm translation unit tests and land missing rows Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover * test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths * test: close lost-intent gaps found in review of the llm translation migration Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised * test: assert provider acceptance in the reasoning effort e2e The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid * test: cover async and stream-end paths the legacy translation suite exercised Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker * test: type the new llm translation e2e calls and mark moved fixture dirs as packages * test: name the reasoning effort e2e models as module constants * test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name * test: block the prompt the shared guardrail denies and drop the reasoning presence check * test: drop the duplicate litellm import left by the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
d6db8e8744
|
fix(gemini): map OpenAI voice names to Gemini prebuilt TTS voices (#45303)
* fix(gemini): map OpenAI voice names to Gemini prebuilt TTS voices * test(gemini): type the voice mapping test helpers * fix(gemini): leave nova out of the voice mapping since Gemini accepts it as given * test(gemini): add integration cells for the OpenAI voice mapping on Gemini and Vertex TTS --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
fa2c8984ba
|
test: move the unit half of 126 mixed legacy files into tests/unit (#45090)
* test: move the unit half of 126 mixed legacy files into tests/unit * test: restore litellm globals that moved tests set * test: finalize migration test cleanup * test: restore original bodies of moved legacy tests The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated * test: drop module imports shadowed by restored local imports * test: assert on LiteLLM output in no-assertion moved tests and isolate leaks Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags * test: drain queued logging tasks before the Phoenix span test The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
e2971e0af4
|
refactor(llms): expose public names for private provider helpers (#45037)
* refactor(litellm): migrate private usage in llms Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve Bedrock batch signature marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): migrate llms private usage symbols Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve UUID Watsonx project IDs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): retarget llms mocks to public names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): limit llms changes to renames and forwarders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): keep GCS mock client patching private Vertex auth methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): assert forwarder arguments and type forwarder signatures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
dfe4df8c8a
|
test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)
* test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
c3a23fe499
|
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
bc2df7cdcd
|
fix(gemini): stop replaying thinking block signatures to Gemini (#44661)
* fix(gemini): stop replaying thinking block signatures to Gemini A thinking block's signature has no provenance, and LiteLLM never fills it from a Gemini response (Google signs text and functionCall parts, which ride provider_specific_fields and the tool call id), so a Claude signature replayed through a mixed model group reached Gemini as a thoughtSignature and Google answered 400 Invalid thought signature on every later Gemini-served turn. The same replay also sent the thinking text a second time as a plain text part. The thinking text now goes out once, as the thought part built from reasoning_content, and no part is built from thinking_blocks * test(gemini): type the parts helper and split its comprehension * test(gemini): cover thinking signature replay on the integration rig Two integration files from the audit of the foreign thought signature fix: 62 wire cells asserting the model turn Google receives on chat, messages and responses across gemini and vertex_ai, streaming and not, SDK and httpx clients, the sad shapes of thinking_blocks, context caching through cachedContents, and 3 chaos cells (a concurrent burst across endpoints, upstream stream drops, a worker SIGKILL mid burst) * test(gemini): read the integration salt from the environment The wire test decrypted Responses ids with a literal salt; tests/integration/_support/process.py boots the proxy with LITELLM_SALT_KEY when it is set, so the test now reads the same variable with the same default --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Thippaluri Yaseen Basha
|
829cba1bf1
|
fix(gemini): forward seed to the Gemini API instead of rejecting it (#43197)
* fix(gemini): forward seed to the Gemini API instead of rejecting it The gemini/ provider left seed out of its supported params, so requests with seed failed with UnsupportedParamsError, or lost the seed silently when drop_params was on. The Gemini API accepts generationConfig.seed and the inherited mapping already translates it, so adding it to the allowlist is enough * test(gemini): assert the forwarded seed without mutating shared state |
||
|
devin-ai-integration[bot]
|
69ad004015
|
refactor(anthropic): rename experimental_pass_through to pass_through (#43329)
* refactor(anthropic): rename experimental_pass_through to pass_through Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): point compact patch targets at renamed pass_through path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
2530255624
|
test: stop CI tests from downloading tokenizer files and images (#43257)
* test: load the embedding base image from a committed 100x100 PNG instead of downloading it * test: move the volcengine embedding test into tests/unit * test: check gpt2 and r50k_base tokenizer parity against committed tiktoken reference files * test: check hub tokenizer selection against an in-memory Hugging Face hub * test: serve image URLs from respx in the gemini tool-result and format-param tests * ci: drop the emptied legacy core-utils test path * test: cover the cohere and anthropic tokenizer paths in the hub tokenizer test * test: fetch every format-param image through respx and check its bytes reach the request * test: drop the gpt2 and r50k_base parity tests, which no litellm path uses * test: drop comments that restate assertions in the format-param test |
||
|
yuneng-jiang
|
5e6dc89ba1
|
test: move tests/test_litellm/llms into tests/unit/llms (#43191)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: keep the job's UNIT_FLAG out of the shard-script tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |