mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-21 10:15:47 +00:00
86 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f96200bcea
|
fix(vertex_ai): bound shared credential cache retention (#48442) | ||
|
|
1ecc782986
|
fix(vertex_ai): reuse rerank credential cache (#48430) | ||
|
|
dc0237ccd7
|
fix(cost): bill gemini audio input and cached audio at flex, priority and batch rates (#48086)
* fix(gemini): price tiered audio token rates * fix: correct Gemini audio pricing catalog * fix(gemini): preserve cached modality pricing --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
66c8f5b1a9
|
fix(llms): join api_base and route with one shared helper (#47535)
* fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * fix(llms): join api_base and route with one shared helper A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base * refactor(llms): move per-provider api_base strips onto join_api_base Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join * ci: run the trailing-slash URL builder test in the llms shard * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell * fix(llms): send every provider route through join_api_base Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash * fix(llms): keep the route skip only where a builder already had it join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path already ends in it, and it is used only where the merge base or the trailing-slash test PR head already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the path so a host named anthropic-* builds /anthropic/v1/messages * fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL * test(llms): run the Gemini RAG trailing-slash case through aingest The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears * test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base * test(integration): cover api_base joins across providers, query strings and named hosts * feat(url_utils): add join_api_base and ensure_api_route helpers * fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host join_api_base and ensure_api_route now keep the // of a base left with only a scheme, so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it * fix(azure_ai): build the count_tokens URL from the messages URL * fix(azure_ai): match anthropic as a whole path segment in the messages URL * fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints Azure AI chat and messages go back to cutting the base at the first /anthropic, as on main, and count_tokens keeps its path check, so a gateway path like /anthropic-proxy builds the same URLs as main. A base that already ends in /v1/messages still gets count_tokens next to it. A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is used as its text again in AgentCore, Bedrock responses and the RAGFlow base, instead of falling back to the default host. get_volcengine_base_url and GigaChat's get_api_base strip a trailing slash again for outside callers. The cache-hit wire test is skipped as a known proxy bug, since one cache hit writes two cache_hit spend rows. * fix(llms): drop a slash before the api_base query and tighten the route test checks * test(llms): annotate new test locals as Final and flatten the builder generator * fix(url_utils): keep a scheme-only api_base hostless so it fails before sending --------- Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com> |
||
|
|
820725661e
|
fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage (#45194)
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first. * fix(proxy): log a provider error frame through an attempt-scoped logging copy The error-frame failure stashed the superseded attempt's usage on the logging object the retry shares, so the retry's success log carried the failed attempt's token counts. The failure now logs through a copy of the logging object with its own model_call_details, and the helper moves into the Anthropic passthrough logging handler so the proxy streaming handler only routes by endpoint type. * fix(proxy): scope the error-frame failure log to streams closed at the frame A stream that ran to its end at the provider's error frame kept logging through the success route before this PR, and the attempt-scoped failure log is now taken only for a stream the router closed at that frame, with the copy made at the close so the retry's deployment never reaches it * fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage A /v1/messages stream dropped after message_start and retried in its model group shares one Logging object across attempts, so the retry's success log inherited the dropped attempt's stashed usage and exception, and a second drop never reached the failure handlers because the first had already marked the object as logged. The dropped attempt now dispatches its failure handlers through an attempt-scoped copy, and the router discards the stashed usage right before it opens the next attempt, on same-group retries and on every fallback hop, so a request that runs out of attempts still bills the last attempt's partial usage. * fix(router): dispatch chat stream failures through the attempt-scoped logging copy The router now clears a dropped attempt's stashed usage before every fallback hop, so the chat stream wrapper's queued failure log must read that usage from its own copy of the logging object, like the /v1/messages path already does. The stash test asserts through a real success log, the cooldown test reads a real Router's cooldown list, and the integration helpers carry full types. * test(vertex): give the stream wrapper's logging stub an attempt_scoped_copy * test: route the pass-through logging mock's attempt copy to itself and read the callback sink through a reader The pass-through failure test stubbed dispatch_failure_handlers on a spec'd MagicMock, which the attempt-scoped copy no longer reached, so the logging worker never finished. The integration tests kept earlier sink batches in a caller-owned list the helper mutated; a small reader keeps them instead. * test(router): use real Logging in fallback cooldown test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): preserve superseded retry usage for billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): bill retry input after prior stream usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): track retry dispatch for terminal billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): use deterministic retry timestamp Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): avoid real clock in fallback tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): remove duplicate streaming logger import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): retain error-frame usage for exhausted retries * test(proxy): fix retry billing expectations and helper annotations * fix(proxy): preserve failure estimate without carried usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): audit retry helpers and fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): fix retry fixture timestamps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): observe retry stash at attempt boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): keep terminal failure attribution when dropped attempts log on copies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish a dropped attempt's payload only while it is still the current attempt Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover spend-log encoding error propagation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): skip adapter failure dispatch when the wrapped stream already logged it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): cool down the final fallback attempt when its failure callbacks were blocked Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): import Final in adapter regression Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop spend-log encoding expectation superseded by main's poison isolation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): finish the wrapped stream's failure log before surfacing the error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): wait for the wrapped stream's failure log to finish like the inline dispatch did Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): wait for the executor-run sync failure callback separately from the async caller boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): pin pass-through stream fixture start times Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload before awaiting failure callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload from the failure handler after baseline invalidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(logging): drop unused discard_partial_usage_for_failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
045fb7ba84
|
feat(vertex-ai)!: support gemini 3.1 flash tts (#31915)
* feat(vertex-ai): support gemini 3.1 flash tts
* fix(vertex-ai): keep TTS metadata compatible with the cost map guard
* fix(vertex-ai): preserve existing Gemini TTS models and voice mappings
* fix(vertex-ai): recognize provider-stripped TTS model names
* test(vertex-ai): match current speech HTTP error response
* fix(vertex-ai): preserve nested speech locale and voice
* fix(vertex-ai): use provider speech media type in proxy
* fix(gemini): preserve TTS dispatch and nested speech settings
* test(gemini): run TTS regressions in CI shards
* fix(vertex-ai): account for Gemini Cloud TTS audio cost
* fix(vertex-ai): bill Gemini TTS without optional audio decoder
* test(gemini): cover TTS mapping and billing edge cases
* test(vertex-ai): cover Gemini voice normalization in provider shard
* fix(vertex-ai): keep Gemini TTS routing with generic provider metadata
* chore(vertex-ai): remove obsolete lint suppressions
* fix(gemini): strip audio format from speech configuration
* chore(vertex-ai): remove obsolete mutation suppressions
* fix(vertex-ai): preserve FLAC billing and wrapped audio headers
* fix(vertex-ai): reject unmeasurable FLAC before billing
* fix(vertex-ai): reconcile speech support with current provider mappings
* fix(vertex-ai): validate speech mappings and usage types
* fix(vertex-ai): preserve speech mapper override forwarding
* fix(gemini): map voice aliases across speech request forms
* fix(vertex-ai): send only formats the speech bridge cannot serve to Cloud TTS
Gemini TTS detection now goes by model name, so newer Vertex Gemini TTS
models keep working, and only mp3, opus, alaw, mulaw and other container
formats move to Cloud Text-to-Speech. wav and pcm requests stay on
generateContent as on main.
A Cloud TTS response whose duration cannot be measured now returns the
audio unbilled with a warning instead of raising after a paid call. FLAC
handling is removed because Cloud TTS has no FLAC encoding.
The cost map, the google_genai mappers and the proxy error path go back
to main, since those changes were outside this feature.
* fix(vertex_ai): keep Gemini TTS formats, region and instructions on routes that serve them
Only mp3, opus, alaw and mulaw go to Cloud Text-to-Speech. Every other
format gets a local 400 that lists the supported ones, before Google is
called.
Cloud TTS uses the deployment's us or eu regional host. Other regions
and a custom api_base get a 400 that says to request wav or pcm.
instructions is sent as the Cloud TTS style prompt, and a request with
no voice keeps the Gemini model.
get_usage_object is back to main's code, so dict usages still go
through Usage.__init__, and the proxy no longer labels pcm16 or
linear16 output as WAV.
* fix(vertex_ai): send Gemini TTS to every regional Cloud TTS host and keep speak tags as text
Cloud Text-to-Speech serves Gemini TTS on a host named after each Vertex
location, so a non-global vertex_location now uses
{location}-texttospeech.googleapis.com instead of only the us and eu hosts.
Google answers for regions it does not serve.
Gemini TTS voices reject SSML, so input that contains <speak> is sent as
text for Gemini TTS models. Chirp keeps the SSML auto-detect.
Removes the proxy alaw and mulaw media-type entries. Cloud TTS returns
those encodings with a WAV header, so the response is labeled audio/wav
and the entries never applied.
Adds a regression test that custom per-token pricing bills the usage a
speech response reports.
* fix(vertex_ai): map Gemini TTS regions to Google's Cloud TTS locations and keep a bad voice from cooling down the deployment
US and EU Vertex regions go to the us and eu Cloud TTS hosts, northamerica-northeast1 to its own, global or unset to global, and any other region gets a local 400. A Cloud TTS 404 on the Gemini path, such as a misspelled voice, comes back as a 400 so the router keeps the deployment serving wav. Usage reads the sent request from the response instead of the logging object.
* fix(vertex_ai): default Gemini TTS Cloud formats to a Gemini voice and validate speakers
Parse only usage fields from the sent Cloud TTS request, so a numeric speakingRate no longer fails after Google bills it.
Send the Gemini voice Kore when no voice is named, and reject Cloud TTS voice names unless drop_params is set.
Reject malformed speaker entries unless drop_params is set, and translate Cloud TTS speaker fields on wav and chat audio.
Use the Vertex default region when no location is set, so mp3 and wav share a region.
* test(vertex_ai): cover two-speaker Gemini TTS mp3 through aspeech with fake credentials
Replaces two tests that patched the Vertex TTS config's own token and URL methods.
* fix(vertex_ai): reject malformed Gemini TTS voices and keep beta Chirp and Lyria params unmapped
* fix(vertex_ai): drop unknown Gemini TTS voice keys on wav and pcm instead of rejecting them
* fix(vertex_ai): reject unknown Gemini TTS voice keys on wav, pcm, and chat audio unless drop_params is set
* fix(vertex_ai): honor every drop_params setting in the Gemini TTS voice key check
Global drop_params, batch row and deployment drop_params on Vertex batch
file uploads now drop unknown voice keys. A voice object with both
speechConfig and speech_config gets a 400 for the ignored speech_config.
* fix(gemini): drop unknown nested voice fields under drop_params and detect TTS by served model
* fix(gemini): filter unknown speech config fields with fixed-depth helpers and pass audio params through unchanged
* fix(gemini): camel-case snake replicated voice keys so drop_params keeps them
* fix(vertex-ai): reject voice fields Cloud TTS cannot send unless drop_params is set
* fix(vertex-ai): reject wrong-typed Cloud TTS voice values and check speakers as sent
Gemini TTS on mp3, opus, alaw, or mulaw now answers 400 for a voice field whose value Cloud TTS cannot send, such as a list languageCode or a string voiceConfig, and for nested voice fields under a speaker that also names speakerId. drop_params still drops them.
* fix(vertex-ai): reject wrong-typed speech config wrappers and speakerId overriding a prebuilt voice on Cloud TTS
* fix(vertex-ai): keep speakers and voice_config beside a speech_config wrapper on Cloud TTS
* fix(vertex-ai): pick Cloud TTS voices from the same speech config the WAV path sends to Gemini
* fix(files): carry the deployment's drop_params on model= file uploads so Vertex batch rows drop unknown voice keys
* fix(gemini): keep a structured speech voice when the model is a model_alias_map alias for a Gemini TTS model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
|
11b5d1a817
|
feat(gemini): set the long decode budget for requests with no output limit from LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS (#47542)
* feat(gemini): LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS sets the long decode budget for requests that send no limit * test(gemini): cover a default long decode budget set by LITELLM_GEMINI_DEFAULT_MAX_OUTPUT_TOKENS * fix(gemini): apply the default output budget only to requests asking for text output * refactor(gemini): apply the default output budget once in the async request transform * fix(gemini): leave the default output budget off non-chat models and a snake_case generation_config * fix(gemini): skip the default output budget for audio-only models and continuation resumes Lyria and other Gemini chat models whose cost map row lists no text output answer 400 when sent a maxOutputTokens, and a request that resumes from a continuation token answers 400 when a limit is added beside it. Both now go out without the default budget. * fix(vertex_ai): skip the Gemini default budget for a model a deployment's model_info overrides, and pass odd generationConfig shapes through * fix(vertex_ai): cap the Gemini default budget per deployment at the loaded cost map and its own model_info limit * test(vertex_ai): wrap a long test signature under the 120 character limit * fix(vertex_ai): keep a direct call's default output budget under the loaded cost map limit A litellm.Router deployment's model_info writes onto the shared cost map row, so a later direct call with no model_info read that raised value and Vertex answered 400. The direct call now caps at the lower of the loaded row and the live row, so litellm.register_model can still lower the cap. * fix(vertex_ai): ignore a deployment output limit that is not positive A model_info.max_output_tokens of 0 or -1 capped the default budget at that value, and Google answers 400 to a limit that is not positive. It now counts as unreadable, so the cost map limit applies. * fix(gemini): never continue a request asking for more than one candidate in camelCase * fix(gemini): keep continuing a request whose candidate count is null or 0 |
||
|
|
f768fb32ef
|
fix(streaming)!: send the include_usage chunk with an empty choices array (#47109)
* fix: Send the include_usage chunk with an empty choices array Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Skip chunks with no choices when reading streams Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Strip usage from plain-iterator async stream chunks Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Expect empty choices on the usage chunk in the gemma tracing test Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage The usage chunk now has choices=[] only when the caller set stream_options.include_usage itself. When the OpenAI handler, the proxy's always_include_stream_usage setting, or the harness added include_usage on the caller's behalf, the chunk keeps its legacy single empty-delta choice, so clients that read choices[0] on every chunk keep working. The proxy and harness mark their injection with the private _litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy. Plain-iterator streams keep usage on their chunk when usage was not requested, and the LLM Shield trailing flush ignores chunks with no choices. * fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them. * fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route * refactor(router): move the stream usage injection check into router_utils * fix(router): read stream_options from pydantic deployment params when marking injected usage Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices. * fix(proxy): keep the stream usage injection marker out of stored request bodies * test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths * test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
c29a4f56a3
|
fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later (#47645)
* fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later Google ignores temperature, top_p and top_k on Gemini 3.6 Flash, Gemini 3.5 Flash-Lite and every later Gemini release, and says a later generation will return 400 for them. The model map now carries supports_sampling_params: false on those rows, and the Gemini and Vertex transformation reads the flag the way the Anthropic and Bedrock paths already do, dropping the params under drop_params and otherwise raising the client-side 400 that names drop_params BREAKING CHANGE: a request that sends temperature, top_p or top_k to a flagged Gemini model without drop_params now gets a 400 instead of a 200 whose sampling params Google silently ignored Fixes #47015 * fix(gemini): gate the passed-through top_k where the request body is built and keep allowed_openai_params working top_k is a provider-specific kwarg that never reaches the OpenAI param mapping, so the flag has to be applied while the Gemini request body is built, the same boundary Anthropic uses. The mapping now skips a flagged sampling param instead of raising, because the shared unsupported-param check already returns the 400 that names drop_params, and a param the caller lists in allowed_openai_params is forwarded verbatim as that 400 promises * fix(gemini): filter top_k out of the generation config instead of mutating optional_params * test(gemini): type the sampling-flag twin test and keep its locals out of the loop body * fix(vertex_ai): reject or drop sampling params on batch rows for flagged Gemini models The batch JSONL wrapper calls map_openai_params directly and skips get_optional_params, so a flagged model's temperature, top_p and top_k vanished from the uploaded row with no 400. Batch rows now get the same contract as a real-time call: a 400 naming drop_params, or the knobs dropped when litellm.drop_params is on * fix(vertex_ai): ignore null sampling knobs on batch rows for flagged Gemini models A batch row carrying temperature: null, top_p: null or top_k: null means no override, the same as on a real-time call, so the row is no longer refused with a 400 and the null knobs are simply not forwarded |
||
|
|
f3e1307ed9
|
fix(vertex_ai): log Gemini 3 sampling deprecation once per model per worker (#40868)
* fix(vertex_ai): log Gemini 3 sampling deprecation once per process Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): bound the Gemini 3 sampling warn-once cache and test it by behavior * fix(vertex_ai): log the Gemini 3 sampling deprecation once even when first calls race Pin the warned-model cap to its own constant so the shared LRU size knob cannot turn the dedupe off, and serialize the cached call so concurrent first requests for one model log it once * fix(vertex_ai): stop logging the Gemini 3 sampling deprecation for new models once 64 are remembered * fix(vertex_ai): give each mapped Gemini 3 model its own sampling notice and share one for unmapped names * fix(vertex_ai): key the Gemini 3 sampling notice on the loaded catalog so runtime-priced names share one notice --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
7e8be61fa0
|
test: fix flaky team delete, refusal sink, loop-blocking, Moyai tamper and model-cost leak tests (#47135)
* test(integration): stabilize auth and team delete checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): make loop-blocking and tampered-signature checks deterministic Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): isolate model-cost state and synchronize loop progress Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): fail loop-progress handshakes when the blocking call never starts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
208cac49f0
|
fix(vertex_ai)!: send integer enums to Gemini instead of dropping them (#44908)
* fix(vertex_ai): send integer enums to Gemini instead of dropping them
Gemini rejects enum values that are not strings, so _fix_enum_types removed
every non-string enum. Integer enums are now sent as type INTEGER with
format "enum" and string values, which Gemini accepts and enforces.
* fix(vertex_ai): keep converted integer enums when a tool is reused
The schema is edited in place, so a tool definition passed again already
carries the converted enum. Keep an integer enum that is already in the
format "enum" form instead of dropping it, and type the new locals.
* fix(vertex_ai): use Sequence at runtime in the enum cast
* fix(vertex_ai): treat integer-string enum values as an integer enum
A reused schema already holds its integer enum as strings. Instead of
reading `format` to recognise that, accept integer strings as well as
integers on an integer-typed enum and send both in Gemini's form. This
drops the untyped `schema.get("format")` read.
* fix(vertex_ai): map Gemini tools from a copy so a fallback keeps the caller's schema
Gemini tool mapping rewrote the caller's tools in place, so a router fallback to another provider received the Gemini-converted integer enum (string values with format enum) and returned the integer as a string
* fix(vertex_ai): send integer enums only to Gemini 3 and newer, and only ones Gemini accepts
gemini-2.5-flash stalls for minutes on a larger integer enum, so Gemini 2.x
keeps main's behavior and drops it. An enum outside int64 or longer than 1000
values is dropped too, since Gemini answers 400 for those. A nullable integer
enum written as a type list now keeps its values like the anyOf form does
* fix(vertex_ai): drop integer enum strings longer than int64 instead of failing the request
* fix(vertex_ai): drop integer enums when a request's enums pass Gemini's size limit, and send int enum members as numbers
Gemini limits the total size of all enums in one request, so the per-enum 1000-value cap still let several large integer enums through to a 400. Integer enums are now sent only while every enum value in the request adds up to 2000 characters or fewer, and an (int, Enum) member goes out as its number instead of its name
* fix(vertex_ai): send only integer enums Gemini returns exactly, and count them in Gemini token counts
* fix(vertex_ai): send Gemini integer enums without format enum, which made gemini-3.8-flash think much longer
Also skip the response cache in the repeated-request integration test, since copying tools makes the second identical request a cache hit
* fix(vertex_ai): keep dropping integer enums in Vertex batch rows, which Vertex batch cannot parse
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
|
91724bb0a7
|
fix(responses)!: count input tokens the way inference sends the request (#47363)
* fix(count-tokens): translate chat tool use before Claude token counts Responses and chat tool use (function tools, function_call items, tool messages) is now converted the way Claude chat inference converts it before the Anthropic, Azure AI and Vertex partner counters call count_tokens. Translation errors answer 400 before any provider call. Anthropic-shaped requests and /v1/messages/count_tokens keep their body as sent. * test(count-tokens): compare Claude count bodies with inference for Responses tool use * fix(responses): forward the caller's Responses body to OpenAI token counts OpenAI deployments now get the caller's /v1/responses/input_tokens body as sent, minus LiteLLM-only fields, so OpenAI's own count and errors come back. previous_response_id is checked for ownership like inference and routes to the deployment that served it * fix(responses)!: count Claude Responses bodies the way inference sends them Non-native Responses counts now run the body through the inference bridge, so text.format, tool_choice, reasoning, built-in tools, and file_search count the way inference sends them on Anthropic, Azure AI, Vertex, and Bedrock. The OpenAI native count body uses the drop_params-resolved tool fields. * fix(responses): honor drop_params and replay previous turns in Claude counts Claude input_tokens counts now read the same drop_params setting inference merges from the deployment, router defaults, and request, so a forced tool_choice downgrades to auto under drop_params. previous_response_id replays the stored turns through inference's session loader, and chat messages carrying fields beyond role and content go through the provider's chat translation * fix(responses): keep proxy-only request keys out of OpenAI counts and map chat tools in Bedrock counts * fix(count-tokens): import on Python 3.11 and keep stored prompts in OpenAI counts under drop_params * fix(responses): leave tool_choice out of counts when drop_params drops every tool * fix(count-tokens): count Mantle options, drop cache marks, and reject input inference cannot send The Bedrock Mantle fallback now counts thinking, tool_choice and output_config like the runtime count. Claude Responses counts leave out cache_control marks again, so more than four marks count instead of failing. Counts answer 400 before any provider call for user blocks chat inference refuses and for Responses tools with no Chat Completions equivalent, instead of dropping the block or counting a placeholder tool. Adds the wire tests for these paths * fix(count-tokens): count Responses tools the way the inference bridge sends them A Claude count now hoists additional_tools input items into tools, strips a malformed tool flag under drop_params and rejects it without, and leaves tools out when the deployment lists tools in additional_drop_params, so the count matches the body Claude inference sends for the same request * fix(count-tokens): keep Bedrock runtime counts option-free and send OpenAI the provider file id * test(count-tokens): capture the Anthropic count request at the HTTP layer instead of patching the handler * fix(count-tokens): ignore a caller-sent file id mapping and keep non-string file ids as sent |
||
|
|
06e3e32c88
|
fix(vertex_ai): reuse known context caches instead of listing on every request (#46857)
Vertex AI and Gemini context caching now remember each cache they create or find until 10s before its expireTime, so a repeat cache-marked request goes straight to generateContent instead of listing every cachedContents first. Concurrent async requests for one prefix share one lookup or create When generation rejects a remembered cache with a 400, 403 or 404 that names the cached content, the request rebuilds the cache and sends once more. Any other 400, 403 or 404 drops the record without a retry Fixes #36395 |
||
|
|
270d108ace
|
fix(anthropic): json_schema envelope edge cases on the Anthropic json tool path (#47478)
* test(anthropic): cover json_schema envelope edge cases on the Anthropic json tool path * fix(anthropic): unwrap required-only keys, walk past tool-named envelopes, keep non-ASCII text, flush cut streams, check Vertex output_format * fix(anthropic): send held json tool text on cut Bedrock Invoke streams and drop a non-object output_format under drop_params A Bedrock Invoke stream that ends before the json tool block closes now sends the held text and closing brace, as the direct Anthropic stream already did. A non-object output_format on the Anthropic messages route and its providers is dropped under drop_params and otherwise forwarded, so the provider answers its own 400. Vertex Claude names the proxy setting in its 400 and reads drop_params through the shared helper. Adds integration cases for the json_schema envelope and the cut stream. * fix(anthropic): drop a non-object output_format on chat, unwrap a whole envelope on a cut stream, escape lone surrogates Chat completions to Anthropic, Azure AI, Bedrock Invoke and Vertex now drop a non-object output_format under drop_params, matching /v1/messages. A stream cut after a complete held envelope is unwrapped like a normal end. A json_schema answer holding a lone surrogate keeps its \u escape so the response can still be encoded * fix(anthropic): join a split answer when the stream ends in its last block A cut during the second json tool block now runs the held block through the same flush as a block stop, so it joins the streamed keys or the earlier split blocks instead of sending two objects back to back. The Vertex output_format fold drops its unreachable drop_params branch, since both parent configs drop a non-object value first. * test(vertex): expect the LiteLLM 400 for a non-object output_format on /v1/messages |
||
|
|
d29200fa05
|
fix(vertex_ai): bill per-character text embeddings on Google's billableCharacterCount (#47460)
Vertex text embedding predict responses report metadata.billableCharacterCount, the unit Google bills text-embedding-005 and text-multilingual-embedding-002 on. Carry it into usage.prompt_tokens_details.character_count and, when the model carries input_cost_per_character, bill that count instead of the token count so the prompt is charged under one unit. Per-token embeddings (gemini-embedding-001) are unchanged. Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
6162184b1c
|
fix(managed_files): send provider file ids when creating a fine-tuning job from managed files (#46552)
* fix(managed_files): send provider file ids when creating a fine-tuning job from managed files Fine-tuning job creation with a managed training file sent the gateway's managed id and the provider-prefixed deployment model to OpenAI, which rejected the job. The pre-call hook now builds the file id mapping for training_file and validation_file, the deployment hook swaps both, the SDK strips the provider prefix from the model, and the create response returns the caller's own file ids * fix(managed_files): route managed fine-tuning creates only to deployments holding every file Fine-tuning creates now narrow routing the same way managed batches do, so a deployment added after the upload is never picked. Adds a two-worker chaos integration test and runs tests/unit/fine_tuning in the endpoints shard. * fix(managed_files): keep fine-tuning creates on the model group when no deployment holds the files A managed file used on a model group it was never uploaded for kept failing with the provider's 400 before the routing filter. The filter turned that into a 429 no-deployments error, so fine-tuning now falls back to the whole group in that case while batch input files keep routing only to deployments that hold them. * test(fine_tuning): cover the provider prefix strip for OpenAI and Azure creates through the proxy * test(fine_tuning): start each multi-deployment case from an empty upstream log * test(fine_tuning): boot both proxies of the late deployment cell on one scratch database --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
|
b51b9b7d43
|
fix(anthropic)!: answer 400 for a malformed tool_result text on /v1/messages and count_tokens (#47347)
* fix(anthropic): answer 400 for a malformed tool_result text on the Responses bridge and count_tokens A tool_result text block whose text is null, a number, or missing made /v1/messages and /v1/messages/count_tokens answer 500 on OpenAI and Azure models. Both routes now answer 400 naming the block, and drop the block when drop_params is on. * fix(anthropic): keep a missing tool_result text as empty text, bound the error, honor body drop_params on count_tokens * fix(anthropic): apply the tool_result text check on the chat/completions bridge and every count route The chat/completions bridge used for Azure, Gemini and other providers now answers the same 400 for a null or numeric tool_result text and drops the block under drop_params, so a model group that mixes OpenAI with another provider answers the same way on every deployment. Gemini and Vertex count_tokens apply the same check, count_tokens turns on drop_params for Claude Code and Codex the way /v1/messages does, and a tool_result without a tool_use_id is checked too * fix(proxy): answer 400 for a malformed tool_result text on count_tokens models without a count API Providers with no token counting API fall back to the local tokenizer, which counted a null or numeric tool_result text as fine while /v1/messages on the same model answers 400. The fallback now runs the same check, unless drop_params applies to the deployment or the request * fix(proxy): honor request drop_params on the local count fallback when no deployment resolves * test(integration): cover non-string tool_result text on both /v1/messages bridges and count routes * fix(anthropic): keep a dropped tool_result on Mistral and reject it before a guardrail rewrite Under drop_params a tool_result whose only text block was null or a number became an empty tool message, and Mistral's empty-message filter removed it, so Mistral answered 400 for a missing tool reply. The filter now only removes empty assistant messages, which is what Mistral rejects. A guardrail that rewrites structured_messages wrote the dropped text back as "", so a null or numeric text was silently blanked even without drop_params. The rewrite path now answers the same 400 as the bridges unless the request or global drop_params is set. * test(integration): cover the Mistral empty tool reply and the guardrail rewrite tool_result text check * fix(anthropic): keep a malformed tool_result text through a guardrail rewrite and keep Mistral's null-message drop A guardrail that rewrites structured_messages now writes the caller's null or numeric tool_result text back in place, so the route's bridge and the deployment's drop_params decide after routing, the same as with no guardrail. The rewrite path no longer answers its own 400 before routing. Mistral drops a null-content system or user message again, as main does, and only a tool reply is exempt from that filter so a dropped tool_result keeps its tool call. * test(mistral): expect empty system and user turns dropped and the empty tool reply kept * perf(anthropic): restore kept tool_result text parts in one linear pass * test(integration): cover the tool_result text restore across tool_results, the chat bridge, and a batch upload behind a rewriting guardrail * fix(anthropic): let guardrails scan malformed tool_result text and honor drop_params on native Anthropic A rewriting guardrail now sees a non-string tool_result text as its JSON text and the write-back restores the caller's value by the sanitized tool_use_id when the guardrail left it unchanged. Native Anthropic drops the bad block under drop_params on /v1/messages and count_tokens, and Mistral keeps an empty tool reply only when it answers a tool call, sent as empty text. * fix(anthropic): compare a kept tool_result text to its scanned form once and align the wire tests with drop_params * test(integration): cover native Anthropic drop_params resolution, guardrail scanning of numeric tool_result text, and the Mistral orphan tool message * fix(count_tokens): honor the router's default drop_params when counting a malformed tool_result text * test(count_tokens): send the tool_result count tests through a real request body instead of a mocked parser * test(count_tokens): send the native Claude count test through a real request body * fix(anthropic): keep each non-string tool_result text when tool_use_ids sanitize to the same id * fix(anthropic): keep non-ASCII characters in the guardrail's view of a non-string tool_result text * test(gemini): mock the count_tokens upstream with respx instead of patching the client factory * fix(mistral): keep an empty tool reply only for an open call, and forward non-object message items on the native drop An empty tool reply re-answering a call the conversation already moved past made Mistral reject the request, and a tool call id that is not a string crashed the filter. Native Anthropic under drop_params assumed every message item was an object. * fix(mistral): find answering tool replies in one pass instead of copying the open call set per reply * fix(mistral): ignore dropped empty placeholders when tracking open tool calls * fix(mistral): drop an empty tool reply when another reply to the same call carries the result * refactor(anthropic): share tool_result text helpers to shrink the diff * fix(anthropic): answer 400 for a non-string tool_result text on the OpenAI Responses bridge |
||
|
|
16645cc963
|
fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162)
* fix(router): bound fake-streamed calls by the request timeout, not stream_timeout A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream. * test(timeout): cover stream_timeout on live streams only and the router's timeout split * fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management * fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface A faked stream now takes the longer of timeout and stream_timeout, so a caller who raised stream_timeout to cover a slow fake stream keeps that headroom. Azure o-series dispatch is mirrored when deciding whether a stream is faked. text_completion, /v1/messages and /v1/responses through the Router now get timeout and stream_timeout separately, like chat calls. * fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages * fix(router): report and log the timeout each stream was bounded by Native /v1/messages and /v1/responses now record the timeout they applied, so x-litellm-timeout and the cooldown guard read the real limit instead of the raw request value. A prompt-managed model also passes on only the caller's or its own timeout, so the target deployment's timeout still wins. * fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit * fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses * fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit * test(integration): cover faked, live and converted stream timeout limits per endpoint * test(router): regression tests for faked-stream timeouts |
||
|
|
ef1d3959a0
|
fix(count_tokens)!: answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens (#47163)
* fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens * fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests Vertex has no count-tokens route for Mistral or codestral models, so they now get the same 400 as other partner models without a count API instead of a 404 that points operators at IAM. * fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens * fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model * fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors * fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer * fix(count_tokens): pass through an OpenAI provider error that omits param or code * fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route * test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy * test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes |
||
|
|
f8217d2a03
|
fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count (#46843)
* fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count /v1/messages runs clear_tool_uses context edits before it sends an Anthropic body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted the body unedited, so a request with older tool results cleared counted about 2.8x what inference bills. The count now runs the same history and context-edit step inference runs, honors additional_drop_params on the deployment, and skips compact edits so a count never makes a paid summary call. * fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs Run compact edits in the count without the summary call so trigger checks and history slicing match /v1/messages, honor a request-level additional_drop_params, keep the full body when any deployment in the model group drops context_management, and answer 400 naming a block Gemini cannot count when an edit trigger fails on it * fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check * fix(count_tokens): log a failed edit in the count at debug, since the send already logs it * fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does * fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits. * fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name * fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to * fix(messages): drop an invalid context_management spec under drop_params so the send matches the count * fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes. * test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code |
||
|
|
859529ccbc
|
fix(realtime)!: keep turn_detection.type when a transcript guardrail disables auto-response (#44939)
* fix(realtime): keep turn_detection.type when a transcript guardrail disables auto-response * test(realtime): type the GA session.update helper and mark its locals Final * test(realtime): expect typed turn_detection on the voice guardrail session updates * test(realtime): accept either gate path for the client's first voice session.update * fix(gemini): drop VAD tuning when the guardrail turns auto-response off * fix(gemini): check create_response without a redundant isinstance * fix(vertex_ai): keep the client's VAD tuning on Vertex when auto-response is off * fix(realtime): keep the transcript-guardrail gate typeless on xAI so it does not auto-answer * fix(realtime): replace any client turn_detection with the untyped gate on xAI xAI answers on its own under a typed server_vad even with create_response off, so a client's typed turn_detection must not reach xAI while the transcript guardrail gates the turn. * fix(realtime): keep a client's early turn_detection and hide the guardrail cancel's error A session.update sent before session.created reached OpenAI ahead of the proxy's own gate, so the gate replaced the client's turn_detection. The proxy now re-sends the client's gated turn_detection after the gate when the two differ. The guardrail's response.cancel now carries a litellm_guardrail_ event_id, and the error OpenAI returns for it (response_cancel_not_active when nothing is answering) no longer reaches the client. * fix(realtime): send the guardrail gate before a client's early session.update and let Muse and Chirp fall back to server_vad under drop_params * fix(realtime): hide only the cancel errors that echo the guardrail's own response.cancel ids Track the event ids of the guardrail cancels this session sent and drop a backend error only when its error.event_id is one of them, so a client's own event id with the same prefix still gets its error. Cover the scripted response.cancel reply in the integration upstream, update the stale list-mode test to the typed gate, and mark the xAI cancel error cell as an expected failure, since xAI replies with its own event id. * test(integration support): deliver the disconnect a real server sends after the client goes away The scripted upstream now keeps reading frames after a reply hits a closed socket, so the unit test's fake ASGI receive has to end with the websocket.disconnect message a real server delivers. * fix(realtime): delete the blocked transcript item before the guardrail's refusal turn OpenAI GA answered the blocked question in some refusal turns because the blocked user item stayed in the conversation. The raw OpenAI-protocol path now sends conversation.item.delete for it right after the cancel. Provider config backends have no such event, so that path no longer passes an item id. Also bound the guardrail integration file by the shared graceful-stop timeout and make the xAI cancel-error xfail require an AssertionError. * fix(realtime): keep the blocked item on xAI, which cannot delete an audio input item * test(realtime): assert the xAI voice block sends no conversation.item.delete upstream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
a844dadcd2
|
feat(gemini): allow disabling thought summaries while keeping thinking on (#36460)
* feat(gemini): allow disabling thought summaries while keeping thinking on Both reasoning-effort mappers hardcoded includeThoughts to True whenever thinking was on, so thinking ON with summaries OFF was unreachable even though that is the google SDK's own default. Summaries are the narration rather than the reasoning, and for callers that discard them they are pure added latency. Adds litellm.gemini_include_thoughts_default, defaulting to True so nothing changes for anyone who does not set it. It gates the includeThoughts=True branches in the thinkingBudget mapper, the thinkingLevel mapper, and the anthropic-style thinking param mapper. The disable and none paths still force includeThoughts=False: with thinking off there is nothing to summarise. * test(gemini): assert thought-summary flag through the outbound Gemini request * feat(gemini): let a request's thought summary ask override the gateway default thinking.display (summarized, updates, omitted) and a reasoning_effort object carrying summary now decide includeThoughts for that request; gemini_include_thoughts_default applies only when the request says nothing * feat(gemini): let a thought summary ask on /v1/messages and /v1/responses override the gateway default * fix(gemini): read a text gateway default as a boolean and leave display omitted to the flag Normalize gemini_include_thoughts_default through normalize_drop_params so a YAML or env value written as "false" or "0" turns thought text off and a null or unknown value keeps the default on. thinking.display "omitted" no longer counts as a per-request ask, so the gateway flag decides, matching /v1/messages. Adds proxy integration cells for chat, /v1/messages and /v1/responses that prove the flag, every per-request ask, streaming, and a burst across workers. * test(gemini): match the streamed Responses usage, which omits null token details * feat(gemini): read the AI SDK reasoning summary alias and thinking.summary as chat asks * fix(gemini): keep the logged reasoning_effort unchanged and key the response cache on the summary alias A dict reasoning_effort next to the AI SDK reasoning_summary alias now counts as a summary ask, and the alias is part of the cache key so a plain request and an alias ask never share a reply * fix(caching): key the response cache on the summary alias wherever the request carries it The cache key now reads reasoning_summary and reasoningSummary through the same reader as the summary fold, so an alias inside extra_body gets its own key too * test: type the new summary-ask unit tests and mark their locals Final * test: annotate the remaining summary-ask unit tests the PR touches --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
1b1ddef1b2
|
fix: preserve non-null anyOf branches when stripping null types for Vertex (#41683)
* fix: preserve non-null anyOf branches when stripping null types for Vertex * refactor: build anyOf branches immutably and drop explanatory comments * style: apply ruff format * fix: annotate anyOf list construction for the LIT002 budget * fix(vertex_ai): send a null-only anyOf as one nullable null instead of raising * test(vertex_ai): pin empty anyOf passthrough * fix(vertex_ai): forward a non-list anyOf unchanged and add wire cells for the null fold * fix(vertex_ai): forward a non-object anyOf branch for Google to judge instead of crashing * fix(vertex_ai): fold null out of type arrays and union branches before sending to Gemini --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
dba86bec28
|
fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them (#46767)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
|
7090588bdf
|
feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240)
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions * fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel * feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing early-access row, carrying the same limits and capabilities. The AI Studio row mirrors the early-access entry; the Vertex row differs only in provider and pricing source. Prices follow once Google publishes them. Extend the mapped metadata test to cover all three ids, that each routes to its provider, that its backup copy matches, and that the public rows carry the same limits as the early-access row. * feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops * chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap * fix(gemini): follow continuation hops in a loop instead of recursion The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls * fix(gemini): bill Google Search grounding from every continuation request Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain * fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch * chore(gemini): name the type errors the dual-mode stream iterator trips The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up * fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment * fix(gemini): keep one response id across a continued answer A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key * test(gemini): add integration cells for long decode continuation Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once * fix(vertex_ai): fetch the access token again before each continuation request A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged * fix(vertex_ai): start each continuation stream with an empty parser buffer When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async * fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way * fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes * fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not * fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0 A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer Adds the audit's wire tests for both, which fail 24 cells before this change * test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed * fix(vertex_ai): log a continued answer's raw response as one object A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch * test(integration): check a continued answer logs one merged raw response A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch * test(vertex_ai): cover a follow-up tool call and its thought signature in long decode * test(gemini): pin that a one-request answer logs Google's body unchanged |
||
|
|
17f621e116
|
fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836)
* fix(vertex_ai): route Model Garden models through the generateContent adapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vertex_ai): annotate generateContent route test parameters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): type litellm_params in get_vertex_ai_model_route * fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions Streamed Model Garden calls through the generateContent adapter now request stream usage where the deployment supports it, keep cached-token details, and look up the provider-relative cost key, so they bill like the non-stream call. The adapter forwards router metadata, passes upstream errors through with their status, countTokens falls back to local counting for non-Gemini routes, and the native Gemini config is limited to the Gemini routes. * fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing Streamed generateContent through the chat adapter now ends with the upstream usage, honors always_include_stream_usage false and the caller's stream_options, counts one failure per failed call, and routes Vertex deployments by their base_model like chat does * fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage * test(google_genai): expect the trailing usage chunk on bridged OpenAI streams The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone. * fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk * fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent * test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire * test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
08a24d391a
|
fix(count_tokens): count system and tools on Vertex Gemini, fall back to Mantle on Bedrock 404 (#46430)
* fix: align token counting with inference context Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve Vertex Gemini count message shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: forward Google countTokens tools in shapes every provider counter accepts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: add package marker for Vertex count-token unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): ignore deployment system_instruction when counting tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): forward system and tools on Google AI Studio, keep strict on Anthropic tools Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(count_tokens): ruff format gemini count handler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): return Mapping from count request body builders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): truncate long Anthropic tool names on Vertex count, update stale strict test expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): simplify Gemini count context and tests Put the Vertex Gemini count branch back in the same shape as main and the partner model branch, keeping only the system and tools context, the provider error mapping and the gemini tokenizer default Share one error response helper between the Vertex and AI Studio callers, drop the unread kind fields and the extra validation of map_function output, and say Gemini instead of Vertex AI in the error that AI Studio users also see Tighten the tests by removing a stray parametrize on a helper, sharing the SDK count args, asserting the exact forwarded system and tools, and folding the Anthropic tool shape tests into one table * fix(count_tokens): reuse inference tool translation for Vertex Gemini count Vertex and AI Studio count now convert tools with the same functions inference uses, translate_anthropic_tools_to_openai and the Responses tool converter, before VertexGeminiConfig.map_function. Unnamed and Anthropic hosted tools now count the same declarations inference sends instead of being dropped Tools that cannot be converted, such as a non list tools value, a non object tool or a schema that is not a JSON object, now return a 400 before any request instead of being counted without them --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
958cba7016
|
fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results (#38284)
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
|
c00706218b
|
fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097)
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(vertex_ai): keep yield outside the JSON decode guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): surface BaseLLMException raised while streaming a request body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(http_handler): only follow __cause__ when unwrapping streamed body errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): avoid recursive streamed error unwrapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows Keep main's error path for a directly raised provider error and only unwrap one found behind a transport error. Add the parser column to invalid JSON errors, and cover the real aiohttp chain in the _handle_error test * fix(vertex_ai): report every row conversion error as a 400 naming its line A message with no role, a non-string role, a bad temperature, or a litellm BadRequestError raised while converting a row still returned a 500, and the async upload lost the 400 for litellm's own exceptions. The stream now turns any error from one row's conversion into a 400 naming that line. _handle_error only unwraps an httpx.TransportError, which is what the aiohttp transport raises around a failing request body, so other callers keep their own status. A file that starts with a UTF-8 byte order mark now uploads, and provider messages start with 'Batch input file line N' like the proxy check. * fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line A chat row whose body is missing or not an object now raises a 400 naming its line instead of uploading a blank prompt, and a non-object body on line 1 no longer crashes the GCS object name lookup. Rows too deeply nested or too long to parse return a 400 from both the SDK and the proxy's own batch file check. JSON error columns now count leading whitespace. * fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line * test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos * fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact * test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
35970c3eac
|
fix(vertex): report an unreadable credentials file as a read failure (#40173)
* fix(vertex): report an unreadable credentials file as a read failure Tell a path apart from inline JSON by shape, not os.path.exists(). Fixes #40166 * test(vertex): narrow pytest.raises to the expected message * fix(vertex): log the credentials path instead of returning it to the caller * refactor(vertex): log the credential source decision where it is made * fix(vertex): handle paths open() rejects with a bare ValueError * fix(vertex): redact the credentials path before it reaches the log * test(vertex): cover the brace fallback and non-text credentials file * test(vertex): exclude the unreachable exhaustiveness guard from coverage * refactor(vertex): read the failure's fields off the matched object CodeQL's py/uninitialized-local-variable does not follow keyword capture patterns in a match statement and reported `path` and `reason` as possibly unbound. Narrowing on the class and reading the attributes says the same thing without a suppression. * fix(vertex): accept a read-only mapping in extract_aws_params * fix(vertex): keep encoded credentials out of the log and cover the legacy route * fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust * fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file * fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message * test(vertex): integration cells for credentials file and inline value failures * test(vertex): drive the Rust messages cell through Vertex Claude --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
47e4ed88c2
|
fix(anthropic): keep refusal stop_reason and stop_details on the logged response (#46104)
* fix(anthropic): keep refusal stop_reason and stop_details on the logged response * fix(anthropic): keep the refusal stop-fields boundary tolerant of a non-dict upstream body * test(anthropic): use Anthropic's documented refusal example as the unit-test fixture * fix(streaming): keep provider fields on async stream cache hits and received_model_id on synthesized finish chunks * fix(caching): replay cached completion_tokens_details on stream cache hits and skip non-object cached provider fields * test(integration): cover Anthropic-family refusal stop_details on the wire, the cache and under chaos * fix(caching): skip a cached completion_tokens_details that fails validation on stream replay * fix(anthropic): keep refusal stop fields on a JSON-mode non-stream response Also adds the round 3 audit cells for JSON-mode refusals, the proxy response serialization, and wrong-typed cached usage details on both stream replays * test(anthropic): expect content_filter on streamed JSON tool refusals after main's stream change --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
12cce6c372
|
fix(anthropic): check Vertex forced-tool support on the real model, skip non-string user ids, drop speed where unsupported (#46337)
* fix(anthropic): check Vertex and Bedrock forced-tool support on the real model The Vertex and Bedrock invoke Claude transforms swapped the model for a claude-3-sonnet stand-in before calling the shared Anthropic mapping so response_format would take the tool-based path. Every model-keyed check in that mapping then ran against the stand-in, so the synthetic json tool was forced on models whose map entry says forced tool use is unsupported and the request 400ed upstream. Bedrock patched the one symptom with a backstop, Vertex had nothing. The shared mapping now asks a uses_native_structured_output hook for the structured-output decision and the two overrides answer it with their provider-scoped lookup, so the real model reaches every check and the stand-in, the restore dance and the Bedrock backstop go away. * fix(anthropic): skip non-string metadata user ids instead of raising _valid_user_id ran re.match straight on litellm_params metadata.user_id, so an integer id raised TypeError and the proxy answered 500. The user param path already skips non-strings, so the validator now does the same for any caller: a non-string id is simply not forwarded. * fix(utils): drop speed on every provider without fast mode speed rides into get_optional_params as a provider-specific kwarg, so the only gate for it lived in the Anthropic transform and Bedrock converse, OpenAI, Azure and the extra_body legs forwarded it to a 400. The gate now runs once in get_optional_params before provider-specific kwargs are copied, so every provider drops it with drop_params or rejects it with a clean 400, and a direct Anthropic fast-mode model keeps it. * fix(utils): gate speed only under drop_params and keep it for litellm_proxy Raising on speed when drop_params is off would turn requests that tolerant OpenAI-compatible backends accept into 400s, and dropping it for litellm_proxy would disable fast mode on a downstream proxy that supports it * test(anthropic): derive the forced-tool regression model from the cost map and drop new docstrings * test(utils): pin that speed is untouched without drop_params and annotate new test locals * test(integration): lock the Vertex and Bedrock invoke json tool to the real model's forced-tool support Vertex sends tool_choice only when the model accepts forced tools, reasoning_effort maps the same with and without response_format, and Bedrock invoke keeps sending the json tool unforced |
||
|
|
c581c85559
|
fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude (#45871)
* fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude Vertex AI Claude deployments that emulate response_format through a forced tool sometimes nest the schema object under an extra key such as `parameters` or `input`, and that envelope reached message.content with a 200. The converter now unwraps exactly one nested object that matches the schema, leaving a payload that is itself schema-shaped alone. Native structured-output rows sent the deprecated top-level `output_format`, which Vertex rejects with a 400. The Vertex request now carries it as `output_config.format`, with an explicit caller value winning. * fix(anthropic): unwrap tool-emulated json_schema envelopes on streams and tool-name wrappers * test(anthropic): expect the json tool description through a shared constant * fix(anthropic): drop the json tool description that makes sonnet-5 omit required keys * fix(anthropic): keep schema-accepted answers whole and decide held streams at the second key An answer the json_schema accepts is returned as is before any envelope rule runs, the tool-name wrapper is unwrapped only when the schema rejects it, a malformed schema turns the unwrap off, a streamed answer whose first key is a schema property named name or tool_name waits for its second key before streaming, the held fragments go through a StringIO, and a prefix that stays undecided past 4 KB is held to the end * test(anthropic): keep the stream family cells to providers whose stream iterator gets the schema * fix(anthropic): unwrap json tool envelopes by structure instead of schema validation The unwrap now reads only the schema's top-level property names. A lone undeclared key holding an object, alone or beside a name equal to the json tool's own name, is the envelope, and the unwrap repeats while the inner object shares no key with the schema. jsonschema and referencing are no longer imported on the request path. The stream holds only an answer that opens with the tool's name or with an undeclared key holding an object, and a schema without top-level properties streams as it arrives. * fix(anthropic): walk json tool envelopes iteratively * fix(anthropic): keep json tool arguments too deep to decode instead of raising * fix(anthropic): unwrap json tool envelopes beside undeclared string labels * fix(anthropic): keep json answers whole on schemas that allow extra keys and stream held text as sent A schema that allows undeclared keys and requires none now keeps an undeclared wrapper key on both the stream and non-stream paths. When no envelope is found, the held stream text is now sent exactly as the model wrote it. On Vertex /v1/messages, output_format now wins over output_config.format when both are set, which matches the Anthropic adapter. * test(anthropic): cover label envelopes, dynamic-key schemas, as-written held streams and output_format precedence --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
0ebec123fd
|
chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)
* chore(typing): validate provider payloads at the boundary to clear 464 basedpyright Any errors Replaces untyped reads of decoded JSON in 48 provider, guardrail and proxy modules with one pydantic validation at the point the payload enters the module, so every value read afterwards carries a real type. reportAny 6111 -> 5663 and reportExplicitAny 1416 -> 1400. No basedpyright, ruff-strict or type-discipline rule is higher repo-wide or in any single file, and no cast, ignore or suppression marker was added. * chore(typing): replace 47 Any annotations that every caller already satisfies with object Each annotation here is a parameter or return type whose value is only stored, compared, logged or forwarded, so object describes it exactly and no function body changes. A site was kept only if no basedpyright, ruff-strict or type-discipline rule rose in any file that references it. reportExplicitAny 1400 -> 1353 and reportAny 5663 -> 5655. * chore(typing): type upstream responses and loose annotations across providers, proxy and integrations Second pass of the Any reduction. Decoded upstream JSON is validated once at the provider boundary and read through typed views afterwards, annotations that every caller already satisfies move from Any to a concrete type, and the nine search transformations validate each result item on its own so error text no longer carries the item index. No cast, no type: ignore, no suppression marker and no budget file edit * ci(unit): run the brave, google_pse, nlp_cloud and opensandbox tests in the providers shard |
||
|
|
1587faeeb9
|
test(unit): wait on the work itself instead of fixed sleeps (#46261)
* test(unit): wait on the work itself instead of fixed sleeps in 21 flaky files Replace sleeps that raced background logging, cache writes, refreshes and thread work with either awaiting that work directly (logging worker flush, the batch logger's flush_queue, the tasks the call spawned, thread joins) or an Event the test's own fake sets when it runs. Each wait keeps a 5s hang guard that never decides pass or fail. Tests that need a fake clock, real sockets, or measure latency are left unchanged, and no production code moves * test(unit): await the tasks each completion spawns instead of yielding once The logging tests yielded once with asyncio.sleep(0) so the task that enqueues the success log could run before the worker flush. That relied on the enqueue happening before the task's first await. Snapshot the running tasks around each acompletion call and await the ones it spawned, so the guarantee holds however many steps sit before the enqueue |
||
|
|
fdab308646
|
fix(vertex_ai): keep Gemini cache creation token count out of request kwargs (#46152) | ||
|
|
0eff31ae4e
|
fix(gemini): bill context cache creation tokens from cachedContents responses (#46144)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
b24b68e576
|
fix(gemini): price tool-use prompt tokens as input (#46021)
* test(gemini): tool-use prompt tokens must be priced as input Signed-off-by: Delweng <delweng@gmail.com> * fix(gemini): price tool-use prompt tokens as input toolUsePromptTokenCount was folded into usage.prompt_tokens but left out of prompt_tokens_details.text_tokens, and generic_cost_per_token bills input from text_tokens, so URL context / code execution / File Search tool tokens were counted but never charged. Search-grounded responses keep adding zero via the grounding-aware billable count, and promptTokensDetails absent still leaves text_tokens None so billing falls back to the full prompt_tokens Signed-off-by: Delweng <delweng@gmail.com> --------- Signed-off-by: Delweng <delweng@gmail.com> |
||
|
|
87478c7b6d
|
fix(gemini): do not inject temperature for Gemini 3+ models when omitted (#45107)
* fix(gemini): do not inject temperature for Gemini 3+ models when omitted map_openai_params added temperature=1.0 to every Gemini 3+ request that did not set one, on both the gemini and vertex_ai routes. Google is deprecating temperature, top_p and top_k for Gemini 3+ and says upcoming models will reject them, so this default turns a request that never set temperature into one that will fail An omitted temperature now falls back to Google's own default, which their docs give as 1.0 for all Gemini 3 models. Explicit temperature, top_p and top_k are still forwarded Fixes #38663 * test(gemini): annotate sampling regression tests * test(gemini): drop the injected temperature from Gemini translation cases The nine Gemini base cases for chat completions, responses and messages send no temperature, so the provider request no longer carries the removed 1.0 default. Each new body was sent to the real Gemini API for gemini-3.5-flash, gemini-3.8-flash and gemini-3.1-pro-preview and returned 200, and the stored provider responses are unchanged * test(gemini): drop the default-temperature wording from the thinking level test docstring * test(gemini): cover omitted and explicit sampling on both Gemini routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
a7a3a6b375
|
test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940)
* test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e * test: fix disk cache collection, ungate saturation check, drop RSS memory tests * test: assert one delivered log per completion in Datadog and LangSmith checks * test: drop SDK-direct e2e copies and align cost-map path filter cases --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
|
e0398455d0
|
test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759)
* test: migrate selected llm translation contracts * test: migrate anthropic completion translation tests * test: migrate bedrock and cohere translation tests * test: tighten bedrock translation migration * test: migrate azure and openai model translation tests * test: migrate openai and small-provider translation tests * test: migrate gemini and vertex translation tests * test: migrate optional params and rerank translation tests * test: migrate huggingface, json and openrouter translation tests * test: fix openrouter integration expectations * test: migrate databricks, xai, cometapi and fireworks translation tests * test: migrate ovhcloud, prompt and small provider translation tests * test: split reasoning effort grid into unit and e2e * test: migrate router and realtime guardrail translation tests * test: migrate google interactions translation tests * test: migrate a2a, evals, skills and hosted vllm translation tests * test: remove remaining llm translation suite * test: tighten migrated llm translation unit tests and land missing rows Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover * test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths * test: close lost-intent gaps found in review of the llm translation migration Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised * test: assert provider acceptance in the reasoning effort e2e The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid * test: cover async and stream-end paths the legacy translation suite exercised Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker * test: type the new llm translation e2e calls and mark moved fixture dirs as packages * test: name the reasoning effort e2e models as module constants * test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name * test: block the prompt the shared guardrail denies and drop the reasoning presence check * test: drop the duplicate litellm import left by the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
|
1dbde0845e
|
test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758)
* test: move guardrails, image, audio, batches, search legacy tests to unit * test: restore the custom guardrail and remaining chunk 1 unit rows * test: move the ocr matrix to e2e * test: drop comments from moved tests and fix vertex ocr credentials * test: move litellm utils and unified google legacy tests to their tiers * test: move logging callback and pass-through unit legacy tests to their tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: move store_model_in_db legacy tests and start the admin ui migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: finish the proxy job rows and drop their CircleCI jobs Moves the remaining multi-instance, openai endpoint, otel, pass-through and top-level proxy tests into tests/integration with scripted upstreams, rewrites the weak replacements to assert the outbound wire and exact responses, and removes the build_and_test job group this PR owns from CircleCI * test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names * test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist * test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in * test: type the e2e beta header, prompt cache and secret manager values * test: assert the missing model error message on the OpenAI routes * test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks * test: make the langfuse fixture directory a package for the layout check * test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane * test: keep the ocr e2e pdf fixture next to the test * test: put meta below covers on the streaming tool search e2e * test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures * test: count only OCR calls in the OCR matrix recorder * test: pin OCR parse bodies and billed cost, group new e2e cases into classes * ci: restore the python image anchor dropped in the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
a2e334febe
|
test: migrate local_testing SDK and provider tests out of build_and_test (#45760)
* test: migrate local_testing SDK and provider tests out of build_and_test * test: cover sagemaker region, hf template and nova paths end to end Region rows now go through litellm.completion on the sagemaker/ handler that the legacy tests exercised, the prompt template test mocks the Hugging Face tokenizer config instead of switching to custom_prompt_dict, streaming tests assert role, finish reason and body, and Nova single turn and streaming run through litellm.completion with a recorded request body * test: tighten migrated integration tests for local OpenAI routes, timeouts and air-gapped boot Split the catch-all providers migration file by behavior. The fake endpoint route tests now hit the owned upstream directly, as the legacy harness tests did. LM Studio asserts the outbound key, path and stripped model. The embedding timeout test asserts the proxy maps a deployment timeout to a 408 and forwards the read timeout to the provider client, instead of timing out the test client The Azure streaming tool-call test also asserts the second identical stream is served from the Redis cache with the same reassembled tool call The Docker network test always skipped in CI because the security job has no litellm/litellm image and no egress to pull one, and it tested a published image rather than the checkout. It now boots the checkout proxy with every outbound HTTP connection refused and asserts readiness, the bundled cost map fallback and that the cost map host is the only egress attempted * test: tighten the migrated e2e rows for local_testing SDK and provider tests Drop the duplicated Anthropic covers marker and the Anthropic cell wrongly claimed by the Bedrock and Vertex cache tests. Keep the file-wide Anthropic model as it is on main. Send the 2023-06-01 version header to Anthropic through the deployment's extra_headers so the header actually reaches the provider, and mark the creation-token case 1h then 5m like the legacy test did Reuse the Bedrock file's own registration for the pydantic response test, swap the hand-built PNGs for the shared cat.jpg fixture, assert the SageMaker Nova image answer names the cat and that top_logprobs=3 is honored, and register the SageMaker Nova opt-in markers the same way as the other opt-in markers * test: restore lost assertions in migrated test_main tests Restore the role alternation check in test_completion_anthropic_hanging, the embedding and streaming response-header checks, the logit_bias drop check and the gemini system and safety settings checks. Add an azure 429 retry-after test for test_exception_with_headers, trigger the router fallback with mock_timeout like the legacy test, and stop the mock timeout tests sleeping 3s * test: restore lost legacy assertions in migrated vertex ai unit tests * test: tighten migrated cost, streaming and embedding tests and scope the hosted mock guard * test: strengthen migrated anthropic, azure, bedrock and cohere provider tests Assert full request bodies and usage instead of presence checks, restore the dummy-tool and custom_llm_provider assertions the legacy tests made, add the async and per-model variants the legacy parametrization covered, split streamed tool arguments across deltas, and move the Azure chat streaming tests to the azure.py mirror file * test: tighten migrated llms provider tests and cover legacy intents lost in the move Strengthen the migrated fireworks, gemini, mistral, together, openai text completion, openrouter, perplexity and openai error tests so they assert what the legacy tests proved (cost, provider tag, usage-only-on-last-chunk, function_call rebuild, reasoning param, sync/async/stream error fields). Replace the huggingface and jina tests that doubled litellm code with respx end-to-end tests. Add the tests the pre-existing mapped tests never covered: cohere 400 mapping, token-array text completion, hosted_vllm/llamafile/lm_studio env api base embeddings, watsonx embedding auth, ollama ssl_verify client and ollama vision prompt * test: cover residual legacy paths for anthropic tool streams, watsonx async stream, ollama http images and caller async embedding clients * test: restore single request assertions in the deep_infra and azure_ai embedding tests * test: drop a no-op streaming cost flag from the openai streaming cost test * test: cover residual legacy-test lines in caching, main, utils, replicate, vertex and bedrock invoke * test: cover streamed audio assembly, choice membership, untranslated responses events and the stream log guard * test: pin cost rates in unit tests, use sdk for cohere e2e, drop vertex llama e2e * test: send the vertex maps e2e request through the openai sdk * test: drop the local_testing jobs that select no tests after the migrations * test: wait for the streamed cache entry and keep custom Mistral rates test-local * test: keep untouched Mistral test lines as they were --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
|
cf8bc755a7
|
fix(vertex_ai): parse Vertex timestamps as UTC so created_at is not shifted by the host timezone (#45550)
* fix(vertex_ai): parse Vertex timestamps as UTC so created_at is not shifted by the host timezone convert_vertex_datetime_to_openai_datetime parsed the trailing-Z Vertex and GCS timestamps into a naive datetime, so .timestamp() read them as host local time and every created_at it produced was off by the host UTC offset (about 7h ahead on a PDT host). Attach UTC before converting, and accept the whole-second and nanosecond forms Google's RFC 3339 timestamps also use, which used to raise. Refs LIT-9383 * test(vertex_ai): type the UTC epoch test, guard tzset, build the rig config without mutation |
||
|
|
9b53ee878f
|
perf(ci): isolate and shard integration suites (#44146)
* perf(ci): distribute unit cases and reuse security sweep connections * revert(ci): reject first speed trial after new failures * perf(ci): avoid repeated test discovery and parsing * perf(ci): isolate parallel security workers * revert(ci): restore measured security execution isolation * perf(tests): reuse source scans and deduplicate cache cleanup * perf(ci): restore parallel security with safe log batching * test(ci): isolate fixture state and schema comparisons * test(ci): make schema and scheduler fixtures deterministic * test(ci): drain queued logging before fixture teardown * test(ci): preserve logging delivery without idle worker restarts * test(ci): keep guardrail sinks available during parallel bursts * test(ci): isolate full proxy lifecycle from gateway imports * test(ci): preserve first failure responses and sweep evidence * test(ci): restore documentation URL settings between cases * test(ci): isolate open MCP tools in private gateways * test(ci): group private MCP cases and handle exited workers * test(ci): retain proxy exits and MCP failure details * test(ci): repair merge interactions with main Key the schema shard's full clone on artifact-name, since main removed the unit-flag input. Give each owned Redis its own log file again. Hand the realtime transcription tests main added a logging object whose dispatch handlers are awaitable, so the per-test logging drain no longer waits on a MagicMock task forever * perf(ci): shard owned integration cases and close lifecycle leaks * fix(ci): qualify shard execution and harden interrupted cleanup * fix(ci): bound worker supervision and own fault traffic * perf(ci): reuse unparametrized fixtures across bounded case groups * fix(ci): preserve boot fault allowances and complete exit records * test(ci): tolerate startup retries and retain readiness details * perf(ci): balance integration shards by recorded duration * fix(ci): load migration database helpers only for workers |
||
|
|
d6db8e8744
|
fix(gemini): map OpenAI voice names to Gemini prebuilt TTS voices (#45303)
* fix(gemini): map OpenAI voice names to Gemini prebuilt TTS voices * test(gemini): type the voice mapping test helpers * fix(gemini): leave nova out of the voice mapping since Gemini accepts it as given * test(gemini): add integration cells for the OpenAI voice mapping on Gemini and Vertex TTS --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
cddc7cde97
|
test: fix stale and flaky tests across CircleCI, GHA and Buildkite (#45530)
* test(integration): poll the partition lock witness off the event loop The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting * test(e2e): retry the reasoning turn until the model emits a reasoning item Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion * test(e2e): retry a Bedrock Converse model once when it reports no cache tokens Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged * test(e2e): poll until a deleted stored response stops being retrievable Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx * test(e2e): give the Together prompt cache five primed attempts Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed * test(e2e): allow 30s for the idle RSS reading at session start A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged * test(e2e): scope the MCP submission check to its own card The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card * test(router): wait for the primary success record before counting it The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs * test(proxy): give the spend-log cleanup run a 1s budget A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves * test(autorouter): release the slow token count at 2.05s instead of 2.3s The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack * test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting * test(integration): give the two worker-kill chaos tests the owned-proxy time budget Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests * test(integration): compare regional image responses without their created timestamp Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared * test(integration): count only the S3 logger's flush task The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels * test(integration): send guardrail timeout probes eight at a time All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits * test(integration): count a killed Arize chaos worker as gone once it is a zombie psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget * test(integration): send the Typesafe connection test its real model and endpoint The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly * test(integration): keep DB-stored models out of the usage-routing Redis read test The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off * test(vertex_ai): prove the batch upload streams by laziness instead of peak memory The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests * test(integration): keep the alias test_connection call as a known bug * test: type the counting mapper and the Bedrock rerun helper * test: annotate the new test locals as Final and build them as tuples * test(vertex_ai): prove pull-driven transforms with the garbage tail alone |
||
|
|
56bafc29cf
|
feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305)
* feat(gemini): accept file content blocks with video_metadata on multimodal embeddings * test(gemini): annotate the embedding mock helper and keep the type comment short * fix(gemini-embeddings): keep file blocks through a partial embedding cache hit and validate video_metadata strictly * test(caching): type the partial-hit cache test and pin its clock * fix(gemini-embeddings): accept the Files API URI /v1/files returns as a file block's file_id * fix(gemini-embeddings): 400 on bad input shapes, drop unknown block keys under drop_params, cache multi-block inputs A bare object `input` answered 500 from both the caching handler and the transformation; both now answer 400. An unresolved `files/` reference on vertex_ai/ answered a ValueError 500; it now answers 400 naming the gemini/ provider. An empty `format` passed through to the provider; it now answers 400 naming file.format. Unknown block keys (`detail`, an unknown video_metadata key, a top-level block key) are dropped under drop_params, global or per-request, and still answer 400 without it. The embedding cache counted file blocks against `max_messages`, so a request with 5 or more blocks was never cached; file blocks no longer count. * fix(caching): answer 400 for an object embedding input on the cache lookup * test(gemini-embeddings): annotate the new embedding tests with return types and Final * test(gemini-embeddings): annotate the transformation and files batch tests with return types and Final * test(gemini-embeddings): audit file content blocks on the wire, in the cache, in batch uploads and under chaos Integration cells for gemini/ and vertex_ai/ embedding file blocks with video_metadata: the batchEmbedContents and embedContent wire shapes, format overrides, nested and repeated inputs, every malformed block answering 400 before any provider call, drop_params at the request, deployment and YAML levels, the OpenAI SDK sync and async clients, Redis cache fills, hits, partial hits and the metadata in the key with zero-spend hit rows, chat, responses and messages controls, the vertex_ai/ batch JSONL upload, a mixed fast/slow/malformed/dropped burst and a worker kill. The wire peer now reads chunked request bodies, which the streamed GCS media upload sends * test(integration): treat an upload the client abandons before its terminating chunk as a disconnect, never a stored request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
3571b5dc72
|
fix(vertex_ai): forward the inline-tools-2026-09-15 beta to Vertex and Anthropic (#45297)
* fix(vertex_ai): forward the inline-tools-2026-09-15 beta to Vertex and Anthropic * test(vertex_ai): trim the inline-tools beta test docstrings and type their locals * test(vertex_ai): add audit cells for inline-tools beta forwarding on Vertex Anthropic * test(vertex_ai): send legal whitespace in the hostile beta header audit cell * test(vertex_ai): close the SDK clients in the inline-tools audit cells --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |