mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-04 07:27:51 +00:00
15 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Gopal Kamineni
|
a7030a5a2c
|
fix(vertex_ai): batch Claude models in Anthropic's native row shape (#45715)
* fix(vertex_ai): batch Claude models in Anthropic's native row shape
Claude deployments on Vertex were filed under publishers/google/models/<claude>
when the batch input was uploaded, so create_batch asked Vertex for a Google
model that doesn't exist and got a bare 404 on batchPredictionJobs. The rows
were also written as Gemini GenerateContentRequests, which Vertex rejects for
Claude even with the right model path
Claude uploads now go under publishers/anthropic/models/<model> and each line
becomes {"custom_id", "request": <Anthropic Messages body with
anthropic_version vertex-2023-10-16>} using the existing Vertex Claude
transform. Output rows in the Anthropic message shape are converted back to
OpenAI batch rows (custom_id from the row, errors as failed rows), native rows
are costed from Anthropic usage, and create_batch fails fast with a 400 when a
Claude job targets the global location, which Vertex doesn't serve for
partner model batches
Fixes #45671
* test(vertex_ai): cover malformed Claude output rows and rows without custom_id
* fix(vertex_ai): build Claude batch rows with chat's param mapping and surface Vertex batch create errors
Claude batch rows now go through get_optional_params like Vertex Claude chat, so unsupported params, top_k, response_format and extra_body follow drop_params. Tool names Claude would rename get a 400 at upload. A refused batch create returns Vertex's own status and message.
* fix(vertex_ai): price Claude batch output rows without a request echo, keep @ version ids, refuse a json_tool_call user tool, keep Vertex error headers
* fix(vertex_ai): keep httpx.HTTPStatusError on Vertex batch create errors and apply deployment chat params to Claude batch rows
* fix(vertex_ai): keep Vertex's status code on batch create errors through the proxy
* fix(vertex_ai): keep Vertex's error body on batch create errors and require custom_id on Claude lines
* fix(vertex_ai): keep the old batch create error text and Google object names
A failed Vertex batch create keeps httpx's error line first and adds Vertex's body after it, and logs the status and body again. Only Claude object names keep the @ version, so Google models are stored under the same names as before
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
yucheng-berri
|
788868a52f
|
fix(security)!: build proxy-only failure logs from an allowlist and drop caller guardrail info (#48599)
* fix(proxy): build proxy-only failure logs from an allowlist and ignore request-time GCS credentials * fix(proxy): drop caller metadata from proxy-only failure logs metadata and litellm_metadata leave the proxy-only failure-log allowlist, so a rejected request can no longer point its failure log at its own Langsmith or Langfuse host through nested metadata. * fix(proxy): keep allowlisted metadata in proxy-only failure logs * test(proxy): isolate proxy-only failure log metadata tests * fix(logging): ignore non-list request tags when building the logging payload A guardrail-rejected request with metadata tags set to a non-list value crashed the standard logging payload, so its spend row lost model attribution. * fix(proxy): keep router attribution and drop unauthenticated tags in proxy-only failure logs A request rejected before routing now logs the configured deployment and model group even when its metadata or litellm_metadata carries its own values. Requests rejected at auth no longer write caller tags into failure logs and tag spend. * fix(proxy): always strip client guardrail info and mirror failure attribution into litellm_metadata allow_client_pricing_override no longer keeps a caller-supplied standard_logging_guardrail_information on the chat or pass-through paths, since its guardrail_cost is billed. Proxy-only failure logs now also carry key and team attribution in litellm_metadata. * refactor(proxy): keep the proxy-only failure-log fixes inside the type gates Types the picked helpers so basedpyright and the type-discipline gate stay at main's counts. No runtime change. * fix(proxy): keep the WS frame guardrail-info strip and string interaction tags as sent The guardrail-info key left the pricing strip list, so the Responses WS frame guard now drops it by name. get_request_tags no longer raises on string tags, so background interaction polling checks for them before keeping them as sent. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Emerson Gomes
|
045fb7ba84
|
feat(vertex-ai)!: support gemini 3.1 flash tts (#31915)
* feat(vertex-ai): support gemini 3.1 flash tts
* fix(vertex-ai): keep TTS metadata compatible with the cost map guard
* fix(vertex-ai): preserve existing Gemini TTS models and voice mappings
* fix(vertex-ai): recognize provider-stripped TTS model names
* test(vertex-ai): match current speech HTTP error response
* fix(vertex-ai): preserve nested speech locale and voice
* fix(vertex-ai): use provider speech media type in proxy
* fix(gemini): preserve TTS dispatch and nested speech settings
* test(gemini): run TTS regressions in CI shards
* fix(vertex-ai): account for Gemini Cloud TTS audio cost
* fix(vertex-ai): bill Gemini TTS without optional audio decoder
* test(gemini): cover TTS mapping and billing edge cases
* test(vertex-ai): cover Gemini voice normalization in provider shard
* fix(vertex-ai): keep Gemini TTS routing with generic provider metadata
* chore(vertex-ai): remove obsolete lint suppressions
* fix(gemini): strip audio format from speech configuration
* chore(vertex-ai): remove obsolete mutation suppressions
* fix(vertex-ai): preserve FLAC billing and wrapped audio headers
* fix(vertex-ai): reject unmeasurable FLAC before billing
* fix(vertex-ai): reconcile speech support with current provider mappings
* fix(vertex-ai): validate speech mappings and usage types
* fix(vertex-ai): preserve speech mapper override forwarding
* fix(gemini): map voice aliases across speech request forms
* fix(vertex-ai): send only formats the speech bridge cannot serve to Cloud TTS
Gemini TTS detection now goes by model name, so newer Vertex Gemini TTS
models keep working, and only mp3, opus, alaw, mulaw and other container
formats move to Cloud Text-to-Speech. wav and pcm requests stay on
generateContent as on main.
A Cloud TTS response whose duration cannot be measured now returns the
audio unbilled with a warning instead of raising after a paid call. FLAC
handling is removed because Cloud TTS has no FLAC encoding.
The cost map, the google_genai mappers and the proxy error path go back
to main, since those changes were outside this feature.
* fix(vertex_ai): keep Gemini TTS formats, region and instructions on routes that serve them
Only mp3, opus, alaw and mulaw go to Cloud Text-to-Speech. Every other
format gets a local 400 that lists the supported ones, before Google is
called.
Cloud TTS uses the deployment's us or eu regional host. Other regions
and a custom api_base get a 400 that says to request wav or pcm.
instructions is sent as the Cloud TTS style prompt, and a request with
no voice keeps the Gemini model.
get_usage_object is back to main's code, so dict usages still go
through Usage.__init__, and the proxy no longer labels pcm16 or
linear16 output as WAV.
* fix(vertex_ai): send Gemini TTS to every regional Cloud TTS host and keep speak tags as text
Cloud Text-to-Speech serves Gemini TTS on a host named after each Vertex
location, so a non-global vertex_location now uses
{location}-texttospeech.googleapis.com instead of only the us and eu hosts.
Google answers for regions it does not serve.
Gemini TTS voices reject SSML, so input that contains <speak> is sent as
text for Gemini TTS models. Chirp keeps the SSML auto-detect.
Removes the proxy alaw and mulaw media-type entries. Cloud TTS returns
those encodings with a WAV header, so the response is labeled audio/wav
and the entries never applied.
Adds a regression test that custom per-token pricing bills the usage a
speech response reports.
* fix(vertex_ai): map Gemini TTS regions to Google's Cloud TTS locations and keep a bad voice from cooling down the deployment
US and EU Vertex regions go to the us and eu Cloud TTS hosts, northamerica-northeast1 to its own, global or unset to global, and any other region gets a local 400. A Cloud TTS 404 on the Gemini path, such as a misspelled voice, comes back as a 400 so the router keeps the deployment serving wav. Usage reads the sent request from the response instead of the logging object.
* fix(vertex_ai): default Gemini TTS Cloud formats to a Gemini voice and validate speakers
Parse only usage fields from the sent Cloud TTS request, so a numeric speakingRate no longer fails after Google bills it.
Send the Gemini voice Kore when no voice is named, and reject Cloud TTS voice names unless drop_params is set.
Reject malformed speaker entries unless drop_params is set, and translate Cloud TTS speaker fields on wav and chat audio.
Use the Vertex default region when no location is set, so mp3 and wav share a region.
* test(vertex_ai): cover two-speaker Gemini TTS mp3 through aspeech with fake credentials
Replaces two tests that patched the Vertex TTS config's own token and URL methods.
* fix(vertex_ai): reject malformed Gemini TTS voices and keep beta Chirp and Lyria params unmapped
* fix(vertex_ai): drop unknown Gemini TTS voice keys on wav and pcm instead of rejecting them
* fix(vertex_ai): reject unknown Gemini TTS voice keys on wav, pcm, and chat audio unless drop_params is set
* fix(vertex_ai): honor every drop_params setting in the Gemini TTS voice key check
Global drop_params, batch row and deployment drop_params on Vertex batch
file uploads now drop unknown voice keys. A voice object with both
speechConfig and speech_config gets a 400 for the ignored speech_config.
* fix(gemini): drop unknown nested voice fields under drop_params and detect TTS by served model
* fix(gemini): filter unknown speech config fields with fixed-depth helpers and pass audio params through unchanged
* fix(gemini): camel-case snake replicated voice keys so drop_params keeps them
* fix(vertex-ai): reject voice fields Cloud TTS cannot send unless drop_params is set
* fix(vertex-ai): reject wrong-typed Cloud TTS voice values and check speakers as sent
Gemini TTS on mp3, opus, alaw, or mulaw now answers 400 for a voice field whose value Cloud TTS cannot send, such as a list languageCode or a string voiceConfig, and for nested voice fields under a speaker that also names speakerId. drop_params still drops them.
* fix(vertex-ai): reject wrong-typed speech config wrappers and speakerId overriding a prebuilt voice on Cloud TTS
* fix(vertex-ai): keep speakers and voice_config beside a speech_config wrapper on Cloud TTS
* fix(vertex-ai): pick Cloud TTS voices from the same speech config the WAV path sends to Gemini
* fix(files): carry the deployment's drop_params on model= file uploads so Vertex batch rows drop unknown voice keys
* fix(gemini): keep a structured speech voice when the model is a model_alias_map alias for a Gemini TTS model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
ryan-crabbe-berri
|
c29a4f56a3
|
fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later (#47645)
* fix(gemini)!: honor supports_sampling_params on the Gemini and Vertex path and flag Gemini 3.6 Flash, 3.5 Flash-Lite and later Google ignores temperature, top_p and top_k on Gemini 3.6 Flash, Gemini 3.5 Flash-Lite and every later Gemini release, and says a later generation will return 400 for them. The model map now carries supports_sampling_params: false on those rows, and the Gemini and Vertex transformation reads the flag the way the Anthropic and Bedrock paths already do, dropping the params under drop_params and otherwise raising the client-side 400 that names drop_params BREAKING CHANGE: a request that sends temperature, top_p or top_k to a flagged Gemini model without drop_params now gets a 400 instead of a 200 whose sampling params Google silently ignored Fixes #47015 * fix(gemini): gate the passed-through top_k where the request body is built and keep allowed_openai_params working top_k is a provider-specific kwarg that never reaches the OpenAI param mapping, so the flag has to be applied while the Gemini request body is built, the same boundary Anthropic uses. The mapping now skips a flagged sampling param instead of raising, because the shared unsupported-param check already returns the 400 that names drop_params, and a param the caller lists in allowed_openai_params is forwarded verbatim as that 400 promises * fix(gemini): filter top_k out of the generation config instead of mutating optional_params * test(gemini): type the sampling-flag twin test and keep its locals out of the loop body * fix(vertex_ai): reject or drop sampling params on batch rows for flagged Gemini models The batch JSONL wrapper calls map_openai_params directly and skips get_optional_params, so a flagged model's temperature, top_p and top_k vanished from the uploaded row with no 400. Batch rows now get the same contract as a real-time call: a 400 naming drop_params, or the knobs dropped when litellm.drop_params is on * fix(vertex_ai): ignore null sampling knobs on batch rows for flagged Gemini models A batch row carrying temperature: null, top_p: null or top_k: null means no override, the same as on a real-time call, so the row is no longer refused with a 400 and the null knobs are simply not forwarded |
||
|
Max Azatian
|
208cac49f0
|
fix(vertex_ai)!: send integer enums to Gemini instead of dropping them (#44908)
* fix(vertex_ai): send integer enums to Gemini instead of dropping them
Gemini rejects enum values that are not strings, so _fix_enum_types removed
every non-string enum. Integer enums are now sent as type INTEGER with
format "enum" and string values, which Gemini accepts and enforces.
* fix(vertex_ai): keep converted integer enums when a tool is reused
The schema is edited in place, so a tool definition passed again already
carries the converted enum. Keep an integer enum that is already in the
format "enum" form instead of dropping it, and type the new locals.
* fix(vertex_ai): use Sequence at runtime in the enum cast
* fix(vertex_ai): treat integer-string enum values as an integer enum
A reused schema already holds its integer enum as strings. Instead of
reading `format` to recognise that, accept integer strings as well as
integers on an integer-typed enum and send both in Gemini's form. This
drops the untyped `schema.get("format")` read.
* fix(vertex_ai): map Gemini tools from a copy so a fallback keeps the caller's schema
Gemini tool mapping rewrote the caller's tools in place, so a router fallback to another provider received the Gemini-converted integer enum (string values with format enum) and returned the integer as a string
* fix(vertex_ai): send integer enums only to Gemini 3 and newer, and only ones Gemini accepts
gemini-2.5-flash stalls for minutes on a larger integer enum, so Gemini 2.x
keeps main's behavior and drops it. An enum outside int64 or longer than 1000
values is dropped too, since Gemini answers 400 for those. A nullable integer
enum written as a type list now keeps its values like the anyOf form does
* fix(vertex_ai): drop integer enum strings longer than int64 instead of failing the request
* fix(vertex_ai): drop integer enums when a request's enums pass Gemini's size limit, and send int enum members as numbers
Gemini limits the total size of all enums in one request, so the per-enum 1000-value cap still let several large integer enums through to a 400. Integer enums are now sent only while every enum value in the request adds up to 2000 characters or fewer, and an (int, Enum) member goes out as its number instead of its name
* fix(vertex_ai): send only integer enums Gemini returns exactly, and count them in Gemini token counts
* fix(vertex_ai): send Gemini integer enums without format enum, which made gemini-3.8-flash think much longer
Also skip the response cache in the repeated-request integration test, since copying tools makes the second identical request a cache hit
* fix(vertex_ai): keep dropping integer enums in Vertex batch rows, which Vertex batch cannot parse
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
c00706218b
|
fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097)
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(vertex_ai): keep yield outside the JSON decode guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): surface BaseLLMException raised while streaming a request body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(http_handler): only follow __cause__ when unwrapping streamed body errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): avoid recursive streamed error unwrapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows Keep main's error path for a directly raised provider error and only unwrap one found behind a transport error. Add the parser column to invalid JSON errors, and cover the real aiohttp chain in the _handle_error test * fix(vertex_ai): report every row conversion error as a 400 naming its line A message with no role, a non-string role, a bad temperature, or a litellm BadRequestError raised while converting a row still returned a 500, and the async upload lost the 400 for litellm's own exceptions. The stream now turns any error from one row's conversion into a 400 naming that line. _handle_error only unwraps an httpx.TransportError, which is what the aiohttp transport raises around a failing request body, so other callers keep their own status. A file that starts with a UTF-8 byte order mark now uploads, and provider messages start with 'Batch input file line N' like the proxy check. * fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line A chat row whose body is missing or not an object now raises a 400 naming its line instead of uploading a blank prompt, and a non-object body on line 1 no longer crashes the GCS object name lookup. Rows too deeply nested or too long to parse return a 400 from both the SDK and the proxy's own batch file check. JSON error columns now count leading whitespace. * fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line * test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos * fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact * test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
cddc7cde97
|
test: fix stale and flaky tests across CircleCI, GHA and Buildkite (#45530)
* test(integration): poll the partition lock witness off the event loop The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting * test(e2e): retry the reasoning turn until the model emits a reasoning item Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion * test(e2e): retry a Bedrock Converse model once when it reports no cache tokens Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged * test(e2e): poll until a deleted stored response stops being retrievable Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx * test(e2e): give the Together prompt cache five primed attempts Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed * test(e2e): allow 30s for the idle RSS reading at session start A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged * test(e2e): scope the MCP submission check to its own card The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card * test(router): wait for the primary success record before counting it The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs * test(proxy): give the spend-log cleanup run a 1s budget A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves * test(autorouter): release the slow token count at 2.05s instead of 2.3s The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack * test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting * test(integration): give the two worker-kill chaos tests the owned-proxy time budget Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests * test(integration): compare regional image responses without their created timestamp Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared * test(integration): count only the S3 logger's flush task The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels * test(integration): send guardrail timeout probes eight at a time All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits * test(integration): count a killed Arize chaos worker as gone once it is a zombie psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget * test(integration): send the Typesafe connection test its real model and endpoint The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly * test(integration): keep DB-stored models out of the usage-routing Redis read test The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off * test(vertex_ai): prove the batch upload streams by laziness instead of peak memory The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests * test(integration): keep the alias test_connection call as a known bug * test: type the counting mapper and the Bedrock rerun helper * test: annotate the new test locals as Final and build them as tuples * test(vertex_ai): prove pull-driven transforms with the garbage tail alone |
||
|
devin-ai-integration[bot]
|
56bafc29cf
|
feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305)
* feat(gemini): accept file content blocks with video_metadata on multimodal embeddings * test(gemini): annotate the embedding mock helper and keep the type comment short * fix(gemini-embeddings): keep file blocks through a partial embedding cache hit and validate video_metadata strictly * test(caching): type the partial-hit cache test and pin its clock * fix(gemini-embeddings): accept the Files API URI /v1/files returns as a file block's file_id * fix(gemini-embeddings): 400 on bad input shapes, drop unknown block keys under drop_params, cache multi-block inputs A bare object `input` answered 500 from both the caching handler and the transformation; both now answer 400. An unresolved `files/` reference on vertex_ai/ answered a ValueError 500; it now answers 400 naming the gemini/ provider. An empty `format` passed through to the provider; it now answers 400 naming file.format. Unknown block keys (`detail`, an unknown video_metadata key, a top-level block key) are dropped under drop_params, global or per-request, and still answer 400 without it. The embedding cache counted file blocks against `max_messages`, so a request with 5 or more blocks was never cached; file blocks no longer count. * fix(caching): answer 400 for an object embedding input on the cache lookup * test(gemini-embeddings): annotate the new embedding tests with return types and Final * test(gemini-embeddings): annotate the transformation and files batch tests with return types and Final * test(gemini-embeddings): audit file content blocks on the wire, in the cache, in batch uploads and under chaos Integration cells for gemini/ and vertex_ai/ embedding file blocks with video_metadata: the batchEmbedContents and embedContent wire shapes, format overrides, nested and repeated inputs, every malformed block answering 400 before any provider call, drop_params at the request, deployment and YAML levels, the OpenAI SDK sync and async clients, Redis cache fills, hits, partial hits and the metadata in the key with zero-spend hit rows, chat, responses and messages controls, the vertex_ai/ batch JSONL upload, a mixed fast/slow/malformed/dropped burst and a worker kill. The wire peer now reads chunked request bodies, which the streamed GCS media upload sends * test(integration): treat an upload the client abandons before its terminating chunk as a disconnect, never a stored request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mubashir1osmani
|
8b68c3cd09
|
fix(vertex_ai): keep legacy bucket_name in credential resolution and add GCS_BATCH_BUCKET_NAME env var (#42803)
* fix(vertex_ai): map legacy bucket_name to gcs_bucket_name and add GCS_BATCH_BUCKET_NAME env var * refactor(router): keep legacy bucket_name as a credential field instead of a validator Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): pass the RAG corpus bucket to the file upload instead of hopping through GCS_BUCKET_NAME * fix(vertex_ai): accept existing_file_id in the RAG Engine store step so ingest() runs end to end --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
5e6dc89ba1
|
test: move tests/test_litellm/llms into tests/unit/llms (#43191)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: keep the job's UNIT_FLAG out of the shard-script tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3fa688223d
|
fix(vertex_ai): translate /v1/responses batch rows through the Responses-to-Chat bridge (#43042)
* fix(vertex_ai): translate /v1/responses batch rows through the Responses-to-Chat bridge Vertex batch uploads treated every non-embeddings JSONL row as a chat completions body, so a /v1/responses row lost its input and reached GCS as a blank text part. Route detection now recognizes /v1/responses rows and bridges them to chat through the same Responses-to-Chat bridge the real-time path uses. That bridge call moves out of the Bedrock files transformation into a shared helper both providers call, forwarding the record's fields as sent, like real time, instead of validating them against the SDK TypedDicts whose required keys clients omit. * chore(batches): type the Vertex responses test helper and drop the quoted input cast * fix(batches): translate developer messages to system on Vertex and Bedrock batch rows like real time --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
f4f1a75a9c
|
test(vertex_ai): run the files peak-memory guards without coverage tracing (#42914)
The new CircleCI tests pipeline (#42773) runs tests/unit under pytest-cov on CPython 3.12.2, where coverage traces every line through sys.settrace. The two tracemalloc peak comparisons in test_vertex_ai_files_streaming.py drive 8000-row payloads through both pipelines and slow from ~10s to over 3 minutes under that tracer, so both hit the 90s pytest-timeout on every run. Mark them no_cover so pytest-cov pauses tracing for just these two. Their assertions are unchanged and every other test in the file still reports coverage. |
||
|
yuneng
|
8554d1f827 |
test: prepare reference input before tracing path-sourced peak
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng
|
e3911c71f7 |
test: make path-sourced streaming peak test differential
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng
|
e1556ce32b |
test: migrate legacy provider tests to tests/unit (wave 2, phase 13)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |