mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
307 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
f95e466bb7
|
test(integration): fix three more CircleCI integration reds from test setup (#46510)
OpenRouter basic translation expected the pre-#44788 response shape. Read refusal, logprobs and system_fingerprint from the fixture at their new positions. The team model TPM test required exactly one of three concurrent requests to be served. The pipelined limiter can reject all of them, which is an accepted trade-off, so assert no over-admission and that a lone retry is served when the burst got nothing. The Voyage host tests count every host their proxy dials. The responses cost poller could dial api.openai.com for background-response rows that other tests left in the shared worker database, so turn polling off in those proxies. |
||
|
devin-ai-integration[bot]
|
47e4ed88c2
|
fix(anthropic): keep refusal stop_reason and stop_details on the logged response (#46104)
* fix(anthropic): keep refusal stop_reason and stop_details on the logged response * fix(anthropic): keep the refusal stop-fields boundary tolerant of a non-dict upstream body * test(anthropic): use Anthropic's documented refusal example as the unit-test fixture * fix(streaming): keep provider fields on async stream cache hits and received_model_id on synthesized finish chunks * fix(caching): replay cached completion_tokens_details on stream cache hits and skip non-object cached provider fields * test(integration): cover Anthropic-family refusal stop_details on the wire, the cache and under chaos * fix(caching): skip a cached completion_tokens_details that fails validation on stream replay * fix(anthropic): keep refusal stop fields on a JSON-mode non-stream response Also adds the round 3 audit cells for JSON-mode refusals, the proxy response serialization, and wrong-typed cached usage details on both stream replays * test(anthropic): expect content_filter on streamed JSON tool refusals after main's stream change --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
0915691d32
|
test(integration): remove six setup causes of CircleCI integration reds (#46490)
Drain the worker's fake-upstream request log when a test's gateway fixture starts, so traffic left by earlier tests on the same xdist worker no longer shows up in a test's first observation read Generate the fine-tuning rig's provider file and job ids per fixture build, so a rebuilt module fixture in the same worker no longer collides on the managed object unique constraint Wait for every wedged team_id and team_alias invalidation publish to fail before taking the recovery marker, now that team updates publish both keys Pin the cooled-down priority scheduler request to the connection that observed the cooldown, since a sibling worker reads cooldowns from Redis on an interval Simulate upstream outages and the held budget peer on ports the test keeps owning, instead of releasing a port another worker can bind Start a peer proxy for the accounting group, which has a test that needs one |
||
|
Mateo Wang
|
457b5d26c7
|
fix(anthropic): stream json_schema answers as content on Bedrock Mantle and Claude Platform, and join split answers (#46331)
* fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude
Vertex AI Claude deployments that emulate response_format through a forced
tool sometimes nest the schema object under an extra key such as
`parameters` or `input`, and that envelope reached message.content with a
200. The converter now unwraps exactly one nested object that matches the
schema, leaving a payload that is itself schema-shaped alone.
Native structured-output rows sent the deprecated top-level `output_format`,
which Vertex rejects with a 400. The Vertex request now carries it as
`output_config.format`, with an explicit caller value winning.
* fix(anthropic): unwrap tool-emulated json_schema envelopes on streams and tool-name wrappers
* test(anthropic): expect the json tool description through a shared constant
* fix(anthropic): drop the json tool description that makes sonnet-5 omit required keys
* fix(anthropic): keep schema-accepted answers whole and decide held streams at the second key
An answer the json_schema accepts is returned as is before any envelope rule runs, the tool-name
wrapper is unwrapped only when the schema rejects it, a malformed schema turns the unwrap off, a
streamed answer whose first key is a schema property named name or tool_name waits for its second
key before streaming, the held fragments go through a StringIO, and a prefix that stays undecided
past 4 KB is held to the end
* test(anthropic): keep the stream family cells to providers whose stream iterator gets the schema
* fix(vertex_ai): unwrap claude json_schema answers nested under up to three wrapper keys
A tool-emulated json_schema answer was unwrapped only when the real object sat one key down. In live QA, 1 of 58 answers (/v1/responses, claude-sonnet-5) came back as {"parameter": {"value": {...}}} and reached the client still wrapped
The unwrap now follows single-key wrappers up to three levels down and stops at the first object that has a key the schema declares. That object is returned only when it has every required key. A payload wrapped deeper than three levels, a level with more than one key, and a level whose key is one of the schema's own property names all come back exactly as they arrived
The streamed path needs no change because the buffered answer goes through the same unwrap when the tool block closes, and a test now pins that both paths agree
* fix(bedrock): unwrap streamed claude json_schema answers on claude platform and mantle
A streamed json_schema answer on bedrock/claude_platform and Bedrock Mantle Claude reached the client as tool_calls deltas holding the json_tool_call wrapper, with finish_reason tool_calls and no content. The shared HTTP handler built the async stream iterator without json_mode, and neither stream path passed the response_format schema
The handler now asks the provider config for the iterator through get_model_response_iterator_for_request, which carries json_mode and the request's optional params on both the sync and async paths. Claude Platform and Mantle override it to hand the json tool schema to the Anthropic stream iterator, so the streamed answer is the bare schema object with finish_reason stop, the same as the non-streamed one
Databricks is the only other provider on this handler whose iterator reads json_mode, so its async Claude json mode stream now returns the answer as content like its sync stream already did
The stream family integration test covers claude_platform_unmapped and bedrock_mantle again
* fix(anthropic): unwrap json tool envelopes by structure instead of schema validation
The unwrap now reads only the schema's top-level property names. A lone
undeclared key holding an object, alone or beside a name equal to the json
tool's own name, is the envelope, and the unwrap repeats while the inner
object shares no key with the schema. jsonschema and referencing are no
longer imported on the request path. The stream holds only an answer that
opens with the tool's name or with an undeclared key holding an object, and a
schema without top-level properties streams as it arrives.
* fix(llms): keep other providers' async streams as they were
The request-aware iterator hook passed json_mode on async streams to every provider on the shared HTTP handler, which changed Databricks and any other provider whose iterator reads it. The default hook now makes the same two calls the handler made before, so only Claude Platform and Bedrock Mantle change, and the Databricks test that pinned the wider change is removed
* fix(anthropic): walk json tool envelopes iteratively
* test(llms): pin which stream iterators get json_mode from the shared handler
A provider config that does not override the request-aware iterator hook still gets json_mode on a sync stream and still does not get it on an async stream, the same as before the hook existed
* fix(anthropic): keep json tool arguments too deep to decode instead of raising
* fix(anthropic): unwrap json tool envelopes beside undeclared string labels
* fix(anthropic): keep json answers whole on schemas that allow extra keys and stream held text as sent
A schema that allows undeclared keys and requires none now keeps an undeclared wrapper key on both the stream and non-stream paths. When no envelope is found, the held stream text is now sent exactly as the model wrote it. On Vertex /v1/messages, output_format now wins over output_config.format when both are set, which matches the Anthropic adapter.
* fix(anthropic): join a json_schema answer the model splits into one tool block per field
With tool emulation the model sometimes answers with one json_tool_call block per schema field, each naming the field and holding its value. The non-stream path returned only the first block and the stream sent every block's text back to back, so the client got one field or invalid JSON. Both paths now build one schema object from those blocks, and any other set of blocks keeps today's output
* refactor(anthropic): set the streamed finish reason in one expression
* test(anthropic): cover label envelopes, dynamic-key schemas, as-written held streams and output_format precedence
* fix(anthropic): unwrap a streamed values wrapper like the non-stream answer
* fix(anthropic): join streamed split answers on schemas without required keys, keep client tool calls and length on json_schema streams
* fix(anthropic): join streamed split answers that carry text labels on schemas without required keys
* test(bedrock): script the wrapped json_schema stream with the wrapper Bedrock Mantle sends
* test(bedrock): script the wrapped json_schema stream with the wrapper Bedrock Mantle sent raw
* fix(anthropic): join a split json_schema answer whose fields all hold the same string
* fix(anthropic): join a split json_schema answer sent as one partial object per field
* fix(anthropic): join field blocks after an empty streamed answer without a leading comma
|
||
|
Mateo Wang
|
389403fede
|
fix(proxy): check previous_response_id ownership on /v1/responses/compact (#46408)
* fix(proxy): check previous_response_id ownership on /v1/responses/compact * test(proxy): cover the unmanaged response id escape hatches on every response id route * test(proxy): audit compact previous_response_id ownership Add integration cells for the native /v1/responses/compact routes covering owner, admin, cross-tenant, unmanaged, malformed, duplicated and retained ids, provider errors, the response cache, id-security settings, an upstream outage, a held upstream, a worker kill and a proxy restart. Unskip the provider wire cell whose bug this branch fixes. * test(proxy): cover the no signing key path on every response id route |
||
|
Mateo Wang
|
6b7c77b7d3
|
fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355)
* fix: lower aiohttp keepalive timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http): restore the 15s keep-alive default and apply it to the Rust client 15s is aiohttp's own default and what every release before v1.78.0 used. The Rust HTTP client reads the same AIOHTTP_KEEPALIVE_TIMEOUT variable, so its default moves with it. * test(integration): cover upstream connections dropped after the keep-alive idle window --------- Co-authored-by: nate <nate@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Rick
|
4f3cb48dc1
|
feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos (#43706)
* feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos
DashScopeVideoConfig maps litellm's OpenAI Videos API onto Alibaba Model
Studio's async task API: create hits
/api/v1/services/aigc/video-generation/video-synthesis and status and content
poll /api/v1/tasks/{task_id}. It is registered for dashscope and for the
qwencloud and qwen_ai_platform brand aliases, following the same
get_dashscope_family_*_config pattern image generation uses. Unlike the
Sora-shaped providers the request body is nested, so prompt and media go under
input and everything else goes under parameters. X-DashScope-Async is always
sent, because DashScope rejects the call outright without it
OpenAI params translate to DashScope's contract. seconds becomes duration, and
size yields both a ratio and a named resolution tier (1280x720 gives 16:9 plus
720P), since sending only the ratio leaves the request on the default 1080P and
silently bills at that rate. input_reference becomes a single media entry,
base64 encoded into a data URI for file inputs. Which media type a model takes
(first_frame or, on the reference-to-video models, reference_image) is read
from provider_specific_entry.dashscope_video_reference_input via
get_model_info rather than hardcoded; models without it predate the media
array and get the flat img_url field. A caller-supplied media array always
wins, as that is the only way to express multi-role combinations
The create call is the only billed call, so it bills what DashScope will
deliver: the requested duration and tier, or the documented 5 second and 1080P
defaults when they are omitted. Smart duration (duration -1) lets the service
pick the length after that call, so it is rejected rather than recorded at
zero, with a 400. Task creation and task lookup both report failures as a 200 carrying
top-level code and message with no output, so that shape is raised instead of
surfacing as a queued video with an empty id. Status responses re-encode the
deployment model from the polled id, so follow-up polls and downloads route to
the same deployment. Timestamps parse as UTC+8. DashScope publishes no list,
delete or remix endpoint, so those raise rather than guess at a URL
The end-to-end tests drive the real avideo_generation() entrypoint through an
httpx MockTransport and assert on the JSON body httpx actually encoded. That is
what catches a mapper that drops input_reference before the request is built,
or a body fragment json cannot serialize, neither of which a test calling the
transform directly would see
Both model_prices maps gain wan3.0-video, wan3.0-video-prime, wan2.7-t2v,
wan2.7-i2v, wan2.7-r2v and the HappyHorse 1.0 and 1.1 t2v, i2v and r2v models,
for dashscope and the qwencloud and qwen_ai_platform aliases, priced per output
second by resolution tier from Model Studio's USD rate cards. dashscope and
qwen_ai_platform default to the Beijing host and use its rate card; qwencloud
defaults to the international host and uses Singapore's. The three
*_API_BASE_VIDEO overrides are documented in BerriAI/litellm-docs#1920
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(dashscope): fail loudly on video responses that cannot be read
Addresses the review on the video generation PR. A body that is not the
documented task shape, an output block missing task_id or task_status, an
undocumented task_status and a missing output all raise DashScopeVideoError
now, instead of defaulting into a queued video with an empty id that callers
poll forever. A JSONDecodeError from a gateway response no longer escapes as
a generic connection error.
task_id and task_status are required on _TaskOutput, and a missing output
raises even when the body carries no code or message.
Download failures move into the 4xx range, since a 5xx has the proxy retry
them: 409 while the task is still running, 400 for a task that failed, 404
for an id that expired. The status a provider reported is kept when it
reported one, so an in-body failure that arrives on a 200 is billed as the
400 it is rather than as a retryable connection error.
characters and image are only dropped when drop_params is set; the params
that are rebuilt into the body or consumed by the caller (model, prompt,
user, extra_*) stay silently skipped, since the proxy stamps user from the
caller's end-user header.
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(dashscope): clear the lint gates after merging main
* fix(dashscope): accept file tuples and paths as video input_reference
video_generation() types input_reference as FileTypes, which includes
(filename, content[, content_type[, headers]]) tuples and pathlib paths,
but the DashScope encoder only read bytes and file objects, so those
image-to-video calls raised before reaching DashScope. Bare strings stay
URLs and are never opened as paths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dashscope): snap size to a supported ratio and read file tuples and paths
* test(dashscope): assert whole mapped values and bill against hand-written rates
The tier pricing test read its expected rate from the same map the cost
calculator uses, so a wrong rate passed. Billing is now checked against
fixed rates driven through the status response usage, while the map test
only asserts every supported tier has its own rate. Size mapping tests
compare the full mapped dict, and the single-test wiring class is a
plain function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(videos): download content through the caller's client and stop retrying dead tasks
Two Cursor Bugbot findings on the DashScope video PR.
The DashScope artifact lives on OSS, so transform_video_content_response makes a
second request. It built its own cached client, discarding an injected client and
its ssl_verify. Thread the client through the content response the way
transform_video_status_retrieve_response already does, and use it in the three
providers that fetch from a second URL (DashScope, RunwayML, fal.ai). The
remaining providers accept it and read the response body, matching how they
already accept an unused client on status retrieve.
A FAILED or CANCELED task reported without a code or message fell past the
in-flight check into the 502 branch, so the proxy retried a task that can never
produce a video. Raise 400 for a terminal status instead.
* fix(dashscope): answer 404, not 409, when a Wan video is downloaded before it finishes
A download that lands while the task is still PENDING or RUNNING answered 409. That
reads as a client error, which was the intent, but litellm's retry predicate counts 409
as retryable, so the proxy polled DashScope twice more before giving up and then counted
the caller's own early download against the model's error rate
404 is what the OpenAI videos API documents for content that is not ready, and it is the
one status in that family the retry predicate leaves alone. The rest of the mapping is
unchanged: 400 for a task the provider reported as failed, 404 for an expired task id,
502 when a succeeded task carries no artifact
The download tests now drive avideo_content and assert the caller sees NotFoundError
rather than an internal server error
* fix(dashscope): route a video poll back to the model group that created the task
The create response encoded the upstream model name into the returned id. Both the
dashscope and qwen_ai_platform prefixes serve the same model names from their own hosts,
so a proxy with a deployment per region had deployments that only the model group told
apart. Encoding the bare model name resolved a poll to whichever deployment the router
lists first, which sent it to the other region's host with that region's key and a task
id that region never issued. That is the reroute the earlier proof records as the
dashscope and qwen_ai_platform poll ids coming back tagged qwencloud
The id now carries the model group the create was routed to, which is what the proxy
resolves a poll back to, and falls back to the model name for a direct SDK call that has
no router
The regression test drives a real router with one deployment per region, under a
different prefix and under the same one, and asserts the poll carries the key of the
region that created the task
* fix(dashscope): read the routed model group from litellm_metadata
The router stamps the model group it picked into litellm_metadata. metadata only
carries a copy when the caller sent no metadata of its own, and every proxy request
sends some, so a proxied create still encoded the bare model name and the poll
resolved to whichever region's deployment the router lists first
The regression test now also creates the video with caller metadata, which fails
without this change
* fix(dashscope): reject unbillable video requests and keep fal and runway on their own clients
* fix(dashscope): show the caller why a video task is gone or not ready instead of the bare upstream body
* fix(dashscope): check video request values before the billed create and honor per-call drop_params
Type check documented DashScope parameters so a bad seed answers 400 instead of a billed failed task,
pass the call's and deployment's drop_params into the video mapping, answer 400 for an expired task id so
the deployment is not cooled down, read DASHSCOPE_API_BASE for video, keep polling SUSPENDED tasks, and
refuse to map size onto models litellm has no price for.
* fix(dashscope): carry the requested model name in video ids so alias and wildcard keys can poll
A key scoped to an alias or to a name a wildcard serves could create a
video and then got a 403 on the poll and the download, because the id
carried the group that served the call (wan3, qwencloud/*) and the key
was never given that name. The id now carries the name the router was
asked for on that hop, and the router resolves an alias, routing group
or wildcard name in an id back to itself when no deployment name or
provider model matches it first
* fix(dashscope): keep a routing group's video polls on the member that created the task
A routing group spans several model groups, so an id carrying the group's name
load-balanced its polls, and some left for another region's host and key with a
task id that region never issued. The id now carries the member that served the
task when the request named a routing group, and keeps the requested name for a
model group, an alias, or a wildcard match.
* fix(dashscope): carry the requested and served model in video ids so group-scoped keys poll in the creating region
* fix(proxy): pin a video's served group only after key and team aliases resolve
* test(integration): audit DashScope video create, poll, download, routing and chaos paths
* fix(proxy): keep catch-all video polls on their provider and count served polls against the requested group
* test(routing): expect a forged served group to be dropped from the polled video id
* test(integration): cover rate-limited routing group polls and catch-all polls of DashScope videos
---------
Co-authored-by: AaronHowell <237895480@qq.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
ryan-crabbe-berri
|
12cce6c372
|
fix(anthropic): check Vertex forced-tool support on the real model, skip non-string user ids, drop speed where unsupported (#46337)
* fix(anthropic): check Vertex and Bedrock forced-tool support on the real model The Vertex and Bedrock invoke Claude transforms swapped the model for a claude-3-sonnet stand-in before calling the shared Anthropic mapping so response_format would take the tool-based path. Every model-keyed check in that mapping then ran against the stand-in, so the synthetic json tool was forced on models whose map entry says forced tool use is unsupported and the request 400ed upstream. Bedrock patched the one symptom with a backstop, Vertex had nothing. The shared mapping now asks a uses_native_structured_output hook for the structured-output decision and the two overrides answer it with their provider-scoped lookup, so the real model reaches every check and the stand-in, the restore dance and the Bedrock backstop go away. * fix(anthropic): skip non-string metadata user ids instead of raising _valid_user_id ran re.match straight on litellm_params metadata.user_id, so an integer id raised TypeError and the proxy answered 500. The user param path already skips non-strings, so the validator now does the same for any caller: a non-string id is simply not forwarded. * fix(utils): drop speed on every provider without fast mode speed rides into get_optional_params as a provider-specific kwarg, so the only gate for it lived in the Anthropic transform and Bedrock converse, OpenAI, Azure and the extra_body legs forwarded it to a 400. The gate now runs once in get_optional_params before provider-specific kwargs are copied, so every provider drops it with drop_params or rejects it with a clean 400, and a direct Anthropic fast-mode model keeps it. * fix(utils): gate speed only under drop_params and keep it for litellm_proxy Raising on speed when drop_params is off would turn requests that tolerant OpenAI-compatible backends accept into 400s, and dropping it for litellm_proxy would disable fast mode on a downstream proxy that supports it * test(anthropic): derive the forced-tool regression model from the cost map and drop new docstrings * test(utils): pin that speed is untouched without drop_params and annotate new test locals * test(integration): lock the Vertex and Bedrock invoke json tool to the real model's forced-tool support Vertex sends tool_choice only when the model accepts forced tools, reasoning_effort maps the same with and without response_format, and Bedrock invoke keeps sending the json tool unforced |
||
|
Mateo Wang
|
d81a5719ad
|
fix(cache): keep cache hits free of unset fields, and follow intentional behavior changes in CircleCI suites (#46359)
* test(integration): follow count_tokens error surfacing and decisions pricing in CircleCI suites Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): restore Responses input_file PDF pricing coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep integration AGENTS guide unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cache): preserve provider field presence on cache hits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect credential display name in response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): align OpenInference refusal field expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect Akto request ingestion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): expect missing tool error wording Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect no refusal field in cursor chat wire Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a93fee9f6c
|
feat(decisions): add Laya and Bespoke providers to the Add Model form (#45260)
* feat(ui): add decisions and classifier providers to the Add Model form Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): use literal upstream decisions paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: drop duplicate TYPESAFE provider fields entry superseded by main's TypeSafe Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): drop stray blank line in provider_info_helpers test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): type the OSS decisions provider fields test * fix(decisions): keep Laya and Bespoke env keys on their configured server Route laya and bespoke Decisions deployments through the same oss_connection and validate_oss_model rules the pass-through and the classifier router use, so LAYA_API_KEY and BESPOKE_API_KEY only go to LAYA_API_BASE and BESPOKE_API_BASE, an api_base with credentials, a query, or a non-HTTP scheme answers 400, and an unknown checkpoint name answers 400 before any upstream call * fix(decisions): send the Laya and Bespoke env key to a form-added model on the env server A deployment added through Add Model always carries an API base, so the env key only reached deployments with no base. The env key now goes to a deployment whose base matches LAYA_API_BASE or BESPOKE_API_BASE, an empty deployment key counts as no key, and the Bespoke placeholder is the name Ollama serves. * test(decisions): cover the env-base classifier case in the complexity router test * fix(decisions): match the env server with or without /v1 and point Ollama users at bespoke/nimble * fix(decisions): call /v1/systemone once from classifiers and pass-through on a /v1 base * fix(decisions): strip /v1 from the API base path only, so a host named v1 still works --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
2f3ccbe7c7
|
feat(decisions)!: send images to Cloudflare Clef through its images field (#45734)
* feat(decisions): send images to Cloudflare Clef OpenAI-format input_image parts and the System One images field now reach Cloudflare clef and clef-flash in its images field, with remote URLs downloaded and inlined first. Providers that cannot take images drop them under drop_params and answer 400 otherwise. * fix(decisions): follow drop_params for images and use the call timeout for image downloads * fix(decisions): drop a malformed images entry on its own under drop_params * fix(decisions): refuse images next to input whatever drop_params says * fix(decisions): bound clef image fetches, honor drop_params for images, and log caller URLs Refuse more than 4 images for Cloudflare clef before any download, keep the caller's timeout as one deadline across image fetch retries, accept images next to input under drop_params, validate image strings and objects, and pass the caller's image URLs (not inlined bytes) to logging callbacks. * fix(decisions): check input_image URLs and clef image sizes * fix(decisions): accept data URL media type parameters and check clef image types * fix(decisions): count clef images after dropping unsupported types and flag clef vision --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
1aee1a21ed
|
fix(databricks): send a single plain text block as a string for non-Claude models (#45686)
* fix(databricks): send a single plain text block as a string for non-Claude models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(databricks): simplify single-text-block helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(databricks): clarify json_object validation wording Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): integration cells for single text block content shape Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): rename list-content cells to stays_list Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): expect plain string content for bridged Databricks messages Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): cover the customer's Unity Catalog model from #45617 and #45618 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(databricks): drop casts from _collapse_single_text_block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): assert the whole outgoing messages list in the collapse unit test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): cover system joins, fallback, SDK clients, and outage burst for one-block content --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
b37eac29f0
|
fix(anthropic): cache tool result screenshots on the /v1/messages to Responses bridge (#46324)
* fix(anthropic): put a tool_result cache breakpoint on its hoisted images on the Responses bridge A marked tool_result whose images get moved into a trailing user message now carries the breakpoint on the last moved image instead of the function_call_output, so the prefix OpenAI caches includes the images * test(anthropic): cover a marked tool_result with two screenshots on the Responses bridge * fix(anthropic): keep every tool result screenshot inside the breakpoint that covers it on the Responses bridge A marked tool_result's screenshots now go right after its output, so earlier unmarked screenshots land inside the breakpoint and later tool outputs stay after it. A cache_control mark on a block inside a tool_result is read too. * fix(anthropic): cover only the screenshots up to a mark inside a tool result, and drop raw breakpoints under drop_params A cache mark on a text block inside a tool result now covers that result only up to the marked block, the way Anthropic caches it, so screenshots after the marked text stay outside the breakpoint. A mark on the tool result itself still covers every screenshot in it A deployment that reads no prompt cache breakpoints now drops raw prompt_cache_breakpoint marks, top-level and inside tool results, when drop_params is on, and still forwards them as sent without it * fix(anthropic): keep raw breakpoints under drop_params for models that take them * docs(anthropic): say a mark inside a tool result covers its whole text and documents --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
cfa98e2f40
|
test(integration): cover pass_through endpoint contracts (#44690)
* test(integration): cover pass_through endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): run configured endpoint contracts on one proxy worker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove pass-through streams, SDK spellings, endpoint renames and bedrock stream bodies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover pass-through wire behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover Vertex and Gemini passthrough authentication Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Prove pass-through runtime method, credential-less Anthropic, stream barriers and Vertex strip layers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(pass_through): gate unquoted Bedrock stream targets and assert configured endpoint part types and methods after rename Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert pass-through query pairs so duplicate params fail Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
1047a450bf
|
test(integration): pin assistants and evals endpoint contracts (#44734)
* test(integration): cover assistants_evals endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): replace exception assertions flagged by ruff Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(openai): type assistant run options Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): cover assistant and eval contract paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(openai): keep assistant run options unvalidated Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin assistants and evals wire contracts as BUG skips Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover every eval route, full responses, image parts and full SDK stream payloads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): relay upstream eval run model, send reasoning_effort, cover SDK root alias, done frame and client disconnect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): name reasoning_effort in the run BUG skip Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): use a non-null marker thread and name the eval run model restamp in its BUG skip Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): mark disconnect cases with a messages.create the proxy relays on main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(assistants_evals): cover thread message shapes, raw run options, header leaks and /v1 eval bases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect the OpenAI /cancel path for eval run cancel Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
5261a2cc06
|
test(integration): cover responses endpoint contracts (#44733)
* test(integration): cover responses endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten responses contract tests per review Isolate team-only response-id ownership with a same-user different-team key, send every denial through the OpenAI SDK on all three aliases, make the previous_response_id affinity check deterministic, assert the whole cancel response, prove the Cursor stream text, -fast suffixes and include_obfuscation strip, add SDK compact cases, restore router pre-call checks after the affinity test, and inline the root-level responses_test_support helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for every proxy worker before routing to new deployments Restore the cross-worker convergence wait dropped when the shared helper was inlined, so the affinity and retrieve-routing tests cannot pass before the alternative deployment is loaded on a multi-worker gateway Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover Responses ownership and wire contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): address responses contract review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect the provider model in the conversation passthrough body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expand responses endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): restore streamed response retrieval Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for proxy workers after deployment setup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): track readiness per proxy worker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
60ee966885
|
test(integration): cover anthropic_messages endpoint contracts (#44704)
* test(integration): cover anthropic_messages endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close anthropic_messages review gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove azure beta filtering and the safeguards allowlist Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): compare the forwarded anthropic-beta token-for-token and pin the truncated stream bug Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: strengthen endpoint wire contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: follow count_tokens upstream error relay from main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: follow device-code refresh token response from main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0ca9f8185c
|
test(integration): cover files_batches endpoint contracts (#44703)
* test(integration): cover files_batches endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): split raw cloud-storage delete test by caller Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop mutable polling state in content download test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): dispatch request spellings without rebinding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): import assert_never from typing_extensions for python 3.10 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten files_batches wire and access assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tidy files_batches access tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover repeated target models, upload expiry, batch aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove files streaming path, SDK model denial and admin GCS delete Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): give the streaming head the wire barrier's full budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: strengthen files and batches wire coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: drop duplicate parametrize from the model-routed upload helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: pass the SDK base url in the skipped repeated-plain upload test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): allow batch output metadata lookup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: avoid draining file content requests before wait Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c9a1d9b026
|
test(integration): cover small_inference endpoint contracts (#44654)
* test(integration): cover small_inference endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert full search tool listing for scoped keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep moderation and search tool peers stateless Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover filename-agnostic OCR uploads, list queries, object rerank documents and search tool authorization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): build search tool and moderation proxy configs without mutating the loaded yaml Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover the /ocr alias, model search spelling and Cohere v2 max_tokens_per_doc, clean up non-admin search tool rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover octet-stream OCR uploads and BUG-skip the Python requests boolean spelling * test(integration): pin Exa passthrough of native fields and caller contents Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
32f0d680d2
|
test(integration): cover google_genai endpoint contracts (#44680)
* test(integration): cover google_genai endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten google_genai contract assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover non-admin agent api_base guard and exactly-once spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expand Google GenAI route contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover adapter responseJsonSchema, tool-loop history, /models alias and anthropic stream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin raw SSE id/retry framing and agent list/versions key guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): strengthen Gemini endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
903f04dd46
|
test(integration): cover vector_stores endpoint contracts (#44739)
* test(integration): cover vector_stores endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): strengthen vector store route contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): update vector store route contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop owner-update BUG skip for admin-only vector store routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vector_stores): cover the managed-file fallback, owner delete and per-request retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): probe vector store worker readiness by deployment id Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin vector store retry header and create routings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(integration): ruff format vector store managed file tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
17c860293e
|
test(integration): cover realtime endpoint contracts (#44689)
* test(integration): cover realtime endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): reject duplicate multipart parts and cover the transcription_sessions alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop the type ignore and check the aliased transcription response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin realtime multipart part types, replayed transcription session, undeclared session fields and budget refusal path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): move the refusal path note next to the expected message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert realtime refusal status before parsing and pin the replayed transcription session whole Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): sort the realtime webrtc test imports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pick the transcription mint and token with one Final each * test(integration): pin realtime transcription mint, replay, and redeem contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(integration): ruff format the realtime webrtc test * test(integration): drop the redundant comment on the realtime SDK calls test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): type the realtime client secret refusals table Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(realtime): tighten endpoint contract assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
d3cc77def4
|
test(integration): cover chat_completions endpoint contracts (#44688)
* test(integration): cover chat_completions endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin per-call prompts and both n=2 chat choices Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover mixed path-model auth, string guardrail rejection and typed text-completion forms Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop the prompt-spellings cache workaround and cover Azure SDK text completions on the deployments alias --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
35124f3b49
|
test(integration): cover images endpoint contracts (#44653)
* test(integration): cover images endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): add SDK-driven image edit contract Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover image route aliases, repeated mask[] and variations route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover AzureOpenAI images.edit alias and extra_body on gpt-image generation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
1c0f722aae
|
test(integration): cover embeddings endpoint contracts (#44660)
* test(integration): cover embeddings endpoint contracts * test(integration): cover azure, vllm, nebius token arrays and flag bedrock token-array bugs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin path-over-body model routing for azure embeddings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin embeddings base64 relay, flat token arrays and api-key-only auth Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> |
||
|
devin-ai-integration[bot]
|
c581c85559
|
fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude (#45871)
* fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude Vertex AI Claude deployments that emulate response_format through a forced tool sometimes nest the schema object under an extra key such as `parameters` or `input`, and that envelope reached message.content with a 200. The converter now unwraps exactly one nested object that matches the schema, leaving a payload that is itself schema-shaped alone. Native structured-output rows sent the deprecated top-level `output_format`, which Vertex rejects with a 400. The Vertex request now carries it as `output_config.format`, with an explicit caller value winning. * fix(anthropic): unwrap tool-emulated json_schema envelopes on streams and tool-name wrappers * test(anthropic): expect the json tool description through a shared constant * fix(anthropic): drop the json tool description that makes sonnet-5 omit required keys * fix(anthropic): keep schema-accepted answers whole and decide held streams at the second key An answer the json_schema accepts is returned as is before any envelope rule runs, the tool-name wrapper is unwrapped only when the schema rejects it, a malformed schema turns the unwrap off, a streamed answer whose first key is a schema property named name or tool_name waits for its second key before streaming, the held fragments go through a StringIO, and a prefix that stays undecided past 4 KB is held to the end * test(anthropic): keep the stream family cells to providers whose stream iterator gets the schema * fix(anthropic): unwrap json tool envelopes by structure instead of schema validation The unwrap now reads only the schema's top-level property names. A lone undeclared key holding an object, alone or beside a name equal to the json tool's own name, is the envelope, and the unwrap repeats while the inner object shares no key with the schema. jsonschema and referencing are no longer imported on the request path. The stream holds only an answer that opens with the tool's name or with an undeclared key holding an object, and a schema without top-level properties streams as it arrives. * fix(anthropic): walk json tool envelopes iteratively * fix(anthropic): keep json tool arguments too deep to decode instead of raising * fix(anthropic): unwrap json tool envelopes beside undeclared string labels * fix(anthropic): keep json answers whole on schemas that allow extra keys and stream held text as sent A schema that allows undeclared keys and requires none now keeps an undeclared wrapper key on both the stream and non-stream paths. When no envelope is found, the held stream text is now sent exactly as the model wrote it. On Vertex /v1/messages, output_format now wins over output_config.format when both are set, which matches the Anthropic adapter. * test(anthropic): cover label envelopes, dynamic-key schemas, as-written held streams and output_format precedence --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
365c4885de
|
fix(openai): carry cache_control to prompt_cache_breakpoint on chat completions to gpt-5.6 (#46101)
* fix(openai): carry cache_control to prompt_cache_breakpoint on chat completions to gpt-5.6 A cache_control mark on a system, developer, user or tool message sent to /v1/chat/completions now reaches an OpenAI deployment that reads prompt_cache_breakpoint (gpt-5.6+ on api.openai.com, or one opted in through prompt_cache_options) as a prompt_cache_breakpoint on the same content part, instead of being stripped. Marks on assistant turns and tool definitions are still dropped, and deployments that do not read breakpoints keep today's behavior. * test(openai): rewrite chat prompt_cache_breakpoint wire test as sent-vs-received literals Each test now shows the request the caller sends and the exact body OpenAI receives as plain JSON, with one helper that returns the upstream body. Drops the per-test marker plumbing and the spend-log check, which did not test the cache mark translation. * test(openai): cover a marked chat message with null content on the breakpoint path A system, developer, user or tool message that carries cache_control but whose content is neither a string nor a list of parts is handed to the existing strip untouched instead of raising inside the transform. The new case pins that the request still goes out with the mark stripped and no breakpoint added. * refactor(openai): reuse the cache control hook's breakpoint placement on chat completions The chat transform had its own copy of how a message-level cache_control lands on a part for OpenAI's prompt_cache_breakpoint. It now calls one hook helper that the injection path also uses, so which roles and part types can carry a breakpoint is decided in one place. A message-level mark on list content now lands on the last part OpenAI accepts a breakpoint on, matching injection. * test(e2e): cover cache_control reaching gpt-5.6 explicit prompt caching on chat completions Four live cases through the proxy: a marked system part, a marked string system message, a marked user part and a marked tool result. Each must write the cache on the first call and read it on the second. * fix(openai): parse content parts instead of a banned TypeGuard when renaming cache marks The strict ruff gate bans typing.TypeGuard as unverified narrowing. The part rename now validates an accepted part with pydantic before handing it to the shared rename helper. |
||
|
ryan-crabbe-berri
|
7bd108139f
|
fix(openai): carry cache_control to prompt_cache_breakpoint on /v1/responses to gpt-5.6 (#46105)
* fix(openai): carry cache_control to prompt_cache_breakpoint on /v1/responses to gpt-5.6 Native /v1/responses used to strip every Anthropic cache_control mark, so a gpt-5.6 deployment in explicit mode never wrote a cache. Marks on system, developer and user parts and on function_call_output now become prompt_cache_breakpoint on the same block when the deployment reads breakpoints, and are stripped exactly as before everywhere else * fix(openai): read str providers and base_url in the prompt cache breakpoint gate Responses configs generated from a provider JSON file, and openai_like itself, name their provider as a plain str, so .value crashed every native Responses request through them. The gate also only read api_base, while responses() keeps a caller's base_url as an extra field on the params, so an injected mark on a custom host was rewritten to a breakpoint * test(openai): cover the pass-through branches of the prompt cache breakpoint swap A part type OpenAI does not cache keeps no breakpoint and an item mark skips it, and an item or content of a shape OpenAI will reject reaches the strip untouched instead of failing inside the proxy * fix(prompt_caching): keep a client-written prompt_cache_breakpoint when a cache_control mark lands on the same block |
||
|
devin-ai-integration[bot]
|
74e5dc9747
|
fix(proxy): keep provider refusal, logprobs and system_fingerprint in chat completions (#44788)
* fix(proxy): keep provider refusal, logprobs and system_fingerprint in chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): handle missing chat completion roles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): restore throwaway Usage skip in ModelResponse Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): handle explicit fields on custom response dumps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: update refusal expectations for LIT-9235 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(lint): suppress type discipline violations with reasons Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: ruff format proxy utils Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): keep basedpyright counts at merge-base for azure dump and ModelResponse Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): keep an explicit null system_fingerprint passed to ModelResponse Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Krrish Dholakia
|
719c44db3f
|
fix(proxy): raise on provider count_tokens errors instead of falling back to tiktoken (#46121)
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
yuneng-jiang
|
ec58b889cc
|
test(integration): isolate shared proxy state behind six order-dependent flakes (#46172)
Each test below passed or failed on the same commit depending on test order or port timing in the CircleCI integration workflow - readiness probe only polls a port the launched proxy owns, so a stolen port no longer sends GET /health/integration-readiness to another test's wire server - team update tests seed fresh member users, since a proxy admin relocation adds team members to the destination org and leaked org_b membership into the shared INTERNAL_USER - realtime whisper tests wait until every proxy worker serves only the new gpt-realtime-whisper deployment, via a per-worker deployment id route - configured fallbacks test runs on its own proxy and database, because /config/update persists router settings and cannot unset fallbacks, which left [] and changed the no-fallback error text other tests assert - otel excluded services tests each get a private Redis so a sibling proxy cannot warm the budget registry cache the tenant span depends on - Langfuse audit upstream uses unique response ids instead of len(targets), which could repeat under concurrency and drop SpendLogs rows |
||
|
devin-ai-integration[bot]
|
b6df3d7fe5
|
fix(responses): number bridged chat tool calls 0, 1 and pin the remaining bridge shapes (#46109)
* test(responses): pin bridged commentary plus final answer as one chat choice Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): pin reasoning, two-message and two-tool-call bridge shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): number bridged chat tool calls 0, 1 instead of all 0 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4b7b8efd4c
|
fix(responses): fold bridged Responses output into one chat choice (#46099)
* fix(responses): fold bridged Responses output into one chat choice A Chat Completions request bridged to /responses got one choice per Responses message item, so a reply shaped [message, message, function_call] put the tool call on choices[1] and clients reading only choices[0] never saw it. Fold every message text, annotation, reasoning item and tool call into choices[0] Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): drop mutable annotations and the extra cast in the choice fold Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): remove a cast-ok marker that suppressed nothing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): join folded message texts with a newline Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(responses): join folded message texts with no separator so stream and non-stream match Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4fae3c64bb
|
feat(sagemaker)!: inference components and OpenAI-compatible embeddings (#44985)
* feat(sagemaker): inference components and OpenAI-compatible embeddings
Send model_id on a SageMaker embedding deployment as the invoke call's
InferenceComponentName, sync and async, and keep it out of the request body.
Add the sagemaker/openai/<endpoint> route, which sends the OpenAI embeddings
request shape ({"input": [...]} plus extra_body fields) and reads
data[].embedding and usage from the response for containers that serve the
OpenAI embeddings API. Plain sagemaker/<endpoint> deployments keep the
Hugging Face shape
* refactor(sagemaker): type validate_environment messages and cover base64 embeddings
* fix(sagemaker): name model_id in inference component errors and reject a non-string model_id on embeddings
* fix(sagemaker): fall back to the error text when SageMaker sends a null message
A ClientError whose Error.Message is null (SageMaker's InternalFailure shape) reached the
inference component hint as None and crashed with a TypeError; the message now falls back
to the error's own text on the chat and embedding paths
* test(sagemaker): audit matrix for inference-component embeddings
* fix(sagemaker): read the error status and text through one shape-checked helper
botocore's ConnectionClosedError carries response=None, so the except blocks that
read e.response.get(...) crashed with an AttributeError and the proxy answered 400
"Invalid request format" for a dropped connection. client_error_details validates
the botocore error shape with pydantic and falls back to 500 plus the error's own
text for any other shape (a missing response, an httpx Response, a plain exception).
* fix(sagemaker): drop the inference component hint from chat and derive the SDK wire expectation from the request
The chat except blocks never see SageMaker's text: HTTPHandler.post raises from
raise_for_status before the handler's own SagemakerError line, so the hint check ran
on httpx's generic "Client error '400 Bad Request' for url" line and could not match on
any chat path. The hint now names litellm.embedding only and the chat blocks keep the
status and text they had, without the dead check. The two OpenAI SDK wire tests pinned
the SDK's default encoding_format; they now expect whatever the SDK sent.
* refactor(sagemaker): meet main's per-rule lint and type ceilings
Main started gating every lint rule on its merge-base count and added LIT015
(frozen pydantic models) after this branch was cut, so the merge with main
pushed LIT001, LIT011, LIT015, BLE001 and reportArgumentType past their
ceilings. The response models are frozen with tuple fields, the OpenAI
embedding response is built with model_copy instead of writing into the
caller's object, the invoke_endpoint arguments are a TypedDict, and
optional_params and extra_body are validated into Mapping[str, object] before
they are read
* refactor(sagemaker): pick the embedding config in one place and stop pinning upstream error text
The openai/, Voyage and Cohere checks lived in both get_model_config and
get_supported_openai_params, so they now share one lookup. Tests compare error
messages against the error's own text, or against botocore's and json's own
wording, instead of copying that wording, and assert whole messages where the
text is ours
* fix(caching): key direct calls on model_id so inference components don't share cached embeddings
* test(sagemaker): name the fake-boto3 SDK test class for what it covers
* test(sagemaker): kill and stop chaos workers through the owned-proxy harness
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
ryan-crabbe-berri
|
c2d0726a08
|
fix(anthropic): carry cache_control to prompt_cache_breakpoint on the /v1/messages to Responses bridge (#46082)
A caller hitting /v1/messages with a cache_control mark and an OpenAI gpt-5.6 or newer deployment behind it lost the mark when the request was rebuilt as a Responses request, so OpenAI never cached at the boundary the caller chose. The bridge now rewrites each marked system, user and tool_result block to prompt_cache_breakpoint on the same block when the deployment reads breakpoints, keeps the system prompt as a developer message so its mark survives, and still strips the mark for deployments that do not read it The hook's gate becomes public so the bridge and the hook share one answer for which deployments read breakpoints, and the one-block swap lives next to the existing breakpoint helper in prompt_templates |
||
|
devin-ai-integration[bot]
|
ca4169d20a
|
feat(providers): add CoralBricks as a native provider (#44700)
* feat(providers): add CoralBricks as a native provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): harden cost map consistency tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): type cost helpers, read cost maps from disk, cover streaming spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): drop Final on loop-rebound names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): add the provider audit cells, the Add Model browser cell and the README row * test(coralbricks): derive UI spend from model info, type browser rows, always release held upstream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): release held upstream before the wire server joins its threads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): number forwarded chats with itertools.count instead of a mutable list Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(coralbricks): drop retired glm-5.3-flash-fp4 and test against deepseek-v4.1-flash-fast-fp4 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): pin retries, await the respawned worker, read the UI rate from the catalog The retry cells now set num_retries on the deployment, since the CI rig posts router_settings.num_retries 0 after readiness and the router default of 2 no longer applies there. The SIGKILL chaos cell waits for the respawned worker's startup before the owned proxy is stopped and derives its timeout from the harness budget. The Add Model spec waits for the catalog the form loads, reads the expected rate from it, and finds the deployment through /v1/model/info, which does not paginate * fix(coralbricks): price the current model names and the new DeepSeek input rate CoralBricks renamed glm-5.3-fp4 to glm-5.3-fast and deepseek-v4.1-flash-fast-fp4 to deepseek-v4.1-flash-fast (the fp4 names stay as deprecated aliases), and cut DeepSeek V4.1 Flash input from $0.30 to $0.01 per 1M on 2026-10-08. Rates read from https://www.coralbricks.ai/api/public/models on 2026-10-10; a live cache write billed 6014 tokens at 9e-08 each, so the cache-write rate stays input plus cacheWritePerM. The test config builders now build new dicts instead of editing the parsed YAML, and the cost-map test locals carry Final. * fix(coralbricks): keep pricing for the deprecated fp4 slugs CoralBricks still serves * fix(coralbricks): list the catalog rows for wildcard deployments and derive the test's input limit from the row * fix(coralbricks): price the deprecated glm-5.3 and deepseek-v4.1-flash slugs --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mayank-affirm
|
56a1e94a31
|
fix(bedrock_mantle): map the OpenAI-shaped context overflow envelope too (#41747)
* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(model_hub): surface model_info.description in model group info and Model Hub UI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): aggregate model group description without in-place mutation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(health): skip background health check DB writes when the latest-row read fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock_mantle): map the OpenAI-shaped context overflow envelope too Mantle reports context overflow in two envelopes. #37862 covered the structured `validation_error` one carrying token counts ("prompt tokens (N) exceed model maximum (M)"). Mantle also returns the OpenAI-shaped body: {"error":{"code":"context_length_exceeded","message":"Your input exceeds the context window of this model. ...","param":"input", "type":"invalid_request_error"}} That passes the existing `invalid_request_error` gate but matches neither the token-count pattern nor `is_error_str_context_window_exceeded`'s substring list, so it fell through to a generic BadRequestError. Clients such as Claude Code only reactive-compact on a recognised overflow phrase, so an overflowing request surfaced as an ordinary error with no recovery. Return the same normalised "prompt is too long: ..." message for this envelope when it carries no counts, keeping the numeric wording when the provider supplies them. * fix(bedrock_mantle): map the streamed context overflow and keep its 400 on /v1/messages streams * fix(anthropic_adapter): keep only the context overflow's 400 on streamed /v1/messages so other provider 4xx still fall back * test(anthropic_adapter): wrap a parametrize line over the 120 char limit * test(integration): audit Bedrock Mantle context overflow mapping across routes, clients, and fallbacks * test(integration): move the streamed /v1/messages overflow spend row into a BUG-skipped test * test(integration): keep every chaos cell in the burst and skip only the missing spend row * fix(bedrock_mantle): answer 400 on an in-stream context overflow The Mantle overflow mapper built its ContextWindowExceededError on the original exception's response, so an overflow that Mantle reports inside a 200 stream (the chat-mode token-count envelope, which the OpenAI chunk parser wraps as a 500) kept that 500 and the streamed /v1/messages route answered HTTP 500 api_error instead of 400 invalid_request_error. The mapper now builds the error on a 400 response that keeps the original headers and request, and the mapping test covers an in-stream 500 and 503. The integration suites pin the streamed /v1/messages overflow to the pre-content 400 route under a retry budget and to the message_start plus invalid_request_error event without one, drop the chaos carve-out that expected no spend row, and give the fallbacks file a timeout mark sized to the proxy's graceful stop. * fix(bedrock_mantle): keep the provider's 400 body and match the overflow code only as error.code --------- Co-authored-by: milan <milan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
a86ad9ebf8
|
feat(decisions): support Clef-omni audio and video (#45811)
* feat(decisions): support Clef-omni audio and video * fix(decisions): accept Clef-omni media types with parameters and cover the proxy routes The clip patterns now mirror Cloudflare's input schema, so a data URL or content_type carrying parameters like audio/webm;codecs=opus is accepted. The four proxy routes get media tests of their own, the backup cost map is synced with the new Clef-omni rows, and the docs page moves to the docs repo as a follow-up. * fix(decisions): refuse parameterized media-type data URLs and refresh the OpenAPI snapshot Cloudflare answers 422 to a data URL whose media type carries parameters even though its schema only requires the data:audio/ prefix, so LiteLLM now refuses those with a 400 before the call. The object form keeps passing parameters through, which Cloudflare accepts. The lazy OpenAPI snapshot and the dashboard API types are regenerated for the new audio and videos fields * fix(decisions): gate media by cost-map flags, treat null clips as none, drop unsupported clips before validation * fix(decisions): keep the SDK's positional order and gate media flags on evaluation-mode models * fix(decisions): gate media on both decisions model modes and move the audit cells onto the harness helpers * test(decisions): cover the Fireworks System One router in the media refusal cells --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sukhdeep
|
87478c7b6d
|
fix(gemini): do not inject temperature for Gemini 3+ models when omitted (#45107)
* fix(gemini): do not inject temperature for Gemini 3+ models when omitted map_openai_params added temperature=1.0 to every Gemini 3+ request that did not set one, on both the gemini and vertex_ai routes. Google is deprecating temperature, top_p and top_k for Gemini 3+ and says upcoming models will reject them, so this default turns a request that never set temperature into one that will fail An omitted temperature now falls back to Google's own default, which their docs give as 1.0 for all Gemini 3 models. Explicit temperature, top_p and top_k are still forwarded Fixes #38663 * test(gemini): annotate sampling regression tests * test(gemini): drop the injected temperature from Gemini translation cases The nine Gemini base cases for chat completions, responses and messages send no temperature, so the provider request no longer carries the removed 1.0 default. Each new body was sent to the real Gemini API for gemini-3.5-flash, gemini-3.8-flash and gemini-3.1-pro-preview and returned 200, and the stored provider responses are unchanged * test(gemini): drop the default-temperature wording from the thinking level test docstring * test(gemini): cover omitted and explicit sampling on both Gemini routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
088184c24b
|
feat(fireworks_ai): add Decisions support for accounts/fireworks/routers/rune on /v1/systemone and /v1/decisions (#46057)
* feat(fireworks_ai): add Decisions support for accounts/fireworks/routers/rune on /v1/systemone and /v1/decisions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(fireworks_ai): point the rune cost row at the public Fireworks docs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(decisions): patch litellm.callbacks with monkeypatch in the Fireworks cached-cost test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
7efce7f8e0
|
test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)
* test(integration): give chaos and timing tests a harness API and gate raw signals Chaos tests killed and suspended proxy workers with raw os.kill and psutil calls and bounded elapsed time with fixed literals, so they raced the uvicorn worker healthcheck and failed on slow CI boxes OwnedProxy now offers kill_worker and replacing_worker, which return only once a replacement answers readiness with its own pid, and suspended, which is refused unless the proxy was launched with a lenient worker healthcheck. timing.py bounds durations against a measured control or the injected delay. run.py measures the proxy import time once and derives readiness deadlines and the per-test timeout from it, writing boot-calibration.json check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time asserted under a numeric literal) for tests/integration outside _support, and every call site is migrated so both report zero with no suppressions * test(integration): steer the assistants wire burst onto both workers The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split * test(integration): restore product-enforced timing bounds lost in the migration Some TQ011 migrations dropped or loosened a bound whose time is the behaviour under test. The websocket session cap only kept its floor, so a cap closing at 300 s would have passed, and several timeouts were bounded by the injected stall instead of the configured product value assert_enforced_within bounds an elapsed time by the configured value plus explicit slack, with an optional floor. The websocket cap, the MCP validation timeout, the guardrail litellm_params timeout, the prisma command timeouts, the baseline projection lock timeout and the key metadata statement timeout use it. The device code refusal keeps its 15 s margin, and the langfuse retried prompt fetch is now bounded by a cold fetch of a steady prompt as well as by the ignored retry-after * fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat in a ping on the killed worker for up to that long before respawning it. It now uses the graceful stop budget, and a worker kill returns only after the victim is reaped, so pooled connections no longer race the dead worker's socket. Several tests on main picked the multiprocessing resource tracker as their victim and never killed a worker. Now that they kill a real one, presidio counts streams cut by the death, delete_model waits for the config reload that removes a deleted model from a replacement worker, and the wedged coordination Redis test kills before the wedge and waits for every parked publish to fail before resuming Redis. The redis cluster test imports the harness through the tests package so it collects in the unit job, and the per-model fallback rig exposes the owned proxy it now needs. * test(observability): only require S3 deliveries the killed worker could not have buffered A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
c0ca4a51f7
|
fix(bedrock_mantle): send max_completion_tokens for gpt-5.6 and gpt-6 chat completions (#43982)
GPT reasoning-series models (gpt-5.x, gpt-6) on Bedrock Mantle chat completions now send max_completion_tokens, carrying a deployment or client max_tokens over with its value. Every other Mantle chat model keeps sending max_tokens. |
||
|
devin-ai-integration[bot]
|
a7a3a6b375
|
test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940)
* test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e * test: fix disk cache collection, ungate saturation check, drop RSS memory tests * test: assert one delivered log per completion in Datadog and LangSmith checks * test: drop SDK-direct e2e copies and align cost-map path filter cases --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
3090f4b23e
|
fix(ci): isolate integration workers and repair test fixtures (#45863)
* test(integration): wait for customer spend rows and use owned OpenRouter cost * ci: isolate integration workers and clean up resources after crashes * docs: describe isolated extensions workers and shard census * fix(ci): restore integration service and fixture guarantees * fix(integration): preserve replay IDs and native route fixtures * fix(tests): scope integration harness imports to each unit case * fix(ci): preserve fixtures and wait for owned workers * fix(ci): isolate nested runs and telemetry cases * fix(ci): make worker readiness and fault tests deterministic * chore(ci): drop MCP fixture repair already covered by main * chore(ci): keep the local catalog fix from main --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
1dbde0845e
|
test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758)
* test: move guardrails, image, audio, batches, search legacy tests to unit * test: restore the custom guardrail and remaining chunk 1 unit rows * test: move the ocr matrix to e2e * test: drop comments from moved tests and fix vertex ocr credentials * test: move litellm utils and unified google legacy tests to their tiers * test: move logging callback and pass-through unit legacy tests to their tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: move store_model_in_db legacy tests and start the admin ui migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: finish the proxy job rows and drop their CircleCI jobs Moves the remaining multi-instance, openai endpoint, otel, pass-through and top-level proxy tests into tests/integration with scripted upstreams, rewrites the weak replacements to assert the outbound wire and exact responses, and removes the build_and_test job group this PR owns from CircleCI * test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names * test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist * test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in * test: type the e2e beta header, prompt cache and secret manager values * test: assert the missing model error message on the OpenAI routes * test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks * test: make the langfuse fixture directory a package for the layout check * test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane * test: keep the ocr e2e pdf fixture next to the test * test: put meta below covers on the streaming tool search e2e * test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures * test: count only OCR calls in the OCR matrix recorder * test: pin OCR parse bodies and billed cost, group new e2e cases into classes * ci: restore the python image anchor dropped in the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
ca0ff24726
|
fix(databricks): call the ai_decide AI Function route for decisions and the auto-router (#45722)
* fix(databricks): call the ai_decide AI Function route for decisions and the auto-router Databricks serves ai_decide at POST <host>/api/2.0/ai-functions/ai-decide, not on a serving endpoint's /invocations route. The decisions provider and the auto-router classifier now accept only the ai_decide model, post to that route without a model field, unwrap the response envelope, and map a noul answer's probability to the System One noul field. * test(databricks): bound the megabyte model name test by scaling, not wall time * test(databricks): assert the whole OpenAI-format decision and floor the liveliness bound * test(decisions): move the Databricks OpenAI-format wire row to the ai_decide route * fix(databricks): keep serving endpoint decisions beside the ai_decide route * test(databricks): assert the whole classifier request body in the test routing case * test(databricks): correct System One spend call type Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
5390199d63
|
test: model the team membership lookup in MCP DCR admission fixtures and assert decisions call types per route (#45917)
Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a2e334febe
|
test: migrate local_testing SDK and provider tests out of build_and_test (#45760)
* test: migrate local_testing SDK and provider tests out of build_and_test * test: cover sagemaker region, hf template and nova paths end to end Region rows now go through litellm.completion on the sagemaker/ handler that the legacy tests exercised, the prompt template test mocks the Hugging Face tokenizer config instead of switching to custom_prompt_dict, streaming tests assert role, finish reason and body, and Nova single turn and streaming run through litellm.completion with a recorded request body * test: tighten migrated integration tests for local OpenAI routes, timeouts and air-gapped boot Split the catch-all providers migration file by behavior. The fake endpoint route tests now hit the owned upstream directly, as the legacy harness tests did. LM Studio asserts the outbound key, path and stripped model. The embedding timeout test asserts the proxy maps a deployment timeout to a 408 and forwards the read timeout to the provider client, instead of timing out the test client The Azure streaming tool-call test also asserts the second identical stream is served from the Redis cache with the same reassembled tool call The Docker network test always skipped in CI because the security job has no litellm/litellm image and no egress to pull one, and it tested a published image rather than the checkout. It now boots the checkout proxy with every outbound HTTP connection refused and asserts readiness, the bundled cost map fallback and that the cost map host is the only egress attempted * test: tighten the migrated e2e rows for local_testing SDK and provider tests Drop the duplicated Anthropic covers marker and the Anthropic cell wrongly claimed by the Bedrock and Vertex cache tests. Keep the file-wide Anthropic model as it is on main. Send the 2023-06-01 version header to Anthropic through the deployment's extra_headers so the header actually reaches the provider, and mark the creation-token case 1h then 5m like the legacy test did Reuse the Bedrock file's own registration for the pydantic response test, swap the hand-built PNGs for the shared cat.jpg fixture, assert the SageMaker Nova image answer names the cat and that top_logprobs=3 is honored, and register the SageMaker Nova opt-in markers the same way as the other opt-in markers * test: restore lost assertions in migrated test_main tests Restore the role alternation check in test_completion_anthropic_hanging, the embedding and streaming response-header checks, the logit_bias drop check and the gemini system and safety settings checks. Add an azure 429 retry-after test for test_exception_with_headers, trigger the router fallback with mock_timeout like the legacy test, and stop the mock timeout tests sleeping 3s * test: restore lost legacy assertions in migrated vertex ai unit tests * test: tighten migrated cost, streaming and embedding tests and scope the hosted mock guard * test: strengthen migrated anthropic, azure, bedrock and cohere provider tests Assert full request bodies and usage instead of presence checks, restore the dummy-tool and custom_llm_provider assertions the legacy tests made, add the async and per-model variants the legacy parametrization covered, split streamed tool arguments across deltas, and move the Azure chat streaming tests to the azure.py mirror file * test: tighten migrated llms provider tests and cover legacy intents lost in the move Strengthen the migrated fireworks, gemini, mistral, together, openai text completion, openrouter, perplexity and openai error tests so they assert what the legacy tests proved (cost, provider tag, usage-only-on-last-chunk, function_call rebuild, reasoning param, sync/async/stream error fields). Replace the huggingface and jina tests that doubled litellm code with respx end-to-end tests. Add the tests the pre-existing mapped tests never covered: cohere 400 mapping, token-array text completion, hosted_vllm/llamafile/lm_studio env api base embeddings, watsonx embedding auth, ollama ssl_verify client and ollama vision prompt * test: cover residual legacy paths for anthropic tool streams, watsonx async stream, ollama http images and caller async embedding clients * test: restore single request assertions in the deep_infra and azure_ai embedding tests * test: drop a no-op streaming cost flag from the openai streaming cost test * test: cover residual legacy-test lines in caching, main, utils, replicate, vertex and bedrock invoke * test: cover streamed audio assembly, choice membership, untranslated responses events and the stream log guard * test: pin cost rates in unit tests, use sdk for cohere e2e, drop vertex llama e2e * test: send the vertex maps e2e request through the openai sdk * test: drop the local_testing jobs that select no tests after the migrations * test: wait for the streamed cache entry and keep custom Mistral rates test-local * test: keep untouched Mistral test lines as they were --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
yucheng-berri
|
5f9dd38867
|
feat(guardrails): add decision model guardrail with Admin UI flow (#45663)
* feat(guardrails): add decision_model guardrail for decisions-API models Adds a new guardrail provider that scores request and response text with a decisions-API model (e.g. Jev) against named predicate checks. Each check carries its own threshold and action (block or log); a flagged block check rejects the request with a 400 naming the flagged checks, and call failures honor unreachable_fallback (fail_open or fail_closed). Ships four preset checks for the Admin UI (prompt injection, jailbreak, system prompt extraction, data exfiltration) served from /guardrails/ui/add_guardrail_settings, plus a Decision Model step in the add-guardrail form filtered to evaluation-mode models. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): fail closed on verdict errors and drop comments in decision_model * feat(guardrails): cap decision_model input with head and tail elision * fix(guardrails): list only block checks in decision_model 400s and tighten inputs * feat(guardrails): honor timeout, cap checks at MAX_DECISION_QUESTIONS, and reject duplicate check names * fix(guardrails): decorate decision_model apply_guardrail with log_guardrail_information * fix(guardrails): suppress untyped-decorator pyright error on log_guardrail_information import * feat(guardrails): screen each text in overlapping chunks with a shared decisions-call semaphore Replace head/tail elision with per-text screening: each non-empty text is sent to the decision model on its own, split into overlapping chunks when it exceeds max_input_chars, and all calls run under a per-event-loop semaphore shared across requests (max_concurrent_decision_calls, default 8). Check verdicts aggregate the max probability across calls; a missing predicate answer on a Block check blocks under fail_closed and only warns under fail_open, while log-only checks are never flagged for it. The 400 detail now lists flagged check names only. The provider combobox also matches Decision Model on jev, typesafe, prompt injection, jailbreak, and decision * fix(ui): align decision model custom check rows with preset rows * fix(ui): move decision model custom check remove button to the row end * style(ui): run prettier on decision model check files * fix(ui): inline decision model evaluation-mode filter after lens module removal * fix(guardrails): keep decision model detections when a chunk call fails * fix(ui): require an enabled decision model check before create * fix(guardrails): screen decision model texts sequentially like presidio * feat(guardrails): replace decision model presets with provider picker and per-question test * fix(ui): render decision model question test as a full-width row * feat(ui): add run-history test section to decision model form * fix(ui): prevent threshold overlap and show test shortcut * feat(guardrails): default decision model threshold to 0.7 * test(ui): expect decision model default threshold 0.7 in characterization payload * feat(guardrails): screen tool call arguments with the decision model * refactor(guardrails): freeze the decision model config model * feat(guardrails): list only evaluation-mode deployments in the decision model picker * feat(guardrails): hide the decision model questions table until a question exists * style(ui): format DecisionModelFields * style(ui): format decisionModelQuestion tests * refactor(guardrails): type the tool call function extraction * refactor(guardrails): build tool call texts without a mutable accumulator * refactor(ui): share isDecisionMode for the decision model picker * fix(guardrails): screen prior Responses function call arguments in the decision model guardrail The Responses input handler leaves function_call arguments out of texts and tool_calls, so a prompt injection in a prior call's arguments passed unscreened. Assistant tool calls from structured messages are now screened too, deduplicated against the request's tool calls * test(integration): send instructions with the decision model guardrail check Checks require instructions, so creating the guardrail returned 422 and the test never sent a chat request * fix(ui): score the decision model test preview the way the guardrail does An unanswered block question now shows Block, disabled questions no longer count, and a run is hidden once the model or a question's text changes since it was scored * fix(ui): clear the decision model error on pick and list questions under Add a question Picking a model after a failed Create left "Select a decision model" on screen, because the form only validates on submit. The field now revalidates when it already shows an error. The Questions table and its heading appear only once a question exists, below the Add a question box * test(ui): move the decision model form tests into an integration test file They render the whole AddGuardrailForm, so they belong in a .integration.test.tsx file. Plain fields are now filled with fireEvent.change, and the two existing form test files go back to main apart from the modelHubCall mock the form now needs * test(guardrails): build the decision model test params without mutating a local * fix(guardrails): reject list checks outside decision_model and honor streaming scan settings - LitellmParams now rejects a list `checks` for any guardrail other than decision_model, so a Bedrock guardrail with list checks gets 422 again instead of being stored and failing every request - The decision model guardrail reads streaming_end_of_stream_only and streaming_sampling_rate, so a streamed post_call response can be scored once at end of stream instead of every 5 chunks - max_concurrent_decision_calls is described as a per-worker limit * fix(ui): match guardrail providers like the default picker filter - The provider search uses the Base UI collator filter on the label and the aliases, so searches such as presidio_pii or llm_as_a_judge match again, as they did before the alias filter - Model groups from /model_group/info are parsed defensively and after the providers are populated, so a malformed response cannot empty the provider list * fix(ui): edit decision model questions in place and show passing test chips in green Questions start as an Add question button. Each click adds a blank question card with its name, text, action and threshold, editable until the guardrail is created, mirroring the per-model budget editor. Create and Next reject a blank or duplicate enabled question. Test run verdicts and chips use the success tone for Pass, the error tone for Block and the warning tone for a flagged Log only question * fix(ui): drop the per-question enable checkbox from the decision model form A question is either in the list or removed with its X, so every listed question is sent and scored. The card now labels Name and Question the way it labels Action and Threshold * fix(ui): give the question name field the full card width * fix(ui): show the provider logo on the selected decision provider * fix(ui): show the full threshold slider track on decision model questions * feat(ui): add decision model cards to the guardrail garden Adds a Decision Model section to the guardrail garden with TypeSafe Jev, Perplexity, Microsoft, OpenAI and Databricks cards. Each card opens the create form scoped to that provider's decision models. The create form drops the Decision Provider picker and lists every decision model on the proxy with its provider logo, prompts to add a model when none is available, and says that test runs are billed and logged. * fix(ui): shorten the decision model test hint to one line * fix(ui): prefill the decision model from the garden card's provider * chore(ui): regenerate the dashboard API types after merging main --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0126c8db73
|
fix(sagemaker): keep AWS credentials out of the sagemaker_chat request body (#45724)
* fix(sagemaker): keep AWS credentials out of the sagemaker_chat request body sagemaker_chat and sagemaker_nova built the container body from every optional param, so aws_access_key_id, aws_secret_access_key, aws_session_token and the other aws_* auth params, plus model_id and sagemaker_base_url, were sent in the JSON body to the endpoint container. The request is still SigV4-signed with the same credentials, which stay in optional_params for signing * fix(sagemaker): limit the sagemaker_chat body filter to aws_ params model_id and sagemaker_base_url are not credentials, so they stay in the body as before and the change sticks to what the fix is about * test(sagemaker): assert the whole sagemaker_chat request body * fix(sagemaker): filter the sagemaker_chat body with the shared AWS auth param list * test(sagemaker): cover the sagemaker_chat body on every endpoint, credential source, and outage --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |