mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-18 04:44:37 +00:00
303 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
devin-ai-integration[bot]
|
859529ccbc
|
fix(realtime)!: keep turn_detection.type when a transcript guardrail disables auto-response (#44939)
* fix(realtime): keep turn_detection.type when a transcript guardrail disables auto-response * test(realtime): type the GA session.update helper and mark its locals Final * test(realtime): expect typed turn_detection on the voice guardrail session updates * test(realtime): accept either gate path for the client's first voice session.update * fix(gemini): drop VAD tuning when the guardrail turns auto-response off * fix(gemini): check create_response without a redundant isinstance * fix(vertex_ai): keep the client's VAD tuning on Vertex when auto-response is off * fix(realtime): keep the transcript-guardrail gate typeless on xAI so it does not auto-answer * fix(realtime): replace any client turn_detection with the untyped gate on xAI xAI answers on its own under a typed server_vad even with create_response off, so a client's typed turn_detection must not reach xAI while the transcript guardrail gates the turn. * fix(realtime): keep a client's early turn_detection and hide the guardrail cancel's error A session.update sent before session.created reached OpenAI ahead of the proxy's own gate, so the gate replaced the client's turn_detection. The proxy now re-sends the client's gated turn_detection after the gate when the two differ. The guardrail's response.cancel now carries a litellm_guardrail_ event_id, and the error OpenAI returns for it (response_cancel_not_active when nothing is answering) no longer reaches the client. * fix(realtime): send the guardrail gate before a client's early session.update and let Muse and Chirp fall back to server_vad under drop_params * fix(realtime): hide only the cancel errors that echo the guardrail's own response.cancel ids Track the event ids of the guardrail cancels this session sent and drop a backend error only when its error.event_id is one of them, so a client's own event id with the same prefix still gets its error. Cover the scripted response.cancel reply in the integration upstream, update the stale list-mode test to the typed gate, and mark the xAI cancel error cell as an expected failure, since xAI replies with its own event id. * test(integration support): deliver the disconnect a real server sends after the client goes away The scripted upstream now keeps reading frames after a reply hits a closed socket, so the unit test's fake ASGI receive has to end with the websocket.disconnect message a real server delivers. * fix(realtime): delete the blocked transcript item before the guardrail's refusal turn OpenAI GA answered the blocked question in some refusal turns because the blocked user item stayed in the conversation. The raw OpenAI-protocol path now sends conversation.item.delete for it right after the cancel. Provider config backends have no such event, so that path no longer passes an item id. Also bound the guardrail integration file by the shared graceful-stop timeout and make the xAI cancel-error xfail require an AssertionError. * fix(realtime): keep the blocked item on xAI, which cannot delete an audio input item * test(realtime): assert the xAI voice block sends no conversation.item.delete upstream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Rick
|
f81e657903
|
feat(volcengine): add Doubao-Seedance video generation via the OpenAI /v1/videos surface (#46156)
* feat(volcengine): add Doubao-Seedance video generation via the OpenAI /v1/videos surface
New VolcEngineVideoConfig maps litellm's OpenAI Videos API onto Volcengine
Ark's contents/generations task API: create hits
/api/v3/contents/generations/tasks and returns a cgt-... task id, status and
content poll /api/v3/contents/generations/tasks/{id}, list maps the same
collection (limit -> page_size, Ark's filter.* through extra_query), delete
maps DELETE on the task URL, and edit/extension reuse create with a
reference_video content item since Ark models both as ordinary generation
tasks. Ark's own edit and extension require ratio adaptive (edit also
duration -1), so those are pinned unless the caller overrides them. Remix has
no Ark equivalent and raises.
The body is flat apart from a typed content array: the prompt is a text item
and every image/video/audio input is a url item carrying a role, which is the
only way to express reference-to-video, edit and extension jobs, so a
caller-supplied content array wins outright. input_reference becomes a
first_frame item, base64 encoded into a data URI for file inputs.
OpenAI params translate per Ark's contract: seconds becomes duration and size
yields both a ratio and a named resolution tier, with the ratio snapped onto
the set Ark accepts (854x480 reduces to 427:240, which Ark rejects) and the
tier taken from the shortest side. Emitting only the ratio leaves the request
on the model default and bills at that default's rate.
Billing reads the delivered duration and resolution rather than the requested
ones, because duration -1 and ratio adaptive both let Ark deliver something
else. Ark bills per output token (tokens = width * height * 24 * seconds /
1024), so the cost map carries per-second rates by resolution tier, derived
from the console's CNY per million token list rates (2.5 70, 2.0 46, 2.0-fast
37, 2.0-mini 23 without a video input) at the repo's 7.0 CNY/USD convention.
Requests that do carry a video input bill at the lower rate instead, which
litellm's per-second tiers cannot express, so those are overestimated.
usage.total_tokens rides along for invoice reconciliation.
Ark echoes the delivered geometry only as a resolution tier plus a ratio, and
the 2.0 family's rendered frame does not follow the tier name: its 16:9 "480p"
canvas is 864x496, not the 854x480 the tier implies, and its other ratios
differ from the tier's nominal shortest side too. A size reconstructed from
the tier alone therefore reports pixels the model never produces and scales
the 480p rate against the wrong pixel count, so both the reported size and
the 2.0 480p rates come from the measured frame (a per-family table, sourced
from the Seedance 2.0 Mini model card and re-verified against a live gateway).
The create response reports the requested duration and resolution, since only
the create, edit and remix calls are costed and Ark's create response carries
no usage block, so without it every video logs zero spend. A caller that
leaves seconds off is not free either: Ark runs its own default length, which
is 5s for the 2.0 family, so that is what the create response bills, while
the 2.5 family's default is -1 (Ark picks) and stays unbilled until the
status response. The create response also echoes the requested seconds and
the frame the job will be rendered at.
Editing and extending need the source video as a URL: Ark's task query is a
GET while litellm's edit prefetch hook only POSTs, so a task id cannot be
resolved into one. The model comes from an explicit extra_body.model, else
the encoded video id, else the model the router already resolved, with
litellm's provider prefix dropped.
The credential's api_base is the Anthropic-compat surface
(/api/compatible), so the base is cut at the first /api/ marker rather than
by a /api/v3 suffix before the video paths append their own version.
The cost-map schema test's exception list gains the 2.0 4K tier, whose
$1.28/second rate exceeds its $1 sanity bound, alongside the existing
runwayml/seedance2 entry.
* fix(volcengine): answer 4xx when a Seedance video is downloaded before it finishes
Downloading a task that is still queued or running raised a bare ValueError, which the
video content handler synthesizes into a 500. The proxy then retried the same poll twice
and reported the caller's own early download as a gateway fault, and the router went
looking for a model group fallback for a job that was simply unfinished
The download path now raises VolcEngineVideoError carrying the status the OpenAI videos
API documents for content that is not ready (404), 400 for a task the provider reported
as failed, 404 for an expired task record, and 502 for a succeeded task with no artifact.
None of them are 5xx, so the router stops retrying them
Regression tests drive avideo_content and assert the caller sees NotFoundError or
BadRequestError instead of an internal server error
* fix(volcengine): clear the type-discipline and override findings on the Seedance video path
The module annotated four mutable collections and carried five `# mutable-ok` markers on
lines that suppressed nothing, so the type-discipline gate counted LIT001 x4 and LIT013
x5 against the PR. The content-item and task-dump helpers hand back read-only views of
what they build, so they now say Mapping. VideoObject.error is declared as a dict by the
type it feeds, so that one marker moves onto the annotation it actually suppresses
transform_video_status_retrieve_response also dropped the client keyword the base class
declares, which basedpyright reported as an incompatible override
* fix(volcengine): bill every Seedance video at create and keep its polls on the deployment that made it
Ark bills a task when it finishes while litellm bills it at create, so a request
whose cost is not known then is rejected: Ark-picked length (duration -1), frames,
a video input, and with it edit and extension. An omitted length is sent as 5s,
since the 2.5 family otherwise lets Ark pick it and logged $0.
Video ids now carry the requested and the served model group, read from
litellm_metadata, so polls and downloads go back to the deployment that created
the task. The routing and input_reference helpers move out of the DashScope
config into a shared module both providers use.
A download before the render finishes answers 400, unreadable or unmapped task
bodies answer 502, malformed seconds, size, parameters and Ark fields raise
unless drop_params is set, an unpriced resolution tier is refused, and the
rendered frame table moves into the cost map.
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(volcengine): reject a text item in content so the guarded prompt is the only text sent
Input guardrails scan data["prompt"], so a caller could pass a harmless prompt
for scanning and a different, unscanned text item in content, which this branch
kept and the guarded prompt was dropped. The prompt now always leads the content
array and a second text item answers 400.
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(volcengine): route video to VOLCENGINE_API_BASE when the deployment sets no api_base
* fix(volcengine): let a model with no tier prices keep its resolution instead of rejecting every tier
* fix(volcengine): answer 400 for malformed or unknown create fields, send a 720p default tier, and stop the list cursor
* fix(volcengine): map size to the closest tier by pixel count, reject draft, remix and list cursors, map user to safety_identifier, read ARK_API_BASE
* fix(volcengine): drop unsupported create values under drop_params, take OpenAI's object input_reference, default to 4 seconds, price endpoint ids by base_model
* fix(videos): unwrap nested file tuples without recursion
* fix(volcengine): price Seedance tiers at Ark's own per-resolution token rates
Ark prices 1080p and 4k per token above and below the 480p/720p rate, so the
single 46 CNY per million tokens the rows were derived from billed 2.0 1080p
and 2.5 1080p about 10% low and 2.0 4k 77% high. Ark's per-resolution table
and its own 16:9 examples agree: 1080p 2.48 CNY/s, 4k 5.05 CNY/s for 2.0 and
3.74 CNY/s for 2.5 1080p, which is 51, 26 and 77 CNY per million tokens.
Rewrites the tier test to check each rate against the rendered pixels at that
tier's own price instead of assuming every tier scales from the base rate,
and drops the 4k cost exception that only existed for the old rate.
Co-Authored-By: Bytechoreographer <noreply@bytechoreographer.com>
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(volcengine): unpack Ark parameters sent through extra_body
* fix(volcengine): ignore the extra_body keys litellm maps itself
The shared request builder merges the caller's raw extra_body back over the
mapped params, so a key this adapter maps itself comes back a second time as a
field of its own, and the unknown field check turned the supported request
into a 400.
|
||
|
Ayush Shekhar
|
7930c0e434
|
feat(router): mid-stream fallback continuation for chat completions (#41127)
* feat(router): continue chat streams on fallback after a mid-stream break A chat-completions stream that breaks after content has been delivered was re-raised and the fallback deployment never ran (#40404), while the Responses-API path already continues via a prefilled assistant turn. Add the same for chat completions behind `enable_mid_stream_fallback_continuation` (opt-in, async-only), so a post-content break re-enters the fallback chain with the partial text as an assistant prefill instead of surfacing the error. Continuation is only attempted when it is safe: the stream emitted plain assistant text (no tool/function calls, thinking blocks, reasoning items, audio or images), the request is not constrained output (response_format or a forced tool_choice) and not merge-reasoning mode, and the fallback target's model supports assistant prefill. The target check runs as a deployment pre-call filter, so a chain with no prefill-capable deployment empties and the original error is surfaced rather than a duplicated or rejected request being sent. Plain reasoning_content is treated as out-of-band and does not block a continuation, matching the Responses-API path; Anthropic thinking is excluded because its signed thinking blocks cannot ride a text-only prefill. CustomStreamWrapper now latches whether a disqualifying delta was streamed and carries it on MidStreamFallbackError so the router can decide without re-scanning chunks. Default behavior is unchanged. * fix(router): harden mid-stream continuation eligibility and prefill folding Address review of the mid-stream continuation: - keep response_format={"type": "text"} eligible; only json_object / json_schema and other structured formats decline, since "text" is the unconstrained default - fold a new partial into an existing trailing assistant prefill instead of appending a second one, so a nested mid-stream break stays a single prefill turn even on providers that do not merge consecutive assistant messages - split the deployment prefill-capability lookup into two readable steps - cover the merge_reasoning decline branch, the text response_format pass, and the prefill-folding path with tests * fix(router): address review of mid-stream continuation - register the continuation deployment filter as a process-global singleton instead of tracking it per router, so discarding one router no longer removes the type-deduplicated filter that other live routers still depend on - disqualify reasoning_content: it reaches the caller as visible reasoning that a text-only prefill cannot carry, so a reasoning-capable fallback would re-derive it and produce an incoherent response - move MID_STREAM_CONTINUATION_KWARG to constants and lazy-import the filter class in the router, breaking the module-level import cycle CodeQL flagged - trim the added docstrings to the repository comment policy * fix(router): keep mid-stream continuation within the type-discipline budget The LIT* type-discipline gate counts mutable-collection use the ruff pass does not see. Annotate the reads with read-only Sequence/Mapping views, compare response_format against a frozen MappingProxyType sentinel, and mark the genuinely mutable litellm message and deployment-list boundaries with mutable-ok reasons, so LIT001 and LIT002 stay under their limits. * fix(router): clear basedpyright operator error and the import-cycle alert - narrow the accumulated stream delta content to str before appending, so the shared helper no longer trips reportOperatorIssue (str += object) - keep the continuation marker in the pre-call-check module and lazy-import it at the two router call sites instead of routing it through constants, so the router has no module-level dependency that closes an import cycle * fix(router): make the mid-stream continuation marker unforgeable The deployment filter is process-global and read the continuation marker as a truthy value from request_kwargs. Since the proxy can forward arbitrary request-body fields into the router, a client could send `"_mid_stream_continuation": true` on a normal request to filter a mixed model group down to prefill-capable deployments and steer their prompt onto that provider. Set a private sentinel object internally and check it by type instead of truthiness: a JSON request body cannot construct one, so a forged flag is ignored. Adds a regression test that forged client values (true, "true", 1, a dict) leave the deployment list untouched. * feat(router): honor per-deployment model_info for continuation prefill capability The deployment filter only read supports_assistant_prefill from the cost map, so a model configured with `model_info: {supports_assistant_prefill: true}` on the deployment (the usual way to declare a capability for a model that is not in the cost map, or to override it) was wrongly skipped. Check the deployment's model_info first (True or False both authoritative), then fall back to the cost map. Adds tests for the override in both directions. * fix(router): bound mid-stream continuation to the caller's output allowance A continuation handed the fallback the full max_tokens / max_completion_tokens even though part of the answer had already streamed, so the total could exceed the caller's limit and repeated interruptions could re-grant the allowance on every hop, amplifying spend. Reduce each configured ceiling by the tokens already emitted before re-entry, and decline the continuation (surface the original error) when the allowance is exhausted. The reductions compound across hops because the trimmed ceiling is what the next hop sees. Also add the missing explanatory comment on the model_info fall-through except. * refactor(router): extract mid-stream decline to stay within the complexity budget The output-ceiling branch pushed stream_with_fallbacks past the C901 limit. Both decline paths (ineligible after content, and exhausted allowance) re-raised the original provider error the same way, so pull that into _raise_original_mid_stream_error. It removes the duplicated branching, drops the function back under the complexity ceiling, and reads clearer. * fix(cost-map): mark Claude Sonnet 4.6 as not supporting assistant prefill The Anthropic API and Bedrock both reject an assistant-last message on claude-sonnet-4-6 with "This model does not support assistant message prefill", so the continuation filter must not route a mid-stream continuation to it. Flip the flag on every claude-sonnet-4-6 entry and on databricks' opus-4-6 entry, whose anthropic sibling already says false. The prefill check tests now register synthetic cost-map entries instead of pinning real models, which broke when claude-3-opus left the map. * refactor(router): type the streaming fallback kwargs and drop narration comments Removes an unused mutable-ok marker (LIT013) and the explanatory comments Greptile flagged, and types initial_kwargs on both streaming fallback iterators as dict[str, Any] like their Responses siblings so the new continuation helpers take a typed mapping and the basedpyright budget stays at or below main. * fix(router): run post_call_rules over the joined text of a continued stream Each provider stream wrapper validates only the fragment it produced, so a rule that trips on the primary's prefix plus the continuation's text never fired. The router now feeds litellm.post_call_rules the emitted text plus every continuation chunk before yielding it. Also drops a narrating test-helper docstring. * refactor(router): read a deployment's declared prefill support without an empty except * test(router): cover _raise_original_mid_stream_error directly for the router coverage gate * fix(router): fold the continuation prefill into a trailing assistant turn A request that already ends in an assistant message took the partial output as a second assistant turn, which prefill providers reject. The partial now extends that turn in place, with prefix set, so the continuation request never ends in two assistant messages. * fix(router): keep content blocks on a trailing assistant turn when extending it with the partial output A trailing assistant message whose content is a list of blocks was stringified before the partial output was appended, so the fallback received the Python repr of the blocks as its prefill. The partial now lands as one more text block, and a content shape that cannot be extended surfaces the original stream error instead of a garbled continuation. * fix(router): continue the trailing text block in place when extending a prefill with content blocks Anthropic strips trailing whitespace per content block, so appending the partial output as a new text block after an assistant turn that ends in "Sure, " would send the prefill as "Sure,here". The continuation now extends the last text block itself and only opens a new text block when the trailing block is not text (an image, say), so the prefill reads exactly as the text the caller already received * test(router): type the prefill check's deployment helper * fix(router): keep a continued stream on the primary's identity and widen what can continue Continuation chunks now keep the primary stream's id and created time and drop the fallback's second role delta. Requests with n > 1 no longer continue. Prefill support reads model_info.base_model and litellm_params.base_model before the deployment model. enable_mid_stream_fallback_continuation is runtime-updatable through update_settings and registers the prefill filter. The bridge rebuild of MidStreamFallbackError keeps the inner stream's disqualifier. A fallback that repeats the prefill is not doubled in the continuation input. * fix(router): decline mid-stream continuation after a content filter, merged reasoning, or a plain trailing assistant turn Also count the partial stream's completion tokens in the continued stream's usage, accept string values for enable_mid_stream_fallback_continuation, and list the new router helpers in the router coverage check * fix(router): drop re-emitted whitespace at the continuation join and sum both legs' usage consistently * fix(router): keep the full prefill so a nested break keeps the separator * fix(router): bill both calls of a continued stream and fix Bedrock Converse, Databricks, and Vertex DeepSeek prefill routes * test(router): integration cells for mid-stream fallback continuation Covers the continued stream the caller gets, the prefill turn and reduced max_tokens the fallback receives, the prefill-capability filter, the runtime toggle, repeated requests through the response cache, and the spend row of a continued stream * fix(router): log a continued stream under the caller's id, price each cut on its own prompt, and keep prefill flags per deployment * fix(router): price and log each cut leg before its continuation reuses the logging object Price the cut stream's partial and let its failure log finish (bounded) before the fallback overwrites a shared proxy logging object, and skip continuation when an output guardrail is registered or a request fallback carries its own messages. * test(router): integration cells for the output-guardrail and request-fallback declines and the cut call's failure log Cover a registered post_call guardrail and a request fallback carrying its own messages keeping the original error, an input-only guardrail leaving continuation on, and the cut call's failure event naming its own deployment and price. * fix(router): skip mid-stream continuation when a request fallback overrides request params A request-body fallback entry that sets anything besides model (stream, messages, max_tokens) changes the request the prefill continuation relies on, so a stream: false entry ended the answer with no error. * fix(router): skip mid-stream continuation for a fallback group target that overrides request params The override check read only top-level fallback entries, so a target inside a group's list, such as {"primary": [{"model": "backup", "stream": false}]} in the request, the config, or a key's router settings, still ended the answer with no error. * fix(router): check only the request's own fallback chain for request param overrides Scanning every group's chain turned continuation off for all requests when one unrelated group's target overrode a param. Resolve the chain the router will use for this request, the same way the fallback path does. * fix(router): keep the original error when request fallbacks are malformed The request-param override check resolved a standard-shape chain outside the router's own error handling, so an empty entry or a non-list chain raised StopIteration or TypeError and replaced the original mid-stream error. Skip empty entries and treat a non-list chain as no chain. * fix(router): read deployment-configured constraints, decline deployment-id fallbacks without prefill, strip only the last Bedrock text block * fix(router): read a deployment's tool_choice ahead of router defaults in the continuation check, and restore the renamed prefill test names * fix(router): filter continuation deployments on every routing strategy, read extra_body and fallback deployments' own params The deployment filter now also runs on the sync selection path that usage-based-routing v1 and custom strategies use, one predicate reads n, response_format, tool_choice and merge_reasoning_content_in_choices with extra_body laid over them, fallback deployments whose own config constrains the output are skipped, and continuation declines when the cut deployment's id is not in the router (per-request credential deployments) * fix(router): decline a cut mid-echo, reduce extra_body output limits, and check post_call_rules once on nested continuations --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Marcus
|
642a2ad934
|
fix(prompt-templates): treat initial_prompt_value and final_prompt_value as optional (#39762)
* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(model_hub): surface model_info.description in model group info and Model Hub UI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): aggregate model group description without in-place mutation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prompt-templates): treat initial_prompt_value and final_prompt_value as optional completion() only stores a custom prompt template key when its value is truthy, and custom_prompt() already defaults both prompt values to "". Seven call sites read them back with direct indexing, so a template that sets only roles raises KeyError instead of rendering, on ollama, petals, vllm, predibase, codestral, anthropic text completion, and response_schema_prompt. Five other providers already read defensively, so the same config works on sagemaker and 500s on ollama. Read both keys with .get(key, ""), matching those five providers and the defaults custom_prompt() already declares. Templates that set all three keys render exactly as before. * test(prompt-templates): drop the explanatory docstrings from the optional-keys regression test * fix(health): skip background health check DB writes when the latest-row read fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prompt-templates): read every custom prompt template field through one typed helper Templates that omit roles, initial_prompt_value, or final_prompt_value now render with empty defaults across every provider that reads them, instead of failing with KeyError or TypeError. A field with the wrong type fails with a validation error naming it. * fix(prompt-templates): keep templates that rendered on main working and fail bad fields with a 400 Only the roles a request uses are checked, null fields mean unset, and extra role keys are ignored, so every template main rendered renders the same. A wrong-type field now raises a BadRequestError naming it. initial_prompt_value is declared litellm-owned so it stays out of provider request bodies, like final_prompt_value. The template tests move into the mapped factory test file, and the predibase test asserts the real rendered prompt * fix(prompt-templates): keep bad-template failover and render non-mapping role entries like main A wrong-type template field now raises a 500 InternalServerError naming the field, so the router still retries and cools down the bad deployment. A roles value or role entry that is not a mapping renders unwrapped as it did on main, and role keys are only checked for roles a request uses. * fix(prompt-templates): warn when a custom prompt template roles value is not a mapping * fix(prompt-templates): keep deployments in rotation on a bad template and reject a template with nothing to render * test(integration): audit custom prompt templates across endpoints, providers, and chaos --------- Co-authored-by: milan <milan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Aryan Pardeshi
|
a844dadcd2
|
feat(gemini): allow disabling thought summaries while keeping thinking on (#36460)
* feat(gemini): allow disabling thought summaries while keeping thinking on Both reasoning-effort mappers hardcoded includeThoughts to True whenever thinking was on, so thinking ON with summaries OFF was unreachable even though that is the google SDK's own default. Summaries are the narration rather than the reasoning, and for callers that discard them they are pure added latency. Adds litellm.gemini_include_thoughts_default, defaulting to True so nothing changes for anyone who does not set it. It gates the includeThoughts=True branches in the thinkingBudget mapper, the thinkingLevel mapper, and the anthropic-style thinking param mapper. The disable and none paths still force includeThoughts=False: with thinking off there is nothing to summarise. * test(gemini): assert thought-summary flag through the outbound Gemini request * feat(gemini): let a request's thought summary ask override the gateway default thinking.display (summarized, updates, omitted) and a reasoning_effort object carrying summary now decide includeThoughts for that request; gemini_include_thoughts_default applies only when the request says nothing * feat(gemini): let a thought summary ask on /v1/messages and /v1/responses override the gateway default * fix(gemini): read a text gateway default as a boolean and leave display omitted to the flag Normalize gemini_include_thoughts_default through normalize_drop_params so a YAML or env value written as "false" or "0" turns thought text off and a null or unknown value keeps the default on. thinking.display "omitted" no longer counts as a per-request ask, so the gateway flag decides, matching /v1/messages. Adds proxy integration cells for chat, /v1/messages and /v1/responses that prove the flag, every per-request ask, streaming, and a burst across workers. * test(gemini): match the streamed Responses usage, which omits null token details * feat(gemini): read the AI SDK reasoning summary alias and thinking.summary as chat asks * fix(gemini): keep the logged reasoning_effort unchanged and key the response cache on the summary alias A dict reasoning_effort next to the AI SDK reasoning_summary alias now counts as a summary ask, and the alias is part of the cache key so a plain request and an alias ask never share a reply * fix(caching): key the response cache on the summary alias wherever the request carries it The cache key now reads reasoning_summary and reasoningSummary through the same reader as the summary fold, so an alias inside extra_body gets its own key too * test: type the new summary-ask unit tests and mark their locals Final * test: annotate the remaining summary-ask unit tests the PR touches --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
debanshd
|
1b1ddef1b2
|
fix: preserve non-null anyOf branches when stripping null types for Vertex (#41683)
* fix: preserve non-null anyOf branches when stripping null types for Vertex * refactor: build anyOf branches immutably and drop explanatory comments * style: apply ruff format * fix: annotate anyOf list construction for the LIT002 budget * fix(vertex_ai): send a null-only anyOf as one nullable null instead of raising * test(vertex_ai): pin empty anyOf passthrough * fix(vertex_ai): forward a non-list anyOf unchanged and add wire cells for the null fold * fix(vertex_ai): forward a non-object anyOf branch for Google to judge instead of crashing * fix(vertex_ai): fold null out of type arrays and union branches before sending to Gemini --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4e49394ade
|
fix(anthropic): keep trailing user messages with None or empty list content (#41397)
* fix(anthropic): keep user messages with None or empty list content so requests do not end on an assistant turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): avoid mutable empty-list construction in sanitizer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): avoid rebinding sanitizer input Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): keep a trailing user turn whose content key was dropped litellm.completion strips None fields from every message before the provider transform, so a user message sent with content null reaches the Anthropic converter with no content key and raised KeyError. Read the content with get so that turn gets the same placeholder as an empty list. * fix(anthropic): only fill user turns with empty content and keep their cache_control * fix(anthropic): fill each empty user message in place like an empty string * fix(anthropic): add the placeholder only for an empty turn or a cache breakpoint, in linear time * test(anthropic): cover the tool-result guard with cache_control and widen the linear-time bound * test(anthropic): cover empty user turns through the proxy on chat, responses and messages * test(anthropic): type the empty user turn tests and move the linear-time check to a benchmark --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
a3251a3bfd
|
fix(responses): preserve provider request IDs in streaming failure logs (#45453)
* fix(responses): preserve provider request IDs in streaming failure logs * fix(bedrock): log the provider request id for in-stream errors on bridged Bedrock streams Bedrock models that run through the chat-completions bridge (Claude, converse, invoke) raise in-stream failures from the event-stream decoder as a BedrockError without headers, so their failure logs kept error_provider_request_id null. The decoder now passes the stream's request-id headers to those errors. Both the native Responses iterator and the decoder pass only the PROVIDER_REQUEST_ID_HEADERS subset, so error responses and failure logs pick up the request id and none of the 200 response's other headers. MidStreamFallbackError gets its response at construction instead of mutating it afterwards. --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
b21e659c24
|
fix(proxy)!: count endpoints return errors in their source API shape instead of silent local counts (#46815)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): reject unconvertible blocks nested inside a tool_result * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system * fix(proxy): count endpoints return missing-key and request errors in their source API shape * fix(proxy): drop imports the count error change left unused * fix(proxy): let Entra-authenticated Azure AI Claude count tokens and keep Google count errors in the Google envelope * fix(proxy): send Azure environment keys as x-api-key on Azure AI Claude counts and log unexpected Google count failures * fix(proxy): count with the global OpenAI and Anthropic keys chat already uses * fix(proxy): resolve the OpenAI count base with the key so a global key stays on its gateway * fix(proxy): keep unexpected Google count failures generic so internal details stay out of the response * fix(proxy): resolve the Azure AI Claude count base from the same settings chat reads * fix(proxy): count OpenAI tokens with the workload identity bearer chat already uses * fix(proxy): mint the OpenAI workload identity bearer inside the count error handling * fix(proxy): report a failed OpenAI workload identity exchange on the count endpoints as a 401 --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
David Bochenski
|
f072498304
|
fix(bedrock): grant sts:GetWebIdentityToken and sts:TagGetWebIdentityToken in web identity session policy for claude_platform (#46670)
* fix(bedrock): grant sts:GetWebIdentityToken in web identity session policy for claude_platform Claude Platform on AWS calls STS GetWebIdentityToken (outbound identity federation) with the caller's session before serving inference. The session policy passed to AssumeRoleWithWebIdentity is a permission ceiling and never granted it, so every bedrock/claude_platform request using OIDC web identity auth failed with 403 "no session policy allows the sts:GetWebIdentityToken action", even when the role's identity policy allowed it. Add a ClaudePlatformOutboundIdentityLiteLLM statement granting only sts:GetWebIdentityToken, plus tests that it is granted, scoped to its own statement, and that no sts:* wildcard is introduced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(bedrock): also grant sts:TagGetWebIdentityToken for claude_platform web identity Claude Platform on AWS tags the outbound web identity token it requests, so the caller also needs sts:TagGetWebIdentityToken. Found end-to-end on a deployment: with only GetWebIdentityToken granted, calls failed with "not authorized to perform: sts:TagGetWebIdentityToken on resource: arn:aws:sts::<account>:self"; granting both made /v1/chat/completions and /v1/messages (streamed and not) succeed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(bedrock): only let Claude Platform use the outbound identity STS grant Scope the ClaudePlatformOutboundIdentityLiteLLM session policy statement with aws:CalledViaLast = aws-external-anthropic.amazonaws.com, so a web identity session can no longer call sts:GetWebIdentityToken or sts:TagGetWebIdentityToken directly. Drop the code comment and fix the stale test messages. * test(bedrock): fail when the Claude Platform caller condition reaches other statements --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Dan Lemon
|
c8b73cde84
|
feat(soniox): decode proxy form fields for context, translation and flags (#40605)
* feat(soniox): decode proxy form fields for context, translation and flags Multipart form fields reach the proxy as strings, so Soniox received context and translation as JSON strings and booleans as "true". Decode them at the handler entry point and add typed shapes for the context and translation request fields. * fix(soniox): keep bracket-prefixed free-form context and move types under llms Free-form context such as "[inaudible] consultation" is valid Soniox input, so a string that fails JSON parsing is now forwarded unchanged instead of returning a 400. The Soniox request types live next to the provider code. * test(soniox): exercise SDK-typed context and translation shapes through the decoder * refactor(soniox): derive handler options and provider params without mutating a local dict * chore(soniox): drop explanatory comment on the non-forwarded param set * refactor(soniox): return invalid form bool as a value and raise once in the handler * fix(soniox): validate decoded form fields and keep language with hints Validate structured context before sending it as an object, answer 400 for a bad polling interval or flag unless drop_params is on, nest SDK bracketed extra_body fields, keep the OpenAI language at the head of language_hints, and return the translated text for one-way translation without tags. * fix(soniox): keep the source transcript, rebuild SDK list fields, and reject malformed JSON values * fix(soniox): keep cleanup strings whole and drop the unused translation type * fix(soniox): let explicit language_hints win over language and test the proxy form path --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
79ca820933
|
fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert (#46772)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): reject unconvertible blocks nested inside a tool_result * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
Mateo Wang
|
dba86bec28
|
fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them (#46767)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): fetch Gemini remote media without blocking the event loop * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * chore(count_tokens): give the Gemini count helpers' list returns a reason * fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does * fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400 * test(count_tokens): cover the Gemini count error paths and compaction with a text-block system --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
Emmanuel Acheampong
|
de4d28653d
|
fix(crusoe): update base URL to current endpoint (#33121)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d25b58f744
|
fix(count_tokens): count Anthropic-format requests for OpenAI the way /v1/messages inference converts them (#46759)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * test(count_tokens): expect the converted Responses input in the budgeted-key count integration test --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
mrinal-berri
|
bd60cc8608
|
fix(bedrock): forward Nova Sonic realtime tool calls with their arguments (#46214) | ||
|
Mateo Wang
|
d489c04554
|
fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking (#46741)
* fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking * fix(bedrock): drop unknown thinking keys under drop_params on display updates rebuilds Bedrock Invoke copied every caller thinking key when adding display updates, so an unknown key still reached the provider with drop_params on. Filter that copy the same way, and cover each chat call site plus the global litellm.drop_params path in tests * fix(anthropic): drop budget_tokens from rebuilt adaptive thinking under drop_params * test(anthropic): cover drop_params sources for rebuilt adaptive thinking on /v1/messages and Bedrock Invoke |
||
|
Mateo Wang
|
7090588bdf
|
feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240)
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions * fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel * feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing early-access row, carrying the same limits and capabilities. The AI Studio row mirrors the early-access entry; the Vertex row differs only in provider and pricing source. Prices follow once Google publishes them. Extend the mapped metadata test to cover all three ids, that each routes to its provider, that its backup copy matches, and that the public rows carry the same limits as the early-access row. * feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops * chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap * fix(gemini): follow continuation hops in a loop instead of recursion The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls * fix(gemini): bill Google Search grounding from every continuation request Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain * fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch * chore(gemini): name the type errors the dual-mode stream iterator trips The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up * fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment * fix(gemini): keep one response id across a continued answer A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key * test(gemini): add integration cells for long decode continuation Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once * fix(vertex_ai): fetch the access token again before each continuation request A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged * fix(vertex_ai): start each continuation stream with an empty parser buffer When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async * fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way * fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes * fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not * fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0 A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer Adds the audit's wire tests for both, which fail 24 cells before this change * test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed * fix(vertex_ai): log a continued answer's raw response as one object A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch * test(integration): check a continued answer logs one merged raw response A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch * test(vertex_ai): cover a follow-up tool call and its thought signature in long decode * test(gemini): pin that a one-request answer logs Google's body unchanged |
||
|
Mateo Wang
|
84ff9f84a9
|
fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses (#46720)
* fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses * fix(ui): send detail on playground Responses image parts and code snippets * test(messages): cover image detail on the Responses bridge wire |
||
|
Fede Kamelhar
|
68759a239f
|
fix(oci): drop non-function tools under drop_params and gate reasoning_effort on declared levels (#40732)
* fix(oci): skip non-function tools and gate reasoning_effort on models that reject it Fixes #31449. Two OCI GENERIC request-building defects that made Grok-on-OCI unusable for tool-using coding-agent clients (Codex over the Responses API, Claude Code over the Anthropic Messages API): 1. `adapt_tool_definition_to_oci_standard` raised `OCIError("OCI only supports function tools")` for any tool whose type is not "function", aborting the whole request when a client mixed a built-in/server tool in with its function tools. Non-function tools are now skipped with a warning; if none remain, `tools`/`toolChoice` are omitted from the request instead of sending an empty array. 2. The GENERIC param map forwarded `reasoning_effort` to every non-Cohere model, but OCI rejects it for all served xAI Grok models (`Model grok-4.20 does not support parameter reasoningEffort`) and for OpenAI's non-reasoning models (`Unrecognized request argument supplied: reasoning_effort` on gpt-4o / gpt-4.1); gpt-5, the o-series, gpt-oss-* and Meta/Google models accept it. `_model_supports_reasoning_effort` encodes that, `get_supported_openai_params` stops advertising the param for those models, and `map_openai_params` follows the standard unsupported-param contract (dropped under drop_params, clear error otherwise) so `drop_params` / `additional_drop_params` now work as documented instead of the 400 surfacing from the service. Behaviour verified against OCI GenAI in us-chicago-1 (2026-09-11) per model. The two existing reasoning_effort normalisation tests move from a Grok model to openai.gpt-5, which is the case they exercise. 21 new tests; the OCI suite is 406 passed. * fix(oci): drive reasoning_effort gating from the model catalog and drop tool_choice for skipped tools Review follow-ups on #40732: - `_model_supports_reasoning_effort` now reads the `supports_reasoning` flag of the model's `oci/` catalog entry through `get_model_info` instead of hardcoding xAI / gpt-oss branches. Catalogued models without the flag (xAI Grok, Meta Llama) no longer receive `reasoningEffort`; models absent from the catalog keep the historical pass-through. - Catalog: flag `oci/google.gemini-2.5-{flash,pro,flash-lite}` as reasoning models and add `oci/openai.gpt-oss-120b` / `oci/openai.gpt-oss-20b` (pricing from Oracle's public price list, 128K context per OCI docs). - `_model_uses_max_completion_tokens` routes on the `openai.*` prefix only. OCI-hosted Gemini accepts `maxCompletionTokens` but ignores it (the output cap silently disappears) and Llama rejects it with 400, so the catalog's reasoning flag must not drive that switch. - After tool adaptation, a `FUNCTION` tool_choice naming a tool that was skipped (built-in / server tools) is dropped with a warning instead of being sent to OCI, which rejects the unmatched choice. Verified live against OCI GenAI (us-chicago-1): Grok with mixed tools plus a forced built-in now completes and still calls the function tool; Gemini and gpt-oss forward reasoningEffort with maxTokens honoured; Llama drops it under drop_params; gpt-5 keeps maxCompletionTokens. * style(oci): use an immutable set for the tool-name lookup Keeps the type-discipline gate (LIT002, mutable collections) within its budget; the lookup is read-only. * fix(oci): fail closed when a forced tool_choice names a tool OCI will not receive Review follow-ups on #40732: - When tool_choice forces a function that is not among the adapted tools (a skipped built-in or a misspelled name), send `toolChoice: NONE` instead of removing the choice. Falling back to the service default would let the model call any other supplied function even though the caller forced a specific one, which an agent processing untrusted content must not do. - Move the #31449 regression tests into the mapped test module (`test_oci_chat_transformation.py`) instead of a separate file. Verified live against OCI GenAI: Grok and Llama with a forced skipped built-in now complete with no tool call; a forced kept function is still honoured. * test(oci): point reasoning_effort comments at the in-module gating tests The referenced module was folded into this file. * fix(oci): drop non-function tools only under drop_params and gate reasoning_effort on declared levels Non-function tools now follow the unsupported-param contract: a 400 naming drop_params without it, and dropped with a warning under it, with a forced dropped tool turning tool calls off. reasoning_effort is gated on the model's reasoning_effort_levels catalog entry, so OCI Llama keeps forwarding it and the Grok 4.20 models OCI serves, now in the catalog with their Oracle prices, stop sending a param the service rejects * fix(oci): gate reasoning_effort on oci/xai.grok-4.20, which rejects it * fix(oci): disable tool calls when tool_choice forces a dropped custom tool by name * fix(oci): drop tool_choice that forces a dropped tool and keep a kept function's name * fix(oci): keep Gemini 2.5 Flash Lite unflagged for reasoning so released proxies keep its max_tokens cap * fix(oci): answer 400 when a request depends on a dropped tool and leave Gemini max_tokens routing to its own PR Under drop_params, non-function tools still drop when function tools remain and tool_choice does not force a dropped tool. A request whose every tool is dropped, or whose tool_choice forces one, now answers 400 instead of a tool-less reply the model can fill with invented results, or a swap to another tool. The maxTokens routing change for OCI Gemini goes back to base, since it is outside this fix and an open PR carries it. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
da2f5a498f
|
feat(agents): support Databricks Agent Platform agents in the agent gateway (#45681)
* feat(agents): support Databricks Agent Platform agents in the agent gateway Add a databricks_agent provider that speaks the MLflow ResponsesAgent wire (Responses-shaped input, output_text items, output_text.delta SSE) to Model Serving endpoints (/invocations or the unified /serving-endpoints/responses) and Databricks Apps (/responses), with a PAT or OAuth M2M client credentials Register it through the agents config block or the Admin UI agent catalog and call it through /a2a/<agent> (message/send, message/stream) or model: a2a/<agent> chat completions. The chat route now runs every completion-bridge agent through its own provider, and attaches the minted Databricks OAuth token for both bridge and URL agents. Consumed OAuth settings never reach the provider call * fix(agents): harden databricks_agent input, output, and OAuth handling Non-text content parts answer 400 unless drop_params is set, a reply without an output list answers 502, flat OAuth fields resolve an os.environ api_base, the bridge debug log no longer echoes the request model, routing tests stub only the HTTP boundary, and the catalog description fits the dropdown * fix(agents): merge agent and client extra_headers on the a2a chat route, and reject an empty databricks_oauth block * fix(agents): bridge only databricks_agent agents on the a2a chat route, keep the card url over a client api_base, and require workspace_url for a Databricks App * refactor(agents): read the a2a chat bridge providers from a constants set instead of branching on a provider name * fix(databricks_agent): let the configured credential win, reject malformed client extra_headers, and keep a missing endpoint name empty * test(agents): drive the flat OAuth a2a send through the real bridge * fix(agents): keep an empty databricks_oauth block ignored on url agents, as before this change * fix(agents): accept bare App hosts, env service principals, and os.environ refs on Databricks agents * fix(databricks_agent): address review caveats on chat bridging, params, and auth Map max_tokens and max_completion_tokens to max_output_tokens and pass temperature, top_p, and user, so /v1/messages works without drop_params. Send A2A message metadata only with string values, answering 400 or dropping the rest under drop_params. Join message items with a blank line in both stream and non-stream replies. Cache the env service principal token until shortly before it expires. Keep databricks_agent replies out of the response cache. On the chat bridge, let request fields win over the agent's stored params while pinning the agent's api_base and api_key, and apply the agent's static_headers between client and agent extra_headers, as /a2a does. Resolve os.environ/ only for databricks_agent api_base and api_key instead of every bridge provider. Ignore an unusable databricks_oauth block on URL agents on chat, as at the merge base, instead of answering 400. * test(internal_context): list the databricks agent token cache as process-local * fix(databricks_agent): keep the agent's endpoint, input, and target out of client control extra_body model and input answer 400, or are dropped under drop_params. A client api_base or base_url never replaces the agent's card URL on the bridge. Chat to a URL agent keeps its merge-base call, and an incomplete databricks_oauth block on a URL agent keeps its merge-base ValueError. * test(databricks): expect the merge-base 400 for a client api_base on agent chat * fix(databricks): keep agent-owned inputs and identity headers out of a chat caller's reach * fix(databricks): bound the agent-over-caller merge to two levels so it needs no recursion --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
f470e418ea
|
fix(anthropic): keep signed empty thinking blocks and honor thinking.block_binding (#46342)
* fix(anthropic): treat adaptive thinking as enabled when mapping response_format
is_thinking_enabled only recognised thinking.type=enabled, so a request that
paired thinking.type=adaptive with response_format took the synthetic JSON
tool path with a forced tool_choice, which Anthropic rejects alongside
thinking. The same check also gates the Bedrock Converse and Databricks
forced tool_choice downgrades, so they now see adaptive thinking too.
* fix(anthropic): keep signed thinking blocks with empty text on chat completions
Models that default to display: omitted return signed thinking blocks whose
text is empty, and Anthropic rejects the next turn when they are not replayed
unchanged. is_unsignable_thinking_block dropped them along with unsigned
blocks, and the inline content path required non-empty text, so a tool-use
turn on Opus 4.7+ or the 5 families lost its thinking blocks and 400ed on the
tool-result turn. Only a missing signature now drops a block, and the
mid-conversation system placement mirrors that rule. The /v1/messages strip
helper is left to #41362.
* fix(anthropic): skip the thinking-strip retry when block_binding asks for an error
The /v1/messages retry loop strips thinking blocks and resends on any
invalid-thinking 400, including the binding mismatch a caller opted into
with thinking.block_binding.prefix_mismatch_behavior: error. The retry
decision now sees the request body and surfaces that 400 instead of
overriding the caller's choice. #42888 covers the litellm_params side.
* fix(bedrock): carry thinking.block_binding through the Invoke thinking rebuilds
Bedrock Invoke rebuilt the thinking param from scratch when display: updates
was requested, on both the /v1/messages and chat completions paths, and the
legacy enabled-to-adaptive translation plus the clear_thinking injection did
the same, so thinking.block_binding never reached Bedrock. Every rebuild now
spreads the caller's thinking and only overrides the keys it owns, and
AnthropicThinkingParam types block_binding. The thinking-binding-controls
beta was already on the Bedrock allowlist.
* fix(anthropic): keep forcing the response_format tool under adaptive thinking
This reverts
|
||
|
devin-ai-integration[bot]
|
e344dbd8db
|
feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043)
* feat(proxy): split Azure PTU deployments across teams by share
A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to
ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on,
routing hides the deployment from teams holding no share, the v3 limiter enforces each
share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily
flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team
Usage page report PTU-hours next to tokens.
* fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name
The PTU flag is read from the process environment on every check instead of
through the secret-manager path, so a proxy with a hosted secret manager does
not pay a round trip per request. A team's ceiling now carries the model's
cached-input ratio and settlement charges cache reads at that ratio, matching
how Azure sizes a PTU. A deployment registered for one team is found by its
team_public_model_name for PTU-hours and the ceiling, since its model_name is
rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and
parsed_ptu_shares refuses booleans, strings, and non-string team ids, a
fractional ptu_count reserves nothing, and models_processed counts distinct
deployments rather than holdings.
* fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations
* fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id
* fix(ptu): refuse ptu_shares naming a team that does not exist
* refactor(ptu): keep Azure detection under llms/azure and check share teams with one query
* fix(proxy): drop the reverted TeamIdSearchFilter import after merging main
Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally
* fix(proxy): chunk the ptu_shares team lookup under the IN-list bound
* fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling
* fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling
* fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour
* fix(ui): seed total_ptu_hours in the empty daily activity metadata
* fix(ptu): size usage rows by the name the request used
* fix(ptu): size a team's usage rows by the deployment it is served from
* fix(ptu): never size a team's usage row by another team's reservation
* fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length
* fix(ptu): read a share map only through the terms registration requires
A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and
ptu_effective_from, or one that does not add up to ptu_count, is refused at
registration but was enforced once loaded: every other team got the 400 and
the share holder a ceiling while no flat cost accrued. _deployment_shares now
reads the shares through ptu_terms, so such a row reserves nothing, the same
as before the map was read.
* fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off
|
||
|
ryan-crabbe-berri
|
1f041a3265
|
fix(openai): strip prompt_cache_breakpoint under drop_params for OpenAI models that cannot read it (#46453)
A chat completions request to gpt-4.1 (or any OpenAI model older than 5.6) with drop_params on still sent the marker inside messages and tools. It is now removed there, while drop_params off keeps sending it through and models that read it keep it either way Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
2c4ec9c6fa
|
fix(responses): open faked stream items empty so tool arguments and text arrive once (#46533)
* fix(responses): open faked stream items empty so tool arguments and text arrive once * fix(responses): open faked stream items in progress with a fresh empty list, and open mid-stream guardrail blocks empty * fix(responses): open faked refusals, reasoning summaries, custom tool calls and MCP calls empty so each value arrives once * test(integration): cover the empty-open Responses stream items on every fake, replay and block path Integration cells for the faked /v1/responses stream, the chat and messages bridges, cache replay, guardrail block events and the MCP gateway call. Each opened item starts empty and each value arrives once, checked with raw httpx, the OpenAI SDK and the Anthropic SDK, plus upstream errors, 5 KB values, a client drop, a dropped upstream connection and a worker SIGKILL |
||
|
Taylor Hawkes
|
fd84e97f49
|
feat(providers): add Inference APIs as OpenAI-compatible provider (#41669)
* feat(providers): add Inference APIs as OpenAI-compatible provider Registers https://api.inferenceapis.com/v1 as a JSON-configured provider (chat completions and Responses API). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(providers): list Inference APIs in Add Model and the provider docs index, add request-level tests - provider_create_fields.json entry so the Add Model dropdown test passes - provider_endpoints_support.json (+ backup) entry so the provider docs check passes - tests now capture the outgoing HTTP request (URL, auth header, body) instead of only inspecting registrations - drop a redundant comment Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(providers): inject a recording HTTP session instead of patching httpx.Client Request-level tests now set litellm.client_session to an httpx.Client with a MockTransport, so nothing is monkeypatched on a class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * chore(providers): drop redundant comment on the inferenceapis entry Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * feat(inferenceapis): add cost map rows for the 12 chat models and test the responses route Prices and context windows come from https://inferenceapis.com/pricing and GET /v1/models. Moves the provider test to tests/unit and adds the interactions flag the endpoint matrix now carries. * feat(inferenceapis): price the vendor's chat model aliases and mark /v1/messages supported * fix(inferenceapis): serve /v1/messages natively and keep audio routes off the provider * fix(json providers): send provider-specific params in extra_body and report context overflow as ContextWindowExceededError * fix(exception mapping): match the Inference APIs overflow phrase instead of any context_length_exceeded substring * fix(exception mapping): map context overflow to ContextWindowExceededError for Inference APIs only Providers without their own mapper keep reporting an overflow as BadRequestError, since the router raises a ContextWindowExceededError instead of trying generic fallbacks when context_window_fallbacks has no entry for the group. Adds the integration cells for the provider, its fallbacks, and the extra_body and overflow families. * test(inferenceapis): hold upstream replies across the worker kill in the chaos cell A reply the killed worker sent before SIGKILL can lose its queued spend row, which the product never promised to keep. Holding the upstream until after the kill means only the surviving worker answers, so every 200 must log exactly once. * test(inferenceapis): make the chaos cell kill the quieter worker and always release held replies The survivor now must hold at least one in-flight request, so the burst check cannot pass as 0 == 0, and the held vendor is released on exit so a failed assertion cannot stall the wire server teardown --------- Co-authored-by: TaylorHawkes <857858+TaylorHawkes@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
17f621e116
|
fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836)
* fix(vertex_ai): route Model Garden models through the generateContent adapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vertex_ai): annotate generateContent route test parameters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): type litellm_params in get_vertex_ai_model_route * fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions Streamed Model Garden calls through the generateContent adapter now request stream usage where the deployment supports it, keep cached-token details, and look up the provider-relative cost key, so they bill like the non-stream call. The adapter forwards router metadata, passes upstream errors through with their status, countTokens falls back to local counting for non-Gemini routes, and the native Gemini config is limited to the Gemini routes. * fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing Streamed generateContent through the chat adapter now ends with the upstream usage, honors always_include_stream_usage false and the caller's stream_options, counts one failure per failed call, and routes Vertex deployments by their base_model like chat does * fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage * test(google_genai): expect the trailing usage chunk on bridged OpenAI streams The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone. * fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk * fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent * test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire * test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
08a24d391a
|
fix(count_tokens): count system and tools on Vertex Gemini, fall back to Mantle on Bedrock 404 (#46430)
* fix: align token counting with inference context Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve Vertex Gemini count message shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: forward Google countTokens tools in shapes every provider counter accepts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: add package marker for Vertex count-token unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): ignore deployment system_instruction when counting tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): forward system and tools on Google AI Studio, keep strict on Anthropic tools Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(count_tokens): ruff format gemini count handler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): return Mapping from count request body builders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): truncate long Anthropic tool names on Vertex count, update stale strict test expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): simplify Gemini count context and tests Put the Vertex Gemini count branch back in the same shape as main and the partner model branch, keeping only the system and tools context, the provider error mapping and the gemini tokenizer default Share one error response helper between the Vertex and AI Studio callers, drop the unread kind fields and the extra validation of map_function output, and say Gemini instead of Vertex AI in the error that AI Studio users also see Tighten the tests by removing a stray parametrize on a helper, sharing the SDK count args, asserting the exact forwarded system and tools, and folding the Anthropic tool shape tests into one table * fix(count_tokens): reuse inference tool translation for Vertex Gemini count Vertex and AI Studio count now convert tools with the same functions inference uses, translate_anthropic_tools_to_openai and the Responses tool converter, before VertexGeminiConfig.map_function. Unnamed and Anthropic hosted tools now count the same declarations inference sends instead of being dropped Tools that cannot be converted, such as a non list tools value, a non object tool or a schema that is not a JSON object, now return a 400 before any request instead of being counted without them --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
606ecf56de
|
fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals (#46633)
* fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove an OpenAI-format box answer reaches the caller as a box Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): give the box wire test a deployment that advertises supports_box_questions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Sushit Lal Pakrashy
|
958cba7016
|
fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results (#38284)
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Rick
|
71e508b65d
|
feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos (#43705)
* feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos MinimaxVideoConfig maps litellm's OpenAI Videos API onto MiniMax's V2 task API. Create, status, list, content and delete hit /v2/video_generation and /v2/query/video_generation. Remix is not exposed: MiniMax's /v2/video_regeneration only upscales a finished 768P task to 2K and ignores any prompt, so it raises instead of returning a video that silently drops the edit OpenAI params translate to MiniMax's contract: seconds becomes duration, size reduces via gcd to a valid ratio, and text-to-video requests default to ratio 16:9, resolution 768P and duration 5 when unset. input_reference becomes a first_frame content item, base64 encoded into a data URI for file inputs, and multimodal reference media passes through a content array in extra_body. Caller content is validated strictly as image, video or audio items, and the scanned prompt is always the one text item sent, so text can't reach MiniMax past the guardrails; anything else in content returns a 400. api_base shares MINIMAX_API_BASE with the chat config (a trailing /v1 or /v2 is stripped because the video API carries its own version). The list endpoint pages by number with no cursor or sort, so after and an ascending order are rejected with a 400 rather than ignored, and page_num passes through extra_query. Responses validate into frozen pydantic models at the boundary, video ids are provider-encoded so status and content route back to MiniMax, and create reports duration_seconds and video_resolution in usage for cost tracking End-to-end tests drive the real avideo_generation() entrypoint through an httpx MockTransport and assert on the encoded request body, which is what catches a mapper that drops input_reference before the request is built Both model_prices maps gain minimax/MiniMax-H3 and minimax/MiniMax-H3-Max, priced per output second by resolution tier from MiniMax's pay-as-you-go list (H3 $0.08 at 768P and $0.13 at 2K, H3-Max $0.05 at 480P and $0.08 at 768P). MiniMax also bills input images beyond a per-model free allowance (5 for H3, 2 for H3-Max), so each entry carries input_cost_per_image and provider_specific_entry.minimax_free_input_images, and the create response reports the full charge as provider_reported_cost_usd. Reference video input is rejected with a 400, because MiniMax bills its duration only after the create call that litellm bills Co-authored-by: AaronHowell <237895480@qq.com> * fix(minimax): fail closed on invalid video params and malformed task responses Addresses review on #43705: - A create, status or delete response without a task id (or a task) is a 502 instead of a queued object with an empty id - An unmapped task status is a 502 instead of defaulting to queued - seconds/duration must be a positive whole number and size a positive WIDTHxHEIGHT or W:H, otherwise 400; a 0x0 size no longer divides by zero and an unparseable duration no longer falls back to the billed default - characters and image are rejected with UnsupportedParamsError unless drop_params is set, instead of being dropped silently - Downloading a task that is not ready or failed is a 400 that keeps its message through the handler, instead of a ValueError surfaced as a retried 500 - The video download goes through safe_get / async_safe_get Co-authored-by: AaronHowell <237895480@qq.com> * test(minimax): cover the video download path and invalid-param branches The download now goes through safe_get, so the happy path and the finished-task-without-a-url branch are asserted, alongside the non-object parameters rejection and the remix response stub. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> * fix(minimax): return 400 for video remix and clear new lint gate counts * fix(minimax): send file-tuple images and keep input_reference next to content media * fix(minimax): answer 404 when an H3 video is downloaded before it finishes A download that lands while the task is still queued or running answered 400. That is a client error, which was the intent, but 404 is what the OpenAI videos API documents for content that is not ready, and it is the status the sibling DashScope and Volcengine video paths now answer, so a client polling all three reads one shape A failed task keeps its 400, since that is not a timing mistake the caller can retry away. The download test drives avideo_content and asserts the caller sees NotFoundError rather than an internal server error * fix(minimax): answer 400 for unsupported video params, early downloads and unknown ids A create param MiniMax cannot honor (seed, n, quality and the like) was dropped without a word. It now answers 400 unless drop_params is on, like other providers A download before the video is ready and a status call for an unknown task id now answer 400. The router cools a deployment down on a 404 and retries a 5xx, and neither is a deployment fault. DashScope on main answers 400 for the same case Input images past the free allowance are now billed through the shared video cost path, so a deployment with custom pricing bills them too. Status usage no longer carries MiniMax's token counts, list sets has_more to false since MiniMax pages by number, and the price map stops listing video as an input modality * fix(minimax): keep the polled video id, honor deployment drop settings, and match chat's key order Status re-encodes the id with the polled id's model when MiniMax omits it, drop_params and additional_drop_params never reach MiniMax and listed keys are dropped, MINIMAX_API_KEY wins over litellm.api_key, create sets created_at, and a content variant other than the video answers 400 * fix(minimax): list a task without a model under the routed group MiniMax leaves `model` off some listed tasks. The caller polls the ids a list hands back, and the proxy resolves a poll by the model in the id, so an id encoded with an empty model no longer routes back to the deployment that served the task. Fall back to the group the router was asked for. * fix(minimax): honor deployment drop keys inside parameters and bill images at the map rate when a deployment prices only seconds A field the deployment lists in additional_drop_params no longer reaches MiniMax when it arrives inside parameters, and a deployment that sets only its per-second price now bills reference images past the free allowance at the cost map's per-image rate instead of $0 * fix(minimax): keep deployment-dropped fields out of the body when they arrive through extra_body * fix(minimax): answer 400 for a video resolution with no price unless drop_params is on * fix(minimax): send the default resolution when a deployment drops resolution --------- Co-authored-by: AaronHowell <237895480@qq.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Happy <yesreply@happy.engineering> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Wang Yunzhuo
|
a2f792634e
|
fix: preserve explicit provider for async image, audio, and realtime routes (#36346)
* fix(images): preserve explicit provider for async generation * test(images): prove a bare model name reaches its explicit provider on the async path * fix(images): validate async provider hints instead of reading untyped kwargs Read custom_llm_provider and api_base in aimage_generation through a TypeAdapter over a small TypedDict, so the provider lookup gets typed values and the basedpyright gate stays within its merge-base ceiling * fix: answer bad provider hints with a 400 and keep the explicit provider for async speech and transcription * fix: keep the explicit provider for realtime and cover the audio bad-hint 400 * fix: keep the explicit provider on the realtime HTTP routes and read base_url in async audio * fix: stop async image generation from rejecting a base_url it never reads * fix(audio): skip base_url validation when api_base is set on async speech and transcription * test(integration): cover a declared provider with a bare model on image, audio, and realtime routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c00706218b
|
fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097)
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(vertex_ai): keep yield outside the JSON decode guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): surface BaseLLMException raised while streaming a request body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(http_handler): only follow __cause__ when unwrapping streamed body errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): avoid recursive streamed error unwrapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows Keep main's error path for a directly raised provider error and only unwrap one found behind a transport error. Add the parser column to invalid JSON errors, and cover the real aiohttp chain in the _handle_error test * fix(vertex_ai): report every row conversion error as a 400 naming its line A message with no role, a non-string role, a bad temperature, or a litellm BadRequestError raised while converting a row still returned a 500, and the async upload lost the 400 for litellm's own exceptions. The stream now turns any error from one row's conversion into a 400 naming that line. _handle_error only unwraps an httpx.TransportError, which is what the aiohttp transport raises around a failing request body, so other callers keep their own status. A file that starts with a UTF-8 byte order mark now uploads, and provider messages start with 'Batch input file line N' like the proxy check. * fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line A chat row whose body is missing or not an object now raises a 400 naming its line instead of uploading a blank prompt, and a non-object body on line 1 no longer crashes the GCS object name lookup. Rows too deeply nested or too long to parse return a 400 from both the SDK and the proxy's own batch file check. JSON error columns now count leading whitespace. * fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line * test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos * fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact * test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Raushan kumar
|
f9d45f8a43
|
fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)
* fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kolade Fajimi
|
35970c3eac
|
fix(vertex): report an unreadable credentials file as a read failure (#40173)
* fix(vertex): report an unreadable credentials file as a read failure Tell a path apart from inline JSON by shape, not os.path.exists(). Fixes #40166 * test(vertex): narrow pytest.raises to the expected message * fix(vertex): log the credentials path instead of returning it to the caller * refactor(vertex): log the credential source decision where it is made * fix(vertex): handle paths open() rejects with a bare ValueError * fix(vertex): redact the credentials path before it reaches the log * test(vertex): cover the brace fallback and non-text credentials file * test(vertex): exclude the unreachable exhaustiveness guard from coverage * refactor(vertex): read the failure's fields off the matched object CodeQL's py/uninitialized-local-variable does not follow keyword capture patterns in a match statement and reported `path` and `reason` as possibly unbound. Narrowing on the class and reading the attributes says the same thing without a suppression. * fix(vertex): accept a read-only mapping in extract_aws_params * fix(vertex): keep encoded credentials out of the log and cover the legacy route * fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust * fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file * fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message * test(vertex): integration cells for credentials file and inline value failures * test(vertex): drive the Rust messages cell through Vertex Claude --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ErenAta16
|
bc58a5857b
|
fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema (#34653)
* fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema `translate_anthropic_tools_to_openai` assigned the caller's `input_schema` straight into `function_chunk["parameters"]`, then merged the tool's remaining top-level keys into that same dict. Because the assignment aliased rather than copied, keys such as `display_width_px` on a computer tool were written back into the caller's schema. That matters because callers reuse the tool list across passes: the pass-through guardrail translates the tools, and the request is then translated again for the real call. The first pass pollutes the source and the second forwards a schema carrying non-schema keys. Shallow-copying fixes it. `translate_anthropic_output_format_to_openai` in the same file already copies its schema for this reason, so this brings the two into line. Rebased onto current litellm_internal_staging; the original branch had gone to conflict on surrounding drift while the line it fixes is unchanged. * fix(anthropic/adapter): drop the inline comment and test the strict tool shape that Anthropic rejects * fix(anthropic/adapter): deep copy input_schema so downstream schema edits never reach the caller's tool A shallow copy still shared nested properties with the caller, and the Gemini, vLLM and watsonx bridges strip nested additionalProperties in place, so a fallback to Claude sent a strict tool Anthropic rejects. A deep copy also keeps a null or malformed input_schema as sent, as before this PR * fix(anthropic/adapter): deep copy the incoming tools so downstream edits never reach the caller's tools Copying only input_schema left the extra keys merged into parameters, such as input_examples, and OpenAI-format function tools shared with the caller, so Gemini's strict strip still edited the caller's tool before a fallback to Claude * fix(anthropic/adapter): keep translating tools that deepcopy cannot copy A tool passed as a frozen mapping, or with a schema nested past Python's recursion limit, made copy.deepcopy raise, so the request failed before it reached the provider. Copy each tool on its own and keep the original object when the copy fails, which is how these tools were handled before * fix(anthropic/adapter): copy tools through JSON so the Responses bridge and frozen tools keep the caller's schema * test(anthropic/adapter): parse stub bodies with json and cover the tool copy fallback * test(anthropic/adapter): close the connection after the malformed tools request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Chloe Lu
|
897bfea38d
|
fix(anthropic): keep signed empty thinking blocks on /v1/messages requests (#41362)
The request-side sanitizer on /v1/messages dropped every thinking block whose text is empty, even with a valid signature. Claude Opus 4.8 / Fable 5 / Fable 5.1 return exactly that shape by default (thinking.display omitted): the signature is the preserved reasoning. Replaying the assistant turn as returned therefore lost the model's reasoning on every multi-turn request, silently (the provider answers 200 without it), and left preserved-thinking enforcement (thinking.block_binding) nothing to bind. Anthropic accepts an empty thinking block that carries a signature and binds it to its prefix; only an empty block with no signature (the LIT-6357 case, a turn a non-Anthropic model served through the bridge) is rejected with "each thinking block must contain thinking". Use the signature-aware predicate is_empty_unsigned_thinking_block, which #38809 already applied on the emit side, for the request side too. The unsigned case is still dropped. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
3fec5075fa
|
feat(typesafe): support Jev 1.14 images, box questions, and long-context headers (#46596)
* feat(typesafe): support Jev 1.14 images, box questions, and long-context headers * fix(cost): drop the typesafe/jev-1.14.0 entry main already prices * fix(typesafe): send images only with the images header and box questions only to models that answer them TypeSafe takes assets and input_image parts only when the request carries TypeSafe-Danger-Zone-Images: true, so without it they follow drop_params like any other provider that cannot take images. Box support now comes from a supports_box_questions cost-map flag set on jev-1.14.0 and jev-preview, so jev-1.13.0 and jev-latest refuse or drop a box question before the call. A malformed provider_specific_header answers 400 instead of 500. * fix(typesafe): allow null box confidence and validate asset references and base64 * fix(typesafe): gate box questions on the images header, drop unreadable assets under drop_params, forward opt-in headers on the pass-through --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
efd8274315
|
feat(xai): bill grok voice realtime sessions (#46600)
* feat(xai): bill grok voice realtime sessions * chore: revert unrelated formatting churn in test_cost_calculator.py * refactor(xai): tidy realtime usage meter and tests * fix(xai): read push-to-talk mode from client session.update only * fix(xai): clamp decoded audio bytes and lock push-to-talk before content --------- Co-authored-by: kerry <kerry@berri.ai> |
||
|
Kerry Lu
|
0d741a9816
|
fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them (#46611)
* fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them * fix: satisfy the type discipline and basedpyright gates for the thinking block index * chore: sync the OpenAPI snapshot and schema.d.ts with the thinking block index field --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d6e6c6f26c
|
fix(responses): map Anthropic tool search to OpenAI server tool search (#46441)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models * fix(responses): define the tool_search_call output item locally for older openai SDKs * fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500 * fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out A replayed Claude tool search call that the caller answered with a tool message kept its tool_result in the user turn while its tool_use was dropped from the assistant turn, so Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when no tool message answers it. The Responses bridge now strips defer_loading from every tool whenever no tool search tool is sent, whether the caller sent none or the route cannot run it, since a deferred tool with no search tool can never be loaded. Namespace members had started carrying the flag to every bridge provider, which providers that reject unknown tool keys answered 422. The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit. * fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried * fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator * fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic A function tool whose allowed_callers is not a list of strings now answers 400 instead of 500. The guardrail scan reads tools without malformed flags, so the route decides between a 400 and a drop under drop_params. When the caller sends a native Anthropic search tool beside tool_search, only the native one goes out. Databricks Claude no longer counts as an Anthropic request route. A previous_response_id follow-up off Anthropic also drops a search call the client replayed as a function_call with a srvtoolu_ id, and its output. The answered tool id scan in the Anthropic message factory runs once per assistant run instead of once per message. A null created_by on tool_search_call and tool_search_output output items is left out of the serialized response, so a cache hit returns the same keys as the first call. * fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan The scan translated tool_search into Anthropic's regex search tool on every route, so a guardrail returning only function tools removed tool_search from native OpenAI requests and OpenAI answered 400 for the deferred tools * fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai * fix(anthropic): keep answered tool search replays paired with their answers * fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails * fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object * fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent * fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id * fix(responses): answer 400 to a tool_search execution that is not server or client OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed execution is dropped like the other malformed tool flags, which leaves a server search * test(responses): cover the 400 and drop_params paths for a malformed tool_search execution * fix(responses): map Anthropic tool search to OpenAI server tool search Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): sync generated tool search API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): type tool search mapping predicates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): validate tool-search mappings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): use the openai SDK ToolSearchToolParam for the mapped tool search * fix(responses): keep deferred chat tools beside a tool search on the responses bridge Restore the hand-rolled OpenAIToolSearchToolParam, since openai 2.20.0 (the supported floor) has no ToolSearchToolParam, and carry defer_loading from chat function tools onto the Responses request when a tool search tool is present, which OpenAI requires for tool_search * fix(responses): leave the bridge's defer_loading carry to its own PR #46439 owns keeping defer_loading on chat function tools sent through the Responses bridge, so this PR keeps only the tool search mapping and the openai 2.20.0 compatible tool type * fix(openai): keep tool search mapping to the OpenAI API and drop it where the model or history cannot take it * fix(openai): narrow the tool search drop to Anthropic's search tool, drop it when nothing is deferred, and run the wire tests against the OpenAI host * fix(openai): pass malformed tools to OpenAI as sent and keep a caller's own tool_search under drop_params * perf(openai): scan Responses history for Claude search calls only when Anthropic's search tool is sent * test(responses): cover the Anthropic tool search mapping to OpenAI Responses across routes, settings, and chaos * test(integration): define the fallback cell's deployments in the gateway config so both workers load them * test(integration): keep the worker-kill burst held past the replacement wait * fix(responses bridge): count an Anthropic search tool as hosted tool search on the chat bridge main's #46439 refuses defer_loading when no tool_search tool is in the request, and the chat bridge only counted {"type": "tool_search"}, so the Anthropic bm25 or regex tool this PR maps to OpenAI's tool search was refused before the mapping ran Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect the chat bridge to map the search tool now that defer_loading survives --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
d0fa05076d
|
fix(xai): name a completed xAI batch's results file and stop looking it up on xAI's files API (#45559)
* fix(xai): report a completed xAI batch's results as its output file details * fix(xai): keep a short results download and per-call timeouts out of the reported file details * fix(xai): describe a batch's results file from the batch alone and let the batch job fill its size Reading an xAI batch id as a file now costs one batch read and never downloads the results: the file reports a 0 byte placeholder, the results filename and the registration time. The managed proxy fills in the size the batch cost job measures when it downloads the results, keeps a known size when a provider reports none, and drops the per-process results download cache. * fix(managed_files): keep a measured batch output size when a refresh races the batch job A refresh that read the row before the batch cost job stored the measured size could save the provider's unsized details over it. The refresh now reads the row back from the writer and saves only while the row still holds what it read, retrying when another write landed in between. The new tests drive the real xAI and OpenAI retrieve paths through a fake HTTP upstream. * fix(managed_files): guard the refresh save with the row the request already read The conditional save first compares against the row the request read, so a refresh adds no database read unless another write landed in between; only then does it read the row back from the writer and try again. * fix(managed_files): save a refreshed file object with one guarded write and no extra database reads A refresh that loses its guarded write to a newer one now leaves that row alone and drops this worker's cached copy, so the next read loads the newer row instead of retrying against the database on the request path * fix(managed_files): retry the batch cost job's measured size after a read refreshes the row first The batch cost job re-reads the row from the writer and saves again when its guarded write loses to a concurrent refresh, so its measured size is never dropped. Reads without a measured size still make one write and no re-read * fix(managed_files): size a concurrently refreshed row instead of restoring the batch job's placeholder copy When the batch cost job loses its guarded write and the row it re-reads holds refreshed details, it applies its measured size to those details, so the results filename and any size the provider reported are kept * fix(managed_files): keep a batch output's earlier created_at when a refresh replaces its placeholder A refresh saved over a stored row now keeps the earlier of the two creation times, so an xAI placeholder keeps the time the proxy registered it instead of the read time, while a provider's own earlier creation time still wins * test(integration): an xAI placeholder row keeps its registration time after the proxy refreshes it * test(integration): drop the cached output row so the placeholder cell reads the seeded database row * test(integration): drop the cached output row without requiring it to still exist * fix(managed_files): limit the batch output size fill, guarded save and kept created_at to xAI rows * fix(xai): keep the batch id as the batch output filename Managed xAI batch output files keep the filename main already returns (the batch id) instead of renaming them to <batch id>_results.jsonl. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
c0663d5202
|
fix(anthropic)!: translate the OpenAI tool_search tool on /v1/chat/completions for Claude models (#46389)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models
* fix(responses): define the tool_search_call output item locally for older openai SDKs
* fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields
Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500
* fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out
A replayed Claude tool search call that the caller answered with a tool message kept its
tool_result in the user turn while its tool_use was dropped from the assistant turn, so
Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when
no tool message answers it.
The Responses bridge now strips defer_loading from every tool whenever no tool search tool is
sent, whether the caller sent none or the route cannot run it, since a deferred tool with no
search tool can never be loaded. Namespace members had started carrying the flag to every
bridge provider, which providers that reject unknown tool keys answered 422.
The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit.
* fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact
A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried
* fix(anthropic): translate the OpenAI tool_search tool on /v1/chat/completions for Claude models
* fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups
With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does
A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result
The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator
* fix(bedrock): answer 400 for the OpenAI tool_search tool on Converse unless drop_params is set
* fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic
A function tool whose allowed_callers is not a list of strings now answers 400
instead of 500. The guardrail scan reads tools without malformed flags, so the
route decides between a 400 and a drop under drop_params. When the caller sends
a native Anthropic search tool beside tool_search, only the native one goes out.
Databricks Claude no longer counts as an Anthropic request route.
A previous_response_id follow-up off Anthropic also drops a search call the
client replayed as a function_call with a srvtoolu_ id, and its output. The
answered tool id scan in the Anthropic message factory runs once per assistant
run instead of once per message. A null created_by on tool_search_call and
tool_search_output output items is left out of the serialized response, so a
cache hit returns the same keys as the first call.
* fix(anthropic): send one search tool and judge tool search on the real model for chat tool_search
Several OpenAI tool_search items now collapse into one tool_search_tool_regex
tool, since Anthropic answers 400 when two tools share a name. A native
Anthropic search tool the caller already sent wins over the OpenAI item.
Bedrock invoke and Vertex AI now decide tool search support on the real model
before tool-based response_format swaps in a stub model name, so a Claude 5.5
request with response_format no longer gets a 400 on a route that runs tool
search.
* fix(anthropic): name the proxy drop_params setting in the tool_search 400
* fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan
The scan translated tool_search into Anthropic's regex search tool on every
route, so a guardrail returning only function tools removed tool_search from
native OpenAI requests and OpenAI answered 400 for the deferred tools
* fix(anthropic): run tool search on azure_ai, gate it on Databricks Claude, and strip deferral without recursion
Anthropic's Foundry docs list no tool search exclusion for either hosting
option, so the claude-tool-search rule now fills azure_ai. Databricks Claude
models route their tools through the same unsupported-tool-search gate, so a
tool_search tool answers 400 or drops under drop_params instead of a 500.
_without_deferral strips two fixed levels instead of walking recursively.
* fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai
* fix(anthropic): keep answered tool search replays paired with their answers
* fix(anthropic): drop Claude tool search calls from history sent to other routes, name the fix in the 400
A second turn that carries Claude's srvtoolu_ search call and its result now has both removed
before completion() sends it to a route that does not build an Anthropic request, including the
responses bridge. The tool search 400 now names model_info supports_tool_search for a model the
cost map does not know, and bedrock/invoke for a Converse model that runs tool search there.
* fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails
* fix(chat): keep answered Claude tool search calls in history sent to other providers
Only a srvtoolu_ search call with no tool result is dropped before a
non-Anthropic route, so an answered pair keeps the caller's shape where
OpenAI and Gemini take it. The history check no longer copies the list
through a TypeAdapter.
* fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object
* fix(chat): drop unanswered Claude tool search calls only on routes that reject them
OpenAI, Azure OpenAI and Bedrock Converse answer 400 for a srvtoolu_ call
with no tool result, while Gemini answers 200. Keep the history as sent
everywhere else so routes that already work see what the caller sent.
* fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent
* test(integration): cover chat tool_search routing, history drop and load on the wire
Audit round 1 of 3 at
|
||
|
Mateo Wang
|
52fc37888b
|
fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444)
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): give the defer_loading casts a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): read replayed tool-search and namespace fields through typed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep tool_search items on a turn with no message or tool call Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep every reasoning item on a search-only turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): leave content unset on a search-only turn so replay adds no empty message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): replay tool search items only with tool_search, count replayed tools in the allowlist, keep namespace off guardrails * fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses Under drop_params, defer_loading is dropped when no tool_search tool is sent, so the 400 it would cause upstream goes away as before. A non-bool value answers 400 without drop_params and is dropped with it. A null tool-level value falls back to the nested function value. A request carrying a tool_search tool now bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort none, since chat completions rejects the tool_search type. * fix(responses bridge): replay tool search items in output order and count replayed tools in tool policy Store the ordered output prefix through the last tool search item, reasoning included, so a second reasoning item no longer makes OpenAI reject the replay. The shared chat-to-Responses converter now replays tool search items and namespace only when the bridge opts in, so Noma, the complexity router and guardrail payloads keep their shape. The tool policy guardrail and the allowlist count replayed tools only when the request can send them. * fix(responses bridge): forward defer_loading only beside a tool_search tool defer_loading only means something with tool_search, and backends without it reject the key (xAI answers 403). A true value with no tool_search tool now answers 400 naming drop_params and is dropped under drop_params; a false value there is a no-op and is dropped. * fix(tool_loop): keep tool search items and namespace between rounds * fix(responses bridge): check replayed tool search loads when a deployment adds tool_search * fix(chat): drop litellm response fields before strict OpenAI-compatible providers A chat reply from the Responses bridge carries provider_specific_fields on the message and on each tool call, plus reasoning_items. Groq, Fireworks, Mistral and Together reject unknown fields, so sending that reply back to them as history failed with a 400 or 422. Strip them in the shared helper and use it in those providers. * fix(caching): keep reasoning_items and tool search items on response cache hits A cached bridge response lost reasoning_items on a non-streaming hit and lost both reasoning_items and provider_specific_fields (tool_search_items) on a streamed hit, so the next turn could not replay the search or the reasoning around it * test(integration): cover tool search item replay on the chat to Responses bridge Proxy, SDK, authorization and chaos cells for returning and replaying tool_search_call and tool_search_output items and the tool call namespace * test(integration): add response cache twins for the tool search guardrail cells Each guardrail cell now has a twin where the answer is already in the Redis response cache: a key opted out of global guardrails caches a replay that loads a policy-blocked tool, the same replay without guardrails is cached before block mode sees it, rewrite mode is not served the unrewritten conversation's cached answer, and an allowed replay is served from the cache. * fix(responses bridge): forward defer_loading beside a client-executed tool_search The bridge now replays a client tool_search_call and the client's tool_search_output through provider_specific_fields.tool_search_items, so a deferred function beside a client tool_search keeps defer_loading like it does beside a hosted one * test(responses bridge): pin the client tool_search turn and its replay, streamed and not * fix(responses bridge): strip nested namespace for strict providers and keep streamed reasoning_items on text replies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep dropping defer_loading beside a client-executed tool_search Forwarding defer_loading beside a client tool_search made GPT stop with a search call the chat client has to answer by hand, so a plain question that used to get tool calls got none. Keep main's behavior of dropping it there. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nate Armstrong
|
23025c92ba
|
fix(anthropic): emit one tool_use block per call when a /v1/messages stream chunk carries several (#46492)
Providers like Mistral put every parallel tool call, with full arguments, into one
chat-completions chunk. The /v1/messages adapter opened one tool_use block for the
first call and concatenated every call's arguments into it, so clients got invalid
JSON like {...}{...}. _CombinedChunkSplitter now splits such a chunk into one chunk
per call. The first call streams live and calls opened behind it are held, gathering
their fragments by index, until a non-fragment chunk arrives or the stream ends
Fixes #44029
|
||
|
devin-ai-integration[bot]
|
314f17986a
|
fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models (#45882)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models * fix(responses): define the tool_search_call output item locally for older openai SDKs * fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500 * fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out A replayed Claude tool search call that the caller answered with a tool message kept its tool_result in the user turn while its tool_use was dropped from the assistant turn, so Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when no tool message answers it. The Responses bridge now strips defer_loading from every tool whenever no tool search tool is sent, whether the caller sent none or the route cannot run it, since a deferred tool with no search tool can never be loaded. Namespace members had started carrying the flag to every bridge provider, which providers that reject unknown tool keys answered 422. The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit. * fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried * fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator * fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic A function tool whose allowed_callers is not a list of strings now answers 400 instead of 500. The guardrail scan reads tools without malformed flags, so the route decides between a 400 and a drop under drop_params. When the caller sends a native Anthropic search tool beside tool_search, only the native one goes out. Databricks Claude no longer counts as an Anthropic request route. A previous_response_id follow-up off Anthropic also drops a search call the client replayed as a function_call with a srvtoolu_ id, and its output. The answered tool id scan in the Anthropic message factory runs once per assistant run instead of once per message. A null created_by on tool_search_call and tool_search_output output items is left out of the serialized response, so a cache hit returns the same keys as the first call. * fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan The scan translated tool_search into Anthropic's regex search tool on every route, so a guardrail returning only function tools removed tool_search from native OpenAI requests and OpenAI answered 400 for the deferred tools * fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai * fix(anthropic): keep answered tool search replays paired with their answers * fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails * fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object * fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent * fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id * fix(responses): answer 400 to a tool_search execution that is not server or client OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed execution is dropped like the other malformed tool flags, which leaves a server search * test(responses): cover the 400 and drop_params paths for a malformed tool_search execution * test(responses): cover a Claude search call replayed to Bedrock Converse with cache marks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4428c4c78f
|
feat(cost-map): add scx-ai DeepSeek-V4.1-flash, GLM-5.3 and Kimi-K3 (#46496)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: bhuvan2134686 <bhuvankakkar@gmail.com> |
||
|
ryan-crabbe-berri
|
6c78089d34
|
fix(bedrock): use base_model for Claude capability checks on application inference profile ARNs (#46569) | ||
|
ryan-crabbe-berri
|
e889d9d653
|
fix(bedrock)!: replay redacted_thinking on Converse and keep its data as Claude's own string (#46532)
* fix(bedrock): send redacted_thinking back to Converse as reasoningContent.redactedContent Replaying an assistant turn that carried a redacted_thinking block to Bedrock Converse turned it into an empty unsigned reasoningText, which the assistant content builder then dropped as blank text. Bedrock returned 200 so callers never noticed. Map redacted_thinking to redactedContent in both the thinking_blocks and inline content paths of the sync and async builders * fix(bedrock): drop blank redacted_thinking data and keep non-str signatures signed on Converse replay * fix(bedrock): replay redacted_thinking to Converse only when the target is Claude Nova, Llama, DeepSeek and gpt-oss reject reasoningContent.redactedContent with a 400, so a Claude history replayed to them through a Router fallback or model switch keeps dropping the redacted block the way main does. Claude is resolved from the Bedrock base model of the target, and an application inference profile ARN counts as Claude when its deployment's base_model names a Claude model. Adds integration contracts for the replayed Converse request shape, stream and non-stream, with a tool call, and for non-Claude and ARN targets. * test(e2e): cover Bedrock Converse replaying redacted_thinking live through the proxy Turn 1 triggers redacted thinking on Claude Sonnet 4.6, turn 2 replays it, and a live edge between the proxy and Bedrock checks the replayed Converse body carries the same reasoningContent.redactedContent. Covers nonstream, stream and a tool call round trip * fix(bedrock): validate litellm_params before reading the deployment base_model litellm_params arrives as an untyped dict, so passing it straight into the typed helper grew reportUnknownArgumentType. Validating it into Mapping[str, object] first keeps the read typed without growing the type-check gate * fix(bedrock): wrap non-Converse redacted_thinking data and let a named model id win over base_model * test(bedrock): cover redacted_thinking minted outside Converse, model_id and the sync Router base_model path * fix(bedrock): match bare claude- base_model names without reading the untyped cost map * fix(bedrock): keep redacted_thinking data as Claude's own string on every route Converse hands redactedContent back base64-wrapped, and that wrapper used to leak out as data, so a Converse turn replayed to Invoke or the Anthropic API was rejected. Converse replies now decode it, Converse requests wrap it exactly once, and the Anthropic-shaped builders unwrap histories saved with the old wrapping. Responses API encrypted reasoning is no longer sent to Claude on Converse, which rejected it with a 400 |
||
|
Mateo Wang
|
9ba4cc4fb3
|
fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440)
* fix(anthropic): keep tool_search_tool_result so the search call replays with its result Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): drop a client tool_result for a search call replayed with its server result * fix(anthropic): replay a server tool call the client answered as the client sent it * fix(anthropic): keep a server tool result under modify_params, count client-answered marks, and replay it from run_tool_loop modify_params no longer adds a placeholder answer for a server tool call whose result is kept, unless the call ends the request. The cache control hook counts a mark on a server tool call the client answered, since that call goes out as a plain tool_use. run_tool_loop sends provider_specific_fields back and runs only the client's tool calls. * fix(anthropic): keep a server tool result when another tool answer follows, scope run_tool_loop's replay, and keep code_interpreter_results off tool search turns * test(anthropic): audit cells for tool search result round trips, replay, cache marks, pass-through logging and chaos --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |