mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-12 08:42:18 +00:00
665 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
f296c26d4e |
Merge remote-tracking branch 'origin/main' into litellm_chatgpt_team_subscriptions
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
LiteLLM Rust DB / db-queries (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
# Conflicts: # litellm/router_python_core.py # tests/unit/llms/chatgpt/responses/test_chatgpt_responses_transformation.py # tests/unit/proxy/credential_endpoints/test_endpoints.py # tests/unit/test_router/test_router_helper_utils.py # tests/unit/test_utils.py # ui/litellm-dashboard/src/components/model_add/CredentialModal.tsx # ui/litellm-dashboard/src/components/model_add/CredentialsPanel.tsx |
||
|
devin-ai-integration[bot]
|
461b56b14b
|
fix(vertex_ai): bill gemini image text, thinking and priority tokens at their own rates (#51017)
* fix(vertex_ai): bill gemini image text, thinking and priority tokens at their own rates Co-authored-by: kerry <kerry@berri.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(images): bill unitemized output as text when no images return Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost_calculator): price image calls only by the tier the provider served --------- Co-authored-by: kenneth <kenneth@berri.ai> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
paveln83
|
9ebbcc7710
|
feat(anthropic): forward message threads on native /v1/messages (#50976)
* feat(anthropic): forward message threads on native /v1/messages Keep the thread body field on native Anthropic legs, add the message-threads-2026-08-12 beta for it, reject it with a 400 on legs that cannot continue a stored thread, and relay error.details of an upstream Anthropic error so thread_not_found reaches SDK thread helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(anthropic): keep a stale thread 404 from cooling the deployment and type the thread beta ignores Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: sort the cooldown handler imports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): dedupe the message-threads beta map rows and annotate test finals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): bind message threads to the key that created them and answer unsupported continues with thread_not_found Every virtual key on a proxy shares one Anthropic credential, so forwarding thread alone let any key continue another key's thread by its message id. A new default proxy hook binds each thread reply id to its virtual key, or to the signed-in identity when there is no key, and answers a continue naming an id it did not bind with the 404 thread_not_found the SDK replays on. Legs without message threads now serve a create without the thread and answer a continue with that 404 before any upstream call, instead of answering it with no history. * fix(proxy): keep a cached message thread reply with the key that got it first A response cache hit replays the original reply id, and binding it again handed the thread to the second key, so the creator's continue got thread_not_found. * fix(anthropic): keep message threads bound on every route and stable across turns Check the thread binding on every route that carries thread except pass-through, keep bindings in their own cache so they never evict rate-limit counters, add no tool or message derived betas on thread turns, answer a create on a leg without threads with a 400 thread_unsupported_request unless drop_params is set, and forward threads on Claude Platform on AWS * fix(anthropic): reject message threads on /v1/messages passthrough deployments A deployment that forwards /v1/messages untranslated to an OpenAI-compatible server inherited the native config's thread support, so `thread` reached a server that keeps no thread state. It now answers the same 400 thread_unsupported_request on create and 404 thread_not_found on continue as a translated leg * fix(anthropic): resend a thread's create betas on continues and check bound ids on pass-through A thread create now gets the betas LiteLLM derives from its tools and messages again, so an advisor tool create that answered 400 on the previous commit answers 200. Anthropic fingerprints the beta set, so the proxy keeps the anthropic-beta header the create sent upstream in the thread binding and adds it to every continue on /v1/messages. It reads that header from the upstream request, since success logging overwrites the logged request details before the success hook runs A continue on the /anthropic pass-through route of a thread the proxy bound to another key now gets the 404 thread_not_found. Ids the proxy never bound still pass through, as before * fix(anthropic): continue unbound threads off /v1/messages and keep every anthropic-beta on chat and Responses * fix(anthropic): skip non-string anthropic-beta list entries instead of raising * test(anthropic): keep the header merge regression test clear of main's appended tests * fix(anthropic): add a thread's create betas to the client's own on continues and keep base header merging on chat and Responses * fix(anthropic): answer an unbound thread id on chat and Responses with the 404 and drop the stale beta-join tests * fix(proxy): check a thread continue sent inside extra_body against its owner Azure AI promotes extra_body into the request body, so another key could continue a thread by putting thread there, past the ownership hook that read only the top-level field * fix(proxy): write a thread's first owner atomically and bind creates sent inside extra_body Two workers that both miss a reply's binding now agree on the first key stored in Redis, since the claim uses SET NX and each worker caches the stored winner. A create sent inside extra_body now binds its reply like a top-level create. The redundant Claude Platform supports_message_threads override is removed, since the provider already resolves to the native Anthropic beta list * fix(proxy): include agent_id in the key-less message thread owner * fix(proxy): tag message thread ownership Redis calls with their key family --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yujonglee
|
11af2eb22d
|
refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822)
* refactor(rust_bridge): route every Rust-to-Python call through host modules Rust now reaches Python only through litellm/rust_bridge/host/ (plus the execution driver), and no Python module outside host/ imports it. Library calls that reproduce Python behavior (httpx and OSError error text, huggingface_hub downloads, the random generator, os.environ) go through host functions too. The rule is written into host/AGENTS.md The live litellm.model_cost table, its version and every Python writer move to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only the functions Rust calls. The cost map fetch passes use_http_settings instead of handing Rust an http_settings() snapshot * refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust The Redis bridge asked Python for all of os.environ to read one key. CPython's os.environ writes through to the process environment, so Rust reads it with ProcessEnvironment like the rest of the module. Also drops doc comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): raise the native extension size cap to 55 MB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
joshua-berri
|
076ded32be
|
feat(auto_router): report a session's whole spend and name why savings stay unestimated (#51066)
* feat(auto_router): report a session's whole spend on /auto_router/session Claude Code sends its background calls straight to Haiku under the same session id, so those requests never reached the auto-router session rollup and the status line under-reported the session. Every billed spend row in a caller-supplied session now increments a key-scoped Redis counter whose one-hour expiry each write resets, and the endpoint reports it as total_spend, with spend unchanged. The increment runs where the spend row is accepted, so it counts each row exactly as often as the key's spend counters do. It reads as no total without a shared Redis * fix(auto_router): name the unsupported provider when a turn routes off Anthropic A turn routed to a non-Anthropic tier with no native wire response, such as a Cloudflare DeepSeek model, failed strict native validation and was logged as observation_unavailable, which reads like a broken capture. It now reports unsupported_selected_provider. Turns with a native wire response keep their existing reasons, and no estimate is added, so savings for these turns stay unknown * docs(auto_router): explain when session savings stay unestimated saved_spend on /auto_router/session now says a turn inside the longest prompt-cache TTL of the session's first observed turn, or of its last incomplete observation, is not estimated, so a null there is expected for that window * fix(auto_router): say why a system-only cache prefix has no savings estimate Claude Code puts its cache breakpoint on the system prompt, so the baseline prefix count sends Anthropic a body with no messages, which it rejects, and every turn read as token_count_unavailable. A count that fails only on system or tools prefixes now reports uncountable_system_or_tools_prefix. Gateways that accept the empty body still count it. A live check against claude-opus-5-5 showed a differential count with a placeholder message is off by -2 tokens for system-only and +68 for tools, so these turns stay unestimated and the savings gap for Claude Code is not fixed here * fix(auto_router): declare the session total's Redis key family and test its expiry The session total read and wrote Redis without a service target, so its calls showed up as bare redis spans. Both now run under session_spend_totals. The key no longer hashes the api key, which is already a hashed token, and only digests the session id. A new test drives Redis expiry on a test clock to show an active session keeps its total and an idle one restarts * docs(auto_router): note which turns are still estimated inside the history window Turns that took the identical baseline path, and turns whose whole cached prefix the session had already written, are estimated even inside the first cache TTL window, so saved_spend now says so --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
giladk1221
|
9ef5542339
|
feat(search): add Microsoft Web IQ provider (#50883)
* feat(search): add Microsoft Web IQ provider * fix(search): require HTTPS for Web IQ credentials * fix(search): include Web IQ in the packaged endpoint catalog * fix(search): register Microsoft Web IQ query pricing * test(search): fix Web IQ exception assertion lint * fix(search): type Web IQ response passthrough and search error headers The type-check gate failed on 8 new unknown-type errors in the Web IQ transformation. Build passthrough fields from a typed helper and validate the response models directly, and type the base search error headers as dict[str, str] so subclasses no longer inherit an unknown parameter. * fix(search): raise Web IQ parse errors without widening the base header type Typing the base search error headers as dict[str, str] made the shared HTTP handler report one more argument-type error. Restore the base signature and raise the same BaseLLMException directly from Web IQ. --------- Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> |
||
|
devin-ai-integration[bot]
|
53e1e9b87b
|
fix(vertex_ai): bill gemini image edits by token usage (#51038)
* fix(vertex_ai): bill Gemini image edit token usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(vertex_ai): validate image edit usage before normalization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kenneth <kenneth@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
8207a57d84
|
fix(responses): keep the namespace == name shape for deferred top-level tools (#50602)
* fix(responses): keep the namespace == name shape for deferred top-level tools The OpenAI Agents SDK matches a call to a deferred top-level function (defer_loading=True) only when the function_call carries namespace == name, the shape it reserves for deferred top-level tools. The Responses bridge returned no namespace, so after a native tool search the SDK failed with "Tool <name> not found". A client that adds namespace == name then hit the echo path, which joined it into "<name>__<name>" for the provider. - namespace_tool_name_map maps deferred top-level functions to (name, name), so non-streaming and streaming replies carry namespace == name - the function_call echo does not qualify a name whose namespace equals it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(responses): keep the qualified name for a namespace member that shares its namespace's name A namespace "search" with a member "search" is advertised as the chat tool "search__search" and comes back as name == namespace == "search". The echo rule from the previous commit skipped the join for every namespace == name, so that replay reached the provider as "search", a tool it was never given. The bare-name echo now applies only to the request's deferred top-level tools (deferred_top_level_names), and the reply map uses the same set. A deferred top-level name that a self-named namespace member also uses is left out of both directions, so the namespace member keeps its round trip. The lookup reads defer_loading is True instead of validating it, so callers that only read the input (token counting under drop_params) do not raise on a malformed flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(responses): keep the bare name for a namespace == name echo replayed without request tools Stored history (previous_response_id) is rebuilt with no request tools, so the echo of a deferred call carrying namespace == name used to join to name__name. Only a namespace that holds a member of its own name still joins. The lookup also tolerates a tools value that is not a list, and the comprehensions stay within one for and one if clause. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): read request tools through one validated accessor The deferred-name lookup and the response-side tool maps take the request's tools from one TypeAdapter-backed accessor, so a tools value that is not a list of objects reads as no tools and the type gate does not grow. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): type the request mapping handed to the input transformer transform_responses_api_input_to_messages takes dict[str, object] instead of a bare dict, so the tools accessor's argument is known to the type gate and the instructions value is narrowed to str. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): build the namespace tool map from the validated member names namespace_tool_name_map reads each namespace's members through the same validated accessor as the self-named check, which keeps the type gate's per-rule counts at their merge-base values. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): replay stored turns with the tools they were sent with Stored previous_response_id history is rebuilt with each turn's own tools, so a namespace member that shares its namespace's name keeps the joined name__name form on replay, while a deferred top-level call keeps the bare name. Adds the session handler regression. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep the object entries of a tools list and read them on the streaming path too One accessor hands the request's tools to the request path and the streaming iterator, dropping only the entries that are not objects instead of the whole list, so both paths see the same custom, namespace and deferred tools. Trims the explanatory comments. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): count input tokens with the request's tools in hand /v1/responses/input_tokens hands the request params, tools included, to the input transformer, so an echoed call for a namespace member named after its namespace counts as name__name there too. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): hand the request tools to every input conversion that reads an echoed call The guardrail provenance pass, the prompt caching pre-call check and the Rust precall token count convert Responses input with the request's tools, so a namespace member named after its namespace keeps name__name there as it does on the request path. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep namespace == name for deferred tools loaded by a client-side tool search A tool the client's own tool search loaded is still a deferred top-level tool to the Agents SDK, so the bridge keeps its deferral for the response-side name map and the route strips it again before the provider sees the request. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Homen Shum <317650355+homenshum-cheiron@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
norkelbyte
|
845f109524
|
fix(bedrock): forward x-pass- headers on the /bedrock/model pass-through routes (#50728)
* fix(bedrock): forward x-pass- headers on the /bedrock/model pass-through routes
The /bedrock/model/{model}/invoke, invoke-with-response-stream, converse and
converse-stream routes built the pass-through call from the request body, so
x-pass- headers never reached Bedrock. The processor now hands the caller's
x-pass- headers to the pass-through route, keeps them out of the logged body
snapshot, and on Converse drops the invoke-only guardrail headers when the
deployment sets guardrailConfig and the request metadata header when the proxy
owns request metadata
* fix(bedrock): bill the served service tier on invoke pass-through and scope x-pass- forwarding to the Bedrock routes
* test(bedrock): assert forwarded x-pass- headers on the request Bedrock receives
* test(bedrock): gate the router-model x-pass- forwarding on the request Bedrock receives
* fix(bedrock): bill a caller's forwarded invoke guardrail and keep x-pass- headers out of guardrail vendor payloads
* fix(bedrock): bill a caller guardrail on every invoke model and replace a trace value Bedrock rejects
* fix(bedrock): keep guardrail traces out of chat responses, forward the trace header as sent, and drop credential x-pass- headers
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
9fb29c7e38
|
fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions (#50467)
* fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions
An allowed_tools tool_choice in the flat Responses layout ({"type": "allowed_tools", "mode": ..., "tools": [...]})
answered 400 on litellm.completion, acompletion, Router and the proxy's chat and /v1/responses-to-chat routes. One
parser now reads both layouts into the nested Chat layout, and the existing Responses bridge flattens it back for
/v1/responses, so either layout reaches OpenAI in the shape each endpoint expects
* fix(tool_choice): drop an allowed_tools choice under drop_params and refuse conflicting flat entries
An allowed_tools tool_choice a provider can't enforce, or a malformed one, now drops under drop_params and answers 400 without it, the same rule LiteLLM uses for other unsupported values. That covers providers outside OpenAI and Azure, malformed choices on every provider, and Azure api versions older than 2023-12-01
A flat allowed_tools entry that carries both name and a nested function or custom key answers 400, since the two names can disagree, matching what flattening already refuses
* fix(tool-choice): keep the 400 when LiteLLM runs the tools, report auto after a drop
Under drop_params an allowed_tools choice a provider can't enforce was dropped, so on MCP gateway tools and in
run_tool_loop and arun_tool_loop LiteLLM could run a tool the allow list excluded. Those paths now keep the 400 they
answered before the flat layout was accepted, and a malformed choice stays a 400 on MCP gateway tools.
/v1/responses served through chat echoed a choice it had dropped, on sync, async and streamed responses. It now
reports auto, and a choice the provider received is still echoed as sent.
* fix(tool-choice): refuse an excluded tool right before LiteLLM runs it
Checking the provider up front missed a model alias, an old Azure api_version,
a litellm_proxy hop, and SDK fallbacks. The tool loop and the MCP gateway now
check the allowed_tools list right before they run a tool call, so an excluded
call answers 400 and never runs.
Under drop_params the choice now drops only when the caller runs every tool,
so a provider-run MCP server or built-in keeps the 400. The Azure checks read
the request's api_version, which also fixes the /v1/responses echo there.
* fix(tool-choice): drop the choice under drop_params and hand an excluded MCP call back unrun
The drop now applies to every provider that cannot take an allowed_tools
tool_choice, whatever the request's tools hold. LiteLLM never runs a call the
choice excludes: on MCP paths the whole round goes back to the caller in a 200,
the same shape as require_approval, and the tool loop keeps its 400 because it
only returns final content. A malformed choice stays a 400 before any request
on the tool loop and on chat with MCP gateway tools.
* fix(tool-choice): refuse an allowed_tools entry whose name is not a string
An mcp or other non-function entry with a list or dict name passed the
parser and crashed when the allow list was hashed. The parser now treats
it as malformed on both layouts, so it answers the same 400 as an empty
allow list, or drops under drop_params.
* fix(tool-choice): honor server-level and null-name mcp entries and stop hooks running excluded calls
* fix(tool-choice): match named mcp entries by server label and let built-in entries allow their interception call
* fix(tool-choice): leave a malformed allowed_tools choice unrewritten in the interception hooks
* fix(tool-choice): resolve mcp entries like the gateway and keep the choice on every hook path
Under drop_params a malformed allowed_tools choice now acts as if it was never sent, and only a choice that parses is stored and enforced. Without drop_params it answers 400 before any provider request, on /v1/responses with gateway tools before MCP listing
An mcp entry resolves its label the way the gateway picks servers to list (server name in any case, alias, access group, toolset), and a named entry matches the tool's own name or the gateway's prefixed name. The gateway rewrites mcp entries into function entries in the caller's layout and sends "none" when nothing is left
The web search follow-up keeps an allowed_tools choice, compression's litellm_content_retrieve is never excluded and joins the provider-facing choice, and the native /v1/responses hook loop applies the exclusion check and stores the choice
* fix(allowed_tools): stop web search and code interpreter follow-ups forcing another call
A kept allowed_tools choice now goes to the follow-up in mode auto, so a
required choice no longer loops to the agentic loop limit. The MCP label
scope dataclass uses a default_factory so the module imports on Python 3.11
* fix(allowed_tools): stop the Responses web search follow-up forcing another call
The Responses-shaped follow-up after a web search left tool_choice out of its patch, so the caller's required choice went out again through the chat to Responses bridge and looped to the loop limit. It now carries a kept allowed_tools choice in mode auto, like the chat follow-up
* fix(mcp): scope each MCP entry to its own allowed_tools and keep the allowlist on follow-ups
A label now covers only the tools its own allowed_tools lists, so a second label on the same server no longer widens it. Responses MCP follow-ups, streamed or not, keep an allowed_tools choice in auto mode instead of dropping it. The new Responses tests fake only the upstream MCP listing, so LiteLLM's own discovery runs
* fix(mcp): match an entry's allowed_tools against the tool name the gateway lists
A server with a display name override lists the display name, and the listing
filter matches allowed_tools against it, but the label scope only checked the
original name, so no spelling worked. The scope now also accepts the listed
name and its unprefixed part, like the listing filter. Tests now cover the
whole gateway and toolset entries and a display-named tool.
* fix(mcp): match an entry's allowed_tools only against a server's own prefix, not the first hyphen
* fix(mcp): match a tool no server resolves only by its exact listed name in an entry's allowed_tools
* fix(mcp): refuse a tool whose display name only looks like an allowed name, and let the chat follow-up answer
An MCP entry's allowed_tools now matches a tool only by its exact listed name or a spelling routing
accepts for its real name, so a display name that carries the server's own prefix no longer passes for
another tool. The chat MCP follow-up keeps an allowed_tools choice in auto mode, so the model can answer
with text after the tool ran.
* fix(mcp): match a toolset label's tools by exact native name
A toolset label served any tool whose name a routing spelling of a granted name matched, so a toolset granting the literal name "server-tool" also served the real tool "tool" that only another toolset grants. It now serves a tool only when the toolset grants its exact native name, the same check tool_is_granted uses when the gateway lists tools
|
||
|
devin-ai-integration[bot]
|
c319616c1f
|
Registry audit 2026-11-10, Together, Mistral, Fireworks, Azure AI and Copilot fixes (#51003)
Lowers Together Tev1 4B input price to the pricing page value, adds Mistral Leanstral vision and reasoning flags, and absorbs #50991, #50988 and #50958. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Kerry Lu <326648561+kerry-berri@users.noreply.github.com> Co-authored-by: zoveki <349028329+zoveki@users.noreply.github.com> |
||
|
mrinal-berri
|
9ba551b871
|
feat(proxy): let teams use their own allowed S3 buckets for batch files (#50571)
* feat(proxy): let teams send allowed S3 buckets for batch files and read them back A team's metadata can now list allowed_s3_buckets. Keys on that team may send one of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on POST /v1/batches, and the upload and batch-create handlers check the bucket against the team list plus general_settings.allowed_client_side_s3_buckets. The early auth body check still rejects bucket fields on every other route and method Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri through an internal stored_s3_bucket_name param, so content, retrieve, delete and the batch cost job can read files written to a request-chosen bucket. Callers can never send that param, and files in the model bucket resolve exactly as before * test(proxy): assert outcomes in team bucket tests and type the managed content read Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count * fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix * fix(proxy): keep the request's approved S3 bucket when merging deployment credentials Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none * fix(proxy): keep the request bucket only on upload and batch create, not on reads The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for * fix(proxy): only proxy admins can set a team's allowed_s3_buckets Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes |
||
|
devin-ai-integration[bot]
|
d47c03175b
|
fix(responses): keep unset-effort gpt-5.4/5.5 tool calls on chat completions (#50968)
* fix(responses): keep unset-effort gpt-5.4/5.5 tool calls on chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): explicitly route bridge fixtures to responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kenneth <kenneth@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
zoveki
|
901f914ba2
|
fix(responses): replay Claude tool search on follow-up turns (#50821)
* fix(responses): replay Claude tool search on follow-up turns Claude tool search on /v1/responses now returns a tool_search_output after each tool_search_call, streamed and not, listing the tools the search loaded as the caller sent them A stateless follow-up sends the search back to Anthropic as the server_tool_use and tool_search_tool_result Claude produced, and the Anthropic request drops a replayed search whose loaded tools are no longer sent, so a changed tool list no longer answers 400 The guardrail write-back for SDK output items reads part text through a typed helper, so the new output item type adds no attribute errors * fix(responses): emit Claude's tool search output in stream order The streamed tool_search_output item was announced at the end of the stream with an output index reserved earlier, so output_item.added events arrived as 0, 2, 1 and the OpenAI SDK's stream helper raised IndexError. The item is now emitted when Claude's search result arrives. * fix(anthropic): keep the caller's tool names in tool search results A tool name Anthropic cannot take gets rewritten on the request, so Claude's tool_search_tool_result named the rewritten tool. The response now maps those tool references back to the caller's names, streamed and not, and a replayed search result maps them forward again on the next turn. * fix(responses): replay Claude's tool search only on routes that run it A follow-up sent to a route that cannot run tool search, a Claude model without it included, now leaves out the replayed search the way it did before, instead of sending Claude search blocks that model cannot take. The stateless gate and the stored-turn filter now use the same check that already decides whether the tool_search tool itself is kept A streamed tool_search_call now gets its output_item.done when Claude's search result arrives, so it finishes before its tool_search_output. Clients that build history from done events, like Codex, kept the output ahead of its call * fix(responses): keep a replayed search when the caller sent Anthropic's own search tool A Claude model the cost map does not flag for tool search still runs it when the request carries Anthropic's own tool_search_tool_regex or bm25 tool, which is how turn 1 already decides to keep it. Both replay gates now follow that rule, so a stored follow-up on such a model keeps Claude's search as it did before, and a stateless one replays it * fix(responses): replay a BM25 tool search under its own name * fix(responses): keep a replayed tool search visible to guardrails and prune only missing tools A replayed Claude tool search now reaches pre-call guardrails and the Purview DLP scan with its search arguments, a stale search drops only the tool references the follow-up no longer sends, and the stored previous_response_id path keeps the base gate for routes the cost map does not flag * fix(responses): replay Claude's search on a stateless follow-up only when it still sends a search tool A store false follow-up that sent no search tool still got the search replayed, and Claude then said it called a tool it no longer had * fix(responses): replay a stored Claude search only when the follow-up still sends a search tool A previous_response_id follow-up that sent no search tool still got the stored search, the same gap a store false follow-up had. The stale search pruning now logs the tools it leaves out at debug level. handler.py reads the reported speed from the usage chunk, and the guardrail write-back skips tool search outputs, so the branch adds no type errors. * fix(responses): leave out a stored Claude search kept in server_tool_calls Main stores Claude's server-side tool search on the assistant message in provider_specific_fields.server_tool_calls and tool_results, not in tool_calls. The previous_response_id path only looked at tool_calls, so a follow-up that sent no search tool still replayed the stored search. The session handler now finds the search in either place and drops it from both server_tool_calls and tool_results when the follow-up cannot run a search. * test(responses): give the replayed search guardrail fixture its own call id The guardrail test reused toolu_01Weather, an id that streaming and bridge tests already put in the process-wide TOOL_CALLS_CACHE. When one of those ran first in the same worker, the guardrail write-back took its rebuild path and the test failed for a reason unrelated to the replayed search. * fix(anthropic): keep Claude's replayed tool search on renamed tools and stored follow-ups A search-only assistant turn now gets its search result renamed to the sanitized tool names too, so the stale-tool trim no longer drops it. Stored follow-ups on Claude routes replay the search as base did, trimmed to the tools still sent, and Bedrock invoke sends the tool search beta whenever the history holds a search * test(responses): pin that guardrails scan a replayed Claude search on a follow-up without a search tool * fix(anthropic): keep Anthropic's tool names in chat tool search results A chat history whose search result used the request's tool names (weather.get) got Anthropic's 400 when replayed on a proxy still on the old code, during a rolling deploy or a rollback. Chat completions now returns Claude's search result as main does, and the Responses bridge maps a rewritten name back to the request's tool when it builds tool_search_output. That also fills tool_search_output on Bedrock invoke streams, which carry no name map --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
norkelbyte
|
c90b5e057f
|
fix(openai): download PDF URLs off the event loop and transform async chat messages once (#49176)
* fix(openai): transform async chat messages once, whatever configs were built before * fix(openai): keep tool-result PDF downloads off the event loop and stop Mistral rewriting retried messages * fix(openai): pre-download PDF URLs into the sync transform so async requests match sync Async chat requests now download every PDF URL in user and tool messages at once, then run the sync transform_request on a worker thread that reads those downloads. The request, filename, logged messages, and download count match a sync request, and a failed download raises the sync error with no second attempt. DeepSeek joins the configs that flatten content and skip the up-front download. * fix(openai): skip up-front PDF downloads past a file part the transform rejects * fix(openai): keep system-message PDF downloads off the event loop and stop downloads after a dead PDF URL * fix(openai): let PDF downloads after a dead URL finish and retry rate-limited ones one at a time Cancelling in-flight downloads after a dead PDF URL could abort a proxy worker on uvloop, so the downloads now run to the end in the background. A URL whose up-front download got a 429 downloads again inside the transform one at a time, as on main. A content part whose type is not a string is no longer read as a PDF part, so it reaches the provider instead of raising a TypeError. * fix(openai): download a request's PDF URLs one at a time and stop at the first dead one * fix(openai): prefetch the PDF URLs the sync transform downloads, after role mapping and up to the first unreadable part --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
7bcc46246b
|
feat(cost-map): rolling registry update (claude-mythos-5-2, Azure Mythos 5.2, Friendli GLM-5.1 date, GovCloud Fable 5.2 token counting) (#50777)
* fix(cost-map): drop unsupported Mantle token_count_model from the Bedrock GovCloud Claude Fable 5.2 row * feat(cost-map): add claude-mythos-5-2, absorb Azure Mythos 5.2 rows and Friendli GLM-5.1 deprecation Co-authored-by: Lee-Si-Yoon <76679207+Lee-Si-Yoon@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Lee-Si-Yoon <76679207+Lee-Si-Yoon@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> |
||
|
Krrish Dholakia
|
9e80943c13
|
fix(hosted_vllm): omit empty tools list from chat requests (#50812)
* fix: omit empty tools from chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Fix empty tools only for hosted vLLM chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yujonglee
|
523b3efea2
|
refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501)
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
tavren27
|
cd9f98d095
|
fix(anthropic-messages): apply additional_drop_params before the unsupported param checks (#50753)
* fix(anthropic-messages): apply additional_drop_params before the unsupported-param gates * fix(anthropic-messages): apply additional_drop_params before the unsupported param checks Share the tolerant drop list parser between the Python handler and the Rust host route, clear listed params before the Rust gate, and keep the prompt cache baseline body in step with the real request * fix(anthropic-messages): leave additional_drop_params keys out of the Rust host body * test(anthropic-messages): annotate the new test locals with Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
0021d5493c
|
fix(bedrock): answer 400, not 500, for malformed count-tokens requests (#50708)
* fix(bedrock): answer 400, not 500, for malformed native count-tokens requests The Bedrock count-tokens handler now checks the body before any AWS call and raises a 400 BedrockError for a missing or empty messages list, a message without role or content, or a missing model. The token counter marks that rejection as local, so it skips the Mantle fallback and the framing estimate, and /v1/messages/count_tokens answers invalid_request_error get_bedrock_base_model returns a bare cross-region prefix such as us or global unchanged instead of raising IndexError, so Bedrock judges the model id and its own 400 reaches the caller * fix(bedrock): answer 400 for a malformed message on Mantle token counts * fix(bedrock): limit the region-only model id fix to token counting * fix(bedrock): count a routing-prefixed region-only id and leave other providers' count errors as they were --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Ted Swinyar
|
6a232c90e0
|
fix(ollama_chat): give each streamed tool call its own index (#49183)
* fix(ollama_chat): give each streamed tool call its own index Ollama streams parallel tool calls one per chunk with no top-level index, and the delta numbers the calls of each chunk from zero, so every call left the proxy as index 0 and OpenAI-style clients appended the arguments of each later call onto the first one. Number the calls with a counter that lives for the whole stream instead. Fixes #33678 * fix(ollama_chat): answer 400 when replayed tool call arguments are not JSON A chat history that already holds a merged tool call from an older stream used to fail with a 500 APIConnectionError. It now answers 400 naming the tool call. The streaming tests use Ollama's real tool call shape and drop a case the merge base already passed. * fix(ollama_chat): parse replayed tool call arguments with the shared repair parser Empty, null, and truncated arguments in a replayed assistant tool call now go through parse_tool_call_arguments, so they are forwarded as an object instead of answering 500. Arguments that still are not a JSON object answer 400 naming the call id. * fix(ollama_chat): send {} for null replayed arguments and keep argument text out of the repair warning Replayed tool call arguments of the JSON string "null" used to reach Ollama and return 200. The shared parser turns them into None, so they now go out as {} like empty or missing arguments instead of answering 400. The parser's repair warning logged up to 200 characters of the arguments. It now logs only the tool name, context, and length. * fix(ollama): keep replayed tool arguments out of a rejected replay's error chain * fix(ollama): keep the repair warning's argument preview unless message logging is off The shared repair warning lost its 200-character argument preview for every provider. It now logs the same preview as before by default, and logs redacted-by-litellm instead when turn_off_message_logging is on. * test(ollama): cover short and boundary-length repair warning previews --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Jeevan Babu Gotru
|
9de20b6cf8
|
fix(minimax): forward reasoning_effort instead of dropping it (#49583)
* fix(minimax): forward reasoning_effort instead of dropping it * fix(minimax): forward reasoning_effort instead of dropping it * fix(minimax): gate reasoning_effort on the cost map and map MiniMax-M3.1-Flash Forward reasoning_effort for every MiniMax model unless its cost map entry says it does not reason, so a MiniMax model newer than the map keeps the caller's effort instead of losing it. Add a minimax/MiniMax-M3.1-Flash entry with MiniMax's published pay-as-you-go pricing and its accepted effort levels (low, medium, high, xhigh, max), and a minimax-reasoning-baseline fallback rule that marks unmapped minimax/MiniMax-M* ids as reasoning-capable, so thinking is kept for them too. * refactor(minimax): gate reasoning_effort on supports_reasoning like thinking Forward reasoning_effort when the cost map, or the MiniMax M-series fallback rule, says the model supports reasoning, the same check thinking already uses. An unlisted MiniMax model outside the M series now drops it, with the existing drop warning. * refactor(minimax): build supported params in one expression and type the new tests Build the MiniMax supported params list without mutating it, give the new test helpers precise types, and assert invariants (both cost map files agree, prices are positive and ordered, every declared effort level is forwarded) instead of pinning MiniMax's prices and effort levels as literals. * fix(minimax): treat every unlisted minimax/ model as reasoning-capable Widen the MiniMax fallback rule from M-series ids to the whole minimax/ namespace, matching the W&B rule, so an unlisted MiniMax model keeps reasoning_effort and thinking. Exact cost map entries still win, so mapped speech, video and image models are unaffected. --------- Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> |
||
|
ryan-crabbe-berri
|
dfb1aac8b1
|
fix(cost)!: bill transcriptions by the seconds the provider reports (#50727)
* fix(cost): bill transcriptions by the seconds the provider reports * Update litellm/cost_calculator.py sure Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
zoveki
|
1c807aef19
|
fix(prompt_templates): keep moved private helpers importable from common_utils (#50721)
* fix(prompt_templates): keep moved private helpers importable from common_utils The split moved 83 private helpers out of common_utils, so code importing them by their old path got ImportError. A module __getattr__ now serves each one from its new module. Also renames the new test files to the directory's naming convention * fix(prompt_templates): hide the old-path getattr from type checkers and serve only released names Type checkers stop seeing the module __getattr__, so a typo'd import from common_utils fails lint again. The old path now serves only the 69 moved helpers that common_utils had in a tagged release, since the other 14 were never importable from it * fix(prompt_templates): let patches through the old common_utils path reach moved helpers and warn on old-path use * fix(prompt_templates): sync old-path patches of re-exported helpers and restore moved helpers when warnings are errors * fix(prompt_templates): reset a moved helper on old-path delete instead of removing it, and warn on old-path writes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
quietjules
|
001126cd1c
|
feat(proxy): add OpenSearch index pass-through with per-key read/write grants (#50573)
* feat(proxy): add OpenSearch index pass-through with per-key read/write grants Adds /opensearch/<index>/<api> for LiteLLM vector store indexes on OpenSearch stores. A key or team needs read or write on the index in allowed_vector_store_indexes, and the proxy signs each call with the vector store's AWS identity, so callers never hold AWS credentials The route maps the LiteLLM index name to the store's provider index, refuses any path, query parameter, or body field that could reach another index, and sends the exact signed bytes upstream with only the signed headers * fix(proxy): let OpenSearch _bulk and _msearch NDJSON bodies through auth opensearch-py sends _bulk and _msearch NDJSON with content-type application/json, and the auth-time body read parsed it as one JSON document and answered 400 before the route ran. Auth now skips parsing OpenSearch NDJSON bodies the same way it skips Azure Speech audio. * feat(proxy): accept a _bulk _index naming the path index and allow named search pipelines A _bulk action line may name the index in the URL path, and LiteLLM rewrites it to the real index, which is what opensearch-py's helpers.bulk sends. The search_pipeline query param now goes through so hybrid search works, while an inline pipeline stays refused. Also types the type-checking Router stub with _router_init's parameters again, which make lint's tests/e2e check needs since the Rust core change. * fix(opensearch): answer 401 when the AWS credential source cannot read its token file * refactor(proxy): check OpenSearch request bodies without recursive helpers * fix(opensearch): answer 401 when AWS credentials fail to refresh at signing time * fix(opensearch): keep auth off OpenSearch bodies and widen the read list to index checks Auth no longer parses any OpenSearch request body, so a document field named model, api_base or base_url is written instead of answering 401 or 403. GET and HEAD on the index and HEAD on _doc and _source are reads, each operation checks the exact grant the list names, an index whose store name two stores share answers 400, and an _msearch header that names the path index is sent to the mapped index * fix(opensearch): need read for writes that return documents and keep document fields out of LiteLLM params * fix(opensearch): keep documents out of LiteLLM hooks, keep main's credential statuses, make _msearch admin only, and bill only searches * fix(proxy): bill only an exact /opensearch/<index>/_search path A document read such as /opensearch/docs/_doc/_search ended in /_search and was counted as a billable search. The check now matches the index segment and _search exactly * fix(proxy): only proxy admins set vector store index grants, and bill no OpenSearch document call An index grant now opens document writes through /opensearch, so key generate, update, bulk update, regenerate, team new and update, /user/new, and bulk user creation answer 403 when a non-admin changes allowed_vector_store_indexes, top-level or under metadata. Resending the stored grants is not a change An /opensearch path is matched before the LLM route suffixes, so a document id such as search or embeddings never makes a document call count as a billable request * fix(proxy): treat an empty index grant list from a non-admin as a change An explicit empty top-level allowed_vector_store_indexes passed the admin-only check as unchanged and then replaced the stored grants. It now counts as a change, so a non-admin gets 403. The new grant tests also carry parameter types --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tavren27
|
2f430873ae
|
fix(bedrock): derive the web identity session policy per AWS caller (#50592)
* fix(bedrock): derive the web identity session policy per AWS caller Each AWS caller (Bedrock embeddings, batches, files, SageMaker, Polly, S3 Vectors, and the S3 and SQS loggers) now sends its own STS session policy on AssumeRoleWithWebIdentity, so a role that allows those actions is no longer capped by the Bedrock-only policy. The default Bedrock policy is unchanged, and the credential cache key includes the policy so callers never share a narrower session. * fix(bedrock): let async invoke write its S3 output under the web identity session policy Bedrock writes an async-invoke job's output with the caller's session, so the embedding session policy now allows s3:PutObject only when Bedrock makes the call (aws:CalledViaLast bedrock.amazonaws.com). * fix(opensearch): give OpenSearch vector store search its own web identity session policy * fix(bedrock): give every remaining AWS caller its own web identity session policy Secrets Manager, knowledge base ingestion, model listing, the S3 config file reader, the SageMaker chat handler, and the Comprehend Medical and Transcribe pass-throughs each send a session policy with only the actions they call Bedrock embeddings now use the Bedrock policy, so chat and embeddings share one STS session. That policy adds async-invoke status, S3 reads and writes only through Bedrock, KMS only through S3, and Mantle token counting bedrock:InvokeModel gets its own statement without aws:SecureTransport, since Bedrock checks a Nova Sonic realtime stream as InvokeModel with that key false KMS grants are limited by kms:ViaService, file downloads get a read-only S3 policy, and batches and speech config export skip underscore attributes * test(batches): expect config export to skip every underscore attribute The session policy class attribute is private, so batches config export no longer returns single-underscore names * fix(bedrock): scope knowledge base ingestion role grants to the roles it creates Knowledge base ingestion only creates and passes litellm-bedrock-kb-* roles, so its session policy no longer lets the session create, edit, or pass any other role, the proxy's own included * fix(bedrock): give embeddings and the agent-runtime pass-through their own web identity session policies * fix(bedrock): grant enforced guardrails and the remaining agent-runtime operations over web identity Every web identity ceiling that grants bedrock:InvokeModel now also grants bedrock:ApplyGuardrail, so an account or organization enforced guardrail no longer denies embeddings, the agent-runtime pass-through, or realtime The agent-runtime ceiling grants every POST operation the pass-through forwards, adding flow executions, document content, and ingested document ACLs, and a test derives that list from botocore Only the realtime caller gets bedrock:InvokeModel without the aws:SecureTransport condition, since chat and embeddings never open a bidirectional stream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
432efc9b2d
|
fix(anthropic): exchange Google identity tokens again on refresh and read oidc/ references out of ANTHROPIC_IDENTITY_TOKEN (#50583)
A Google metadata-server identity token carries no jti, so Anthropic lets it be exchanged more than once. The shared on-disk token store still answered a mandatory refresh with the stored token minted from that same assertion, which was the almost expired token the refresh existed to replace. Google references now skip the store, so every refresh is a real exchange and a Kubernetes token file keeps today's cross-worker reuse. ANTHROPIC_IDENTITY_TOKEN was always wrapped as oidc/env/ANTHROPIC_IDENTITY_TOKEN, so a value that was itself an oidc/google/ reference was posted to Anthropic as the literal assertion text. A value starting with oidc/ is now the reference, with oidc/env_path/ still rejected, and a raw token in the variable resolves exactly as before. Fixes #50553 Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
c5ad400f0b
|
fix(anthropic): refresh the WIF bearer token on the connection error retry (#50570)
* Refresh the WIF bearer token on the connection error retry * Only refresh the bearer when the retry goes to the host the token was minted for --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
02d610f1a9
|
fix(anthropic): disable optional default thinking for reasoning effort none (#50541) | ||
|
Kerry Lu
|
9ec3b5fd8b
|
feat(cost-map): move -latest model aliases into the cost map (#50491)
* feat(cost-map): move -latest model aliases into the cost map Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: load the finalized local cost map in the moonshot alias test and assert alias pricing by behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: compare alias cost map rows to their targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(cost-map): give each -latest alias its own alias_of row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): keep alias_of rows out of alias lists and add them to the registry schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit |
||
|
DDT
|
d0c16008c8
|
fix(chatgpt): round-trip encrypted reasoning through /v1/messages (#49076)
* fix(chatgpt): round-trip encrypted reasoning through /v1/messages The chat bridge stored encrypted reasoning on reasoning_items, the Anthropic adapter never read that field, stream chunks that only carried thinking blocks were dropped, and the messages adapter stripped the signed blocks before a chatgpt request could turn them back into reasoning items. Co-authored-by: DDT <1786035110@qq.com> * fix(anthropic): keep Responses reasoning on tool turns and retry once when a backend cannot decrypt it * fix(anthropic): stop logging rejected encrypted reasoning Quote the retry helper return type so importing litellm no longer evaluates a string alias with |. The warning keeps provider, model, status, and error code, and leaves the ciphertext out. * fix(anthropic): keep encrypted reasoning on OpenAI thinking and xAI web search turns, and stream signatures only to Messages clients The keep check now knows when the handler moves an OpenAI thinking turn to Responses, and when web_search_options sends an xAI turn there. The Responses-to-chat stream iterator adds the signed reasoning chunk only when the caller is /v1/messages, so /v1/chat/completions streams stay unchanged. The reasoning retry fires only when the translated request still carries encrypted reasoning. * fix(anthropic): keep each streamed reasoning summary with its own item, and read the Messages marker before the bridge call * fix(anthropic): send openai/ Messages turns again once without encrypted reasoning a backend rejects The openai/ Responses path on /v1/messages had no retry, so a chatgpt/ block left in history answered 400 on every later openai/ turn. That path now sends the turn once more without encrypted reasoning, the same way the chat bridge does. The retry matches only invalid_encrypted_content or a "could not be verified" message naming the content, so the exception mapper's own hint no longer triggers it. The xAI web search keep path is backed out, since xAI sends no encrypted reasoning over the bridge. * fix(anthropic): keep every non-streamed reasoning item for Messages clients and read a deployment's reasoning_summary in the keep check * fix(anthropic): log only the outcome of a turn sent again without rejected reasoning A /v1/messages turn whose replayed reasoning the backend rejects is sent once more without it. The rejected first attempt ran the failure callbacks on the shared logging object, which logged a failure for a turn that then succeeded, kept the retry's own failure from being logged, and freed the parallel-request slot while the retry was still in flight. The first attempt now runs inside resent_on, and the client wrapper skips failure logging and the budget unbind for a failure that matches it. The retry runs outside that context, so its own outcome logs as usual. * test(anthropic): type the parameters of the failure-recording test loggers * fix(anthropic): skip the SDK num_retries retry for an attempt that is sent again * fix(anthropic): stop SDK retries from resending a rejected block or clearing the global num_retries A transient 429 or 500 on a /v1/messages turn that carries a replayed encrypted block was retried by an SDK call's num_retries loop, which sent the block again until every retry was rejected. On async calls the loop then raised the first transient error, so the turn failed instead of being sent again without the block. The retry loop now stops at that rejection and passes it on, so the turn is sent again once without encrypted blocks. A rejected first try on a sync call also cleared the process-wide litellm.num_retries, so later calls in the process were no longer retried. That setting now stays as configured. * fix(anthropic): stop a cancelled SDK retry from starting another call The resent-failure retry predicate returned true for BaseException subclasses such as asyncio.CancelledError, so a cancelled acompletion_with_retries or aresponses_with_retries call could start another provider call. Retry only Exception subclasses again, as tenacity's default did, while still skipping a resent failure. * test(anthropic): type the retry helper parameter in the cancellation test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mukesh Dewangan
|
4ed339b2fc
|
fix(vertex_ai): map missing image on Gemini image edit to BadRequestError (400) (#50244) | ||
|
Mateo Wang
|
709b27e268
|
fix(anthropic): preserve streamed tool argument fragments verbatim (#50470)
* fix(anthropic): preserve streamed tool argument fragments verbatim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): send the schema-less stream test the null properties form providers accept * fix(anthropic): unwrap values when a schema-less stream sends its whole tool input in one piece * fix(anthropic): unwrap only a one-piece values input and never emit Infinity in schema-less streams * fix(anthropic): keep JSON mode answers verbatim when they cannot be written back as strict JSON A tool input holding a number past a float or an integer past Python's digit limit now passes through as the model wrote it on the stream and non-stream paths, instead of coming back with Infinity or raising. One-piece values rows for false, null, a plain object and too-deep nesting pin the remaining unwrap branches. * fix(anthropic): reject non-finite floats through pydantic instead of a raising parse hook * fix(anthropic): keep split JSON mode answers whole when a field cannot be written back as strict JSON * fix(anthropic): keep out-of-range numbers as the model wrote them in JSON mode answers Decode tool input with parse hooks that keep a number past a float or an integer past the digit limit as its source text, and write it back unchanged, so held, wrapped and name and value split answers stay whole instead of falling back to raw blocks. * fix(anthropic): keep numbers that round to zero as the model wrote them in JSON mode answers * refactor(anthropic): encode JSON mode answers without a recursive helper * fix(anthropic): decode deeply nested JSON mode answers without slow recursive validation * fix(anthropic): type the decoded JSON mode answer as object and test non-object answers json.loads output now goes through the existing TypeAdapter(object), which returns it unchanged, so no Any reaches the object guard. New stream and non-stream cases check that a list, string or number answer comes back as written. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Ankit Jha
|
72228db5c6
|
fix(caching): don't cache or replay a reply that has no output (#50057)
* fix(caching): don't cache an Anthropic Messages stream whose content blocks are all empty
A non-Anthropic model routed through the Messages adapter turns an empty reply into a single empty text block. The no-output check counted any content_block_start as output, so the async stream cache stored that empty reply and replayed it on every repeat. Judge each content_block_start and content_block_delta by its payload instead, like chat streams judge a choice: a block carries output when a field other than type holds something besides "", [] or null. Empty replies are no longer stored and stored ones become misses. A bare {} is not treated as empty, since the only block fields that hold one (tool_use input) come with an id and a name
Fixes #49931
* fix(caching): treat thought-signature-only and non-streamed empty replies as no output
* fix(caching): treat whitespace-only replies as no output
* fix(caching): keep whitespace replies and text completion fills cached
* refactor(caching): drop the unused streamed-choice branch of the output check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
HootRock
|
8f19607259
|
fix(proxy)!: preserve Responses SSE event framing (#47980)
* fix(proxy): preserve Responses SSE event framing
* style(proxy): format Responses stream terminator condition
* fix(proxy): validate Responses guardrail event names
* fix(proxy): keep Responses first-event errors and keepalive failures in Responses framing
* test(proxy): use controlled clocks for Responses keepalive failures
* fix(proxy): end late Responses failures with an error event the SDK raises on
Errors that arrive after keepalive pings, and guardrail error strings yielded
mid-stream, now end /v1/responses with a named `error` event carrying the
same error object chat sends, so the OpenAI SDK still raises APIError with a
clean message and provider_specific_fields. Background polling passes the
same framing and records an `error` event as a failed response. The first
data line of a frame is found by splitting on newlines only, so a U+2028 in
an error message no longer hides a first-event error.
* fix(proxy): match OpenAI's Responses error event and stream late passthrough blocks as block responses
* fix(proxy): keep the reason when a background Responses request fails
A background poll of a request a post-call guardrail blocked saved a
null code and message, because the guardrail's error event nests them.
A request whose first chunk was an error polled as completed with no
error. Both now poll failed with the same code and message a late
failure gets.
* test(proxy): type the first-chunk poll test parameters
* fix(proxy): keep first-chunk error detection to lone error frames and handle string poll errors
* fix(proxy): keep the HTTP status on late Responses errors and end Responses streams on error events
A late error body without a code now takes the response's status, an upstream error event counts as the stream's terminal event, and a Responses stream that fails before its first chunk ends with an error event instead of data: [DONE].
* test(proxy): type the first-chunk failure test's parameters
* fix(proxy): keep cursor chat framing and JSON-encode Responses hook errors
* fix(proxy): encode non-dict Mapping Responses chunks as JSON objects
* fix(proxy): encode mappings and models nested in Responses stream chunks as JSON objects
* test(proxy): freeze the hook-output fixture
* fix(proxy): keep masked mappings and odd keys safe in Responses stream frames
Only read-only dict views and pydantic models nested in a hook chunk or error event expand into JSON objects. Other mappings, such as httpx.Headers, keep their own string form so masked values stay masked. Keys JSON cannot take become strings, and a chunk JSON still cannot encode goes out as the plain data frame it was before.
* fix(proxy): never fail a Responses stream frame on a value JSON cannot encode
Only dicts and read-only dict views expand into JSON, at the top level too, so a header mapping yielded as a whole chunk keeps its masked string form. Any failure while encoding a chunk falls back to the plain data frame the base sent, and an error event that cannot encode still goes out with its code and message, so a late block never drops the connection.
* fix(proxy): keep what a nested value's repr hides out of Responses stream frames
* fix(proxy): print values in Responses stream frames the way each base path did
* fix(proxy): send string and number subclasses whose repr shows their value as that value in Responses stream frames
* fix(proxy): print container subclasses and SDK models in Responses stream frames the way base did
* fix(proxy): read hook container storage the way repr does and send error-event models as str()
* fix(proxy): frame streamed Responses reads and send unreadable hook values as base did
Track pre-framed and raw SSE frames so a late failure never follows a terminal event, accept CRLF and CR line breaks in a first named error, frame GET /v1/responses/{id}?stream=true as Responses events, print a whole hook proxy chunk once and fall back to its text, ignore a non-integer status_code in a hook error frame, read a late-block dict subclass through items() when that view differs from its storage, and keep the order and inline form base's repr printed for nested mappings
* fix(proxy): match base on Responses stream edge cases
Read a decimal-string or integral-float status_code as its integer again, so
"429" and 429.0 still map to rate_limit_exceeded. Print each hook value's repr
once and reuse it, so a repr that changes between calls cannot reach the
client. Track only pre-framed sequence numbers from 0 to 2**63-1, so a huge
one cannot break the failure event. Build nested JSON with one frame per level
again, which restores the nesting depth sent as JSON. Count a pre-framed frame
as terminal from its data type, and a named error frame only when its data
carries an error, the way the OpenAI SDKs read them.
* fix(proxy): print hook proxies once and keep base's chunk tracking
Read-only mappings in hook chunks now emit the text from their first print, hook
chunks are read without their error field, only a non-empty error object makes a
framed error event terminal, and framed sequence numbers stay below 2**63-1 while
hook chunks keep base's unbounded tracking
* fix(proxy): print each read-only mapping item once and poll framed error events as failed
A MappingProxyType in a hook chunk or error detail now expands from the
mapping it wraps, so each item is printed once, the way base printed it.
A value whose repr changes between calls stays masked, a value whose repr
works once is kept, and deep nesting keeps one stack frame per level
The background reader takes each frame's type from the same terminal rule
the stream uses, so an event: error frame with an error object polls
failed even when its data names another type or none
* fix(proxy): print self-holding hook values once and poll how the stream ended
A container that holds itself now goes out as the marker its repr prints, like {...}, [...] or mappingproxy({...}), and the dict behind a read-only mapping joins the cycle check, so no item is printed a second time. A dict whose item repr adds a key keeps going after the items already shown, the way dict's repr does, instead of printing the whole dict again
A whole read-only mapping chunk that cannot go out as JSON is printed once by the formatter, so a print that fails fails the stream as it did on the base
The background poll reads the event type from the block it parses, as on the base, and keeps the first error frame the stream treats as terminal anywhere in a chunk. That frame fails the poll only when no response.completed, response.failed or response.incomplete follows it
* fix(proxy): name hook chunk events from the sent JSON and keep polls on frames they cannot read
A mapping chunk now takes its event: line from the JSON it sends, so a masked type value never reaches that line, and a chunk the event model cannot read goes out as a plain data frame instead of failing the stream
A dict whose item repr deletes or swaps keys keeps printing every key not yet shown, and an error code past the integer digit limit reads as no status
The background poll skips a frame whose shape it cannot read, as base skipped frames it could not parse, stops reading a chunk at [DONE], and lets only the last terminal frame in a chunk decide its status
A Responses stream's first chunk answers as a JSON error only for codes 400 and up
* fix(proxy): read every frame of a polled chunk and name model chunks from the sent JSON
A background poll parsed only the first data frame of each chunk, so a terminal event behind an error or delta frame in the same chunk was lost, and the poll stored a status its own stream never ended with. It now reads every frame up to [DONE]
A terminal event whose response is a string or null kept no status and polled completed. Its status now comes from the event type, and final output items that are not objects are skipped, so the rest of the final output is still stored
Model chunks now take their event line from the JSON they send, as mapping chunks already do, so a serializer that drops type no longer sends event: response.completed over {}
* fix(proxy): read polled chunks as SSE events and keep draining past a mid-chunk [DONE]
A background poll split a chunk into data lines, so one event with two data lines parsed as two payloads, and a frame with an integer past the digit limit raised a ValueError the parse guard did not catch, which failed the poll and dropped its spend row. The poll now joins each event's data lines the way an SSE reader does and skips any frame it cannot parse
A [DONE] behind another frame in one chunk ended the poll and left the stream unbilled. Only a chunk that opens with [DONE] ends the poll now, as on main, and the poll stops reading a chunk at its first terminal event, so two terminal events in one chunk poll the first one as main did
* fix(proxy): JSON-encode only plain hook chunks on Responses streams
A hook chunk whose whole tree is builtin JSON types streams as its named
event. Any other chunk keeps the base data frame, so nothing a custom
repr hides reaches the wire. Error event details encode the way the chat
error frame does.
* fix(proxy): skip containers already visited when checking hook chunks for JSON
* fix(responses): keep streaming past an event with no type
The Responses stream iterator read event.type directly when it noted a
yielded event, so an event whose JSON had no type, like a custom hook's
dict that the proxy now sends as JSON, raised AttributeError and ended
the stream. Read it with getattr, as the iterator's other checks do.
* fix(proxy): keep hook dicts with no type on the base Responses frame
A hook dict with no type is not a Responses event, so it gets no event
name either way. Sending it as JSON only made litellm SDKs that read
type on every event stop the stream. Keep such dicts on the frame the
base sends, so every SDK version reads them as before.
* fix(responses): keep a background poll's first terminal status and take a first-chunk error code from its HTTP status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
e9b1c11098
|
fix(xai): honor deployment image pixel pricing (#50490)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
94861ecdf7
|
feat(xai): bill image calls from cost_in_usd_ticks and add grok-imagine-image-2.0 resolution prices (#50481)
* feat(xai): use billed image cost and resolution prices * test(xai): mark image generation tests as a package --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
6b745635c8
|
fix(bedrock): forward Codex agent messages that carry encrypted content intact (#50457)
* fix(bedrock): forward Codex agent messages that carry encrypted content intact Codex multi-agent v2 sends worker messages as agent_message items whose encrypted_content block only the model server can decrypt. Bedrock Mantle and Bedrock Runtime used to 400 on agent_message, so LiteLLM rewrote every one into assistant text, pasting the ciphertext into the prompt and dropping the sender and recipient. Bedrock now decrypts these items itself, so the rewrite left the recipient model reading an opaque token instead of its task. Agent messages with an encrypted_content block now go to Bedrock untouched on both endpoints. The Bedrock configs declare supports_encrypted_agent_messages, and the shared normalizer keys off that flag so the two cannot drift. Mantle's /v1/responses route for open-weight models still rejects agent_message in any form, so it keeps the old rewrite. Plaintext-only agent messages keep the rewrite on every route. * fix(bedrock): only keep worker messages locked for OpenAI GPT models Gemma and Grok on Mantle's OpenAI route never lock Codex worker messages, so Codex puts plain text in the locked slot and Bedrock rejected it with invalid_encrypted_content once the route alone decided to forward it intact. Decide from the model instead, and restore the full-value rewrite assertion. |
||
|
Mateo Wang
|
98acc01e0d
|
feat(videos): add first-class OpenRouter video request fields (#50319)
* feat(videos): add first-class OpenRouter video request fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): remove recursive image parameter fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): preserve EdenAI native parameter passthrough Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): deduplicate frame roles in linear time Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): preserve image selection and dropped parameter semantics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): preserve provider scalar parameters and raw extra body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): retain accepted scalar inputs for Vertex and DashScope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
3f38730d2c
|
refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986)
* refactor(typing): clear basedpyright Any errors (wave 1) * refactor(typing): clear basedpyright Any errors (wave 2) * refactor(typing): clear basedpyright Any errors (wave 3) * refactor(typing): clear basedpyright Any errors (wave 4) * refactor(typing): clear basedpyright Any errors (wave 5) * refactor(typing): clear basedpyright Any errors (wave 6) * fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves * fix(logging): keep callbacks running when litellm_params is a params model should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request Adds behavior tests for the changed lines codecov listed as not run * fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call. * fix: bill regional uplift multipliers float() accepts, bytes included * fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type |
||
|
tavren27
|
2d2290067c
|
fix(bedrock): run realtime guardrails on Nova Sonic sessions (#49833)
* fix(bedrock): run realtime guardrails on Nova Sonic sessions * fix(bedrock): quote the spoken violation message and skip it after a blocked Nova Sonic tool output * fix(bedrock): screen every message role, end Nova Sonic sessions on blocked transcripts, drop unparseable frames * fix(realtime): apply guardrail masks, keep block reasons out of spoken notices, fail closed on guardrail crashes * fix(realtime): screen joined text parts, keep the documented notice, honor key opt-outs Text parts of one message are joined before screening again, so a phrase split across parts is still blocked, and a mask collapses them into one part. The spoken notice is realtime_violation_message or the block reason, quoted only on Nova Sonic. Client-written output_text and text parts are screened. Key and team guardrail opt-outs reach regular and translation realtime sessions. A non-string upstream transcript skips screening as before. * fix(realtime): honor guardrail role scope, clear stale response.create drops, screen split keywords, report Nova guardrail outages * fix(realtime): keep every per-part mask when a joined text masks differently * fix(realtime): screen one joined text per item and pass the session's tenant context to guardrails Multi-part items reach each guardrail once as the space-joined text, as on main, and only an item a guardrail rewrites gets its parts screened again to keep per-part masks. Guardrails now receive the session request data with its metadata, so key and team policies apply, and a ModifyResponseException block is reported as a block. Nova scopes every message item as the user turn it sends to Bedrock. * fix(realtime): catch phrases split mid-word across text parts without re-screening single-part items * fix(realtime): screen each joined text in its own guardrail call and keep a Nova opening message for the retry while it is screened * fix(realtime): screen text parts in a loop so large messages keep their masks * fix(realtime): fail closed on a non-string transcript under a guardrail and give each turn its own guardrail metadata * fix(realtime): fail a non-string transcript only when a guardrail would screen the user's turn --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
de0b389205
|
feat(router): add the Switchyard stage router as an auto router strategy (#50407)
* feat(router): add Switchyard Stage Router strategy * feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes ModelCall.respond and fail async. The classifier is now a tagged union of an LLM judge and a decision model, and decision calls go through the router's adecisions so any decisions provider works. Algorithm misbehavior and library errors route to the capable model instead of failing the request, and the session id plus client session headers are always passed so capable holds apply without a classifier Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): keep the Switchyard router inside the type-check gate The gate's environment did not install the optional switchyard extra, so every libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other optional runtime packages, and imports go through switchyard_rust.libsy, which ships py.typed. Step handling moves into helpers so no Final is rebound in the loop, and the dashboard API types are regenerated for the new routing fields Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): route Switchyard on every endpoint and reject unsafe router saves The router now passes each request's call type to routing strategies, and Switchyard converts the request to chat messages with the existing converter for that call type before translating it. /v1/messages turns with tool or thinking blocks used to fall back to the capable model, and now route on their signals. The Anthropic converter can keep tool_result.is_error on an analysis copy, so Switchyard sees tool failures there, while requests sent to models never carry it Saves of a strategy router that lists itself now fail with a 400 instead of a 500 with a dead row, the capable and efficient models must differ, judge and decision calls are not retried before failing open, the Switchyard routing fields are classified for redaction, and the edit dialog returns the saved router instead of the stale one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): catch self-routing on patch and rename, and trim Switchyard tests The self-routing check now reads the stored params through the decrypting helper and runs when a patch carries no litellm_params, so a PATCH that points a router at itself or renames it onto one of its targets returns 400 instead of a 500 with a dead row. The Switchyard tests share one transcript builder and router fixture, and reformatting noise in unrelated test files is reverted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(ui): make Switchyard a strategy in the existing auto router form Switchyard is now a card in the existing strategy picker. Picking it swaps the tier and classifier fields for the Switchyard fields, and the shared name, team and save button create it through one submit branch, the same flow as the other strategies. The separate add form, the extracted picker and the hidden-form swap are gone, and the edit dialog keeps its own Switchyard form The judge's session_affinity and message_hash_fallback settings are removed, since Switchyard rejects both for the stage router Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): accept partial tool semantics and show server errors on model edit The Switchyard tool semantics field rejected mappings that list only some categories, although the server accepts them. Model PATCH errors were reduced to a generic message, so it now goes through the shared API client like model create and surfaces the server text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): show the routing strategy picker up front in the auto router form Every strategy was hidden behind an Advanced configuration link, so the quick setup read as the only option and never said it creates a heuristic router. The picker now always shows with Heuristics as the labelled default, the quick setup sits under it, and picking another strategy opens that strategy prefilled with the chosen models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): show the server's reason when a model settings or API key update fails Both screens replaced the rejection with a fixed message, so a 400 such as a router routing to itself read only as a generic failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(router): route Switchyard on the Rust router core main replaced the Python router core with the Rust one, which left Switchyard routers registered nowhere. Register them with the other pre-routing strategies, hand each strategy the endpoint the request came through, and evict Switchyard routers on reload and delete like the rest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form Switchyard session state is keyed by a hash of the caller's key hash and session id, and client session headers no longer reach Switchyard unscoped. The add form carries the quick setup models into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the schema reads API-saved configs with nulls and structured decision instructions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(router): make the Switchyard test folder a package * fix(router): read the Switchyard session key hash only from the proxy's own metadata * fix(proxy): keep nested nulls as null when masking deployment params * fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies * fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
simonp9
|
1f15088fc9
|
feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes (#49408)
* feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes
The native Codex catalog now lists the key's other chat and responses models after
ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list
refreshes on Codex's timer. A request for a model configured by exact name on another
provider goes through /v1/responses without the ChatGPT bearer or account header,
while ChatGPT names and wildcard-only matches keep the subscription passthrough.
Reasoning items another provider produced are dropped before forwarding to ChatGPT.
* fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models
Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while
`codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native
/backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no
default, so the provider applies its own default exactly as `codex -m` does
* fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes
Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses
request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may
call /v1/responses, drop unpersisted reasoning ids on the passthrough when store
is not true, keep any slug the caller's ChatGPT account lists on the subscription
via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog
added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard
precedence over a catch-all deployment.
* fix(proxy): keep Codex's native route on the subscription when the account list is unknown
A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the
account's list is known and leaves it out. A failed catalog read is remembered for 60 s and
the Codex version comes only from a User-Agent whose product matches the originator header.
The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows,
and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so
Codex stops refetching or dropping its catalog cache.
LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure
blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge
decodes wrapped thinking blocks with their signature.
The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the
native route checks the chatgpt/ form of a name the key holds only through that group. A
none reasoning effort is dropped only under drop_params. The guide covers app tools, every
routing case, and the ETag, and the catalog logs how many models the byte limit left out.
* fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API
* fix(proxy): keep Codex native turns working when the key's model listing fails
A listing callback that raises no longer turns chatgpt.com's catalog or
responses answer into a 500. The catalog keeps upstream's body and ETag, and
responses keep upstream's X-Models-Etag.
Under drop_params, a `none` reasoning effort is now dropped only when a
candidate deployment does not explicitly support it, so an alias to a model
like openai/gpt-5.4 keeps it.
The account slug list and its unavailable marker are keyed on the account id
plus the caller's ChatGPT Authorization header, so a failed fetch with one
bearer cannot turn off LiteLLM routing for another caller on the same account.
The auth step checks the Codex route before importing the Codex module.
* fix(proxy): keep Codex model switches working across LiteLLM providers
On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400
Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts
A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows
* fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers
Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the
native route now hoists them without flattening, like OpenAI and Azure and
the chat completions bridge. Other native Responses providers still get
flat names, now built with the bridge's namespace_tool_name_map, and each
function_call they return gets its tool name and namespace back in the
non-streaming body and in streamed output_item.added, output_item.done,
and completed/incomplete/failed events. Every other event passes through
unchanged
* fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools
Codex sends its freeform exec tool as a custom tool on every turn, and xAI's
Responses API answers 422 for custom, namespace and additional_tools shapes.
On the Codex native route, a model whose deployments include a native
Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now
asks for the chat completions bridge with use_chat_completions_api, which
already turns custom and namespace tools into functions, hoists
additional_tools, and puts the namespace back on each returned call
OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock
Mantle still get it on their own Responses API with additional_tools hoisted.
The route's namespace flattening and the response-side namespace restore are
removed, since no path sends flattened names anymore
* fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve
A reasoning item another deployment produced, replayed with store false and no blob the
target can decrypt, now reaches the target as its readable summary without id or status,
and is dropped when nothing readable is left. OpenAI answered 400 on the status field and
404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment
replays and stored conversations keep the item whole
* refactor(codex): reach encrypted reasoning stripping through the router registry
The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from
litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry
gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with
encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise
every target must decrypt it
* fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog
* fix(proxy): drop Codex's none effort for an alias whose native entry names no default
* fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop
* test(proxy): map the pinned reasoning efforts field in the router registry stub
* fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases
Codex native routes now send a key to LiteLLM only when admin_only_routes allows it,
a native alias with configured service tiers starts with no tier picked, and the
account slug cache declares its Redis key family.
* fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items
* fix(proxy): check vector store grants on file_search tools inside additional_tools items
A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input.
* fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes
* fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply
* fix(guardrails): check additional_tools items next to top-level function tools in tool_policy
ToolPolicyGuardrail read tools from the request body only when no top-level
function tool existed, so a blocked tool defined in an additional_tools item
reached Fireworks once the Responses transform moved the item into tools after
the pre-call hooks. Check every list on each request, including namespace
members inside items.
* fix(router): serve decryptable_reasoning from the Rust core's optional checks
The merge with main removed the Python router core, so the registry's host has no
optional_callbacks and no routing context, and native Codex requests with encrypted
reasoning raised RustCoreUnsupported. The filter now lives with the optional checks,
which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core
Also give the static litellm.Router a declared constructor. Since the Python core was
deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...)
is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change
* fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns
Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures.
* fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias
* fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases
Auth no longer asks ChatGPT for the account's model list when the bearer is
missing or is the LiteLLM key, so a key sent as the bearer never reaches it.
tool_policy now reads the members of a top-level namespace, which is where
the native route's hoist puts them.
A global alias now reads encrypted content affinity from its target group.
A non-list top-level tools value and a non-string function name now reach the
provider, which refuses them, instead of raising a 500.
The catalog ETag no longer depends on the order a process iterates priced
service tiers in, so workers and pods agree on it.
* fix(proxy): list a team's models in the order the team set them
get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker
* fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models
A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models
and /backend-api/codex/responses only when the JWT auth flow would admit a
direct /v1/responses call for that model, so team_allowed_routes,
admin_allowed_routes and RBAC route limits hold on the handoff too.
The Fireworks Responses transform also leaves additional_tools input items
in place when the tools param is not a list, instead of raising while hoisting
* fix(fireworks): hoist additional_tools when tools is a tuple
litellm.responses passes a tuple tools argument through to the Fireworks
transform unchanged, so the non-list guard skipped hoisting and left the
additional_tools item in the input. Treat tuples like lists and cover both
shapes in the drop_params merge test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
36c64ce85d
|
feat(bedrock): default to native structured output on Opus 5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 (#50221)
* feat(bedrock): enable native structured output by default for selected Claude models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep govcloud and bare model rows off native output Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected native schemas with the JSON tool on default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep the rejection retry on the unforceable-tool native path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected schemas on the Messages path for default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format Messages native output retry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): allow profile-only native output metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): only disable default-native output on schema-rejection retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): type fixture parameters in native output retry tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): drop strict tools after a native-output retry hits the grammar limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format native output fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too. * fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action A role allowed only bedrock:InvokeModel got a 403 on schema streams once these models went native, since native streams call converse-stream while the JSON tool path buffered through converse. The retry now sends that request again as a buffered JSON tool stream, the shape the base sent. The wire test moves its neither-key and opt-in cells to Opus 4.7, its Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native. The batch wire test now checks a thinking schema record on both sides of the contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5 and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8 cells assumed those models were opt-in. * fix(bedrock): move the Claude 5 native output default to a key released proxies do not read Released proxies load the cost map from main and have none of the native output retries, so setting supports_native_structured_output on the Opus 5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide or min/max schemas into 400s on v1.108.1. Those rows keep supports_native_structured_output false and gain bedrock_native_structured_output_by_default true, which get_model_info reports as supports_native_structured_output, so this version behaves as before while released versions keep the JSON tool. * fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it * fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key A registered supports_bedrock_native_structured_output false now beats the native output default, and batch uploads read the deployment's model_info, so an opted-out deployment keeps the JSON tool on every record. The Opus 5 and Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default, so released proxies on main's cost map keep the JSON tool on them. The 64 property batch cutoff now applies only to rows with that key, so models that were already native keep their batch output shape. * fix(proxy): report a deployment's Bedrock native output opt-out on /model/info /model/info merged the deployment's model_info over the backend row, so a deployment that set only supports_bedrock_native_structured_output: false still showed the backend's derived supports_native_structured_output: true. The generic flag is now derived from the merged info unless the deployment set it itself, through the same helper get_model_info uses. * fix(bedrock): keep native output when a denied schema stream is resent buffered A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output. * fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend * fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model * fix(bedrock): answer a buffered invoke stream from the non-streaming reply * fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected * fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main. /model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Alexander Chernov
|
d1cde1002e
|
fix(chatgpt): forward text and max_output_tokens to the Codex backend (#38086)
* fix(chatgpt): keep text and accept string input for the Codex backend
Two request-side defects stop chatgpt/* serving structured output or a string
input.
The allowlist at the end of transform_responses_api_request omits "text". The
chat-to-responses bridge translates response_format into text.format, so the
allowlist discards strict schemas silently and the backend answers with prose
instead of JSON. The backend does honour text.format when it receives it.
Separately the backend rejects a bare string input with
{"detail": "Input must be a list"}, while the Responses API itself accepts
either shape, so the string needs wrapping before it is sent.
Both are request-shaping only, so neither depends on the streaming work in
#31332 or #34095, though those two block the same call and have to be in place
to observe this one end to end.
The "text" case was reported in #24356 and closed by the stale bot without a
fix.
Signed-off-by: Alexander Chernov <alexander@chernov.it>
* fix(chatgpt): annotate the input coercion for the type-discipline gate
The LIT002 budget counts mutable-collection construction, and wrapping the
string input builds a list and a dict, which put the total two over its
ceiling. Both constructions are the wire payload this backend requires, so the
list cannot be replaced with an immutable equivalent; annotate the line with a
reason instead, matching the convention used elsewhere in the tree.
LIT005 freezes reasonless suppressions at zero, so the reason is required
rather than decorative.
Signed-off-by: Alexander Chernov <alexander@chernov.it>
* refactor(chatgpt): bind the coerced input instead of rebinding the parameter
Greptile flagged the parameter rebinding and the explanatory comments against
the repo conventions in CLAUDE.md, which bans rebinding a function parameter
(LIT011) and keeps comments to suppressions, TODOs and genuinely complex logic
Bind a Final local for the coerced input rather than reassigning `input`, and
drop both prose comments; the rationale lives in the commit and the PR body
rather than duplicated at the call site. The mutable-ok suppression stays,
since it is the allowed kind and the list is still constructed
Behaviour is unchanged: 24 tests pass, and the same 4 fail with only the
provider file reverted to base
Signed-off-by: Alexander Chernov <alexander@chernov.it>
* fix(chatgpt): forward max_output_tokens to the Codex backend and return incomplete non-stream responses
The Codex backend accepts and enforces max_output_tokens, so the allowlist now keeps it
instead of dropping it. A structured-output request could otherwise stream until the
backend gave up, ignoring the caller's max_tokens.
A capped request ends with response.incomplete, so the non-stream SSE aggregator now
treats that event as terminal and returns the partial response instead of an error.
---------
Signed-off-by: Alexander Chernov <alexander@chernov.it>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
JeongCheolLee
|
febbe38326
|
fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385)
* fix(anthropic): do not surface server_tool_use as client tool_calls Anthropic server-executed tools (web search, tool search, code execution) were returned in chat completions tool_calls with srvtoolu_ ids, so OpenAI-protocol agent clients treated them as pending work and re-invoked the model until their recursion limit. Move them to provider_specific_fields.server_tool_calls on both the non-streaming and streaming paths. History replay, the /v1/responses bridge and the /v1/messages streaming guardrail read that field, so multi-turn reconstruction, web_search_call output items and the guardrail's fail-closed check on server tool blocks keep working. * fix(responses): return Claude web_fetch calls as web_search_call open_page items * fix(responses): keep Claude server web calls without a result as hosted items A non-streamed Claude turn that calls web_search or web_fetch next to a client tool can return the server call before its result. Build an in_progress web_search_call for it, so the Responses bridge no longer returns it as a client function_call * fix(anthropic): keep pending server calls native and hide server calls from client tool paths Replaying a server call that has no result yet now sends it back as server_tool_use, so Anthropic accepts the next turn. The tool permission guardrail checks server calls again and rewrite mode removes a denied one. The /v1/messages adapter keeps Claude's server blocks, and the Responses bridge never returns a server call as a client function_call. * fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls, so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own server_tool_use instead of running again * fix(anthropic): let guardrails and loggers read Claude's server tool calls again Guardrails, tool policies, tracing loggers and the Responses guardrail translation read only client tool calls after server calls moved to provider_specific_fields. They now read server calls first, then client calls, through one shared helper, and the Responses guardrail treats Claude's web_search_call items as web_search or web_fetch calls. Non-streaming splits server calls by block type, as streaming does. * fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy, streamed or not. Redaction with message logging off hides server call arguments and hosted web call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls. * fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use. * fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy. * test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions * test(guardrails): type the fake Pillar client like its Protocol * fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or web_search still needs that call and its thinking replayed, or Claude loses the fetch result * fix(guardrails): scan replayed Claude server calls and hosted web calls Request guardrails now see and rewrite server_tool_calls replayed in chat messages. Cato and Purview scan Claude web_search and web_fetch calls on Responses output, and a function tool call with an extra custom key is no longer dropped from Cisco, Purview, Repello and PANW scans. * fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks * fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows * test: annotate the new server tool call test locals with Final and type their parameters * test: type the new guardrail test helpers with litellm's tool call types instead of Any * test: narrow the remaining new test helper parameters from bare dict and object * fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls * fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera * fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Siddharth
|
2a4f8b1d19
|
fix(bedrock): drop another model's signed thinking when replaying to a non-Claude Converse target (#50073)
* fix(bedrock/converse): omit reasoning signature for non-Anthropic models (#49969) * test(bedrock): cover signed reasoning replay in mapped tests * fix(bedrock): drop another model's signed thinking when replaying to a non-Claude Converse target A signed thinking block from Claude replayed to a Converse model known not to be Claude, such as GPT-6 Luna or Nova 2, now leaves the request instead of going out as reasoningContent. Bedrock rejects that field on those models, so the whole request answered 400 A target that may still be Claude, such as an application inference profile ARN with no base_model, keeps the signed block, since Claude needs it on a tool-use continuation. Unsigned blocks keep the existing text path The thinking-block error matcher added earlier on this branch is removed, because Converse and non-Claude Bedrock requests never reach that retry * fix(bedrock): keep signed thinking for Claude behind a region prefix litellm does not list ca. and in. inference profiles name Claude after a region prefix get_bedrock_cross_region_inference_regions does not have yet, so they counted as non-Claude and lost their signed thinking. The thinking-only turn test now pins the exact merged and continue-message shapes. * fix(bedrock): limit the region-prefix Claude match to signed thinking and drop a dropped block's cache point * fix(bedrock): drop Claude's bridged reasoning items before a Bedrock Responses API replay * fix(bedrock_mantle): drop litellm-minted reasoning items before a Mantle Responses call * test(integration): expect Bedrock and Bedrock Mantle Responses to drop minted reasoning Bedrock and Bedrock Mantle answer 400 on a reasoning item litellm minted, so the wire test now expects both to drop it, like OpenAI and Azure --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
moe-berri
|
b02bb99b08
|
feat: add OpenRouter-style -latest model aliases for first-party providers (#50315)
* Add first-party floating model aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Align Moonshot alias parameter test with its target Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Share floating alias proxy budgets and rate limits with targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Prove target-only team budgets count and block floating aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Share floating alias quotas with project and organization targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Coalesce colliding floating alias rate limit counters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Fix latest alias Rust fallback and failure accounting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: adapt latest alias checks to typed rate-limit store Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ishaan-j-berri
|
80194a7d70
|
fix(anthropic): preserve Azure Responses reasoning replay (#50024) (#50381)
Fixes #50024 Preserve encoded reasoning history through the existing Azure Responses bridge. |
||
|
ishaan-j-berri
|
a12100b33b
|
fix(anthropic): default Responses JSON schemas through a helper (#50023) (#50375) |