Commit graph

55719 commits

Author SHA1 Message Date
Kerry Lu
137fd7cd66
fix(docs): replace stale docs/proxy/enterprise links with the current docs pages (#49491)
* fix(ui): point the Spend Per entity docs link at the cost tracking page

* fix(docs): replace the remaining docs/proxy/enterprise links with current pages

* revert(terraform): leave the team tags docs link for a code-owned follow-up

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 16:38:37 -07:00
ishaan-j-berri
6105aaa63e
Configure full Bedrock guardrail assessments (#49217) (#49470) 2026-10-31 23:37:45 +00:00
ishaan-j-berri
2751cfcad3
Add guardrails.apply SDK helper (#49217) (#49469) 2026-10-31 23:37:45 +00:00
ishaan-j-berri
8d26b8bba3
Return configured guardrail native findings (#49217) (#49460) 2026-10-31 23:37:45 +00:00
moe-berri
87e32e7c6d
fix(agents): show non-admin callers the proxy URL in agent cards (#49483)
* test(agents): cover backend card URL leak to non-admin callers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): assert backend URLs before card parity

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): scope unit card regressions to one agent

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): isolate card redaction from access policy loading

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): remove unused proxy fixture binding

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): normalize unit test imports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): pass endpoint base URL to redaction helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): cover proxy card deep-copy behavior

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(agents): rewrite non-admin agent card URLs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(agents): simplify proxy card URL helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(agents): drop card mapping adapter import

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(agents): use typed card mapping helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): cover alternate card interfaces and real access checks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(agents): drop alternate backend interfaces from non-admin cards

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): probe the real health check in card URL test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 16:17:11 -07:00
moe-berri
9312a25f18
feat(chat): add deep research with persistent citations (#49508)
* feat(chat): add simple deep research and saved citations

* test(chat): verify ordinary request payloads instead of argument layout
2026-10-31 16:13:45 -07:00
tin-berri
36dbb228d9
fix(streaming): preserve served models in OpenRouter streams (#49498) 2026-10-31 16:12:51 -07:00
tin-berri
ae8666b9d1
fix(http): diagnose aiohttp connection pool saturation (#49499) 2026-10-31 16:12:30 -07:00
moe-berri
bba63c1934
fix(proxy): serve /a2a/{agent_id}/message/stream routes (#49471)
* test(a2a): cover message stream route aliases

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): serve /a2a/{agent_id}/message/stream routes

Co-authored-by: Rustaman Delvianova <96533738+Rustaman1280@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): document A2A stream route schemas

* fix(proxy): use immutable A2A schema mappings

* fix(proxy): map v1 a2a stream route to call types and keep stream alias unbilled

* test(a2a): use real guardrail registry and typed locals in stream route tests

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Rustaman Delvianova <96533738+Rustaman1280@users.noreply.github.com>
2026-10-31 16:11:51 -07:00
Mateo Wang
ddab201d68
test(translation): expect the EOF error frame when an Anthropic json_schema stream is cut (#49521)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 16:04:04 -07:00
Nate Armstrong
22d0b1e443
fix(router): restore four base behaviors the RoutingContext refactor changed (#49395) 2026-10-31 22:58:34 +00:00
moe-berri
a7425101bb
fix(a2a): log authenticated pre-dispatch agent rejections (#49492)
* test(a2a): cover authenticated pre-dispatch denial logging

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(a2a): log authenticated pre-dispatch agent rejections

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(a2a): exercise real permission checks for rejected calls

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 15:53:01 -07:00
shrey
ead8aa82ef
fix(passthrough): fit the Deepgram listen unpriced refusal in a WebSocket close frame (#49502) 2026-10-31 15:49:59 -07:00
moe-berri
3bd75f1258
fix(proxy): treat empty agent and skill search embedding model as not configured (#49485)
* test(agents): cover empty search embedding model settings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(agents): treat an empty search embedding model as not configured

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(tests): sort search integration imports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(tests): format search embedding tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(agents): annotate search test locals and isolate upstream call checks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 15:32:26 -07:00
Kerry Lu
e708343309
test: deflake the json accumulator drain test and the armed routing prefetch test (#49505)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 15:14:33 -07:00
yuneng-jiang
da5ea59667
refactor(ui): remove dead API calls from networking.tsx (#49507)
Deletes 17 exported calls that nothing references anymore. The Old Usage page removal left 10 behind, the move of agent traces to the Lens service left 5, and modelInfoV1Call and getUISettings lost their last callers in earlier refactors

Seven names that are only used inside the file stop being exported. buildModelGroupTestRequest stays exported because networking.test.ts calls it directly

Also drops the two test mocks that still stubbed deleted calls and prunes the lower eslint suppression counts for the file
2026-10-31 22:09:15 +00:00
Mateo Wang
6aa546303e
docs(pr-template): wrap the agent-drafted summary in make-me-human-readable tags (#49494)
* docs(pr-template): wrap the agent-drafted summary in make-me-human-readable tags

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs(pr-template): keep the make-me-human-readable tags visible in rendered descriptions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs(pr-template): mark the agent-drafted summary with a visible Make me human-readable warning block

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Revert "docs(pr-template): mark the agent-drafted summary with a visible Make me human-readable warning block"

This reverts commit 4eb7f666a4.

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 21:48:56 +00:00
moe-berri
580b9266a1
feat(chat): resume replies after browser disconnects (#49461)
* feat(chat): resume background MCP runs after browser disconnects

* feat(chat): background ordinary text conversations

* feat(chat): configure background replies from Settings

* fix(chat): address background response review findings

* test(sdk): finish the HTTP smoke stream explicitly

* fix(chat): preserve background response timing metrics

* test(router): terminate the scripted completion stream
2026-10-31 14:43:24 -07:00
alexcastgonz
d2321581ec
fix(cohere): emit streamed tool calls from v2 tool-call-start and tool-call-delta chunks (#47794)
* fix(cohere): emit streamed tool calls from v2 tool-call-start and tool-call-delta chunks

* fix(cohere): type _parse_tool_call_delta without growing basedpyright counts

The reassigned Final local and the untyped chunk lookups added basedpyright
errors past the base ceiling. Narrow them with casts; per-rule counts for the
file are now at or below the merge-base.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(cohere): parse streamed tool calls with a typed model and map Cohere TOOL_CALL to tool_calls

Read the streamed tool call through a frozen pydantic model instead of casts,
which the strict and type-discipline gates refuse. Map Cohere's TOOL_CALL and
STOP_SEQUENCE finish reasons, so a stream that reads Cohere's message-end still
ends tool_calls. The regression test now drives litellm.acompletion with two
parallel streamed calls and checks the rebuilt calls and the final finish reason

* fix(cohere): close a streamed tool call that sent no arguments with an empty JSON object

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-31 21:14:53 +00:00
simonp9
f7b2cd0000
fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500 (#49237)
* fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500

A list role or an object or list tool_call_id crashed the chat conversion with "unhashable type". Depending on the step that hit it, callers got a 500 or a 400 that never named the field. A failed conversion now answers 400 naming the first non-string role or tool call id, and the prompt caching hook reads a non-string answer id as matching no tool call.

* fix(anthropic): name the field the conversion failed on, and answer 400 for a non-string Responses call_id

The chat field check now runs only when the conversion raised a type error, and
scans roles first, then assistant tool call ids, then tool answer ids, so an
orphan tool answer that modify_params drops is no longer blamed. A tool output
call_id that is not a string on /v1/responses answers 400 naming input[N].call_id,
and the clear_tool_uses polyfill leaves a non-string tool_use_id for the provider

* fix(anthropic): name a non-string role whatever error the conversion raised

An int or bool role fails the conversion with an invalid message error, not a type error, so it lost the named 400. Roles are now named for any conversion failure, while tool call ids stay limited to type errors

* fix(responses): name a non-string message item role by its input index

* fix(responses): name a malformed input field only where litellm reads it, by the caller's index

A tool output call_id is named only when it is an object or a list and the
conversion fails, so an orphan numeric id still converts and another field's
failure is never blamed on it. On the bridge to Claude a non-string role or
function_call call_id is named by its input item, counting items before a
compaction item. A numeric assistant tool call id on chat is named too.

* fix(anthropic): name a tool id only for the error it causes and count Codex tool items in input indexes

A numeric tool answer id that modify_params drops was blamed for any later type or attribute error, so the 400 hid the real cause. Tool ids are now named only for the error their value raises, and a hashable answer id never.

The /v1/responses bridge and its input token count now read the input the caller sent, with only additional_tools hoisted into tools, so input[N] counts those items too.

* fix(anthropic): name a tool id only for the error its own value causes and keep Codex tool item text out of the request

* fix(responses): name a tool output call_id only when the items before it convert

* fix(responses): name a non-string input field only on inference so token counting and batch records keep their answer

* fix(responses): let a client field named caller_input or names_non_string_fields reach the bridge

* test(anthropic): build the non-string id histories without editing them in place

* fix(anthropic): never name an empty role for another field's failure

* fix(anthropic): name an empty list or object role when the conversion fails on it

* fix(anthropic): name a list role under the drop_params cache cap and keep the Bedrock batch line prefix

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-31 21:13:24 +00:00
Nate Armstrong
c43a1addf2
refactor(router): extract PythonCore behind RouterCore and pass the router its config (#49119)
* refactor(router): move litellm/router.py to litellm/router_python_core.py

A pure rename with no content change, so git and review tools track the router body as a move.
The next commit turns it into PythonCore and adds the Router facade back at litellm/router.py

* refactor(router): extract PythonCore behind RouterCore and pass the router its config

The routing logic moves unchanged from litellm/router.py into PythonCore in
litellm/router_python_core.py. Router becomes a facade that builds a RouterConfig, picks a
RouterCore with make_core and delegates every member to it. The core no longer reads or writes
litellm.* directly: process settings come from RouterDynamicConfig, and inference calls, sleeps,
callback lists and the global cache go through PythonHost. A CI check bans litellm.* in the router
modules, and the proxy's construction sites build the config outside the router

Settings are static: the constructor arguments and changeset values are copied when the router is
built or the change applied, and a write to a setting attribute becomes apply(ChangeSet(assign=...)).
A Router subclass's core class also subclasses the facade, so an override's super(Sub, self) works,
and an instance-level patch of a core method is kept on the facade so nested patches unwind.

tests/unit/test_router/core tests the core in isolation: ScriptedHost serves every host operation
(scripted inference answers, recorded sleeps, settings, callbacks, cache), and make_core runs each
test once per core in CORE_BUILDERS
2026-10-31 12:54:23 -07:00
moe-berri
a5f980600c
fix(a2a): answer proxy refusals on /a2a POST routes as JSON-RPC errors (#49476)
* test(a2a): expect JSON-RPC errors for proxy refusals on /a2a POST routes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(a2a): answer proxy refusals on /a2a POST routes as JSON-RPC errors

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(a2a): build JSON-RPC refusal data from typed fields

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(a2a): stamp the OTel server span on JSON-RPC refusals

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(a2a): cover JSON-RPC refusal route in unit tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 12:54:10 -07:00
shrey
7143aeeb86
feat(passthrough): add Deepgram streaming text-to-speech WebSocket route (#49282) 2026-10-31 12:46:40 -07:00
yuneng-jiang
3129e44848
ci: cut the weekly rc branch on Monday 3am PT instead of Friday (#49472) 2026-10-31 12:32:23 -07:00
Mateo Wang
d610dee7d0
docs(pr-template): add a handwritten 1-3 sentence description section (#49475)
* docs(pr-template): mark the summary above Details as human handwritten

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs(pr-template): title the summary as a handwritten 1-3 sentence description

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 19:29:26 +00:00
Rick
ced995df6e
fix(logging): drop a first-token time left over from a previous attempt (#47541)
A request re-dispatched to a fallback keeps the same logging object but gets a
fresh start_time. completion_start_time was only filled when it was empty, so
an attempt that inherited a value from the earlier attempt reported a negative
TTFT. That value reaches both the spend log and the latency-based routing
cache, where one negative sample empties the buffer window and takes the whole
model group down until the cache entry expires.

Ignore a recorded first-token time that predates the attempt's start and fall
back to end_time, the value already used when none was recorded.

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
2026-10-31 12:28:58 -07:00
Mateo Wang
3155aacda4
fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests (#49468)
* fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(caching): return the server tool call id instead of a TypeGuard

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): assert the Gemini cache write carries the system prompt

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 12:27:17 -07:00
yucheng-berri
8d202faec7
test(integration): wait for the stalled-tenant root span and accept a bare POST name (#49406) 2026-10-31 12:24:33 -07:00
moe-berri
7e4403ea66
fix(a2a): send agent litellm_params.api_key as bearer auth on the A2A hop (#49474)
* test(a2a): cover agent api key authentication

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(a2a): authenticate plain agent hops with api key

Co-authored-by: Afnan Mojumder <afnan.mojumder@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Afnan Mojumder <afnan.mojumder@gmail.com>
2026-10-31 12:06:52 -07:00
Kerry Lu
6f86b117b5
fix(moderations): stop the router image check from 500ing on a list input (#49454)
* fix(moderations): stop the router image check from 500ing on a list input

* fix(moderations): give the list cast its cast-ok reason

* fix(moderations): type the responses input guard and keep SDK compaction items

* fix(moderations): import the compaction item through litellm.types and mark the TypeIs import

* fix(moderations): sort the compaction item import

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 12:05:13 -07:00
ishaan-j-berri
941f52f26f
fix(proxy): preserve discovery after budget exhaustion (#48809) (#49463)
* fix(proxy): preserve discovery after budget exhaustion (#48809)

* refactor(proxy): decide budget exemption once (#48809)

* refactor(proxy): centralize request budget policy (#48809)

* fix(proxy): keep custom pass-through budget checks (#48809)
2026-10-31 19:00:51 +00:00
moe-berri
6eddc4a979
fix(a2a): drop wrong 10 minutes comment on DEFAULT_A2A_AGENT_TIMEOUT (#49465)
* fix(a2a): correct DEFAULT_A2A_AGENT_TIMEOUT comment to 100 minutes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(a2a): drop DEFAULT_A2A_AGENT_TIMEOUT duration comment

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 11:56:52 -07:00
Kerry Lu
948445c828
fix(watsonx): read pass-through credentials from use_in_pass_through deployments (#49462)
* fix(watsonx): read passthrough credentials from use_in_pass_through deployments

/watsonx/{endpoint} always called the provider config with api_key=None and api_base=None, so it only ever
worked from WATSONX_API_KEY / WATSONX_API_BASE. It now asks PassthroughEndpointRouter for the proxy-wide
watsonx deployment flagged use_in_pass_through, like the other passthrough routes, and keeps the env
fallback when none is flagged.

Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>

* fix(watsonx): keep env key precedence when no deployment is flagged

Resolve only use_in_pass_through deployments in the route. get_proxy_wide_credentials also falls back to
WATSONX_API_KEY, which put it ahead of WX_API_KEY in generate_iam_token

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
2026-10-31 11:53:25 -07:00
Mateo Wang
ee39de7107
fix(scheduler): read shared queue from Redis so expired dead-replica entries stop blocking waiters (#49387)
* fix(scheduler): observe shared queue expiry before polling

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(scheduler): defer optional Redis exception import

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(scheduler): share one Redis queue read per polling interval across waiters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* revert(scheduler): drop shared Redis queue read

This reverts commit 5fd9b44a85

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 11:43:46 -07:00
yucheng-berri
efc2304b7e
fix(streaming)!: fail a stream that ends without a finish reason (#47665)
* fix(streaming): fail on clean EOF without finish reason

* fix(streaming): cover shared OpenAI-compatible EOF paths

* test(router): complete mocked streaming responses

* fix(streaming): keep the EOF error message and cover Anthropic clean EOF

The request-less RemoteProtocolError tripped exception mapping into
'The .request property has not been set.' for non-OpenAI providers.
Adds Anthropic and generic-chunk unit tests, an integration test across
chat, the /v1/messages bridge and the /v1/responses bridge, and moves
main's token-estimate test onto a finished stream with a sibling that
checks the cut stream still bills the estimate on its failure log.

* fix(streaming): count a finish reason any chunk sent, keep LangGraph async streams whole

Gemini marks every chunk unfinished and a later usage chunk reset
intermittent_finish_reason, so a finished Gemini stream failed at EOF.
LangGraph's __aiter__ opened a new aiter_lines on every wrapper call,
which hit StreamConsumed after the first chunk and now surfaced as a
false EOF error instead of a silent truncation.

* fix(streaming): keep the synthetic stop for a stream that sent no chunks

Gemini's OpenAI-compatible endpoint answers with only [DONE] when
reasoning uses up max_tokens. Failing every openai/azure stream that
sent nothing turned that 200 into a 500, while every other provider
already kept the stop for an empty stream. Adds integration cells for
Gemini's trailing usage chunk, multi-event LangGraph runs, a mocked
deployment and the [DONE]-only answer.

* test(streaming): keep pytest.raises to one statement, drop the spend read for an empty stream

An empty stream writes no spend row on main either, so the [DONE]-only
cell checks only what the client sees.

* fix(streaming): read the chunk's finish reason for the sticky flag

* fix(agentcore): end a stream on messageStop when the runtime sends no metadata or final message

* test(streaming): pin the logged-chunks guard, inject the LangGraph client, use current model names

* fix(agentcore): say why the end-of-stream flags rebind

* test(streaming): cover an AgentCore runtime that ends on messageStop through the proxy

* test(streaming): cover SDK clients, request fallbacks and a concurrent burst for a cut stream

* test(agentcore): inject the async transport instead of patching the aiohttp flag

* test(guardrails): end the decision model chat fixture with a finish reason, as a real stream does

* test(streaming): shape the AgentCore and LangGraph fixtures like captures from a real runtime and server

* test(streaming): keep each stream's ending and land every spend row once through a spend log outage

* test(streaming): end each stream by its provider's finish after a proxy worker is killed mid burst

* test(streaming): run the outage test as one burst before and one during the outage instead of a capped wave loop

* test(streaming): connect the outage test's proxy to the relay without TLS so its trigger sees the spend-log insert on an ssl=on Postgres

* test(streaming): type the new stream EOF unit tests

* test(streaming): a merged reasoning stream cut mid reasoning now fails

Main's think tag test expected a stream that ends mid reasoning without a finish reason to finish with a closed think block. With this PR that stream fails, so the case moves to its own test that expects MidStreamFallbackError with the open think block as the partial content

---------

Co-authored-by: King Star <mcxin.y@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-31 11:29:26 -07:00
Kerry Lu
aea0e4f2d4
test(cost): expect cohere rerank-v4 prompt_tokens from billed_units.total_tokens (#49455)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 11:03:10 -07:00
ishaan-j-berri
54892d6a62
fix(oci): preserve all Gemini system instructions (#48813) (#49458) 2026-10-31 17:49:53 +00:00
joshua-berri
6e27da8af9
fix(router): inspect image content without translating routed requests (#49371)
* fix(router): inspect image content without translating routed requests

* fix(router): preserve screenshot image routing

* test: make monthly budget reset coverage deterministic

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-10-31 10:44:24 -07:00
tin-berri
34b6256d52
fix(streaming): normalize stop after OpenAI-compatible tool calls (#49388) 2026-10-31 10:33:56 -07:00
yucheng-berri
c8c0248ac7
test(integration): size restart-mid-burst timeouts from the proxy boot budget (#49376) 2026-10-31 10:17:34 -07:00
Kerry Lu
bfef2e1e56
fix(messages): price a native /v1/messages body as served unless its provider reports its own cost (#49356)
* fix(messages): price a native /v1/messages body as served unless its provider reports its own cost

* fix(messages): bill the requested speed when a native body as served reports none

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 09:52:39 -07:00
Kerry Lu
7b4bdde96e
fix(cost): keep upstream_usage_cost on the Responses stream logging copy so bridged /v1/messages rows price their tokens (#49365)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 09:52:22 -07:00
yujonglee
1595adfb2f
docs(model-catalog): explain meter, band, channel and When (#49447)
* docs(model-catalog): explain meter, band, channel and When

Add a Concepts section to the model-catalog AGENTS.md with worked key
breakdowns, doc comments on the undocumented litellm-cost price types,
and a cost AGENTS.md that points to the concepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(model-catalog): write concepts as nested lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(model-catalog): spell out band rules and channel fallbacks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 16:09:29 +00:00
Kerry Lu
3cb2e60de0
fix(cost): split a provider-reported total between input and output by the cost map rate ratio (#49357)
* fix(cost): split a provider-reported total between input and output by the cost map rate ratio

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost): expect the kyma reported-cost headers to follow the cost map rate split

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost): keep the kyma test file at its original formatting

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): price the response-cost header split from the served model, not the request alias

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 09:04:50 -07:00
devin-ai-integration[bot]
b2647116ca
chore(cost-map): add openai gpt-6-astra-law from the pricing page (#49444)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 08:56:44 -07:00
ryan-crabbe-berri
8ff495be96
fix(proxy): price OpenAI pass-through /v1/decisions calls like native Decisions (#49392)
The OpenAI pass-through dispatcher kept its own list of endpoints it could price,
separate from the handler, and /v1/decisions was on neither. Calls reached OpenAI
but logged $0 and no tokens.

The handler now owns the list the dispatcher reads, and prices a POST to
/v1/decisions through the same Decisions response type and cost path native
/v1/decisions uses, so the _decisions cost-map rates apply. Reads and sub-routes
under /v1/decisions stay unbilled.

Fixes #48543
2026-10-31 07:34:29 -07:00
joshua-berri
be4f3d36ce
feat(accounting): capture request usage through the durable meter (#49133)
* feat(accounting): capture request usage through the durable meter

* fix(accounting): preserve native Google usage compatibility

* fix(accounting): preserve prepared usage observation on failure

* fix(accounting): exclude sync acknowledgement policy rejections from capture

* test(accounting): verify capture races and failure isolation

* fix(accounting): preserve request support with sync acknowledgements

* fix(accounting): normalize usage metadata before queue admission

* fix(accounting): preserve prior attempt links on successful fallback

* fix(accounting): preserve attempt evidence through retries and failures

* fix(accounting): depend on prior receipts without resubmitting usage

* fix(accounting): preserve pricing inputs and resolved capture attribution

* fix(accounting): persist batch usage within capture bounds

* test(accounting): cover batch dependency failure and backpressure

* fix(accounting): isolate batch persistence and preserve trusted attribution

* fix(accounting): share byte admission across batch producers and writes

* fix(accounting): reserve batch snapshot reads before admission

* fix(accounting): compact repeated batch observations without losing requests

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-10-31 07:32:25 -07:00
Mateo Wang
a713ed6bab
fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them (#47692)
* test(e2e): add gemini tool call id length repro for LIT-8912

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them

* fix(gemini): pair a tool result with its call when the client re-adds the thought signature to the call id

Strands, which the AgentCore harness runs, puts provider_specific_fields.thought_signature back on the
assistant tool call id as <id>__thought__<signature> while its tool result keeps the bare id, so the Gemini
conversion found no matching call and answered 500. Ids now match with the signature suffix stripped.

Also adds thought_signature_in_tool_call_id to the litellm_params inventory test and gives the pass-through
tests' mocked logging objects the litellm_params a real Logging object always has.

* fix(gemini): keep the thought signature on streamed Messages and Responses tool calls and accept a string setting

* fix(gemini): read the thought signature back from Responses input and coerce the per-request setting

A Responses function_call input item now keeps its provider_specific_fields, so a
client that sends the item back gets its real thought signature replayed to Gemini.

A per-request thought_signature_in_tool_call_id of "false" or 0 now keeps ids short,
and a value that is not a boolean answers 400, or is dropped under drop_params,
instead of a 500.

The e2e test now uses the sdk fixture and @meta subjects

* fix(caching): key the response cache on thought_signature_in_tool_call_id

* fix(responses): send thought signature fields only with short ids and drop malformed routing kwargs whole

* fix(main): validate routing kwargs after the locals snapshot so fallbacks and model_list calls keep working

The e2e test now uses gemini-3.8-flash with enough max_tokens to finish thinking before the tool call.

* fix(gemini): keep non-chat routes tolerant of a malformed thought_signature_in_tool_call_id, match tool results by exact id first, and keep the deployment's value when drop_params drops a malformed override

* fix(router): validate strict litellm params through a helper so star imports cannot re-bind a public Final

* perf(router): skip strict validation when a request overrides no deployment param

* fix(responses): keep a dict tool call's thought signature on the streamed function_call done item

* refactor(main): return invalid supplemental params as a value and test the streaming tool call with a real response

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 03:20:42 -07:00
paveln83
febd340120
fix(guardrails)!: keep the upstream error when a /v1/messages stream ends in one (#49088)
* fix(guardrails): keep the upstream error when a /v1/messages stream ends in one

Raw-SSE guardrail stream hooks assembled the whole stream, and the Anthropic
chunk parser raises on an error event, so a stream the upstream ended with
event: error failed assembly. tool_permission then blocked it with a 400 that
hid the upstream error, bedrock and model_armor emitted a could-not-assemble
block, and presidio skipped masking.

Assembly now leaves the error events out, so the delivered content is checked
as usual and a clean stream passes through with its error frame. A rewritten
stream ends with the upstream error instead of a message_delta and
message_stop the upstream never sent. tool_permission also passes a stream of
only error frames, which carries no tool call to check.

* fix(guardrails): do not close a block the failed upstream left open when re-emitting a rewritten stream

* fix(guardrails): check only the finished blocks of a stream cut off by an upstream error

tool_permission and Presidio now assemble an error-ended stream through the
error with an opt-in keyword, leaving out a tool call or thinking block the
upstream cut off. Logging, OpenTelemetry, Bedrock, and Model Armor keep the
old assembly. A rewrite drops the text block stop only when the upstream left
its text block open.

* fix(guardrails): keep a cut-off tool call from the client and stop each rewritten block before the next starts

A failed stream holding a cut-off tool_use is now re-emitted through the rewrite, which leaves the call out and keeps the upstream error, so a denied call never reaches the client. The rewrite leaves its text block open only when it is the last block, and keeps a message_delta the upstream sent before failing. The new tool_permission tests no longer patch should_run_guardrail.

* fix(guardrails): pass a cut-off stream with nothing denied through as sent, minus the cut-off tool call

Re-emitting such a stream through the rewrite sent an empty text block and stopped it, and Claude Code
does not retry a failed stream that shows a finished block, so it showed a mid-response error instead
of retrying. Dropping only the cut-off call's events keeps the stream as the upstream sent it.

* fix(guardrails): strip a cut-off tool call only from a stream the upstream failed

Assembly leaves unfinished blocks out of the check only when the stream carries an upstream error
event, so a stream with an unfinished but allowed tool call and no error frame now reaches the client
as sent instead of losing a call the check already read

* test(guardrails): type the parameters of the new stream error tests

* fix(guardrails): skip SSE data lines that are not JSON objects when reading a stream

A data line holding a bare JSON value such as 5 made the stream helpers call .get on an int, so tool_permission and Presidio answered 500 where main answered 400 or passed the stream through

* fix(anthropic): skip SSE data lines that are not JSON objects when assembling a stream

* test(guardrails): annotate the upstream error stream tests

* fix(anthropic): read a stream's non-object lines and named error frames the same way everywhere

Run the skip for a data line that is not a JSON object through extract_sse_data and the assembler loop, so the usage fallback keeps a stream's tokens and the chat handler keeps its contract.

Presidio now holds a leading non-object frame until the first object frame, so the rest of the stream is still masked or blocked. An event: error frame now counts by its name, so a rewrite keeps it.

* fix(guardrails): refuse an unreadable stream in Presidio and cut every kind of tool call

A non-object JSON line with no Anthropic frame after it now gets a 400 from
Presidio before any data byte, since passing it would send text main refused.
A frame named error stays a normal event when its data is another Anthropic
event. server_tool_use and mcp_tool_use cut off by an upstream error are
dropped like tool_use, and a leading frame with empty data is relayed at once.

* fix(guardrails): refuse what Presidio cannot read past a non-object line and keep non-string error types out of the frozenset

Presidio now refuses a stream assembly cannot read when it holds a non-object
data line, instead of relaying it unmasked. Non-string event types no longer
crash a rewrite, and a CRLF stream that opens with a typed error is no longer
read as one error, so its tool calls are still checked.

* fix(guardrails): refuse a first frame whose type is a list or dict in Presidio

Presidio passed PII at 200 where main answered 500 when the first read was
a JSON object with a list or dict type, or a JSON array with a character cut
at the end of the read. has_unreadable_json_event now reads both as data no
reader of these streams can take, so the first-frame check refuses them.

* test(guardrails): cover Presidio's refusal when a word or null line comes before a Gemini stream

The first read then holds the unreadable line behind one Presidio can pass,
so the check has to look at every frame in it.

* fix(guardrails): treat a stream as error-only only when every data frame is an error

A tool call split over data: lines beside one error frame read as error-only,
so tool_permission passed it, and an SDK joins those lines and reads the call.
The check now joins a frame's data lines and refuses anything but an error.

* fix(guardrails): refuse a stream past an upstream error when a client could read a frame differently

Run 8 of the live risk pass showed each stream reader imitating a client and missing a shape: a
whitespace-led data line, an error key with no type, a split tool call after a plain message_start.
An error frame now has to be plain, and assembly past an error returns None for any frame a client
could read differently, so those streams keep main's refusal.

* fix(guardrails): on a failed upstream, deliver only what tool_permission read

A failed stream now reaches the client through the rewrite that already
serves a denial, followed by the upstream error, even when nothing is
denied. Raw frames could carry a tool call the check never read in shapes
such as a reused block index, mixed line ends, a tool in message_start,
interleaved deltas, or a form feed, and each refusal rule only moved the
search to the next shape.

The error frame goes out as its event name and first data line. An error
is found at CR, LF and CRLF line ends, and the rewrite adds no finished
block when it read no text, so Claude Code still retries the stream.

* fix(guardrails): read an error frame's name as clients do and send the error alone when a rewrite cannot be built

A data-less event: error frame with junk after the name is an unknown event to both SDKs, so a healthy stream
stays raw. A finished tool call whose input is not JSON cannot be rebuilt, so a failed stream sends its upstream
error alone, and a healthy stream still raises as before.

* fix(guardrails): read an error frame as a client dispatches it and keep its JSON whole

Only the last event line of a frame a blank line ended names an error, so a
healthy stream with a later event line or an unterminated error is no failed
stream. Any rebuild failure on a failed stream now sends the upstream error
alone with a warning, a param rule denies an input nested too deep, and a raw
line break inside a JSON error stays escaped so every reader gets it whole.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-31 02:56:20 -07:00
paveln83
27fbcd9895
fix(anthropic): map pause_turn and return refusal explanations on chat and responses (#49192)
* fix(anthropic): map pause_turn and return refusal explanations on chat and responses

pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw
stop_reason in provider_specific_fields. Responses reports a paused turn as
incomplete with max_output_tokens and shows the unrun server search as an
in_progress web_search_call instead of a client function_call.

A refusal's stop_details.explanation now fills message.refusal and the final
stream delta.refusal on chat, and becomes a refusal content part on Responses,
streamed or not.

* fix(anthropic): let a chat client resend a paused turn so Claude continues it

* fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message

Chat cannot say where Claude wrote each thinking block in a paused web search turn, and
Anthropic answers 400 when one moves but accepts the turn without them. A new message after
a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped.

* fix(responses): send the streamed refusal part's type and text in content_part.added

* fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text

A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does

* fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses

* fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going

* fix(responses): mark a paused turn's message item completed

* fix(responses): mark every item of a paused turn completed

* fix(responses): keep a refusal's explanation when a refused turn is sent back

A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed

* fix(responses): scan refusal parts in Responses guardrails

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-31 02:53:59 -07:00