* refactor(router): move litellm/router.py to litellm/router_python_core.py
A pure rename with no content change, so git and review tools track the router body as a move.
The next commit turns it into PythonCore and adds the Router facade back at litellm/router.py
* refactor(router): extract PythonCore behind RouterCore and pass the router its config
The routing logic moves unchanged from litellm/router.py into PythonCore in
litellm/router_python_core.py. Router becomes a facade that builds a RouterConfig, picks a
RouterCore with make_core and delegates every member to it. The core no longer reads or writes
litellm.* directly: process settings come from RouterDynamicConfig, and inference calls, sleeps,
callback lists and the global cache go through PythonHost. A CI check bans litellm.* in the router
modules, and the proxy's construction sites build the config outside the router
Settings are static: the constructor arguments and changeset values are copied when the router is
built or the change applied, and a write to a setting attribute becomes apply(ChangeSet(assign=...)).
A Router subclass's core class also subclasses the facade, so an override's super(Sub, self) works,
and an instance-level patch of a core method is kept on the facade so nested patches unwind.
tests/unit/test_router/core tests the core in isolation: ScriptedHost serves every host operation
(scripted inference answers, recorded sleeps, settings, callbacks, cache), and make_core runs each
test once per core in CORE_BUILDERS
* test(a2a): expect JSON-RPC errors for proxy refusals on /a2a POST routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): answer proxy refusals on /a2a POST routes as JSON-RPC errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): build JSON-RPC refusal data from typed fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): stamp the OTel server span on JSON-RPC refusals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): cover JSON-RPC refusal route in unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(pr-template): mark the summary above Details as human handwritten
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(pr-template): title the summary as a handwritten 1-3 sentence description
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A request re-dispatched to a fallback keeps the same logging object but gets a
fresh start_time. completion_start_time was only filled when it was empty, so
an attempt that inherited a value from the earlier attempt reported a negative
TTFT. That value reaches both the spend log and the latency-based routing
cache, where one negative sample empties the buffer window and takes the whole
model group down until the cache entry expires.
Ignore a recorded first-token time that predates the attempt's start and fall
back to end_time, the value already used when none was recorded.
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(caching): return the server tool call id instead of a TypeGuard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Gemini cache write carries the system prompt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(moderations): stop the router image check from 500ing on a list input
* fix(moderations): give the list cast its cast-ok reason
* fix(moderations): type the responses input guard and keep SDK compaction items
* fix(moderations): import the compaction item through litellm.types and mark the TypeIs import
* fix(moderations): sort the compaction item import
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(watsonx): read passthrough credentials from use_in_pass_through deployments
/watsonx/{endpoint} always called the provider config with api_key=None and api_base=None, so it only ever
worked from WATSONX_API_KEY / WATSONX_API_BASE. It now asks PassthroughEndpointRouter for the proxy-wide
watsonx deployment flagged use_in_pass_through, like the other passthrough routes, and keeps the env
fallback when none is flagged.
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(watsonx): keep env key precedence when no deployment is flagged
Resolve only use_in_pass_through deployments in the route. get_proxy_wide_credentials also falls back to
WATSONX_API_KEY, which put it ahead of WX_API_KEY in generate_iam_token
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(streaming): fail on clean EOF without finish reason
* fix(streaming): cover shared OpenAI-compatible EOF paths
* test(router): complete mocked streaming responses
* fix(streaming): keep the EOF error message and cover Anthropic clean EOF
The request-less RemoteProtocolError tripped exception mapping into
'The .request property has not been set.' for non-OpenAI providers.
Adds Anthropic and generic-chunk unit tests, an integration test across
chat, the /v1/messages bridge and the /v1/responses bridge, and moves
main's token-estimate test onto a finished stream with a sibling that
checks the cut stream still bills the estimate on its failure log.
* fix(streaming): count a finish reason any chunk sent, keep LangGraph async streams whole
Gemini marks every chunk unfinished and a later usage chunk reset
intermittent_finish_reason, so a finished Gemini stream failed at EOF.
LangGraph's __aiter__ opened a new aiter_lines on every wrapper call,
which hit StreamConsumed after the first chunk and now surfaced as a
false EOF error instead of a silent truncation.
* fix(streaming): keep the synthetic stop for a stream that sent no chunks
Gemini's OpenAI-compatible endpoint answers with only [DONE] when
reasoning uses up max_tokens. Failing every openai/azure stream that
sent nothing turned that 200 into a 500, while every other provider
already kept the stop for an empty stream. Adds integration cells for
Gemini's trailing usage chunk, multi-event LangGraph runs, a mocked
deployment and the [DONE]-only answer.
* test(streaming): keep pytest.raises to one statement, drop the spend read for an empty stream
An empty stream writes no spend row on main either, so the [DONE]-only
cell checks only what the client sees.
* fix(streaming): read the chunk's finish reason for the sticky flag
* fix(agentcore): end a stream on messageStop when the runtime sends no metadata or final message
* test(streaming): pin the logged-chunks guard, inject the LangGraph client, use current model names
* fix(agentcore): say why the end-of-stream flags rebind
* test(streaming): cover an AgentCore runtime that ends on messageStop through the proxy
* test(streaming): cover SDK clients, request fallbacks and a concurrent burst for a cut stream
* test(agentcore): inject the async transport instead of patching the aiohttp flag
* test(guardrails): end the decision model chat fixture with a finish reason, as a real stream does
* test(streaming): shape the AgentCore and LangGraph fixtures like captures from a real runtime and server
* test(streaming): keep each stream's ending and land every spend row once through a spend log outage
* test(streaming): end each stream by its provider's finish after a proxy worker is killed mid burst
* test(streaming): run the outage test as one burst before and one during the outage instead of a capped wave loop
* test(streaming): connect the outage test's proxy to the relay without TLS so its trigger sees the spend-log insert on an ssl=on Postgres
* test(streaming): type the new stream EOF unit tests
* test(streaming): a merged reasoning stream cut mid reasoning now fails
Main's think tag test expected a stream that ends mid reasoning without a finish reason to finish with a closed think block. With this PR that stream fails, so the case moves to its own test that expects MidStreamFallbackError with the open think block as the partial content
---------
Co-authored-by: King Star <mcxin.y@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(messages): price a native /v1/messages body as served unless its provider reports its own cost
* fix(messages): bill the requested speed when a native body as served reports none
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(model-catalog): explain meter, band, channel and When
Add a Concepts section to the model-catalog AGENTS.md with worked key
breakdowns, doc comments on the undocumented litellm-cost price types,
and a cost AGENTS.md that points to the concepts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(model-catalog): write concepts as nested lists
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(model-catalog): spell out band rules and channel fallbacks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cost): split a provider-reported total between input and output by the cost map rate ratio
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): expect the kyma reported-cost headers to follow the cost map rate split
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): keep the kyma test file at its original formatting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price the response-cost header split from the served model, not the request alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The OpenAI pass-through dispatcher kept its own list of endpoints it could price,
separate from the handler, and /v1/decisions was on neither. Calls reached OpenAI
but logged $0 and no tokens.
The handler now owns the list the dispatcher reads, and prices a POST to
/v1/decisions through the same Decisions response type and cost path native
/v1/decisions uses, so the _decisions cost-map rates apply. Reads and sub-routes
under /v1/decisions stay unbilled.
Fixes#48543
* test(e2e): add gemini tool call id length repro for LIT-8912
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them
* fix(gemini): pair a tool result with its call when the client re-adds the thought signature to the call id
Strands, which the AgentCore harness runs, puts provider_specific_fields.thought_signature back on the
assistant tool call id as <id>__thought__<signature> while its tool result keeps the bare id, so the Gemini
conversion found no matching call and answered 500. Ids now match with the signature suffix stripped.
Also adds thought_signature_in_tool_call_id to the litellm_params inventory test and gives the pass-through
tests' mocked logging objects the litellm_params a real Logging object always has.
* fix(gemini): keep the thought signature on streamed Messages and Responses tool calls and accept a string setting
* fix(gemini): read the thought signature back from Responses input and coerce the per-request setting
A Responses function_call input item now keeps its provider_specific_fields, so a
client that sends the item back gets its real thought signature replayed to Gemini.
A per-request thought_signature_in_tool_call_id of "false" or 0 now keeps ids short,
and a value that is not a boolean answers 400, or is dropped under drop_params,
instead of a 500.
The e2e test now uses the sdk fixture and @meta subjects
* fix(caching): key the response cache on thought_signature_in_tool_call_id
* fix(responses): send thought signature fields only with short ids and drop malformed routing kwargs whole
* fix(main): validate routing kwargs after the locals snapshot so fallbacks and model_list calls keep working
The e2e test now uses gemini-3.8-flash with enough max_tokens to finish thinking before the tool call.
* fix(gemini): keep non-chat routes tolerant of a malformed thought_signature_in_tool_call_id, match tool results by exact id first, and keep the deployment's value when drop_params drops a malformed override
* fix(router): validate strict litellm params through a helper so star imports cannot re-bind a public Final
* perf(router): skip strict validation when a request overrides no deployment param
* fix(responses): keep a dict tool call's thought signature on the streamed function_call done item
* refactor(main): return invalid supplemental params as a value and test the streaming tool call with a real response
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep the upstream error when a /v1/messages stream ends in one
Raw-SSE guardrail stream hooks assembled the whole stream, and the Anthropic
chunk parser raises on an error event, so a stream the upstream ended with
event: error failed assembly. tool_permission then blocked it with a 400 that
hid the upstream error, bedrock and model_armor emitted a could-not-assemble
block, and presidio skipped masking.
Assembly now leaves the error events out, so the delivered content is checked
as usual and a clean stream passes through with its error frame. A rewritten
stream ends with the upstream error instead of a message_delta and
message_stop the upstream never sent. tool_permission also passes a stream of
only error frames, which carries no tool call to check.
* fix(guardrails): do not close a block the failed upstream left open when re-emitting a rewritten stream
* fix(guardrails): check only the finished blocks of a stream cut off by an upstream error
tool_permission and Presidio now assemble an error-ended stream through the
error with an opt-in keyword, leaving out a tool call or thinking block the
upstream cut off. Logging, OpenTelemetry, Bedrock, and Model Armor keep the
old assembly. A rewrite drops the text block stop only when the upstream left
its text block open.
* fix(guardrails): keep a cut-off tool call from the client and stop each rewritten block before the next starts
A failed stream holding a cut-off tool_use is now re-emitted through the rewrite, which leaves the call out and keeps the upstream error, so a denied call never reaches the client. The rewrite leaves its text block open only when it is the last block, and keeps a message_delta the upstream sent before failing. The new tool_permission tests no longer patch should_run_guardrail.
* fix(guardrails): pass a cut-off stream with nothing denied through as sent, minus the cut-off tool call
Re-emitting such a stream through the rewrite sent an empty text block and stopped it, and Claude Code
does not retry a failed stream that shows a finished block, so it showed a mid-response error instead
of retrying. Dropping only the cut-off call's events keeps the stream as the upstream sent it.
* fix(guardrails): strip a cut-off tool call only from a stream the upstream failed
Assembly leaves unfinished blocks out of the check only when the stream carries an upstream error
event, so a stream with an unfinished but allowed tool call and no error frame now reaches the client
as sent instead of losing a call the check already read
* test(guardrails): type the parameters of the new stream error tests
* fix(guardrails): skip SSE data lines that are not JSON objects when reading a stream
A data line holding a bare JSON value such as 5 made the stream helpers call .get on an int, so tool_permission and Presidio answered 500 where main answered 400 or passed the stream through
* fix(anthropic): skip SSE data lines that are not JSON objects when assembling a stream
* test(guardrails): annotate the upstream error stream tests
* fix(anthropic): read a stream's non-object lines and named error frames the same way everywhere
Run the skip for a data line that is not a JSON object through extract_sse_data and the assembler loop, so the usage fallback keeps a stream's tokens and the chat handler keeps its contract.
Presidio now holds a leading non-object frame until the first object frame, so the rest of the stream is still masked or blocked. An event: error frame now counts by its name, so a rewrite keeps it.
* fix(guardrails): refuse an unreadable stream in Presidio and cut every kind of tool call
A non-object JSON line with no Anthropic frame after it now gets a 400 from
Presidio before any data byte, since passing it would send text main refused.
A frame named error stays a normal event when its data is another Anthropic
event. server_tool_use and mcp_tool_use cut off by an upstream error are
dropped like tool_use, and a leading frame with empty data is relayed at once.
* fix(guardrails): refuse what Presidio cannot read past a non-object line and keep non-string error types out of the frozenset
Presidio now refuses a stream assembly cannot read when it holds a non-object
data line, instead of relaying it unmasked. Non-string event types no longer
crash a rewrite, and a CRLF stream that opens with a typed error is no longer
read as one error, so its tool calls are still checked.
* fix(guardrails): refuse a first frame whose type is a list or dict in Presidio
Presidio passed PII at 200 where main answered 500 when the first read was
a JSON object with a list or dict type, or a JSON array with a character cut
at the end of the read. has_unreadable_json_event now reads both as data no
reader of these streams can take, so the first-frame check refuses them.
* test(guardrails): cover Presidio's refusal when a word or null line comes before a Gemini stream
The first read then holds the unreadable line behind one Presidio can pass,
so the check has to look at every frame in it.
* fix(guardrails): treat a stream as error-only only when every data frame is an error
A tool call split over data: lines beside one error frame read as error-only,
so tool_permission passed it, and an SDK joins those lines and reads the call.
The check now joins a frame's data lines and refuses anything but an error.
* fix(guardrails): refuse a stream past an upstream error when a client could read a frame differently
Run 8 of the live risk pass showed each stream reader imitating a client and missing a shape: a
whitespace-led data line, an error key with no type, a split tool call after a plain message_start.
An error frame now has to be plain, and assembly past an error returns None for any frame a client
could read differently, so those streams keep main's refusal.
* fix(guardrails): on a failed upstream, deliver only what tool_permission read
A failed stream now reaches the client through the rewrite that already
serves a denial, followed by the upstream error, even when nothing is
denied. Raw frames could carry a tool call the check never read in shapes
such as a reused block index, mixed line ends, a tool in message_start,
interleaved deltas, or a form feed, and each refusal rule only moved the
search to the next shape.
The error frame goes out as its event name and first data line. An error
is found at CR, LF and CRLF line ends, and the rewrite adds no finished
block when it read no text, so Claude Code still retries the stream.
* fix(guardrails): read an error frame's name as clients do and send the error alone when a rewrite cannot be built
A data-less event: error frame with junk after the name is an unknown event to both SDKs, so a healthy stream
stays raw. A finished tool call whose input is not JSON cannot be rebuilt, so a failed stream sends its upstream
error alone, and a healthy stream still raises as before.
* fix(guardrails): read an error frame as a client dispatches it and keep its JSON whole
Only the last event line of a frame a blank line ended names an error, so a
healthy stream with a later event line or an unterminated error is no failed
stream. Any rebuild failure on a failed stream now sends the upstream error
alone with a warning, a param rule denies an input nested too deep, and a raw
line break inside a JSON error stays escaped so every reader gets it whole.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): map pause_turn and return refusal explanations on chat and responses
pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw
stop_reason in provider_specific_fields. Responses reports a paused turn as
incomplete with max_output_tokens and shows the unrun server search as an
in_progress web_search_call instead of a client function_call.
A refusal's stop_details.explanation now fills message.refusal and the final
stream delta.refusal on chat, and becomes a refusal content part on Responses,
streamed or not.
* fix(anthropic): let a chat client resend a paused turn so Claude continues it
* fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message
Chat cannot say where Claude wrote each thinking block in a paused web search turn, and
Anthropic answers 400 when one moves but accepts the turn without them. A new message after
a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped.
* fix(responses): send the streamed refusal part's type and text in content_part.added
* fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text
A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does
* fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses
* fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going
* fix(responses): mark a paused turn's message item completed
* fix(responses): mark every item of a paused turn completed
* fix(responses): keep a refusal's explanation when a refused turn is sent back
A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed
* fix(responses): scan refusal parts in Responses guardrails
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
