litellm-router-core implements the router core in Rust: a versioned registry with Python's
deployment ids, selection with native simple-shuffle, the retry, fallback and cooldown loop with
Python's cache keys, and per-attempt executors with a native seam for Messages to Anthropic. The
python bridge exposes it as RouterCoreHandle, whose call loop is step driven so Rust never awaits
Python. RustCore drives that loop for completions, messages, embeddings and responses and serves
everything else from a PythonCore built from the same config. A new router rule in the route
catalog opts in with LITELLM_RUST, and any config the Rust core cannot serve gets PythonCore at
construction
The call loop follows Python's fallback walk (per-target depth, a level raising its own error,
lookup by the current and requested group, "*" after specific chains), stamps retries the way an
exhausted retry loop does, and leaves health writes to the Python attempt, which records them.
RustCore builds its handle from the constructor arguments and the applied changesets and rebuilds
it when either version moves. The core unit suite runs on RustCore too
* refactor(router): move litellm/router.py to litellm/router_python_core.py
A pure rename with no content change, so git and review tools track the router body as a move.
The next commit turns it into PythonCore and adds the Router facade back at litellm/router.py
* refactor(router): extract PythonCore behind RouterCore and pass the router its config
The routing logic moves unchanged from litellm/router.py into PythonCore in
litellm/router_python_core.py. Router becomes a facade that builds a RouterConfig, picks a
RouterCore with make_core and delegates every member to it. The core no longer reads or writes
litellm.* directly: process settings come from RouterDynamicConfig, and inference calls, sleeps,
callback lists and the global cache go through PythonHost. A CI check bans litellm.* in the router
modules, and the proxy's construction sites build the config outside the router
Settings are static: the constructor arguments and changeset values are copied when the router is
built or the change applied, and a write to a setting attribute becomes apply(ChangeSet(assign=...)).
A Router subclass's core class also subclasses the facade, so an override's super(Sub, self) works,
and an instance-level patch of a core method is kept on the facade so nested patches unwind.
tests/unit/test_router/core tests the core in isolation: ScriptedHost serves every host operation
(scripted inference answers, recorded sleeps, settings, callbacks, cache), and make_core runs each
test once per core in CORE_BUILDERS
* test(a2a): expect JSON-RPC errors for proxy refusals on /a2a POST routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): answer proxy refusals on /a2a POST routes as JSON-RPC errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): build JSON-RPC refusal data from typed fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): stamp the OTel server span on JSON-RPC refusals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): cover JSON-RPC refusal route in unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(pr-template): mark the summary above Details as human handwritten
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(pr-template): title the summary as a handwritten 1-3 sentence description
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A request re-dispatched to a fallback keeps the same logging object but gets a
fresh start_time. completion_start_time was only filled when it was empty, so
an attempt that inherited a value from the earlier attempt reported a negative
TTFT. That value reaches both the spend log and the latency-based routing
cache, where one negative sample empties the buffer window and takes the whole
model group down until the cache entry expires.
Ignore a recorded first-token time that predates the attempt's start and fall
back to end_time, the value already used when none was recorded.
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(caching): return the server tool call id instead of a TypeGuard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Gemini cache write carries the system prompt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(moderations): stop the router image check from 500ing on a list input
* fix(moderations): give the list cast its cast-ok reason
* fix(moderations): type the responses input guard and keep SDK compaction items
* fix(moderations): import the compaction item through litellm.types and mark the TypeIs import
* fix(moderations): sort the compaction item import
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(watsonx): read passthrough credentials from use_in_pass_through deployments
/watsonx/{endpoint} always called the provider config with api_key=None and api_base=None, so it only ever
worked from WATSONX_API_KEY / WATSONX_API_BASE. It now asks PassthroughEndpointRouter for the proxy-wide
watsonx deployment flagged use_in_pass_through, like the other passthrough routes, and keeps the env
fallback when none is flagged.
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(watsonx): keep env key precedence when no deployment is flagged
Resolve only use_in_pass_through deployments in the route. get_proxy_wide_credentials also falls back to
WATSONX_API_KEY, which put it ahead of WX_API_KEY in generate_iam_token
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(streaming): fail on clean EOF without finish reason
* fix(streaming): cover shared OpenAI-compatible EOF paths
* test(router): complete mocked streaming responses
* fix(streaming): keep the EOF error message and cover Anthropic clean EOF
The request-less RemoteProtocolError tripped exception mapping into
'The .request property has not been set.' for non-OpenAI providers.
Adds Anthropic and generic-chunk unit tests, an integration test across
chat, the /v1/messages bridge and the /v1/responses bridge, and moves
main's token-estimate test onto a finished stream with a sibling that
checks the cut stream still bills the estimate on its failure log.
* fix(streaming): count a finish reason any chunk sent, keep LangGraph async streams whole
Gemini marks every chunk unfinished and a later usage chunk reset
intermittent_finish_reason, so a finished Gemini stream failed at EOF.
LangGraph's __aiter__ opened a new aiter_lines on every wrapper call,
which hit StreamConsumed after the first chunk and now surfaced as a
false EOF error instead of a silent truncation.
* fix(streaming): keep the synthetic stop for a stream that sent no chunks
Gemini's OpenAI-compatible endpoint answers with only [DONE] when
reasoning uses up max_tokens. Failing every openai/azure stream that
sent nothing turned that 200 into a 500, while every other provider
already kept the stop for an empty stream. Adds integration cells for
Gemini's trailing usage chunk, multi-event LangGraph runs, a mocked
deployment and the [DONE]-only answer.
* test(streaming): keep pytest.raises to one statement, drop the spend read for an empty stream
An empty stream writes no spend row on main either, so the [DONE]-only
cell checks only what the client sees.
* fix(streaming): read the chunk's finish reason for the sticky flag
* fix(agentcore): end a stream on messageStop when the runtime sends no metadata or final message
* test(streaming): pin the logged-chunks guard, inject the LangGraph client, use current model names
* fix(agentcore): say why the end-of-stream flags rebind
* test(streaming): cover an AgentCore runtime that ends on messageStop through the proxy
* test(streaming): cover SDK clients, request fallbacks and a concurrent burst for a cut stream
* test(agentcore): inject the async transport instead of patching the aiohttp flag
* test(guardrails): end the decision model chat fixture with a finish reason, as a real stream does
* test(streaming): shape the AgentCore and LangGraph fixtures like captures from a real runtime and server
* test(streaming): keep each stream's ending and land every spend row once through a spend log outage
* test(streaming): end each stream by its provider's finish after a proxy worker is killed mid burst
* test(streaming): run the outage test as one burst before and one during the outage instead of a capped wave loop
* test(streaming): connect the outage test's proxy to the relay without TLS so its trigger sees the spend-log insert on an ssl=on Postgres
* test(streaming): type the new stream EOF unit tests
* test(streaming): a merged reasoning stream cut mid reasoning now fails
Main's think tag test expected a stream that ends mid reasoning without a finish reason to finish with a closed think block. With this PR that stream fails, so the case moves to its own test that expects MidStreamFallbackError with the open think block as the partial content
---------
Co-authored-by: King Star <mcxin.y@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(messages): price a native /v1/messages body as served unless its provider reports its own cost
* fix(messages): bill the requested speed when a native body as served reports none
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(model-catalog): explain meter, band, channel and When
Add a Concepts section to the model-catalog AGENTS.md with worked key
breakdowns, doc comments on the undocumented litellm-cost price types,
and a cost AGENTS.md that points to the concepts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(model-catalog): write concepts as nested lists
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(model-catalog): spell out band rules and channel fallbacks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cost): split a provider-reported total between input and output by the cost map rate ratio
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): expect the kyma reported-cost headers to follow the cost map rate split
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): keep the kyma test file at its original formatting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price the response-cost header split from the served model, not the request alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The OpenAI pass-through dispatcher kept its own list of endpoints it could price,
separate from the handler, and /v1/decisions was on neither. Calls reached OpenAI
but logged $0 and no tokens.
The handler now owns the list the dispatcher reads, and prices a POST to
/v1/decisions through the same Decisions response type and cost path native
/v1/decisions uses, so the _decisions cost-map rates apply. Reads and sub-routes
under /v1/decisions stay unbilled.
Fixes#48543
* test(e2e): add gemini tool call id length repro for LIT-8912
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them
* fix(gemini): pair a tool result with its call when the client re-adds the thought signature to the call id
Strands, which the AgentCore harness runs, puts provider_specific_fields.thought_signature back on the
assistant tool call id as <id>__thought__<signature> while its tool result keeps the bare id, so the Gemini
conversion found no matching call and answered 500. Ids now match with the signature suffix stripped.
Also adds thought_signature_in_tool_call_id to the litellm_params inventory test and gives the pass-through
tests' mocked logging objects the litellm_params a real Logging object always has.
* fix(gemini): keep the thought signature on streamed Messages and Responses tool calls and accept a string setting
* fix(gemini): read the thought signature back from Responses input and coerce the per-request setting
A Responses function_call input item now keeps its provider_specific_fields, so a
client that sends the item back gets its real thought signature replayed to Gemini.
A per-request thought_signature_in_tool_call_id of "false" or 0 now keeps ids short,
and a value that is not a boolean answers 400, or is dropped under drop_params,
instead of a 500.
The e2e test now uses the sdk fixture and @meta subjects
* fix(caching): key the response cache on thought_signature_in_tool_call_id
* fix(responses): send thought signature fields only with short ids and drop malformed routing kwargs whole
* fix(main): validate routing kwargs after the locals snapshot so fallbacks and model_list calls keep working
The e2e test now uses gemini-3.8-flash with enough max_tokens to finish thinking before the tool call.
* fix(gemini): keep non-chat routes tolerant of a malformed thought_signature_in_tool_call_id, match tool results by exact id first, and keep the deployment's value when drop_params drops a malformed override
* fix(router): validate strict litellm params through a helper so star imports cannot re-bind a public Final
* perf(router): skip strict validation when a request overrides no deployment param
* fix(responses): keep a dict tool call's thought signature on the streamed function_call done item
* refactor(main): return invalid supplemental params as a value and test the streaming tool call with a real response
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep the upstream error when a /v1/messages stream ends in one
Raw-SSE guardrail stream hooks assembled the whole stream, and the Anthropic
chunk parser raises on an error event, so a stream the upstream ended with
event: error failed assembly. tool_permission then blocked it with a 400 that
hid the upstream error, bedrock and model_armor emitted a could-not-assemble
block, and presidio skipped masking.
Assembly now leaves the error events out, so the delivered content is checked
as usual and a clean stream passes through with its error frame. A rewritten
stream ends with the upstream error instead of a message_delta and
message_stop the upstream never sent. tool_permission also passes a stream of
only error frames, which carries no tool call to check.
* fix(guardrails): do not close a block the failed upstream left open when re-emitting a rewritten stream
* fix(guardrails): check only the finished blocks of a stream cut off by an upstream error
tool_permission and Presidio now assemble an error-ended stream through the
error with an opt-in keyword, leaving out a tool call or thinking block the
upstream cut off. Logging, OpenTelemetry, Bedrock, and Model Armor keep the
old assembly. A rewrite drops the text block stop only when the upstream left
its text block open.
* fix(guardrails): keep a cut-off tool call from the client and stop each rewritten block before the next starts
A failed stream holding a cut-off tool_use is now re-emitted through the rewrite, which leaves the call out and keeps the upstream error, so a denied call never reaches the client. The rewrite leaves its text block open only when it is the last block, and keeps a message_delta the upstream sent before failing. The new tool_permission tests no longer patch should_run_guardrail.
* fix(guardrails): pass a cut-off stream with nothing denied through as sent, minus the cut-off tool call
Re-emitting such a stream through the rewrite sent an empty text block and stopped it, and Claude Code
does not retry a failed stream that shows a finished block, so it showed a mid-response error instead
of retrying. Dropping only the cut-off call's events keeps the stream as the upstream sent it.
* fix(guardrails): strip a cut-off tool call only from a stream the upstream failed
Assembly leaves unfinished blocks out of the check only when the stream carries an upstream error
event, so a stream with an unfinished but allowed tool call and no error frame now reaches the client
as sent instead of losing a call the check already read
* test(guardrails): type the parameters of the new stream error tests
* fix(guardrails): skip SSE data lines that are not JSON objects when reading a stream
A data line holding a bare JSON value such as 5 made the stream helpers call .get on an int, so tool_permission and Presidio answered 500 where main answered 400 or passed the stream through
* fix(anthropic): skip SSE data lines that are not JSON objects when assembling a stream
* test(guardrails): annotate the upstream error stream tests
* fix(anthropic): read a stream's non-object lines and named error frames the same way everywhere
Run the skip for a data line that is not a JSON object through extract_sse_data and the assembler loop, so the usage fallback keeps a stream's tokens and the chat handler keeps its contract.
Presidio now holds a leading non-object frame until the first object frame, so the rest of the stream is still masked or blocked. An event: error frame now counts by its name, so a rewrite keeps it.
* fix(guardrails): refuse an unreadable stream in Presidio and cut every kind of tool call
A non-object JSON line with no Anthropic frame after it now gets a 400 from
Presidio before any data byte, since passing it would send text main refused.
A frame named error stays a normal event when its data is another Anthropic
event. server_tool_use and mcp_tool_use cut off by an upstream error are
dropped like tool_use, and a leading frame with empty data is relayed at once.
* fix(guardrails): refuse what Presidio cannot read past a non-object line and keep non-string error types out of the frozenset
Presidio now refuses a stream assembly cannot read when it holds a non-object
data line, instead of relaying it unmasked. Non-string event types no longer
crash a rewrite, and a CRLF stream that opens with a typed error is no longer
read as one error, so its tool calls are still checked.
* fix(guardrails): refuse a first frame whose type is a list or dict in Presidio
Presidio passed PII at 200 where main answered 500 when the first read was
a JSON object with a list or dict type, or a JSON array with a character cut
at the end of the read. has_unreadable_json_event now reads both as data no
reader of these streams can take, so the first-frame check refuses them.
* test(guardrails): cover Presidio's refusal when a word or null line comes before a Gemini stream
The first read then holds the unreadable line behind one Presidio can pass,
so the check has to look at every frame in it.
* fix(guardrails): treat a stream as error-only only when every data frame is an error
A tool call split over data: lines beside one error frame read as error-only,
so tool_permission passed it, and an SDK joins those lines and reads the call.
The check now joins a frame's data lines and refuses anything but an error.
* fix(guardrails): refuse a stream past an upstream error when a client could read a frame differently
Run 8 of the live risk pass showed each stream reader imitating a client and missing a shape: a
whitespace-led data line, an error key with no type, a split tool call after a plain message_start.
An error frame now has to be plain, and assembly past an error returns None for any frame a client
could read differently, so those streams keep main's refusal.
* fix(guardrails): on a failed upstream, deliver only what tool_permission read
A failed stream now reaches the client through the rewrite that already
serves a denial, followed by the upstream error, even when nothing is
denied. Raw frames could carry a tool call the check never read in shapes
such as a reused block index, mixed line ends, a tool in message_start,
interleaved deltas, or a form feed, and each refusal rule only moved the
search to the next shape.
The error frame goes out as its event name and first data line. An error
is found at CR, LF and CRLF line ends, and the rewrite adds no finished
block when it read no text, so Claude Code still retries the stream.
* fix(guardrails): read an error frame's name as clients do and send the error alone when a rewrite cannot be built
A data-less event: error frame with junk after the name is an unknown event to both SDKs, so a healthy stream
stays raw. A finished tool call whose input is not JSON cannot be rebuilt, so a failed stream sends its upstream
error alone, and a healthy stream still raises as before.
* fix(guardrails): read an error frame as a client dispatches it and keep its JSON whole
Only the last event line of a frame a blank line ended names an error, so a
healthy stream with a later event line or an unterminated error is no failed
stream. Any rebuild failure on a failed stream now sends the upstream error
alone with a warning, a param rule denies an input nested too deep, and a raw
line break inside a JSON error stays escaped so every reader gets it whole.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): map pause_turn and return refusal explanations on chat and responses
pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw
stop_reason in provider_specific_fields. Responses reports a paused turn as
incomplete with max_output_tokens and shows the unrun server search as an
in_progress web_search_call instead of a client function_call.
A refusal's stop_details.explanation now fills message.refusal and the final
stream delta.refusal on chat, and becomes a refusal content part on Responses,
streamed or not.
* fix(anthropic): let a chat client resend a paused turn so Claude continues it
* fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message
Chat cannot say where Claude wrote each thinking block in a paused web search turn, and
Anthropic answers 400 when one moves but accepts the turn without them. A new message after
a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped.
* fix(responses): send the streamed refusal part's type and text in content_part.added
* fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text
A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does
* fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses
* fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going
* fix(responses): mark a paused turn's message item completed
* fix(responses): mark every item of a paused turn completed
* fix(responses): keep a refusal's explanation when a refused turn is sent back
A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed
* fix(responses): scan refusal parts in Responses guardrails
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci(circleci): clone the public repo over HTTPS so main jobs stop failing at checkout
The project's SSH checkout key stopped authenticating on Oct 27, so every job's built-in checkout step failed with Permission denied (publickey). The repo is public, so each job now fetches its pipeline's commit anonymously over HTTPS, the way the migration jobs already did
* fix(circleci): let the PR path filter read history after the shallow clone and test the clone step itself
* fix(circleci): clone full history without file contents, retry the fetch, and record the integration schedule
The depth-1 clone left the PR path filter without a merge base, and the unshallow that fixed it pulled about 1.4 GB per filtered job. A blobless fetch gives every job the history the built-in checkout step gave, so the path filter change goes back to main's version. The fetch now gets three tries, the tests run the step under the Windows job's bare bash too, and .circleci/AGENTS.md records the integration schedule's cron
* fix(circleci): stop on the fetch error once the three tries run out, and name the GitHub App fallback as the cause
* fix(circleci): retry the checkout's file download with the fetch, since a dropped download failed the job outside the retry
* fix(circleci): fetch the routing parity base with its file contents, since the blobless clone downloaded each file separately
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count Responses WebSocket turn tokens toward key and tag TPM limits
A native Responses WebSocket turn logs its raw events instead of a usage-carrying
response object, so the v3 rate limiter read no usage and never charged the key
or tag token window. Read the turn's usage from its events the way the cost
calculator does, and charge the TPM-limited tags the socket was opened under.
* fix(proxy): refuse socket turns past TPM and count managed, failed, and realtime usage
A later response.create turn on an open Responses WebSocket now gets a read-only
check of every TPM window it counts toward (key, team, user, org, end user, team
member, model-scoped limits and the socket's tags). A turn with no tokens left gets
the 429 rate_limit_exceeded frame and the socket stays open. The check never
reserves or increments a counter and leaves RPM, budget, PTU and project windows
to their own paths. It runs on native and managed sockets.
Managed turns carry the socket's call id in their metadata, so they charge the
socket's TPM-limited tags. A failed native turn charges the partial usage its
failure row already carries. A socket refused at open by a rate limit gets the
same refusal frame and closes 1008, logged once without a traceback.
/v1/realtime sessions now count toward TPM through the realtime usage combiner,
and the v1 limiter reads socket usage through the same helper
* fix(proxy): count failed socket turns, keep socket usage off team PTU, check agent windows per turn
A native socket turn that ends in response.failed now counts the usage that event reports toward spend and TPM, as the HTTP stream path does. Native socket and realtime usage no longer charge the team PTU counter, matching the merge base. The per-turn TPM check now also reads the agent and agent session windows the socket opened with.
* fix(proxy): start missing legacy user, team, and end-user TPM buckets at zero
* fix(proxy): stop charging a realtime session's whole usage into its close minute
Realtime and live sessions log once at close, so counting them put a long session's tokens into one TPM window and refused calls that stayed under the limit. They count no TPM again, as before this branch. Also covers the legacy v1 count-once fix with an HTTP regression case.
* test(proxy): check each per-turn TPM window on its own
* fix(proxy): send a fail-closed per-turn socket refusal with its 503 status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): trace model list requests with the caller's traceparent
The proxy creates an OTEL SERVER span at the auth boundary whenever a request
carries traceparent, but /v1/models never closed it. Unended spans are not
exported, so callers saw nothing for the model list while inference requests
showed up in the same trace
Route the endpoint through the management span wrapper so the SERVER span is
closed and parented to the caller's trace. The wrapper gains a variant that
skips request and response payloads, since attaching the full model list would
bloat every poll
* refactor(proxy): type the span wrapper with ParamSpec
The strict ruff rules ban new typing.cast imports, so the wrapper keeps the
wrapped endpoint's exact signature via ParamSpec instead of casting the
wrapped callable back to its type. No behavior change
* fix(proxy): keep request bodies off the model list span and tidy its tests
The quiet span wrapper now skips the request body as well as the
response, so no payload reaches the span on success or failure. The
wrapper passes the endpoint's kwargs through instead of copying them,
and reads the caller's key with an isinstance check. The model list
test drives the real endpoint without swapping helpers, and the new
payload test checks both wrappers against a real request body
* fix(proxy): keep the 5xx status on the model list span and trace model retrieve
The span wrapper ended the server span on every failure, so a 500 or 503
from /v1/models lost its status code. It now leaves that span to the
proxy's exception handler, which knows the answered status, and only ends
it itself for an HTTPException, which no handler closes on these routes.
GET /v1/models/{model_id} (the OpenAI SDK's models.retrieve) gets the same
wrapper, so it joins the caller's trace like the listing does.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(decisions): honor a deployment's own supports_audio_input and supports_video_input flags
A Decisions deployment's model_info flags for audio and video now win over
the cost map, per media kind, the same way supports_box_questions already
does. An unlisted model flagged true gets its clips sent upstream, and a
false flag refuses them (or drops them under drop_params). OpenAI-format
providers still refuse clips since their request shape has no field for
them, and the SystemOne request builder no longer re-reads the cost map.
* fix(decisions): ignore media flags on a deployment whose own mode is not decisions
A chat-mode deployment's supports_audio_input describes chat input, so it no longer
turns on audio or video clips for Decisions. The cost map decides for it, as before.
Adds a proxy integration test for flagged, partly flagged, unflagged, and drop_params
deployments on both /v1/decisions and /v1/systemone, and for caller-sent model_info.
* fix(decisions): follow the cost map's mode for a flagged deployment without its own mode
A chat deployment flagged for audio or video usually leaves mode unset, so
its flags described chat input yet still let clips through on /v1/decisions.
When the deployment sets no mode, the cost map's mode for the model now
decides whether its media flags apply, and an unlisted model still takes
them. The audio and video lookups also run only when the request carries a
clip.
* fix(decisions): read the mode a flagged deployment falls back to from the shipped cost map
The router copies each deployment's model_info into the model's shared cost map row, so another deployment's mode there could switch off a flagged deployment's media flags. The fallback now reads the cost map as loaded, untouched by router registrations
* fix(decisions): route clips in a load-balanced group to a deployment that takes them
A Decisions or SystemOne request carrying audio or video now skips group deployments whose own flags or the cost map refuse that media, and keeps the whole group when none takes it, so the 400 still names the media.
* fix(decisions): keep deployments that drop or cannot resolve clips in the media filter
* fix(router): skip clip-refusing Decisions deployments after cooldown and tag filtering
Running the media filter before cooldown turned the 400 a group answered
when its only clip-taking deployment was cooling down into a 429. It now
runs next to the image filter, just before order filtering, in both the
async and sync selection paths.
* fix(router): honor router default drop_params when skipping clip-refusing Decisions deployments
* test(router): list the router default params helper in the router coverage check
* test(bedrock): stop reloading the cost map module in the GovCloud fixture
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* perf(logging): skip the base64 data URI scan and thread hop for text without a data URI
* perf(logging): check data URI markers before the scan and hop only for data URIs
Check for ';', 'data:', and ';base64,' before running the data URI regex, and
offload to a worker thread only when a request holds a data URI. Tests now watch
every regex call and confirm plain text is copied without yielding to the loop.
* perf(logging): keep main's copy and offload, skip the regex for strings without data URI markers
* perf(logging): copy a few large plain-text strings inline instead of hopping to a worker thread
Type the regex watcher test helpers.
* perf(logging): skip the second marker scan when copying a few large plain-text strings inline
The async gate already checks every string for the data URI markers before it copies a few large strings inline, so the copy now keeps those strings as they are instead of scanning each one again. SSE-shaped tool output, which holds both ';' and 'data:', was paying for that scan twice on the event loop.
* perf(logging): keep huge plain-text payloads on the worker thread
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add Hostinger Router as an OpenAI-compatible provider
Adds hostinger_router (https://router.hostinger.com/v1) to the
JSON-configured openai_like providers with chat completions, responses,
messages, and embeddings endpoints, cost-map entries for claude-sonnet-5,
gpt-5.2 and the two text-embedding-3 models (credits converted at
1 credit = $0.01, sourced from the public
https://router.hostinger.com/v1/models catalog), the endpoint-support
matrix, Admin UI add-model form fields and logo, and respx-mocked unit
tests covering all declared endpoints.
Also routes /v1/embeddings for JSON-configured providers that declare it
in providers.json; previously embedding calls for these providers fell
through to LiteLLMUnknownProvider.
* fix(hostinger_router): match prices to the live catalog, drop retired max_tokens, regenerate the endpoint matrix
* fix(hostinger_router): keep api_base detection off, send embeddings headers as HTTP headers, drop generic key fallback
Unprefixed models called with Hostinger's api_base keep their own provider, JSON provider
embeddings no longer borrow OPENAI_LIKE_API_KEY or litellm.api_key, and the openai_like
embedding handler sends extra_headers as request headers instead of in the JSON body
* fix(openai_like): route Hostinger messages and responses per model, keep the configured key header
Native /v1/messages and /v1/responses calls for a JSON provider now also follow the
model's cost-map supported_endpoints, so Hostinger models without the route go
through chat completions instead of a 403. Embedding key headers replace
extra_headers case-insensitively, and pinstripes stops declaring /v1/embeddings.
* fix(openai_like): let embeddings extra_headers override the key's headers as chat does
Embeddings now match chat through the OpenAI SDK, so an extra_headers entry replaces the key's header of the same name in any letter case. The Hostinger capability test checks that the supports_* lookups agree with each cost-map row instead of pinning vendor flags
* fix(openai_like): send numeric embeddings extra_headers as text and keep Hostinger's 1h cache TTL
A config.yaml header like X-Org-Id: 12345 loads as an int, which the embeddings header check rejected with a 500. Int and float values now go out as their decimal text. Hostinger Router accepts and bills a 1h cache_control TTL on /v1/messages, so its provider entry opts into keeping it.
* fix(openai_like): send Hostinger's 1h cache TTL as the 5-minute default so cache writes bill at its charge
* fix(openai_like): route Hostinger natives per model from providers.json, refuse non-text embeddings headers unless drop_params
* fix(openai_like): refuse Host, Content-Length and Transfer-Encoding in embeddings extra_headers unless drop_params
* fix(openai_like): refuse malformed embeddings extra_headers and ignore transport headers
Header names must be HTTP tokens and values printable ASCII on one line, so a
colon in a name or a line break in a value answers 400 unless drop_params drops
it. Host, Content-Length and Transfer-Encoding are ignored, since a 400 for them
sends the call to the deployment's fallback with the same headers.
* fix(openai_like): trim embeddings extra_headers values and drop values HTTP cannot carry
HTTP ignores spaces and tabs around a header value, so values are trimmed
before they are sent. A value that is not printable ASCII on one line after
trimming is dropped with a warning instead of failing the call, since the
proxy adds headers itself from key aliases and forwarded client headers that
the caller cannot fix. A malformed header name or a non-text value still
answers 400 unless drop_params drops it.
* fix(openai_like): ignore hop-by-hop extra_headers and trim line breaks around values
TE, Upgrade, Connection, Keep-Alive, Proxy-Connection and Trailer in
extra_headers are now ignored like Host, Content-Length and
Transfer-Encoding, since litellm builds the request and TE: gzip fails
HTTP/2 while Upgrade makes some upstreams hang. Line breaks at the ends of a
value are trimmed along with spaces and tabs, so a secret read from a file
with a trailing newline is sent instead of dropped. The drop warning now
names the model.
* fix(embeddings): keep one request's extra_headers out of the deployment's headers and ignore Expect
embedding() merged a request's extra_headers into the headers dict it got
from the deployment's litellm_params. The router only shallow-copies those
params, so the merge wrote the caller's headers into the deployment itself,
and later embeddings calls through that deployment sent them too. Build a
new dict instead, and drop the None checks that can no longer be true.
Also treat Expect as a transport header that hostinger_router embeddings
never forwards, and name the model with %r in the dropped-value warning.
* fix(embeddings): keep name and value pairs working in headers and answer 400 for other non-object values
The previous commit merged headers and extra_headers with dict unpacking,
which only takes a mapping, so a list of name and value pairs that
dict.update used to accept now raised a TypeError and the proxy answered
500. Merge through dict() so pairs keep working, and answer 400 for any
other value, or drop it under drop_params, as the litellm convention does
for an unsupported param.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): stop stripping cache_control from the caller's messages
Copy before removing the field, so the caller's own message objects are left alone.
* fix(fireworks_ai): strip cache_control on a copy so a fallback keeps the caller's hints
* fix: keep the caller's request intact on more providers and trim extra Claude cache hints under drop_params
* fix: drop only the oldest extra cache hints inside one message when trimming to the Claude limit
* fix: keep replayed session turns yielding their hints as whole turns while a request's own message keeps its newest hints
* fix(cache_control): count only hints that reach Claude and cap hints on OpenAI-compatible Claude routes under drop_params
* fix(cache_control): keep the capped Claude messages a list in OpenAI-compatible request bodies
* fix(caching): cap Claude hints for string drop_params, a Claude base_model, and Converse tool messages
The 4-hint cap now reads drop_params through drops_unsupported_params on chat, Converse, and Anthropic routes, names Claude by base_model on OpenAI-compatible routes, and counts a Converse tool message's hints as the one cache point it sends.
* fix(caching): trim Claude cache hints in one wire-order pass and reserve a top-level cache_control on OpenAI-compatible routes
Dropping the oldest hint one at a time re-walked the whole request per hint, which took seconds on long tool histories. The cap now drops each message's oldest hints in the order the provider counts them, so a string assistant turn loses its text hint before its tool call hints. OpenAI-compatible Claude routes also keep room for a request-level or extra_body cache_control under drop_params.
* test: drop narrating comments and stacked generator clauses from cache_control tests
* fix(vertex_ai): put every system message in the Gemini context cache so generateContent never sends it beside cachedContent
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): forward the client's own metadata on Anthropic and custom pass-through routes
* fix(proxy): forward only a valid metadata.user_id to Anthropic Messages on pass-through routes
The whole-object forward sent keys Anthropic refuses and reached other
providers, so a request that used to pass could start failing with a 400.
Read the client's user_id after hooks, drop it unless it is a string with no
email or phone number, and send only that key when the upstream URL path ends
in /v1/messages. Share the validator with the translated route, whose email
check now matches any embedded address as Anthropic's does.
* fix(proxy): forward metadata.user_id only to Anthropic upstreams no guardrail scans
Limit the pass-through user_id to Messages create on api.anthropic.com or the
configured Anthropic base, mirror the checks Anthropic answers 400 for, and keep
the field off any request a pre-call guardrail inspects. The translated route
goes back to its own user_id filter.
* fix(proxy): judge user_id forwarding on the whole request and cap it for configured Anthropic bases
* refactor(proxy): pass the user_id check a plain mapping instead of an optional body
* fix(proxy): drop an unprintable metadata.user_id and warn when a refused id is dropped
Anthropic's whitespace is not Python's, so a control character before the at sign hid an email from the filter and turned a 200 into a 400. Fireworks and Moonshot also fail on newline ids. Requiring a printable id covers both, and a refused id now logs a warning that never carries the value.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* build(deps): bump pyasn1 from 0.6.4 to 0.6.5
Bumps [pyasn1](https://github.com/pyasn1/pyasn1) from 0.6.4 to 0.6.5.
- [Release notes](https://github.com/pyasn1/pyasn1/releases)
- [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst)
- [Commits](https://github.com/pyasn1/pyasn1/compare/v0.6.4...v0.6.5)
---
updated-dependencies:
- dependency-name: pyasn1
dependency-version: 0.6.5
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump pyasn1, pypdf and minimatch to clear osv-scan
pyasn1 0.6.4 -> 0.6.5 and pypdf 6.21.2 -> 6.21.4 in uv.lock, with a dated
exclude-newer-package cutoff that admits pypdf 6.21.4 before it clears the
3-day window. minimatch has no fixed 10.x release, so the dashboard's override
and a new vscode-extension override move it to 11.0.2.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): bill Claude web search per request on deployments priced in model_info
A vertex_ai deployment priced in its model_info is registered under its hashed id, so the
web search cost check on the model key never saw "claude" and billed Claude searches with the
Gemini grounding calculator: one search per call, or Gemini's $0.035 default with no search
price. The check now also reads the model the response names.
* fix(vertex_ai): bill token-priced Claude deployments at the served model's search price
A Vertex Claude deployment priced in model_info without a search price now bills each web search at the served model's cost map price, read at cost time and never written into the deployment entry. An explicit search price, zero included, still wins.
* fix(vertex_ai): price web search on the Vertex Claude Sonnet 4.5 and Haiku 4.5 rows
Google bills Claude web search at $10 per 1,000 searches on these models, so token-priced deployments of them no longer bill searches at 0
* fix(cost): bill served-model search price on token-only Anthropic and Azure AI Claude deployments
Token-only Claude deployments on anthropic and azure_ai billed web search at 0 because the deployment entry has no search price. They now fall back to the served model's cost map search price, as vertex_ai already does. Vertex routing reads the served model first, so a Gemini deployment whose id contains claude keeps per-prompt grounding
* fix(proxy): price streamed usage frames under the deployment's model so usage.cost matches the spend row
* test(proxy): give the Vertex streamRawPredict logging doubles the request's model, as a real pass-through Logging has
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(files): default a missing file object type to "file" so OpenAI-compatible servers that omit it stop 500ing
* fix(batches): accept the vLLM production-stack router's batch object
The router leaves out the batch object's "object" field and reports "pending" and "running" statuses, so batch create and retrieve through a hosted_vllm deployment failed validation with a 500. A missing or null object now reads as "batch", pending maps to validating and running to in_progress, and any other unknown value is still rejected
* fix(batches): return OpenAI batch objects from batch list behind the vLLM router
* fix(batches): type the batch list conversion without a TypeVar and refuse an async client on the sync path
* fix(batches): keep batch list items as OpenAI's Batch so an odd item never fails a list
* refactor(types): reuse the shared JSON value alias on the batch type
* fix: return the provider page for an async client in sync batch list and pass through non-string list statuses
* test: inject the SDK's own transport in the sync batch list test so it passes on OpenAI SDK 3.x
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): resolve auto router templates through ids a * catch-all lists over a partial wildcard
* fix(ui): resolve wildcard-listed ids by their routed model in the Simple view and named aliases
A catch-all's fixed model or base_model no longer keys every listed id as that one model, a named alias like claude/* over anthropic/claude-* keys its ids by the model the router sends them to, and the Simple view suggests a pair from those ids too
* fix(ui): credit a wildcard-listed id only through the pattern and provider the router uses
When several named wildcards match a listed id, only the one the router's
specificity sort picks is credited. A captured route counts only when its
provider is one /model_group/info reports for the group, so an alias that
points elsewhere keeps its own name. Paused catch-alls credit nothing, and
a name with its own text before the star captures like the proxy lists it.
* fix(ui): resolve model group aliases before wildcards in auto router templates
An alias that shadows an id the catch-all lists now counts as the model the router sends it to, and paused deployments credit no template model
* fix(ui): wait for aliases, read them as admin only, and skip paused wildcard routes in auto router templates
* fix(ui): let slashless wildcard model names take part in route precedence
The router puts every model_name holding a star into its pattern list, with
or without a slash, so a listed id like opus-5-5 under a "*" catch-all goes
to a more specific opus-* deployment first. The dashboard skipped slashless
names when it picked the route, so a paused opus-* still credited opus-5-5
through the catch-all and a template could fill a tier that answers 403.
The old "derives no pattern from a slashless wildcard model_name" test
described /models listing, not routing, and is replaced by the routed case.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): pin admin-owned team keys to their team's MCP servers and toolsets
A virtual key on a real team owned by a proxy_admin or proxy_admin_viewer, with no MCP list of its
own, currently reaches every MCP server in the registry and any toolset, ignoring its team. These
tests expect it to resolve only what its team grants, like the same key owned by an internal user,
while the admin dashboard session keeps the whole registry.
* fix(proxy)!: bind admin-owned team keys to their team's MCP servers
A virtual key on a real team now resolves MCP servers and toolsets through its team even when its
owner is a proxy_admin or proxy_admin_viewer, the way team model access already binds it. The
unscoped admin view stays for the master key, admin keys with no team, the admin dashboard
session, and admitted user subjects. The toolset grant checks on /toolset/{name}/mcp and on the
Responses litellm_proxy MCP tool use the same rule.
BREAKING CHANGE: a team key owned by a proxy admin or admin viewer no longer reaches MCP servers or
toolsets its team does not grant. Grant them to the team or use a key with no team.
* fix(proxy): answer 403 for a toolset the key cannot use on LLM routes
A litellm_proxy MCP tool on /v1/responses, /v1/chat/completions or /v1/messages that named a toolset
the key's team was never granted was skipped, which left the server filter empty and handed the model
every tool the key could reach. The denied name now stays in the filter, so the request gets the same
403 an unknown server name gets.
Admin credentials on a real team stay bound to it whatever carries them, a lite login session included,
the way the team already bounds their model access. The tests pin that and drop an unused patch.
* fix(proxy): keep a denied toolset out of the scoped key beside a granted one
* test(responses): cover toolset name fallbacks and align MCP admin docstrings
* test(mcp): cover no-team admins' toolset access on LLM routes and drop a dead admin-check clause
* fix(responses): keep a whole-gateway MCP entry serving the key's tools beside a denied toolset
* test(responses): check the tools a whole-gateway request lists beside a denied toolset
* test(responses): keep the toolset lookup real in the whole-gateway listing test
* test(responses): run the toolset gateway tests on a real MCP server manager
* fix(responses): keep denied toolset names out of the MCP server filter
A denied toolset named like a server the key holds was matched to that server
by the loose server-name lookup. Denied names now never reach the filter, and a
request naming only denied toolsets answers 403 directly, each name once
* fix(responses): let a held MCP access group outrank a same-named ungranted toolset and give denied toolsets the generic 403
* test(responses): pin access group and granted toolset precedence for LLM-route MCP names
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): cover text verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity as chat verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses): type the bridged text verbosity lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity validation on the codex tool search follow-up turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text.verbosity on previous_response_id client tool_search follow-up
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(aiohttp): hand out bytes buffered before a dropped connection so a cut stream bills its partial usage
test(integration): stop the CyberArk override test failing when a short config value lands inside ciphertext
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(aiohttp): cover bytes already read before a dropped connection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
PR #49113 dropped this row without a stated reason while AWS still serves the EU
inference profile. Without it, calls fall back to the base Haiku 4.5 row and bill
at the non-regional price, and exact-key lookups in litellm.model_cost fail.
Restores the row unchanged from 7aa0393cb in both the root and backup maps.
Fixes#49312
