* fix(decisions): honor a deployment's own supports_audio_input and supports_video_input flags
A Decisions deployment's model_info flags for audio and video now win over
the cost map, per media kind, the same way supports_box_questions already
does. An unlisted model flagged true gets its clips sent upstream, and a
false flag refuses them (or drops them under drop_params). OpenAI-format
providers still refuse clips since their request shape has no field for
them, and the SystemOne request builder no longer re-reads the cost map.
* fix(decisions): ignore media flags on a deployment whose own mode is not decisions
A chat-mode deployment's supports_audio_input describes chat input, so it no longer
turns on audio or video clips for Decisions. The cost map decides for it, as before.
Adds a proxy integration test for flagged, partly flagged, unflagged, and drop_params
deployments on both /v1/decisions and /v1/systemone, and for caller-sent model_info.
* fix(decisions): follow the cost map's mode for a flagged deployment without its own mode
A chat deployment flagged for audio or video usually leaves mode unset, so
its flags described chat input yet still let clips through on /v1/decisions.
When the deployment sets no mode, the cost map's mode for the model now
decides whether its media flags apply, and an unlisted model still takes
them. The audio and video lookups also run only when the request carries a
clip.
* fix(decisions): read the mode a flagged deployment falls back to from the shipped cost map
The router copies each deployment's model_info into the model's shared cost map row, so another deployment's mode there could switch off a flagged deployment's media flags. The fallback now reads the cost map as loaded, untouched by router registrations
* fix(decisions): route clips in a load-balanced group to a deployment that takes them
A Decisions or SystemOne request carrying audio or video now skips group deployments whose own flags or the cost map refuse that media, and keeps the whole group when none takes it, so the 400 still names the media.
* fix(decisions): keep deployments that drop or cannot resolve clips in the media filter
* fix(router): skip clip-refusing Decisions deployments after cooldown and tag filtering
Running the media filter before cooldown turned the 400 a group answered
when its only clip-taking deployment was cooling down into a 429. It now
runs next to the image filter, just before order filtering, in both the
async and sync selection paths.
* fix(router): honor router default drop_params when skipping clip-refusing Decisions deployments
* test(router): list the router default params helper in the router coverage check
* test(bedrock): stop reloading the cost map module in the GovCloud fixture
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* perf(logging): skip the base64 data URI scan and thread hop for text without a data URI
* perf(logging): check data URI markers before the scan and hop only for data URIs
Check for ';', 'data:', and ';base64,' before running the data URI regex, and
offload to a worker thread only when a request holds a data URI. Tests now watch
every regex call and confirm plain text is copied without yielding to the loop.
* perf(logging): keep main's copy and offload, skip the regex for strings without data URI markers
* perf(logging): copy a few large plain-text strings inline instead of hopping to a worker thread
Type the regex watcher test helpers.
* perf(logging): skip the second marker scan when copying a few large plain-text strings inline
The async gate already checks every string for the data URI markers before it copies a few large strings inline, so the copy now keeps those strings as they are instead of scanning each one again. SSE-shaped tool output, which holds both ';' and 'data:', was paying for that scan twice on the event loop.
* perf(logging): keep huge plain-text payloads on the worker thread
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add Hostinger Router as an OpenAI-compatible provider
Adds hostinger_router (https://router.hostinger.com/v1) to the
JSON-configured openai_like providers with chat completions, responses,
messages, and embeddings endpoints, cost-map entries for claude-sonnet-5,
gpt-5.2 and the two text-embedding-3 models (credits converted at
1 credit = $0.01, sourced from the public
https://router.hostinger.com/v1/models catalog), the endpoint-support
matrix, Admin UI add-model form fields and logo, and respx-mocked unit
tests covering all declared endpoints.
Also routes /v1/embeddings for JSON-configured providers that declare it
in providers.json; previously embedding calls for these providers fell
through to LiteLLMUnknownProvider.
* fix(hostinger_router): match prices to the live catalog, drop retired max_tokens, regenerate the endpoint matrix
* fix(hostinger_router): keep api_base detection off, send embeddings headers as HTTP headers, drop generic key fallback
Unprefixed models called with Hostinger's api_base keep their own provider, JSON provider
embeddings no longer borrow OPENAI_LIKE_API_KEY or litellm.api_key, and the openai_like
embedding handler sends extra_headers as request headers instead of in the JSON body
* fix(openai_like): route Hostinger messages and responses per model, keep the configured key header
Native /v1/messages and /v1/responses calls for a JSON provider now also follow the
model's cost-map supported_endpoints, so Hostinger models without the route go
through chat completions instead of a 403. Embedding key headers replace
extra_headers case-insensitively, and pinstripes stops declaring /v1/embeddings.
* fix(openai_like): let embeddings extra_headers override the key's headers as chat does
Embeddings now match chat through the OpenAI SDK, so an extra_headers entry replaces the key's header of the same name in any letter case. The Hostinger capability test checks that the supports_* lookups agree with each cost-map row instead of pinning vendor flags
* fix(openai_like): send numeric embeddings extra_headers as text and keep Hostinger's 1h cache TTL
A config.yaml header like X-Org-Id: 12345 loads as an int, which the embeddings header check rejected with a 500. Int and float values now go out as their decimal text. Hostinger Router accepts and bills a 1h cache_control TTL on /v1/messages, so its provider entry opts into keeping it.
* fix(openai_like): send Hostinger's 1h cache TTL as the 5-minute default so cache writes bill at its charge
* fix(openai_like): route Hostinger natives per model from providers.json, refuse non-text embeddings headers unless drop_params
* fix(openai_like): refuse Host, Content-Length and Transfer-Encoding in embeddings extra_headers unless drop_params
* fix(openai_like): refuse malformed embeddings extra_headers and ignore transport headers
Header names must be HTTP tokens and values printable ASCII on one line, so a
colon in a name or a line break in a value answers 400 unless drop_params drops
it. Host, Content-Length and Transfer-Encoding are ignored, since a 400 for them
sends the call to the deployment's fallback with the same headers.
* fix(openai_like): trim embeddings extra_headers values and drop values HTTP cannot carry
HTTP ignores spaces and tabs around a header value, so values are trimmed
before they are sent. A value that is not printable ASCII on one line after
trimming is dropped with a warning instead of failing the call, since the
proxy adds headers itself from key aliases and forwarded client headers that
the caller cannot fix. A malformed header name or a non-text value still
answers 400 unless drop_params drops it.
* fix(openai_like): ignore hop-by-hop extra_headers and trim line breaks around values
TE, Upgrade, Connection, Keep-Alive, Proxy-Connection and Trailer in
extra_headers are now ignored like Host, Content-Length and
Transfer-Encoding, since litellm builds the request and TE: gzip fails
HTTP/2 while Upgrade makes some upstreams hang. Line breaks at the ends of a
value are trimmed along with spaces and tabs, so a secret read from a file
with a trailing newline is sent instead of dropped. The drop warning now
names the model.
* fix(embeddings): keep one request's extra_headers out of the deployment's headers and ignore Expect
embedding() merged a request's extra_headers into the headers dict it got
from the deployment's litellm_params. The router only shallow-copies those
params, so the merge wrote the caller's headers into the deployment itself,
and later embeddings calls through that deployment sent them too. Build a
new dict instead, and drop the None checks that can no longer be true.
Also treat Expect as a transport header that hostinger_router embeddings
never forwards, and name the model with %r in the dropped-value warning.
* fix(embeddings): keep name and value pairs working in headers and answer 400 for other non-object values
The previous commit merged headers and extra_headers with dict unpacking,
which only takes a mapping, so a list of name and value pairs that
dict.update used to accept now raised a TypeError and the proxy answered
500. Merge through dict() so pairs keep working, and answer 400 for any
other value, or drop it under drop_params, as the litellm convention does
for an unsupported param.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): stop stripping cache_control from the caller's messages
Copy before removing the field, so the caller's own message objects are left alone.
* fix(fireworks_ai): strip cache_control on a copy so a fallback keeps the caller's hints
* fix: keep the caller's request intact on more providers and trim extra Claude cache hints under drop_params
* fix: drop only the oldest extra cache hints inside one message when trimming to the Claude limit
* fix: keep replayed session turns yielding their hints as whole turns while a request's own message keeps its newest hints
* fix(cache_control): count only hints that reach Claude and cap hints on OpenAI-compatible Claude routes under drop_params
* fix(cache_control): keep the capped Claude messages a list in OpenAI-compatible request bodies
* fix(caching): cap Claude hints for string drop_params, a Claude base_model, and Converse tool messages
The 4-hint cap now reads drop_params through drops_unsupported_params on chat, Converse, and Anthropic routes, names Claude by base_model on OpenAI-compatible routes, and counts a Converse tool message's hints as the one cache point it sends.
* fix(caching): trim Claude cache hints in one wire-order pass and reserve a top-level cache_control on OpenAI-compatible routes
Dropping the oldest hint one at a time re-walked the whole request per hint, which took seconds on long tool histories. The cap now drops each message's oldest hints in the order the provider counts them, so a string assistant turn loses its text hint before its tool call hints. OpenAI-compatible Claude routes also keep room for a request-level or extra_body cache_control under drop_params.
* test: drop narrating comments and stacked generator clauses from cache_control tests
* fix(vertex_ai): put every system message in the Gemini context cache so generateContent never sends it beside cachedContent
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): forward the client's own metadata on Anthropic and custom pass-through routes
* fix(proxy): forward only a valid metadata.user_id to Anthropic Messages on pass-through routes
The whole-object forward sent keys Anthropic refuses and reached other
providers, so a request that used to pass could start failing with a 400.
Read the client's user_id after hooks, drop it unless it is a string with no
email or phone number, and send only that key when the upstream URL path ends
in /v1/messages. Share the validator with the translated route, whose email
check now matches any embedded address as Anthropic's does.
* fix(proxy): forward metadata.user_id only to Anthropic upstreams no guardrail scans
Limit the pass-through user_id to Messages create on api.anthropic.com or the
configured Anthropic base, mirror the checks Anthropic answers 400 for, and keep
the field off any request a pre-call guardrail inspects. The translated route
goes back to its own user_id filter.
* fix(proxy): judge user_id forwarding on the whole request and cap it for configured Anthropic bases
* refactor(proxy): pass the user_id check a plain mapping instead of an optional body
* fix(proxy): drop an unprintable metadata.user_id and warn when a refused id is dropped
Anthropic's whitespace is not Python's, so a control character before the at sign hid an email from the filter and turned a 200 into a 400. Fireworks and Moonshot also fail on newline ids. Requiring a printable id covers both, and a refused id now logs a warning that never carries the value.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): bill Claude web search per request on deployments priced in model_info
A vertex_ai deployment priced in its model_info is registered under its hashed id, so the
web search cost check on the model key never saw "claude" and billed Claude searches with the
Gemini grounding calculator: one search per call, or Gemini's $0.035 default with no search
price. The check now also reads the model the response names.
* fix(vertex_ai): bill token-priced Claude deployments at the served model's search price
A Vertex Claude deployment priced in model_info without a search price now bills each web search at the served model's cost map price, read at cost time and never written into the deployment entry. An explicit search price, zero included, still wins.
* fix(vertex_ai): price web search on the Vertex Claude Sonnet 4.5 and Haiku 4.5 rows
Google bills Claude web search at $10 per 1,000 searches on these models, so token-priced deployments of them no longer bill searches at 0
* fix(cost): bill served-model search price on token-only Anthropic and Azure AI Claude deployments
Token-only Claude deployments on anthropic and azure_ai billed web search at 0 because the deployment entry has no search price. They now fall back to the served model's cost map search price, as vertex_ai already does. Vertex routing reads the served model first, so a Gemini deployment whose id contains claude keeps per-prompt grounding
* fix(proxy): price streamed usage frames under the deployment's model so usage.cost matches the spend row
* test(proxy): give the Vertex streamRawPredict logging doubles the request's model, as a real pass-through Logging has
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(files): default a missing file object type to "file" so OpenAI-compatible servers that omit it stop 500ing
* fix(batches): accept the vLLM production-stack router's batch object
The router leaves out the batch object's "object" field and reports "pending" and "running" statuses, so batch create and retrieve through a hosted_vllm deployment failed validation with a 500. A missing or null object now reads as "batch", pending maps to validating and running to in_progress, and any other unknown value is still rejected
* fix(batches): return OpenAI batch objects from batch list behind the vLLM router
* fix(batches): type the batch list conversion without a TypeVar and refuse an async client on the sync path
* fix(batches): keep batch list items as OpenAI's Batch so an odd item never fails a list
* refactor(types): reuse the shared JSON value alias on the batch type
* fix: return the provider page for an async client in sync batch list and pass through non-string list statuses
* test: inject the SDK's own transport in the sync batch list test so it passes on OpenAI SDK 3.x
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): pin admin-owned team keys to their team's MCP servers and toolsets
A virtual key on a real team owned by a proxy_admin or proxy_admin_viewer, with no MCP list of its
own, currently reaches every MCP server in the registry and any toolset, ignoring its team. These
tests expect it to resolve only what its team grants, like the same key owned by an internal user,
while the admin dashboard session keeps the whole registry.
* fix(proxy)!: bind admin-owned team keys to their team's MCP servers
A virtual key on a real team now resolves MCP servers and toolsets through its team even when its
owner is a proxy_admin or proxy_admin_viewer, the way team model access already binds it. The
unscoped admin view stays for the master key, admin keys with no team, the admin dashboard
session, and admitted user subjects. The toolset grant checks on /toolset/{name}/mcp and on the
Responses litellm_proxy MCP tool use the same rule.
BREAKING CHANGE: a team key owned by a proxy admin or admin viewer no longer reaches MCP servers or
toolsets its team does not grant. Grant them to the team or use a key with no team.
* fix(proxy): answer 403 for a toolset the key cannot use on LLM routes
A litellm_proxy MCP tool on /v1/responses, /v1/chat/completions or /v1/messages that named a toolset
the key's team was never granted was skipped, which left the server filter empty and handed the model
every tool the key could reach. The denied name now stays in the filter, so the request gets the same
403 an unknown server name gets.
Admin credentials on a real team stay bound to it whatever carries them, a lite login session included,
the way the team already bounds their model access. The tests pin that and drop an unused patch.
* fix(proxy): keep a denied toolset out of the scoped key beside a granted one
* test(responses): cover toolset name fallbacks and align MCP admin docstrings
* test(mcp): cover no-team admins' toolset access on LLM routes and drop a dead admin-check clause
* fix(responses): keep a whole-gateway MCP entry serving the key's tools beside a denied toolset
* test(responses): check the tools a whole-gateway request lists beside a denied toolset
* test(responses): keep the toolset lookup real in the whole-gateway listing test
* test(responses): run the toolset gateway tests on a real MCP server manager
* fix(responses): keep denied toolset names out of the MCP server filter
A denied toolset named like a server the key holds was matched to that server
by the loose server-name lookup. Denied names now never reach the filter, and a
request naming only denied toolsets answers 403 directly, each name once
* fix(responses): let a held MCP access group outrank a same-named ungranted toolset and give denied toolsets the generic 403
* test(responses): pin access group and granted toolset precedence for LLM-route MCP names
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): cover text verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity as chat verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses): type the bridged text verbosity lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity validation on the codex tool search follow-up turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text.verbosity on previous_response_id client tool_search follow-up
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(aiohttp): hand out bytes buffered before a dropped connection so a cut stream bills its partial usage
test(integration): stop the CyberArk override test failing when a short config value lands inside ciphertext
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(aiohttp): cover bytes already read before a dropped connection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): bill completion-bridge agents at their own prices on /a2a send and stream
message/send on an agent served through the completion bridge (custom_llm_provider
agents such as LangGraph or databricks_agent) logged a $0, 0-token row, and
message/stream logged no row at all, so cost_per_query and per-token prices never
reached spend.
The bridge send path now records counted usage, the agent's pricing keys, and the
agent id on its logging object, the same way URL agents do. The bridge stream is
wrapped in the A2A streaming logger, which now reads the bridge's artifact event
for the reply text and gets the agent name from the proxy. Streamed calls also
count prompt tokens again, since the message text reader now reads pydantic
messages as dicts.
* fix(a2a): count chunked and Task replies once and skip agent prices on error replies
Bridge streams now follow the A2A append flag per artifact, so chunked
pydantic-ai, watsonx, and AgentCore replies count the whole reply once.
Send replies that are a Task count their artifact text. A JSON-RPC error
reply is logged at cost 0 instead of the agent's price.
* fix(a2a): keep one reply text per stream and bill every artifact and status reply
* fix(a2a): ignore a null error field on streams and run the stream per-token billing test
A stream event whose error field is null no longer zeroes the call's cost. The integration test for per-token stream billing now runs instead of skipping, and the URL agent SDK case skips when the optional a2a-sdk is not installed.
* fix(a2a): bill streams the client drops and skip the bridge's working placeholder
A stream the client closes or cancels mid-reply now logs its spend row at the agent's prices on both the bridge and URL paths, and a stream dropped before the agent sends anything still logs nothing. The bridge's "Processing request..." status no longer counts as reply text, so an empty bridge reply bills no completion tokens, as send does. Adds proxy-level tests that a bridge stream bills under the agent's card name.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality
sub-routers now take a RoutingContext instead of the Router, and read the router only through
registry(), settings(), health() and the context's forwarding methods. Router-free helpers move
from router_utils to litellm_core_utils, outside callers of router state switch to entry points,
and the surface allowlist shrinks to the endpoint methods plus the entry points
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* fix(guardrails): keep request steps when a response pipeline blocks before a later request pipeline
