* feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos
A provider registered in custom_provider_map with a BaseVideoConfig handler now resolves on every
video call type (create, status, content, list, remix, edit, extension, characters), the same way
CustomLLM handlers already resolve on /v1/images/generations. The video entry points and the HTTP
handler stop casting the provider name to LlmProviders, which raised before any lookup ran, and a
provider without a video config gets a 400 instead of a 500.
* fix(videos): type the custom provider lookups so the type gate holds
Pick the CustomLLM for a custom provider through ProviderConfigManager.get_custom_llm_handler,
which narrows the now wider custom_provider_map handler union, read custom_provider_map instead
of the private _custom_providers list, and annotate the untyped custom_llm_provider parameters
on the video entry points
* test(videos): cover a CustomLLM shadowing a built-in video provider name and type the new tests
Mark custom_handler ReadOnly on CustomLLMItem, type the new tests, and assert that a CustomLLM
registered under openai still leaves openai video on the built-in config
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Runs every case through the Python /v1/messages path and the Rust path against one recording upstream and compares dispatch, upstream request, outcome, hidden params, standard logging fields and callback kwargs. Known Rust differences are strict xfails tagged with their inventory ID.
* feat(decisions): add Parallel AI web-grounded decisions
* fix(decisions): name the Parallel model decision
* fix(parallel_ai): restrict environment credentials to trusted decision endpoints
* fix(decisions): keep Parallel evidence fields off other providers and handle missing confidence
Only fields a provider names reach the OpenAI response, so existing providers keep
their base shape. A malformed api_base answers 400 on both credential paths. The
complexity router classifier and the Playground handle a provider that omits
confidence. Regenerates the lazy OpenAPI snapshot and the dashboard API types
* chore(cost-map): declare the Decisions routes on parallel_ai/decision
The provider endpoint files are generated from the cost map, so the row must
name both routes for the generated System One flag to hold
* fix(decisions): report no classifier confidence when a provider omits it
A choice answer with no confidence is a valid classifier answer, so the
complexity router picks its tier and leaves classifier_confidence and the
confidence signal out instead of reporting a different metric in its place.
Also show the Parallel AI logo in the provider picker and add a Parallel
translation fixture captured from the live API for both Decisions routes.
* fix(decisions): keep a token-priced row on token billing when it also sets a query price
A plain input_cost_per_query bills flat only on a row with no token price, and input_cost_per_query_decisions always bills flat, so a custom Decisions row that already bills tokens costs the same as before this PR
* fix(decisions): bill a deployment's query price and accept Parallel bases that end in a route
A proxy request logs through an object built before the router picks the deployment, so a query price on litellm_params never reached Decisions billing. Parallel Search accepts a base ending in /v1/search and shares PARALLEL_AI_API_BASE with Decisions, so Decisions now accepts that base and one ending in /v1/decisions.
* fix(decisions): keep per-query billing and optional confidence on Parallel only
A query price on any other Decisions provider moved that provider off token
billing, and a choice or score answer with no confidence became a 200 with a
null for every provider. Both now apply to Parallel alone, so other providers
bill and fail as they do on main.
A Parallel deployment with explicit zero token rates and a query price read as
free, so every budget check was skipped while it billed per call. It now counts
as priced.
The dashboard key for Parallel is now PARALLEL_AI so a served bundle that
predates the provider still sends parallel_ai as the provider slug, and the
README table lists the provider.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): expand wildcard models for providers with model info but no cost map entry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): stop requiring an api key to list vllm models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve model ids during wildcard expansion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): forward api key for model discovery
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): support bearer auth for model discovery
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): list hosted_vllm models from the server chat uses, with a bounded wait
The hosted_vllm listing now resolves HOSTED_VLLM_API_BASE and HOSTED_VLLM_API_KEY like hosted_vllm chat, instead of the /vllm pass-through's VLLM_API_BASE, and waits at most NON_LLM_CONNECTION_TIMEOUT rather than the completion timeout. vllm/* keeps listing nothing, since its chat runs in process. The router pass-through now forwards the deployment's api_key.
* fix(vllm): skip the placeholder key, honor deployment TLS and headers, and retry failed listings after a minute
* fix(discovery): cache only timed-out model listings, so a fast failure is tried again on the next call
* fix(proxy): list wildcard models off the event loop, so a hung provider no longer stalls other requests
* fix(utils): keep a concurrent call's model listing when another call times out
* fix(proxy): run model discovery on its own thread pool and keep credential lookups off the event loop
* fix(proxy): stop the file credential lookup at the first match before listing wildcard models
The fallback lookup now walks the allowed model names lazily, in the same
order as before, so a hung vLLM server is listed only when no configured
name serves the requested provider. The vLLM auth headers no longer
mutate the caller's headers dict.
* fix(proxy): try every configured wildcard route before listing any wildcard's models in the file credential lookup
* fix(proxy): answer model lists from cached listings first and share one listing per server
* test(vllm): give the self-signed cert fixture fixed validity dates
* fix(proxy): skip wildcard servers a key cannot reach on /v1/model/info
/v1/model/info listed every wildcard deployment and filtered by the key
afterwards, so a key limited to one model waited on every wildcard server and
sent listing calls to servers it cannot use. The listing now skips a wildcard
whose patterns match none of the key's models before it asks the server.
A key or team granted hosted_vllm/* with no deployment of that name now lists
no models, like openrouter/*, since chat answers 400 for every id the
environment server would list.
* fix(proxy): match a wildcard's own row pattern before asking its server
The /v1/model/info pre-filter matched a key's models against both the
deployment name and its litellm_params.model, so a key granted
hosted_vllm/* still asked a vllm-secure/* -> hosted_vllm/* server whose
rows it can never see. It now matches only the pattern expansion names
the rows with, picked by one shared helper.
* test(vllm): cover the vLLM listings and no-slash wildcard aliases the patch left unrun
Adds tests for hosted_vllm get_models with and without a server, vllm/*
never asking a configured server, and /model/info pre-filtering of
wildcard aliases whose name has no slash. Drops a test comment that
only repeated a method signature.
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): bill web search interception follow-ups once under the outer request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): keep web search rounds inside interception follow-ups billed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): keep the outer request as parent for nested interception follow-ups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(spend): drop parent override for nested interception follow-ups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): reproduce comparison response ID ownership and wire failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): guard stale retries and malformed comparison envelopes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): authorize and unwrap comparison response ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): require comparison IDs to follow response security policy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): deny raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): reject raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): enforce managed ownership with unmanaged ID opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date
* fix(proxy): report a deployment's own provider retirement date before the bare model's
An Azure deployment declared as model gpt-4o with custom_llm_provider azure, or as
azure/<deployment> with base_model gpt-4o, now resolves azure/gpt-4o's date instead of
the bare OpenAI row's. The GPT-4 series alias test now reads both cost maps with typed rows
* fix(proxy): read Azure's retirement schedule for azure_ai deployments
An azure_ai deployment with no azure_ai row of its own now reports the azure/<model> date, and a cost-map row wins over the provider-less copy the router registers under the deployment's key, which carries the bare model's date.
* fix(proxy): let custom and registered base_model rows keep their deprecation date
Only the copy the router writes at startup, which has no provider and inherits the bare row's date, gives way to a later candidate
* fix(proxy): report a pinned snapshot's own earlier deprecation date over its base_model's
* fix(proxy): recognize the router's copied cost-map rows by the loaded catalog
* fix(proxy): read Azure's schedule for a provider-prefixed azure_ai base_model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tool_choice): accept OpenAI allowed_tools on chat completions and flatten it for the Responses bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: type the allowed_tools tool_choice fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: raw regex for the allowed_tools provider gate match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tool_choice): move the allowed_tools provider gate out of completion() so pyright can still analyze it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tool_choice): keep the allowed_tools helpers typed so the basedpyright gate stays level
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tool_choice): type the allowed_tools validation boundary with Any instead of casts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tool_choice): widen tool_choice to object at the allowed_tools helpers with a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(tool_choice): drop allowed_tools test docstrings, type the new tests, and revert unrelated reformatting
Also reword the allowed_tools 400 so it no longer blames the flat Responses layout for every malformed choice
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool
Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking
sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model
map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The
GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows
* fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool
Native output without thinking now applies to every Bedrock Claude model that cannot
force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5
needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose.
The invoke chat path keeps the native output_format it built instead of inlining it, and
GovCloud region-path ids read their own false rows.
* fix(bedrock): clean caller output_format schemas before sending them natively on invoke
* fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke
* fix(bedrock): keep the faked Converse stream for native output that replaces the json tool
A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture.
* test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(anthropic): pick beta allowlist row per deployment, add mid-2026 beta rows, surface input_transformations
Foundry-hosted anthropic deployments now filter anthropic-beta by the azure_ai row on /v1/messages and chat completions, the bundled config gains the mid-conversation-system-clear-at-2026-08-21 row plus the tool-changes and output-config betas on azure_ai and vertex_ai from a live probe, and Anthropic's input_transformations array reaches chat completions in provider_specific_fields (non-streaming and the first streaming chunk).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): type the provider_specific_fields source dict and beta row test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): input_transformations reaches chat completions and /v1/messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): read the Anthropic response as a Mapping when building provider_specific_fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): fold deployment-row tests into the filtering file and mirror the new beta in the Rust table
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): drop the duplicated beta tuple and mark filter results Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): forward the seven betas Foundry accepts on the azure_ai beta row
A Foundry-hosted anthropic deployment now filters through the azure_ai row, which
dropped compact-2026-01-12, fast mode, fine-grained tool streaming, inline tools,
oauth, output-128k, and token-efficient tools. Foundry answers 200 to each, and
without compact-2026-01-12 a compaction request answers 400, so the row now allows
all seven on both the Python map and the Rust table
* fix(anthropic): keep pass-through and beta-less requests off the Foundry beta row
An openai/ deployment that passes /v1/messages through to a Foundry host lost
every beta, and a request without betas read the beta table from GitHub.
Only an anthropic deployment on a Foundry host now takes the azure_ai row,
and the check reads the provider and host before the table.
* test(anthropic): point the beta-gated wire legs whose row lacks clear-at at a native Databricks deployment
* style(rust): wrap the azure_ai beta header test case the way rustfmt does
* fix(anthropic): forward unlisted betas to Foundry and keep Foundry calls off the Rust messages route
An anthropic deployment on a Foundry host now drops only the betas the azure_ai row marks
unsupported on /v1/messages, and forwards betas the table does not list as sent, as main did.
cognitiveservices.azure.com counts as a Foundry host. The Rust messages route sends an async call
with a Foundry api_base to Python, which filters by row. A sync call stays on Rust, since Python has
no sync /v1/messages handler. The json_mode chat response keeps input_transformations,
and the first /v1/messages call with betas loads the beta table off the event loop.
* fix(anthropic): run the advisor loop for an anthropic deployment on a Foundry host
The advisor interceptor treated every anthropic deployment as native, so a Foundry-hosted one
received the advisor_20260301 tool and Foundry answered 400. Interceptors now get the
deployment's beta row from resolve_beta_provider, so that deployment runs litellm's advisor
loop the same way an azure_ai deployment does
* test(anthropic): drive the Foundry advisor loop test through recorded HTTP
The test now intercepts the Foundry and Anthropic HTTP calls with respx instead of patching litellm's own sub-call helper, so it checks what Foundry receives: no advisor_20260301 tool and no advisor beta
* fix(anthropic): leave the advisor loop to the providers it ran for on main and tidy the PR's tests
Reverts 083990f6d7, since review found it reaches a pre-existing gap that is now with security engineering.
The beta row tests drive a real HTTP transport, the beta table fetch test checks the fetch thread instead of a 2 second deadline, and the touched test files keep main's formatting
* test(anthropic): give the Foundry beta row logging fixture a fixed datetime start time
* test(anthropic): check that chat completions pick the beta row by api_base
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: Return Anthropic error envelopes on Messages auth failures
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Answer Anthropic envelopes on Messages auth failures without relabelling unmapped statuses
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(proxy): answer the Anthropic error envelope on the /anthropic pass-through
The pass-through relays upstream errors in Anthropic's shape, but the proxy's
own refusals there (bad key, model access, budget) still answered the OpenAI
body, so an Anthropic SDK pointed at /anthropic could not read them
* fix(proxy): answer the Anthropic envelope for 404, 405 and unhandled 500 errors on Messages routes
A disabled Claude Code gateway 404, a wrong-method 405 and an unhandled 500 on the
Messages and /anthropic/ routes kept FastAPI's detail body or the OpenAI shape. The
central HTTP and unhandled-exception handlers now pick the Anthropic envelope by route.
The dashboard playground toast titles Anthropic error types when the body has no code.
* fix(proxy): keep LiteLLM error types and Anthropic envelopes on Messages routes
Error bodies on the Messages routes now carry error.litellm_error_type when the
LiteLLM type differs from the Anthropic type, so clients and the Admin UI keep
causes like budget_exceeded. Unlisted 4xx statuses map to invalid_request_error
per Anthropic's errors page, model-list requests that send anthropic-version get
the Anthropic envelope, and the admission, decompression and request-size
middleware errors use it on these routes too.
* test(proxy): expect litellm_error_type in Messages integration tests and type middleware tests
* fix(proxy): keep handler error bodies unchanged and timeouts as api_error
Errors the Messages and count_tokens handlers already returned in the Anthropic
shape keep their original bodies, so litellm_error_type now appears only on
errors the proxy raises before the handler. A 408 stays api_error as before,
and the playground toast titles the new Anthropic types and ignores long
numbers in front of an embedded error
* fix(proxy): keep the base error types inside the Messages handler and scope toast titles to Anthropic bodies
The wider status table (402 billing_error, 409 conflict_error, 504 timeout_error,
other unlisted 4xx invalid_request_error) now applies only to errors the proxy
raises before the handler. Errors inside the Messages handler, streamed error
frames and the SDK streaming adapters keep the merge base's table.
The dashboard toast reads a status before embedded JSON and titles by Anthropic
type only when the body is an Anthropic error envelope, so other pages keep
their titles.
* test(anthropic): mark the new test variables Final
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): add the Codex file edit and long session rows
The file edit row has Codex change a random value into config.txt with the patch tool and checks the file on disk. The long session row has it follow a chain of random file names one cat at a time. new_workspace now returns the resolved path, because Codex reports patched files by their real path (/private/var on macOS).
* test(e2e): make the Codex long session prove every earlier command output reached the model
* test(e2e): Codex auto-compaction through LiteLLM (#49389)
* test(e2e): add the Codex auto-compaction row
Codex reads a random value, compacts through the gateway with its remote compaction forced on, and is asked for the value after the file holding it is deleted. The row reads the compaction from the thread's rollout file and marks Claude on Bedrock as a strict expected failure, since the Responses-to-chat bridge ignores the compaction_trigger item
* refactor(e2e): validate Codex rollout lines with the models directly
* test(e2e): drive the real Codex CLI through the gateway and add the command round trip row
Adds tests/e2e/codex: a harness that runs codex exec --json with a throwaway HOME and CODEX_HOME, a custom responses provider pointed at the gateway, the model list read from the gateway's /v1/models, the sandbox on and approvals off. Each test registers its setup's deployment via /model/new and calls it with a key scoped to it, both deleted on teardown. The first row checks that a value only a command can print comes back through the gateway on all five setups
* test(e2e): type the Codex self-test event helper as a Mapping
* test(e2e): clean up Codex test keys even when setup fails, and fail runs where Codex exits with an error
* test(e2e): record a step when the Codex harness sets up its home and workspace
The suite rules ask every public harness helper that does something to record a step, so a failed run shows whether it died while setting up Codex or while running it
* Update run.py
wip
* fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): gemini tool-call signature rejected by the backend is resent once with the placeholder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): keep the placeholder resend inside the cache rebuild so either rejection order ends in an accepted body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): resend a cached-prefix signature inline and cover native generateContent
When the rejected signature sits inside a cachedContents prefix, the
placeholder resend now carries the history inline, since replacing the
signature in the request body cannot reach the cache. The native
generateContent handlers get the same one-time resend.
* fix(gemini): send the history inline when Vertex rejects a signature while creating the cache
* fix(gemini): keep the context cache on a signature resend and go inline only when that is rejected too
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(managed_files): keep the upload the managed id encodes when two target models share a deployment
* fix(proxy): skip cost tracking for a file delete routed to a deployment
* fix(managed_files): delete repeat uploads after the upload answers
Run the repeat-copy deletes together in a background task so a slow
provider delete no longer holds the upload's response, log the unbilled
file-delete cost skip at debug, and type the new test helpers
* fix(proxy): run managed file repeat-upload cleanup as a drained internal call
Delete repeat uploads after the record is saved, through the shared background
task list the shutdown drain waits on, and as an internal call so logging,
spend tracking and alerts skip it. Leave litellm_proxy deployments on the old
keep-last behavior, since their per-target copies are distinct files. Revert
the cost tracker skip change, which the internal call makes unnecessary.
* fix(managed_files): delete the copy a managed id names on DELETE and isolate cleanup metadata
* fix(managed_files): read the copy DELETE removes from the id's gateway-written fields
* fix(managed_files): delete the copy a managed id carries as an internal call
* fix(managed_files): pass the parent call id to the cleanup deletes
* fix(managed_files): file each repeat-upload delete under the upload that made its copy
POST /v1/files sets no request call id, so the cleanup delete passed none
and its usage ledger row had no parent. Each copy now keeps the call id of
the router upload that made it, and its delete runs under that id.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Every MCP route (/mcp, /<server>/mcp, /toolset/<name>/mcp, /mcp/proxy) ran LiteLLM auth before looking at the HTTP method, so a bare OPTIONS probe got the RFC 9728 401 OAuth challenge and clients that probe with OPTIONS looped on re-auth. The Streamable HTTP handler now returns 204 with Allow: GET, POST, DELETE, OPTIONS before auth runs. GET, POST and DELETE are unchanged.
Resolves LIT-10231
* refactor(ui): move admin and settings text onto i18n
Route the page-authored English on the admin panel, caching, access
groups, tag management, logging and alerts, and router settings pages
through react-i18next, with 1,183 messages added to en.json. The
rendered English is unchanged
* fix(ui): guard i18n label lookups against prototype-named values
A server value such as "toString" used as a key into a label map
resolved to an Object.prototype member and was passed to t(), which
throws. Check own properties before translating so those values fall
back to the raw string as before
* fix(proxy): record the route's call type on failed speech spend rows
A failed /v1/audio/speech request never puts a logging object on the request
data, so the failure lift returned nothing and the spend row's call_type came
out empty (or whatever the client sent in the body). Without a logging object,
lift the call type the route already resolved, so a failure row records the
same call type a success row on that route records
* fix(proxy): keep the call type of a failure logged under its own route
An MCP tool listing that fails inside a Responses, chat, or messages call
is logged under /mcp/list_tools, so its spend row keeps list_mcp_tools
instead of taking the outer request's call type
* fix(proxy): take a failed row's call type from its endpoint, not a route path guess
* fix(proxy): log the endpoint call type on failed image generation and rerank rows
* fix(proxy): estimate failed-call ledger usage from the route's call type again
The failure usage snapshot now gets the same call type the spend row uses for
its estimate, so a failed Responses call counts input from the newest
compaction item again, as on main.
The endpoint call type tests now go through real routers whose provider
answers 400, instead of asserting on a mocked failure hook.
* test(proxy): type the failed spend row tests and fixture
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Move the page-authored English on the agents, vector stores, prompts,
search tools, memory and skills pages, plus the shared skill detail,
make-skill-public and model selector components, into en.json behind t().
The English on screen is unchanged. Keys are added under agents,
vectorStores, prompts, searchTools, memory, skills, claudeCodePlugins and
modelSelector, inserted at the top of en.json
Strings that could only move with a structural change stay English,
along with API values, code samples, paths and names
Moves the text the /guardrails page writes itself onto react-i18next under a
new top-level `guardrails` block in en.json, placed right before
`guardrailsMonitor`. Catalog data (garden cards, custom code templates, option
lists) stores ParseKeys and translates at render time. Provider and vendor
names, API values, code samples and test payloads stay raw
guardrail_info.tsx and custom_code/CustomCodeModal.tsx cross the 800-line
max-lines cap only because Prettier wraps the new t() calls, so each gets a
max-lines suppression entry instead of a split. The react/no-unescaped-entities
entry for add_guardrail_form.tsx is pruned because its apostrophes moved into
en.json
Route the page-authored English on /mcp-servers, /mcp/oauth/callback and the
shared MCP permission, server selector and BYOK components through t() and
Trans, with 1,094 messages added to en.json under mcpServers,
mcpServerManagement, mcpTools and mcpOAuthCallback. The rendered English is
unchanged.
* fix(proxy): preserve DB settings during config reload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): mark reload settings variable as final
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The unchecked track used --input, which is nearly white in the light theme, so an off switch read as a white knob on a white page. The light unchecked track now uses --muted-foreground, and dark mode keeps its own override
* fix(model-catalog): declare the thinking_on_by_default key
The cost map guard runs main's catalog on a PR's cost map, so the key has to be declared on main before a PR can set it on the Claude 5 rows
* chore(model-catalog): keep the attribute list sorted and document thinking_on_by_default
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): set the regional DeepSeek V3.2 context to the model card 164K and date GovCloud West Llama 3
* fix(bedrock): reserve DeepSeek V3.2 output capacity and date GovCloud West Llama 3
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Apply database general settings only for managed keys, warn about ignored copies, and allow admins to delete stale stored keys
Move intended database-only settings into config.yaml before upgrading, including authentication and access restrictions
* fix(proxy): charge budget windows when the cached key or team expired before the request ended
The post-call spend update reads a key's or team's budget_limits from
user_api_key_cache and returned early on a miss, skipping both the window
counter increment and the LiteLLM_BudgetWindowSpend row. The cached object
expires after user_api_key_cache_ttl (60s by default), so any request that ran
longer than that without another request refreshing the entry was counted in
the key's lifetime spend but never in its budget windows. With budget
reservation disabled nothing else charges the windows, so keys kept running
past their window caps
On a miss the update now reads budget_limits straight from the key or team row.
It does not re-cache the object, since a write from the spend path could race a
/key/delete and put a just-deleted key back into the auth cache
* test(proxy): assert the budget window fallback never re-caches the key or team
* fix(proxy): read a missing key or team row once per cache TTL in the budget window fallback
The master key and UI session tokens are logged under names no key row
carries, so every one of their requests missed the auth cache and read the
database. A missing row is now remembered for the cache TTL in this process
only. A failed read is not remembered, and a row that exists is read fresh on
every miss so a key update is never served stale.
* fix(proxy): skip the budget window row read for a key or team with no windows
The fallback remembered only a missing row, so a key or team that exists but
holds no budget_limits was read from the database on every request that
outlived the key cache. It now remembers both cases under one cache key built
in user_api_key_cache.py, for the same TTL as the auth cache
* fix(proxy): keep no-window markers in their own cache and clear them when an update adds windows
Markers for keys and teams with no budget windows lived in the shared 200-entry auth cache, so a burst of late costs could evict cached teams and users. They now live in a dedicated in-memory cache on UserApiKeyCache, reached by local and broadcast evictions. /key/update and /team/update now evict the marker on every worker, so a late cost after an update charges the windows the update added.
* test(proxy): give the spend counter tests the proxy's real key cache and type the window spend helper
* fix(proxy): keep a row read that an update outran from restoring the no-window marker
* fix(proxy): clear finished no-window reads and evict a marker-shaped user id from both caches
* fix(proxy): apply other spend counters before a window row read and let no-window notes follow the key cache settings
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Widen the huggingface-hub cap from <2.0 to <3.0. The locked version stays
at 1.32.0; only the matching specifier line in uv.lock changes.
Claude-Session: https://claude.ai/code/session_016isSPojcgVpmxSRshXQEPw
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The /deepgram/v1/listen WebSocket refuses nova-3-medical with 1008 because the cost map has no
streaming row for it. Deepgram's pricing page does not list Nova-3 Medical, so the new row uses
the $0.0077/min streaming launch price from Deepgram's 2025-03-03 announcement
deepgram/streaming/nova-3 and nova-3-multilingual were $0.0048 and $0.0058 per minute against
$0.0077 and $0.0092 on deepgram.com/pricing as of 2026-11-02, so they are updated too
Resolves LIT-10393
* fix(responses): log the encrypted response id on streamed /v1/responses spend rows
With response id security on, the stream hook encrypts the resp_ id on every
frame, but logging had already received the SDK's own copy of the terminal
response, so the spend row, the standard logging payload and callbacks got the
managed id (or the chatcmpl id on a bridged stream) the client never saw. The
hook now records the exact id it advertises on the request's logging object,
and the logging layer stamps that id on the logged streamed response
* fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id
Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the
id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new
session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes
to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response,
and a raw provider id still gets a managed id
Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or
<id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's
request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The
background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id
* fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner
A GET or cancel of a stored response now logs a new encrypted id of its own,
so callbacks and spend rows never reuse the create's id, while the owner of
the id still gets back the exact id they sent. An admin reading someone
else's response gets an id bound to the admin, as before. The spend row id
suffix for reads and cancels is gone, so with response id security off the
rows and session history match the default branch. A streamed GET follows
the same rules.
* fix(responses): echo the sent id to every reader that passes the response access check
A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed
the access check, not only when the caller is the user or team the id was issued to. An admin, the master key
or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not
reach the first turn from it, so a bridged follow-up through that id lost the conversation history
Every read and cancel still logs a fresh id bound to the reader, never the create's id
* fix(responses): keep a read or cancel row in session history only when it brings new output
A GET, streamed GET or cancel sent with a session header joins the
create's session, and its stored response repeats the create's output.
The replay then sent the same assistant turn once per read.
Session history now keeps an aget_responses or acancel_responses row
only when its stored Responses output has an item id that no earlier
kept row carried. A polled background response keeps the first poll
that returns the finished output, since its queued create row has none,
and later polls of the same output are dropped. Every other row keeps
today's rule
* fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart
A create row now owns its output items in session history, and a read or cancel row adds only items no
create and no earlier read carried, in one linear pass, so a background response polled while it grows,
or a read that ends before its streamed create, replays the answer once and after its prompt
DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id
like a read does
Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row
id, so a streamed background create and its cost event, which now share the client's id, keep separate
files
The integration tests that decoded the managed id now match the exact id the client received
* fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events
A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh
encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET
Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage
file and unbatched GCS object, so batch retrieves and events with no id keep main's names
An item a session read replays takes the newest read's copy at the first read's place, so an item a
poll caught half written replays finished
Tests pin the client's id over the router's continued response id, and the storage test no longer reads
the clock after the upload
* fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items
A streamed background create and its later cost event share the client's
id, so OpenMeter dropped the second event and DynamoDB overwrote the
create's item. Both now key the cost event as <id>_background_cost, like
its spend row and storage file.
Session history again replays a read's output items that carry no id,
as main did.
* fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras
* fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id
A streamed background create and its later cost event now share the id the
client received, so ClickHouse and DeepEval saw both events under one id.
Route both through logged_event_key, the way the Azure storage, GCS,
OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse
request_id matches its spend row's `<id>_background_cost` and its DeepEval
span gets its own uuid. ClickHouse response_id stays the raw provider id.
* test(deepeval): annotate the background cost span test's locals with Final
* test(responses): run the DELETE wire test that pins the echoed client id
DELETE /v1/responses/{id} now answers with the id the caller sent, so the
skip that marked it as a known bug no longer applies. The test passes on
the /v1, /openai/v1 and bare routes.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(model_info): add supports_continuation capability flag for Gemini long decode continuation
* fix(router): keep a deployment's supports_continuation override on that deployment
A supports_continuation value set in one deployment's model_info was written onto the shared provider/model cost map key, so every other deployment of that model reported it too. List it with the other per-deployment capability flags, and cite Google's long decode page next to the cost map assertion
* fix(cost-map): flag vertex_ai/gemini-4-argon-cyber-permissive with supports_continuation
Google's Vertex Gemini 4 Argon guide gives gemini-4-argon-cyber-permissive the same long decode limit as gemini-4-argon, and LiteLLM already continues its long requests, so the cost map now says so in both copies.
* fix(router): count only a boolean true supports_continuation when building a model group
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): share the CLI version rule, env lockdown and rate limiter in tests/e2e/cli_harness
Moves the npm release rule (now taking the package name), the env allowlist plus fresh HOME, and the per-upstream rate limiter out of claude_code so the Codex suite can reuse them. Claude Code behavior is unchanged, its CI, cron and Dockerfile paths now call the shared resolver with @anthropic-ai/claude-code
* test(e2e): freeze the packument map and note the rate limiter self-test
* test(e2e): race real processes on one rate limiter bucket so a missing lock fails
Eight spawned processes start together and drain a bucket that never refills. With the flock they get exactly its burst between them, without it every attempt is granted
* test(e2e): keep the old Claude Code version resolver path for the Buildkite runner images
The runner Dockerfiles still run tests/e2e/claude_code/pr_gate_version_resolver.py and silently install @latest when it is missing, which drops the three day review window. The old path now forwards to cli_harness/npm_version_resolver.py
* feat(proxy): honor model_info.max_input_tokens in the Codex catalog for stock models
A Codex model catalog entry copied from Codex's stock entry (a stock slug, or an alias whose upstream is one) now takes the max_input_tokens every routable deployment of the group sets as its context_window, capped at the stock entry's max_context_window like Codex's own model_context_window. With nothing set, the stock window stays.
* feat(proxy): honor the smallest max_input_tokens a group's deployments set in the Codex catalog
A deployment without a limit counts as Codex's own window, so a limit on only some deployments can lower the window but never raise it
* fix(proxy): skip a zero or negative max_input_tokens when sizing a Codex catalog window
* test(proxy): stub the max_input_tokens group field and type the Codex window tests
* fix(proxy): skip a max_input_tokens a saved /model/info response copied from the cost map
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(hosted_vllm): keep vLLM's stop_reason and token_ids on chat choices
Non-streamed hosted_vllm chat completions moved vLLM's choice-level
stop_reason and token_ids into provider_specific_fields, so clients
written against vLLM could not find them where vLLM puts them. They are
now also set on the choice itself, and the copy in
provider_specific_fields stays for existing readers.
Fixes#47945
* feat(hosted_vllm): keep stop_reason and token_ids on streamed chunks and proxy stream output
Streamed chat finish chunks and streamed /v1/completions finish chunks now carry vLLM's stop_reason. Each stream chunk keeps the token_ids vLLM sent with it, including held-back empty chunks, and the proxy no longer strips those chunks as empty usage chunks or drops the fields in its fast serializer.
* fix(hosted_vllm): keep the upstream choice index on token-id-only stream chunks
* fix(hosted_vllm): keep vLLM choice fields for null, cache hits, n>1, and streamed completions
Response conversion keeps stop_reason and token_ids on the choice for any OpenAI compatible response, cache replays carry them, finish fields are recorded per choice index, text completion streams keep token_ids, and the proxy passes only these two keys through, nulls included.
* fix(hosted_vllm): send empty text on token-id chunks, keep fields on their own choice, and keep null stop_reason on slow-path streams
Token-id-only chunks now carry an empty content or text string, a finish chunk only takes stop_reason and token_ids from its own upstream choice, content chunks no longer repeat stop_reason, the slow stream serializer keeps a null stop_reason or token_ids, and fields cached under provider_specific_fields come back on the choice
* fix(hosted_vllm): keep stop_reason on the last content chunk, keep both fields in built stream responses, and key the cache on return_token_ids
* fix(hosted_vllm): keep another choice's final token_ids and stop_reason on their own stream chunk
With n above 1, a finish-only vLLM chunk for a choice other than the one the single finish chunk
carries was dropped or taken as choice 0's finish, losing that choice's last token_ids. It now goes
out as an id-only chunk on its own choice index.
* fix(hosted_vllm): send the first choice's finish chunk before another choice's later token-id chunks
* fix(streaming): keep each streamed text completion choice on its own index
* fix(streaming): finish each streamed text completion choice on its own index and null token_ids in redacted logs
* fix(proxy): drop token_ids from guarded streams, redact choice-level provider fields, and keep one message_delta per Anthropic stream
* fix(proxy): skip token-id-only chunks on guarded streams and put stop_reason on the final text chunk
* fix(proxy): drop token_ids from the logged and cached response of a guarded stream
* fix(proxy): drop token_ids under per-chunk stream checks and redact text-completion stream choices
* fix(caching): key return_token_ids and continuation_token sent in extra_body
* fix(streaming): drop token ids from streamed chunks while post_call_rules run
vLLM holds text back behind a possible stop string but sends its token ids at once, so an SDK stream sent ids
before litellm.post_call_rules checked their text. While those rules are set, streamed choices drop token_ids
and chunks that carry only ids are skipped, as the proxy already does ahead of a per-chunk check
* fix(streaming): keep vLLM fields when a stream's only choice index is not 0
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock, gemini, deepseek, databricks, gigachat, anthropic): answer 400 for a hosted MCP tool a route cannot run, drop it under drop_params
A `type: "mcp"` tool sent to a route that rebuilds tools for its provider
(Bedrock Converse and Nova, Bedrock Invoke Claude, Vertex AI Claude, Vertex AI
and Google AI Studio Gemini, DeepSeek, Databricks Claude, GigaChat) was dropped
silently and the request still answered 200. These routes now answer
UnsupportedParamsError (400) naming the model, route and server, and pointing
to the LiteLLM MCP gateway. Under drop_params the tool is dropped with a
warning, other tools are kept, and no empty tools field is sent.
Anthropic, Azure AI Claude and Claude Platform on AWS keep mapping the tool to
the MCP connector. Native Responses providers and raw OpenAI-compatible
forwarders are unchanged.
* fix(mcp): check hosted MCP tools once per request and keep base tool_choice handling
Each route now runs one request-level step before its param mapping. When the drop leaves no tools,
the request sends no tools, tool_choice or parallel_tool_calls. Converse and GigaChat go back to their
base tool_choice and empty-tools handling.
* refactor(deepseek): hold the sent tools in a tuple
* fix(mcp): keep tool_choice for legacy functions when every hosted MCP tool drops
* fix(mcp): answer 400 for MCP servers a route cannot run on Messages, Ollama and the Responses playground
/v1/messages requests with mcp_servers on a chat-adapter or Responses-bridge
route now answer 400 naming each server, or drop it with a warning under
drop_params, instead of being forwarded or silently ignored. Ollama generate
no longer pastes a hosted MCP tool (headers included) into the prompt, and
ollama_chat filters hosted MCP tools the same way. The Admin UI Responses
playground now sends litellm_proxy/mcp references like the other endpoints.
* fix(ollama): leave JSON mode off when drop_params drops every hosted MCP tool
Ollama generate set format json for any request with tools, even after the
hosted MCP check dropped all of them, so a plain answer came back as JSON or
as a call to a tool the model never had. The check now runs before the JSON
mode switch, which stays on only while a tool is left for the prompt.
* fix(messages): honor additional_drop_params for mcp_servers and read the beta map only for hosted MCP tools
* fix(mcp_security): accept gateway server names and toolsets the gateway resolves
The guardrail compared litellm_proxy/mcp/<name> refs against registry keys, which are server ids, so a saved server referenced by name, alias, or toolset name was blocked although the gateway routes it. It now resolves each ref the way the gateway does and still blocks unknown names.
* fix(tool_fields): keep MCP drop warnings on one log line
Server labels, model names, and routes come from the caller, so a line break in one could forge extra log lines. The drop warnings now strip CR and LF from each value.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cohere): map Cohere v2 finish_reason on non-streaming chat responses
* fix(cohere): default a missing Cohere v2 finish_reason to stop instead of raising
A 200 body without finish_reason used to return stop and started raising KeyError after the mapping change, so fall back to COMPLETE
* fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests like their chat templates
Render grok-4-1, grok-4.3 and DeepSeek V4.1 tool schemas the way each chat
template does, and add a per-deployment token_count_tool_framing next to
token_count_framing for the hidden tool preamble a host bills. The Azure AI
Foundry grok-4-1, xAI grok-4.3 and Fireworks DeepSeek V4.1 cost map rows carry
the measured framing, overridable in model_info
* fix(count_tokens): count Azure grok-4-1 Responses requests with their own framing and use the tool framing in place of the base framing
* fix(count_tokens): keep operator framing on tool and Responses counts, frame bridged Responses like chat, follow base_model for tool rendering, count grok tool_choice
* fix(token_counter): count Azure Responses strict default, skip false strict and built-in-only tools, per-kind operator framing
* fix(token_counter): count an explicit null strict like false on Azure Responses
* fix(count_tokens): follow the chat completions bridge flag and Azure's strict default on azure/ deployments
Keep Azure grok-4-1 chat tool counts at or over the bill by raising its tool framing to 302
* fix(token_counter): keep strict false for non-grok templates, coerce the bridge flag, frame DeepSeek API counts
* fix(proxy): frame Responses counts for the api_base router defaults send inference to
* fix(proxy): keep the api_base helper's name so main's /v1/messages count caller resolves
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): resolve Invoke Claude default max_tokens from the base model id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): build Invoke Claude request params without mutating
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): type tool-search normalizer return to clear unknown-type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): resolve Invoke and Mantle Claude default max_tokens from the deployment id, ARN base_model, and Mantle row
* fix(bedrock): default Invoke Claude max_tokens from the deployment's own cap and honor AnthropicConfig
Read the called deployment's model_info.max_output_tokens before the base model's
cost row, skip the shared regional key the router fills from every deployment, and
leave the default to litellm.AnthropicConfig(max_tokens) when it is set.
* fix(bedrock): send the mapped default max_tokens on bedrock_mantle Claude chat
* fix(bedrock): bound a global AnthropicConfig max_tokens by the deployment's own cap
* test(bedrock): annotate the new Mantle default max_tokens tests with Final
* refactor(bedrock): inject the AnthropicConfig max_tokens reader into the Invoke Claude config
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
