Commit graph

56246 commits

Author SHA1 Message Date
mateo-berri
a9779db602 Merge remote-tracking branch 'origin/main' into litellm_decisions_drop_question_cap
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
LiteLLM Rust DB / db-queries (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
# Conflicts:
#	tests/unit/llms/base_llm/decisions/test_base_decisions_transformation.py
#	ui/litellm-dashboard/src/app/(dashboard)/playground/components/systemOneUI/lib/schemas.ts
2026-11-10 18:51:07 -08:00
devin-ai-integration[bot]
4b498d1bba
chore(databricks): add claude-fable-5-2, date glm-5-2 retirement and fix llama-3-1-8b context from the supported-models page (#51076)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 01:08:48 +00:00
devin-ai-integration[bot]
665f56d64a
chore(pricing): sync replicate model prices from replicate.com model pages (#51067)
* chore(pricing): sync replicate model prices from replicate.com model pages

* chore(pricing): sync replicate model prices from replicate.com model pages

* chore(pricing): sync replicate model prices from replicate.com model pages

* chore(pricing): sync replicate model prices from replicate.com model pages

* chore(pricing): regenerate provider endpoint support for new replicate modes

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 01:07:16 +00:00
joshua-berri
076ded32be
feat(auto_router): report a session's whole spend and name why savings stay unestimated (#51066)
* feat(auto_router): report a session's whole spend on /auto_router/session

Claude Code sends its background calls straight to Haiku under the same session id, so those requests never
reached the auto-router session rollup and the status line under-reported the session. Every billed spend
row in a caller-supplied session now increments a key-scoped Redis counter whose one-hour expiry each write
resets, and the endpoint reports it as total_spend, with spend unchanged. The increment runs where the spend
row is accepted, so it counts each row exactly as often as the key's spend counters do. It reads as no total
without a shared Redis

* fix(auto_router): name the unsupported provider when a turn routes off Anthropic

A turn routed to a non-Anthropic tier with no native wire response, such as a Cloudflare DeepSeek model,
failed strict native validation and was logged as observation_unavailable, which reads like a broken
capture. It now reports unsupported_selected_provider. Turns with a native wire response keep their
existing reasons, and no estimate is added, so savings for these turns stay unknown

* docs(auto_router): explain when session savings stay unestimated

saved_spend on /auto_router/session now says a turn inside the longest prompt-cache TTL of the session's
first observed turn, or of its last incomplete observation, is not estimated, so a null there is expected
for that window

* fix(auto_router): say why a system-only cache prefix has no savings estimate

Claude Code puts its cache breakpoint on the system prompt, so the baseline prefix count sends Anthropic a
body with no messages, which it rejects, and every turn read as token_count_unavailable. A count that fails
only on system or tools prefixes now reports uncountable_system_or_tools_prefix. Gateways that accept the
empty body still count it. A live check against claude-opus-5-5 showed a differential count with a
placeholder message is off by -2 tokens for system-only and +68 for tools, so these turns stay unestimated
and the savings gap for Claude Code is not fixed here

* fix(auto_router): declare the session total's Redis key family and test its expiry

The session total read and wrote Redis without a service target, so its calls showed up as bare redis
spans. Both now run under session_spend_totals. The key no longer hashes the api key, which is already a
hashed token, and only digests the session id. A new test drives Redis expiry on a test clock to show an
active session keeps its total and an idle one restarts

* docs(auto_router): note which turns are still estimated inside the history window

Turns that took the identical baseline path, and turns whose whole cached prefix the session had already
written, are estimated even inside the first cache TTL window, so saved_spend now says so

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-10 17:07:11 -08:00
devin-ai-integration[bot]
2605aa1c84
chore(cost-map): add 61 fal_ai image and video rows priced from the fal.ai model pages (second fal-ai sync) (#51077)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 00:39:17 +00:00
Kerry Lu
e2625d0509
fix(router): re-check and reserve per-model rate limits on fallback targets (#50994)
* fix(router): re-check and reserve per-model rate limits on fallback targets

A provider error fallback handed the caller the fallback model group without
its per-model TPM/RPM limits ever being checked. The router gets a
fallback_rate_limit_check hook next to fallback_budget_check, the proxy wires
it to the v3 limiter, which reserves the target model's per-model descriptors
before each fallback hop and skips targets that are over their limit.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(router): dedupe a repeated fallback target before the reserving rate limit check

Both walks now consult the attempted set before the rate limit check, so a
chain like ["B", "B"] reserves B once. The deprecated settings parity test
also learns the new setting.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(router): a bidirectional fallback chain never re-reserves the requested model group

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(router): match the surfaced failure in the fallback reservation tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(rate-limit): keep the admission stash untouched while building fallback descriptors

Building the fallback target's descriptors could rewrite the request stash's
PTU ceiling, which post-call settlement reads for the model the caller named.
The fallback check now hides the stash while it builds, like the frame check.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 16:36:10 -08:00
Mateo Wang
7894ec8434
fix(proxy): drop the beta label from the /v1/messages swagger tags (#51046)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 00:35:13 +00:00
Nurlan Mammadli
285ddf82e1
fix(proxy): allow management_routes keys to call PATCH /model/{model_id}/update (#50979)
The PATCH model update endpoint was missing from the management_routes group, so virtual keys restricted to management_routes got a 403 on it while the legacy POST /model/update worked. Fixes #50918.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 16:32:49 -08:00
moe-berri
4957007b0b
fix(model-prices): use current Hy4 promotional rates (#51072) 2026-11-10 16:31:26 -08:00
Joseph Barker
2c9068dbab
fix(guardrails/rubrik): moderate text-completion prompts and forward rac_agent_id (#49912)
* fix(guardrails/rubrik): moderate text-completion prompts and forward rac_agent_id

_moderate_prompt synthesized a user message from texts for /v1/completions
requests but built the payload from structured_messages, so the webhook
received an empty messages list and the prompt went unmoderated. Pass the
resolved messages into _build_prompt_moderation_payload.

Forward metadata.rac_agent_id to the before_prompt and after_completion
webhooks so agent-scoped policies apply. Only that key is forwarded; the
rest of request_data metadata carries non-serializable objects and inbound
headers

* fix(guardrails/rubrik): also read rac_agent_id from litellm_metadata

/v1/messages, /v1/responses and other LITELLM_METADATA_ROUTES keep proxy
metadata under litellm_metadata, so a caller on those routes may set
rac_agent_id there. Read metadata first, then litellm_metadata, forwarding
only a non-empty string

* fix(guardrails/rubrik): take rac_agent_id from key or team metadata only

The Rubrik webhooks use rac_agent_id to choose which agent-scoped
policies apply. Forwarding it from the request body let a caller pick any
agent's policies, including one a policy exempts. Read it from the
caller's admin-configured key metadata, falling back to team metadata,
via CustomGuardrail._get_admin_metadata. The proxy writes those from the
authenticated key and strips client-supplied user_api_key_* fields from
both metadata dicts, so a caller cannot set or override the value

---------

Co-authored-by: yucheng-berri <yucheng@berri.ai>
2026-11-10 16:30:41 -08:00
Mateo Wang
ebd5a15b22
fix(ci): install tzdata so zoneinfo tests pass on runners without legacy tz aliases (#51057)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 15:52:07 -08:00
shrey
37144b5e27
fix(mcp): challenge a single-server listing whose credential the upstream rejected (#50389) 2026-11-10 15:41:04 -08:00
Kerry Lu
05c04d6261
perf(spend): narrow user and team spend log reads to the owner's api_keys so the (api_key, startTime) index is used (#51026)
* perf(spend): narrow user and team spend log reads to the owner's api_keys so the (api_key, startTime) index is used

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend): satisfy the type discipline gate for the api_key scope helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend): run the api_key lookup under the statement timeout, skip it for search and empty key lists

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 15:04:08 -08:00
taxfree
bc3a69ce35
fix(otel): serialize pydantic Message objects in langfuse_otel, weave_otel and langtrace (#46845)
langfuse_otel, weave_otel and langtrace now encode litellm.Message objects, and any value JSON cannot encode, when they write the prompt, so spans keep their token usage and model parameters. Under LITELLM_OTEL_V2 the payload builder dumps Message objects instead of dropping them
2026-11-10 14:50:39 -08:00
giladk1221
9ef5542339
feat(search): add Microsoft Web IQ provider (#50883)
* feat(search): add Microsoft Web IQ provider

* fix(search): require HTTPS for Web IQ credentials

* fix(search): include Web IQ in the packaged endpoint catalog

* fix(search): register Microsoft Web IQ query pricing

* test(search): fix Web IQ exception assertion lint

* fix(search): type Web IQ response passthrough and search error headers

The type-check gate failed on 8 new unknown-type errors in the Web IQ
transformation. Build passthrough fields from a typed helper and validate
the response models directly, and type the base search error headers as
dict[str, str] so subclasses no longer inherit an unknown parameter.

* fix(search): raise Web IQ parse errors without widening the base header type

Typing the base search error headers as dict[str, str] made the shared
HTTP handler report one more argument-type error. Restore the base
signature and raise the same BaseLLMException directly from Web IQ.

---------

Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
2026-11-10 22:43:36 +00:00
devin-ai-integration[bot]
53e1e9b87b
fix(vertex_ai): bill gemini image edits by token usage (#51038)
* fix(vertex_ai): bill Gemini image edit token usage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(vertex_ai): validate image edit usage before normalization

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kenneth <kenneth@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 14:41:59 -08:00
devin-ai-integration[bot]
ac0d2fcd68
chore(cost-map): add fal_ai image and video rows priced from the fal.ai model pages (first fal-ai sync) (#51035)
* chore(cost-map): add fal_ai image and video rows priced from the fal.ai model pages (first fal-ai sync)

* chore(cost-map): add 18 more fal_ai rows whose page states a per-image, per-megapixel or per-second price despite a generic API unit

* chore(cost-map): add 11 promotional fal_ai video rows at the rate live on 2026-11-10 (promotional rates apply until the promotion ends)

* test: exempt the three fal_ai seedance-2.5/us rows whose 1080p rate is above $1 per second from the cost ceiling check

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 22:40:03 +00:00
Kerry Lu
8207a57d84
fix(responses): keep the namespace == name shape for deferred top-level tools (#50602)
* fix(responses): keep the namespace == name shape for deferred top-level tools

The OpenAI Agents SDK matches a call to a deferred top-level function
(defer_loading=True) only when the function_call carries namespace == name,
the shape it reserves for deferred top-level tools. The Responses bridge
returned no namespace, so after a native tool search the SDK failed with
"Tool <name> not found". A client that adds namespace == name then hit the
echo path, which joined it into "<name>__<name>" for the provider.

- namespace_tool_name_map maps deferred top-level functions to (name, name),
  so non-streaming and streaming replies carry namespace == name
- the function_call echo does not qualify a name whose namespace equals it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(responses): keep the qualified name for a namespace member that shares its namespace's name

A namespace "search" with a member "search" is advertised as the chat tool
"search__search" and comes back as name == namespace == "search". The echo
rule from the previous commit skipped the join for every namespace == name,
so that replay reached the provider as "search", a tool it was never given.

The bare-name echo now applies only to the request's deferred top-level
tools (deferred_top_level_names), and the reply map uses the same set. A
deferred top-level name that a self-named namespace member also uses is left
out of both directions, so the namespace member keeps its round trip.

The lookup reads defer_loading is True instead of validating it, so
callers that only read the input (token counting under drop_params) do not
raise on a malformed flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(responses): keep the bare name for a namespace == name echo replayed without request tools

Stored history (previous_response_id) is rebuilt with no request tools, so the echo of a deferred
call carrying namespace == name used to join to name__name. Only a namespace that holds a member of
its own name still joins. The lookup also tolerates a tools value that is not a list, and the
comprehensions stay within one for and one if clause.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): read request tools through one validated accessor

The deferred-name lookup and the response-side tool maps take the request's tools from one
TypeAdapter-backed accessor, so a tools value that is not a list of objects reads as no tools and
the type gate does not grow.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): type the request mapping handed to the input transformer

transform_responses_api_input_to_messages takes dict[str, object] instead of a bare dict, so the
tools accessor's argument is known to the type gate and the instructions value is narrowed to str.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): build the namespace tool map from the validated member names

namespace_tool_name_map reads each namespace's members through the same validated accessor as the
self-named check, which keeps the type gate's per-rule counts at their merge-base values.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): replay stored turns with the tools they were sent with

Stored previous_response_id history is rebuilt with each turn's own tools, so a namespace member
that shares its namespace's name keeps the joined name__name form on replay, while a deferred
top-level call keeps the bare name. Adds the session handler regression.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): keep the object entries of a tools list and read them on the streaming path too

One accessor hands the request's tools to the request path and the streaming iterator, dropping only
the entries that are not objects instead of the whole list, so both paths see the same custom, namespace
and deferred tools. Trims the explanatory comments.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): count input tokens with the request's tools in hand

/v1/responses/input_tokens hands the request params, tools included, to the input transformer, so an
echoed call for a namespace member named after its namespace counts as name__name there too.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): hand the request tools to every input conversion that reads an echoed call

The guardrail provenance pass, the prompt caching pre-call check and the Rust precall token count convert
Responses input with the request's tools, so a namespace member named after its namespace keeps name__name
there as it does on the request path.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): keep namespace == name for deferred tools loaded by a client-side tool search

A tool the client's own tool search loaded is still a deferred top-level tool to the Agents SDK, so the bridge keeps its deferral for the response-side name map and the route strips it again before the provider sees the request.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Homen Shum <317650355+homenshum-cheiron@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 14:24:15 -08:00
ryan-crabbe-berri
e95bf288f3
feat(cli): connect a per-user GitHub Copilot credential from the command line (#50863)
* feat(cli): connect a per-user GitHub Copilot credential from the command line

lite credentials connect runs the GitHub device login with the caller's own key,
so users can connect Copilot without opening the LiteLLM UI.

* fix(cli): stop connect before the GitHub code expires and explain lost connections

The connect command kept polling until the proxy rejected the expired code with a raw HTTP 400, so it now stops one interval before expiry and prints its own expired message. A dropped connection or timeout while waiting now prints a short rerun message instead of a traceback.

* fix(cli): keep checking for approval until the GitHub code actually expires

Connect stopped one interval early, so an approval in the last seconds was missed. The countdown now starts when the code is requested, which is never later than the proxy's own cutoff, and polling continues while another poll fits.
2026-11-10 22:10:24 +00:00
devin-ai-integration[bot]
0fd6dd4168
fix(proxy): recheck missing users when resolving session grants (#51030)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 21:55:53 +00:00
paveln83
a7696a2bc2
fix(proxy): send the streamed Responses fallback metadata as a typed Responses event (#50804)
* fix(proxy): send the streamed Responses fallback metadata as a typed Responses event

With expose_fallback_errors_to_caller on and include_fallback_errors set, a streamed /v1/responses
call that fell back sent its fallback metadata as a chat.completion.chunk with no type, so the OpenAI
SDK read it as an audio delta event with type None. On the Responses route the frame is now
{"type": "litellm.fallback_metadata", "litellm_fallback": {...}} and goes through the same Responses
framing as every other event, so it gets an event: line whenever its neighbours do. The chat route
keeps its chat chunk unchanged

* test(proxy): stop pinning the Responses stream terminator in the fallback metadata test

Main ends Responses streams without data: [DONE] since #47980, so the test reads the event frames whether or not a terminator follows them, and passes both before and after that change merges

* fix(proxy): keep the top-level model on the Responses fallback metadata event

* fix(proxy): keep the fallback metadata id on the Responses event and omit an empty model

* fix(proxy): leave the top-level model off the Responses fallback event when raw model names are on

The complexity router's return_raw_model_name leaves every other event unstamped, so the fallback event now matches. A new test streams a raw upstream frame before the event to cover how main names it.

* chore: push again after GitHub dropped the previous push event

* fix(proxy): send the Responses fallback event without an event line so older OpenAI SDKs read it

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 21:29:32 +00:00
norkelbyte
845f109524
fix(bedrock): forward x-pass- headers on the /bedrock/model pass-through routes (#50728)
* fix(bedrock): forward x-pass- headers on the /bedrock/model pass-through routes

The /bedrock/model/{model}/invoke, invoke-with-response-stream, converse and
converse-stream routes built the pass-through call from the request body, so
x-pass- headers never reached Bedrock. The processor now hands the caller's
x-pass- headers to the pass-through route, keeps them out of the logged body
snapshot, and on Converse drops the invoke-only guardrail headers when the
deployment sets guardrailConfig and the request metadata header when the proxy
owns request metadata

* fix(bedrock): bill the served service tier on invoke pass-through and scope x-pass- forwarding to the Bedrock routes

* test(bedrock): assert forwarded x-pass- headers on the request Bedrock receives

* test(bedrock): gate the router-model x-pass- forwarding on the request Bedrock receives

* fix(bedrock): bill a caller's forwarded invoke guardrail and keep x-pass- headers out of guardrail vendor payloads

* fix(bedrock): bill a caller guardrail on every invoke model and replace a trace value Bedrock rejects

* fix(bedrock): keep guardrail traces out of chat responses, forward the trace header as sent, and drop credential x-pass- headers

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 21:28:49 +00:00
devin-ai-integration[bot]
1d5e0c4be5
test(integration): keep owned peers off ephemeral ports (#51019)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 21:19:57 +00:00
Mateo Wang
9fb29c7e38
fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions (#50467)
* fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions

An allowed_tools tool_choice in the flat Responses layout ({"type": "allowed_tools", "mode": ..., "tools": [...]})
answered 400 on litellm.completion, acompletion, Router and the proxy's chat and /v1/responses-to-chat routes. One
parser now reads both layouts into the nested Chat layout, and the existing Responses bridge flattens it back for
/v1/responses, so either layout reaches OpenAI in the shape each endpoint expects

* fix(tool_choice): drop an allowed_tools choice under drop_params and refuse conflicting flat entries

An allowed_tools tool_choice a provider can't enforce, or a malformed one, now drops under drop_params and answers 400 without it, the same rule LiteLLM uses for other unsupported values. That covers providers outside OpenAI and Azure, malformed choices on every provider, and Azure api versions older than 2023-12-01

A flat allowed_tools entry that carries both name and a nested function or custom key answers 400, since the two names can disagree, matching what flattening already refuses

* fix(tool-choice): keep the 400 when LiteLLM runs the tools, report auto after a drop

Under drop_params an allowed_tools choice a provider can't enforce was dropped, so on MCP gateway tools and in
run_tool_loop and arun_tool_loop LiteLLM could run a tool the allow list excluded. Those paths now keep the 400 they
answered before the flat layout was accepted, and a malformed choice stays a 400 on MCP gateway tools.

/v1/responses served through chat echoed a choice it had dropped, on sync, async and streamed responses. It now
reports auto, and a choice the provider received is still echoed as sent.

* fix(tool-choice): refuse an excluded tool right before LiteLLM runs it

Checking the provider up front missed a model alias, an old Azure api_version,
a litellm_proxy hop, and SDK fallbacks. The tool loop and the MCP gateway now
check the allowed_tools list right before they run a tool call, so an excluded
call answers 400 and never runs.

Under drop_params the choice now drops only when the caller runs every tool,
so a provider-run MCP server or built-in keeps the 400. The Azure checks read
the request's api_version, which also fixes the /v1/responses echo there.

* fix(tool-choice): drop the choice under drop_params and hand an excluded MCP call back unrun

The drop now applies to every provider that cannot take an allowed_tools
tool_choice, whatever the request's tools hold. LiteLLM never runs a call the
choice excludes: on MCP paths the whole round goes back to the caller in a 200,
the same shape as require_approval, and the tool loop keeps its 400 because it
only returns final content. A malformed choice stays a 400 before any request
on the tool loop and on chat with MCP gateway tools.

* fix(tool-choice): refuse an allowed_tools entry whose name is not a string

An mcp or other non-function entry with a list or dict name passed the
parser and crashed when the allow list was hashed. The parser now treats
it as malformed on both layouts, so it answers the same 400 as an empty
allow list, or drops under drop_params.

* fix(tool-choice): honor server-level and null-name mcp entries and stop hooks running excluded calls

* fix(tool-choice): match named mcp entries by server label and let built-in entries allow their interception call

* fix(tool-choice): leave a malformed allowed_tools choice unrewritten in the interception hooks

* fix(tool-choice): resolve mcp entries like the gateway and keep the choice on every hook path

Under drop_params a malformed allowed_tools choice now acts as if it was never sent, and only a choice that parses is stored and enforced. Without drop_params it answers 400 before any provider request, on /v1/responses with gateway tools before MCP listing

An mcp entry resolves its label the way the gateway picks servers to list (server name in any case, alias, access group, toolset), and a named entry matches the tool's own name or the gateway's prefixed name. The gateway rewrites mcp entries into function entries in the caller's layout and sends "none" when nothing is left

The web search follow-up keeps an allowed_tools choice, compression's litellm_content_retrieve is never excluded and joins the provider-facing choice, and the native /v1/responses hook loop applies the exclusion check and stores the choice

* fix(allowed_tools): stop web search and code interpreter follow-ups forcing another call

A kept allowed_tools choice now goes to the follow-up in mode auto, so a
required choice no longer loops to the agentic loop limit. The MCP label
scope dataclass uses a default_factory so the module imports on Python 3.11

* fix(allowed_tools): stop the Responses web search follow-up forcing another call

The Responses-shaped follow-up after a web search left tool_choice out of its patch, so the caller's required choice went out again through the chat to Responses bridge and looped to the loop limit. It now carries a kept allowed_tools choice in mode auto, like the chat follow-up

* fix(mcp): scope each MCP entry to its own allowed_tools and keep the allowlist on follow-ups

A label now covers only the tools its own allowed_tools lists, so a second label on the same server no longer widens it. Responses MCP follow-ups, streamed or not, keep an allowed_tools choice in auto mode instead of dropping it. The new Responses tests fake only the upstream MCP listing, so LiteLLM's own discovery runs

* fix(mcp): match an entry's allowed_tools against the tool name the gateway lists

A server with a display name override lists the display name, and the listing
filter matches allowed_tools against it, but the label scope only checked the
original name, so no spelling worked. The scope now also accepts the listed
name and its unprefixed part, like the listing filter. Tests now cover the
whole gateway and toolset entries and a display-named tool.

* fix(mcp): match an entry's allowed_tools only against a server's own prefix, not the first hyphen

* fix(mcp): match a tool no server resolves only by its exact listed name in an entry's allowed_tools

* fix(mcp): refuse a tool whose display name only looks like an allowed name, and let the chat follow-up answer

An MCP entry's allowed_tools now matches a tool only by its exact listed name or a spelling routing
accepts for its real name, so a display name that carries the server's own prefix no longer passes for
another tool. The chat MCP follow-up keeps an allowed_tools choice in auto mode, so the model can answer
with text after the tool ran.

* fix(mcp): match a toolset label's tools by exact native name

A toolset label served any tool whose name a routing spelling of a granted name matched, so a toolset granting the literal name "server-tool" also served the real tool "tool" that only another toolset grants. It now serves a tool only when the toolset grants its exact native name, the same check tool_is_granted uses when the gateway lists tools
2026-11-10 13:04:06 -08:00
joshua-berri
894552205b
feat(spend): report guardrail spend separately in daily activity (#51010)
* feat(spend): report guardrail spend separately in daily activity

Model and key spend in /user/daily/activity/aggregated include the cost of
guardrails like Bedrock guardrails, and nothing in the response says how much.
A customer that bills tenants per key and per model could not get model cost
alone, so tenants that pay AWS for their own guardrails were charged for them
twice

Daily spend rows now carry a guardrail_spend column next to spend, and every
breakdown in the daily activity responses exposes it, so model cost is
spend - guardrail_spend. spend keeps its meaning, so budgets still count
guardrail cost as before

The value comes from the request's cost_breakdown.guardrail_cost, capped at
the row's spend because an allowance discount or a repeat job poll can charge
less than the guardrail cost the breakdown records. The delivery ledger
freezes it into the accounting payload next to accounting_savings, since the
stored payload does not keep cost_breakdown, and the legacy daily queue writes
the same value

The migration only adds the column with a 0 default, so rows written before it
read 0

* fix(spend): count guardrail spend on rows billed without a cost breakdown

A request blocked before the LLM call is billed for its guardrails straight from
guardrail_information, so it carries no cost_breakdown and its guardrail_spend
read 0. Both writers now price guardrail_spend from guardrail_information with
guardrail_information_cost, the same function success and failure rows use to add
guardrail cost to spend

Also fills guardrail_spend in the client usage view built from request logs,
qualifies the field description for rows recorded before it existed, and adds
the lock timeout the migration checks require

* fix(ui): avoid a nested ternary when reading guardrail entries from request logs

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-10 13:02:26 -08:00
joshua-berri
8fb2b97d5b
fix(router): sync responses() applies the Router's num_retries and fallbacks (#51011)
* fix(router): sync responses() applies the Router's num_retries and fallbacks

Sync Router.responses() made one attempt and raised, while aresponses()
retried and fell back on the same Router. The sync plan now walks retries
and fallbacks like the async one, stamps the deployment's num_retries on
the error the same way, and the SDK no longer nests its own retries under
the Router's for sync generic calls

Fixes #49938

* fix(router): sync generate_content() retries and falls back through the Router

Reading litellm_metadata in the SDK retry check left sync generate_content
with no retries at all, since its Router plan made one attempt and the inner
completion() call stopped retrying. Sync generate_content and
generate_content_stream now walk the Router's retries and fallbacks like
their async versions, so retries are neither lost nor nested

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-10 12:55:49 -08:00
Mateo Wang
4867a56e9a
fix(proxy)!: reject client-supplied Vertex location and project unless the admin opts in (#47028)
* fix(proxy): reject client-supplied Vertex location and project unless the admin opts in

A request body could carry vertex_location, vertex_project, or their
vertex_ai_* aliases and override the region and project pinned on a
Vertex AI deployment, sending the admin's credentials to a caller-chosen
project or region. Add the four fields to the banned request params so
they need allow_client_side_credentials or configurable_clientside_auth_params.

/vertex_ai/live and /rag/ingest name their Vertex project and location
by design and have no deployment pin to override, so they keep those two
fields while every other banned param is still rejected there

* fix(proxy): close the remaining Vertex target overrides and keep inert fields working

/vertex_ai/live now closes with 1008 when the query names a project or location that differs from a use_in_pass_through deployment's pin, unless that deployment or the proxy opts in

A model group's opt-in now counts only when every deployment in it carries it, and Google-style routes look the opt-in up on the model named in the URL

Vertex target names inside metadata, the live route's vertex_ai_* aliases, and /health/test_connection inheritance go back to the base behavior, since nothing reads them as a target

* test(batches): give the batch FakeRouter the model-list lookup the opt-in check reads

* fix(proxy): pin default_vertex_config on /vertex_ai/live and the proxy's Vertex project on /rag/ingest

* fix(proxy): require the Vertex opt-in on alias targets and fallbacks, and pass the live rewriter flag in main's tests

* test(proxy): give the batch tests' fake router the alias and routed-group lookups

* fix(proxy): close the Vertex opt-in, RAG corpus, and live setup-frame gaps found in review

A Vertex opt-in now counts only when every group the request can reach opts in, router,
context window, and content policy fallbacks and one alias hop included, while a group
with no Google deployment never needs the fields. A model named in the route path is
checked next to the body model.

RAG ingest from a non-admin key refuses a bare or dot-segment corpus id, a
vertex_ai_project or vertex_ai_location naming another target, and a store a non-admin
saved before the upgrade, and reads a corpus named by project number on the proxy's
own project.

/vertex_ai/live closes 1008 on any caller-named project when the proxy pins none, moves
a setup frame sent as a binary message onto the session's target, and resolves the
region from default_vertex_config before the model path.

* fix(proxy): let a live caller name the proxy's configured Vertex project when nothing pins one

With default_vertex_config set but no project pinned, /vertex_ai/live closed every query that named a project,
including the one the admin set in VERTEXAI_PROJECT or DEFAULT_VERTEXAI_PROJECT. A query naming one of those
projects now connects, and any other project still closes with 1008 before a token is minted.

* fix(proxy): keep an admin-created vector store's corpus as saved on RAG ingest

A non-admin ingest into a store an admin created named by project number had its corpus moved onto
the proxy's own project before the admin-store exemption ran, so the upload went to a different
corpus than the admin saved. The exemption now runs first, and a non-admin store still gets the
project-number rewrite and the target check.

* fix(proxy): authorize a RAG corpus moved onto the proxy's project and check client-side opt-ins once per request

* fix(proxy): merge opt-in fallback groups only when their deployments share provider and opt-in list

* fix(proxy): check team public models before allowing target overrides

* fix(proxy): include team wildcard targets in opt-in checks

* fix(proxy): validate overrides against authenticated team routes

* fix(proxy): retain URL model during deferred credential checks
2026-11-10 20:50:17 +00:00
devin-ai-integration[bot]
88dd18f4ab
chore(cost-map): add github_copilot/gemini-4-carbon from the Copilot changelog launch post (#51012)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 20:38:09 +00:00
devin-ai-integration[bot]
63afd25823
fix(cost-map): sync together_ai together/Tev1-4B-experimental input price with /v1/models (#51013)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 20:36:41 +00:00
devin-ai-integration[bot]
c319616c1f
Registry audit 2026-11-10, Together, Mistral, Fireworks, Azure AI and Copilot fixes (#51003)
Lowers Together Tev1 4B input price to the pricing page value, adds Mistral Leanstral vision and reasoning flags, and absorbs #50991, #50988 and #50958.

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Kerry Lu <326648561+kerry-berri@users.noreply.github.com>
Co-authored-by: zoveki <349028329+zoveki@users.noreply.github.com>
2026-11-10 11:56:15 -08:00
devin-ai-integration[bot]
263281ac89
feat(cost): bill batch cached image input tokens for OpenAI image models (#50997)
* feat(cost): bill batch cached image input tokens for OpenAI image models

* fix(cost): bill batch cached audio and image tokens without a text cache price

* fix(cost): honor modality cache prices in the half-price batch fallback

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
2026-11-10 19:49:15 +00:00
joshua-berri
f0d234c4c6
fix(router): let an explicit routing tag outrank the encrypted content pin (#50975)
* fix(router): let an explicit routing tag outrank the encrypted content pin

A Responses follow-up that replays encrypted reasoning minted on one
deployment, but asks for another with a routing tag, failed with "Not
allowed to access model due to tags configuration". The affinity check
now runs the router's tag filter on the deployment it is about to pin,
and on encryption-boundary peers. When the filter would reject them, it
releases the pin and strips the reasoning the tagged deployment cannot
decrypt, so the request routes there. Requests the tag filter accepts
behave as before.

Fixes #50926

* fix(router): strip reasoning against only the boundary peers the tags keep

When tags rejected some encryption-boundary peers, reasoning was still
judged against every peer, so a blob only a rejected peer could decrypt
reached the tagged one. Pin and strip against the tag-kept peers.

* fix(router): mark the deployment TypeVar Final

* fix(router): keep every boundary peer for the router's tag check

Returning only the tag-kept peers hid tags the later tag check needs to
fail open, so a request that passed the affinity check was refused.
Return all boundary peers and judge the reasoning against the kept ones.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-10 11:18:30 -08:00
mrinal-berri
9ba551b871
feat(proxy): let teams use their own allowed S3 buckets for batch files (#50571)
* feat(proxy): let teams send allowed S3 buckets for batch files and read them back

A team's metadata can now list allowed_s3_buckets. Keys on that team may send one
of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on
POST /v1/batches, and the upload and batch-create handlers check the bucket against
the team list plus general_settings.allowed_client_side_s3_buckets. The early auth
body check still rejects bucket fields on every other route and method

Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri
through an internal stored_s3_bucket_name param, so content, retrieve, delete and the
batch cost job can read files written to a request-chosen bucket. Callers can never
send that param, and files in the model bucket resolve exactly as before

* test(proxy): assert outcomes in team bucket tests and type the managed content read

Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count

* fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back

A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix

* fix(proxy): keep the request's approved S3 bucket when merging deployment credentials

Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none

* fix(proxy): keep the request bucket only on upload and batch create, not on reads

The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for

* fix(proxy): only proxy admins can set a team's allowed_s3_buckets

Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes
2026-11-10 11:06:17 -08:00
devin-ai-integration[bot]
59a3e75368
test(anthropic): scope structured output captures to inference routes (#50983)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 18:53:04 +00:00
yuneng-jiang
906578df39
test(integration): recognize the model list's lock in the held write helper (#50974)
The registry lock fix (#50933) makes add_deployment and set_model_list wait
for a held delete on the same lock, taken inside HostedModelList as
self._lock. The helper only knew lines spelled with
self._deployment_index_lock, so it never saw those two writes reach the lock
and timed out after 10 seconds.
2026-11-10 10:52:27 -08:00
Kerry Lu
2624f7caea
fix(cost-map): mark the azure_ai Claude rows that bill web search as supporting it (#50960)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 10:42:31 -08:00
devin-ai-integration[bot]
d47c03175b
fix(responses): keep unset-effort gpt-5.4/5.5 tool calls on chat completions (#50968)
* fix(responses): keep unset-effort gpt-5.4/5.5 tool calls on chat completions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(responses): explicitly route bridge fixtures to responses

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kenneth <kenneth@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 18:33:27 +00:00
devin-ai-integration[bot]
5299914964
test(http2): drain client traffic after sending GOAWAY (#50972)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 18:31:57 +00:00
Kerry Lu
43a4b465e9
fix(cost-map): give the Gemini 4 Argon rows the shared Gemini 3 search free tier (#50959) 2026-11-10 18:30:00 +00:00
shrey
e6db6206d1
fix(mcp): accept comma-separated vendor scope grants on the connect check (#50717) 2026-11-10 10:23:50 -08:00
yuneng-jiang
3539c0afc9
test(integration): cap the OAuth metadata lookup in the mixed capabilities subscription case (#50867)
The second server in this case is a plain MCP SDK server registered as
oauth_delegate. The gateway's first catalog listing waits for that server's
OAuth metadata lookup, which never resolves and runs for the full
MCP_METADATA_TIMEOUT of 10 seconds. The MCP SDK client also gives
server/discover 10 seconds, so the case passed or failed on about a tenth
of a second of head start.

Setting LITELLM_MCP_METADATA_TIMEOUT to 5 for this case keeps the lookup
well inside the client limit. The case still asserts that subscription
changes are delivered and tools are refreshed across both servers.
2026-11-10 18:11:16 +00:00
paveln83
90f37e113c
fix(proxy): send a prompt-only call_endpoint token count to every provider count API (#50783)
* fix(proxy): count a prompt-only call_endpoint request with the provider on every counter

POST /utils/token_counter?call_endpoint=true with only a prompt sent the
prompt to the provider count API on Gemini and Vertex only. Anthropic,
OpenAI, Bedrock and Azure AI (Claude) deployments answered a local
estimate instead. Every counter now gets the prompt as one user message,
so a provider error surfaces the same as the messages form, and the
framed stand-in after a failed provider count frames the prompt as one
user turn, matching the messages form

* fix(proxy): count a call_endpoint prompt like a user message on providers without a count API

* fix(proxy): count a call_endpoint prompt as a turn only on chat deployments

A prompt-only /utils/token_counter request with call_endpoint=true now goes
to the count API as one user message only when the deployment serves chat
(mode chat, responses, or unknown). Embedding rows and models with no
deployment keep the raw local count.

The Anthropic, Azure AI, and OpenAI count calls now honor the deployment's
timeout, and a count error with an empty body names its status code.

* fix(proxy): fall back to router_settings.timeout for count calls and answer 408 on a count timeout

Count calls to Anthropic, Azure AI, and OpenAI now pick their timeout the way
a chat call to the same deployment does: the deployment's timeout, else
router_settings.timeout, else request_timeout. A count call that times out
answers 408 instead of a 500 processing error.

* fix(proxy): count a prompt as a turn only where a count API exists and give count calls chat's timeout order

* fix(proxy): read the count call timeout through the router settings snapshot

The token counter now asks llm_router.settings().timeout_for() for the chat timeout order instead of a new Router member, so the router surface does not grow

* fix(router): answer timeout_for on every router settings snapshot

The proxy reads the count call timeout through llm_router.settings().timeout_for(), but only the Rust core's snapshot had that method, so a plain RouterSettings (the unit tests' router stub) raised AttributeError. timeout_for now lives on RouterSettings and reads the core's _core_deployment_call_timeout, like the snapshot's other members

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 18:09:19 +00:00
Kerry Lu
c1abd490b2
feat(customers): reject unknown fields on /customer/new and /customer/update instead of dropping them (#50965)
* feat(customers): reject unknown fields on customer create and update instead of dropping them

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(customers): derive the budget hint from the budget request fields and fix the create example

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 18:08:21 +00:00
Kerry Lu
c229851924
feat(ui): customer portal (beta) surfacing the existing customer endpoints and customer usage (#50807)
* feat(ui): Customer Portal (Beta) surfacing the existing customer endpoints and customer usage

Adds a Customer Portal mode to the dashboard (product switcher entry, Customers and
Chargebacks nav) built only on endpoints and data the proxy already serves.

Customers: list ranked by this month's spend from the customer daily-activity
aggregate, status (active, over budget, blocked), search, Create Customer through
/customer/new (id, alias, monthly max_budget, tpm_limit, rpm_limit), Edit alias and
budget through /customer/update, Block and Unblock. Customer detail shows the keys
that sent traffic as that customer this month from the api_key_breakdown of the
same aggregate, read only.

Chargebacks: per month, grouped by customer, customer and key, or customer and
model, with CSV export.

All strings go through i18n. Pure helpers (customerForms, customerPortalData,
chargebackRows) carry unit tests, and the ViewSwitcher integration test covers
entering and leaving the portal.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): refresh the customers list after editing or blocking from customer detail

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): name the expected objects in the customer portal tests so the inline-object lint budget holds

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): mock next/navigation in the navbar test now that ViewSwitcher routes with useRouter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): format customer portal money and limits through the format keys instead of inline formatters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): note that customer blocking needs blocked_user_check and neutralise formula prefixes in the chargeback CSV

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 10:00:20 -08:00
devin-ai-integration[bot]
3e3842046d
test(budgets): expect per-model tpm and rpm limits in budget info (#50967)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 17:59:48 +00:00
Nate Armstrong
461d3407bf
fix(router): hold the registry lock across router-core writes and mirror reads (#50933)
* fix(router): hold the registry lock across router-core writes and the mirror reads at their positions

A registry lookup returns positions in router-core's list and the host reads its mirror at them,
while an upsert wrote router-core and read the mirror back unlocked. A read on another thread in
between got an IndexError past the mirror, or after a removal the neighbouring model's entries

* test(router): cover the registry lock's read and write paths

Move the concurrent-read regression into the registry writes characterization, drop the
explanatory lock comments and the unused litellm-model lookup
2026-11-10 17:58:46 +00:00
Kerry Lu
0b070d25d3
feat: add claude mythos 5.2 to vertex ai cost map (#50940)
* Add Claude Mythos 5.2 to Vertex AI cost map

Copies the vertex_ai/claude-fable-5-2 and @default rows as
vertex_ai/claude-mythos-5-2 and @default. Anthropic's Mythos 5.2 model
page lists Google Cloud as a platform and says it has the same
specifications and pricing as Fable 5.2, and Google publishes no Mythos
row yet, so the new rows cite Anthropic's page as their source.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Bill Mythos 5.2 on Vertex through the cost path in tests

Adds cost_per_token checks with cache reads, 5m and 1h cache writes and
the regional uplift so the Vertex Mythos 5.2 rows are proven to charge
what Fable 5.2 charges, and annotates the new test locals with Final.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Read the regional_endpoint_uplift_multiplier key in the Vertex Mythos uplift test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 09:35:37 -08:00
tavren27
1035971a05
fix(decisions): keep box questions on a deployment when a same-model sibling turns them off (#50724)
* fix(decisions): keep a same-model deployment's false box-question flag off the shared cost-map row

* fix(router): take a withdrawn supports_box_questions true off the shared row on update and delete

A deployment's true reached the shared backend row, but a later false no longer
overwrites it, so updating or deleting that deployment left box questions on for
its unflagged siblings until restart. The row now gets the catalog's value back
unless another served deployment of the model still sets true.

* fix(router): count every live router and set_model_list when withdrawing a shared box-questions true

* fix(router): match box-questions siblings by their cost-map row and withdraw on retire

A sibling spelled with its provider registers onto the same litellm.model_cost row, so the withdrawal now compares rows instead of backend keys. Retiring a router takes back the true its deployments put on rows no other live router still sets. Withdrawals are deduped per backend so a large set_model_list stays cheap.

* fix(router): take back a box-questions true an aborted set_model_list registered

A set_model_list that fails partway rolls back the deployment list, but a true the
aborted build already wrote to the shared cost-map row stayed there, so unflagged
same-model deployments kept sending box questions. The rollback now withdraws the
attempted entries too

A withdrawal now only rewrites rows that still hold true, so an entry the failed build
never reached gets no row written for it, even on a model the catalog does not list

The delete wire test now uses a decisions-mode catalog backend whose catalog lists no
box questions, since an unflagged deployment only reads the shared row for a
decisions-mode model

* fix(router): take back an env-named box-questions true a failed set_model_list registered

The rollback took flags back from the raw model_list entries, so a deployment whose model came from os.environ/ was looked up under the reference instead of the model it registered onto, and its true stayed on the shared row

It now takes them back from the entries the failed build reached, whose os.environ/ values are already read

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 17:26:46 +00:00
devin-ai-integration[bot]
c2d7a99ecd
test(cost): sync vertex 3.1 pro price fields and bedrock converse usage shapes (#50956)
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 09:24:50 -08:00
Kerry Lu
4714aac42b
feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter (#50857)
* feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(rate-limit): build model scope settlement targets from one owner tuple

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 09:22:18 -08:00