The template tests hardcoded the model names the presets happened to ship
with, so editing autorouter_presets.json to name newer models turned every
preset red in the fixtures and hung six waitFor calls
* feat(ui): add Test Routing to the auto router create form
Route a test prompt through the complexity-router config on screen before the router
is saved, showing the model it lands on and the same decision trace the Logs page renders.
Adds POST /auto_router/test_routing, which classifies with the live pre-routing hook and
sends nothing to the routed model.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): reset the routing test modal on reopen and expose /auto_router on the UI backend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): enforce caller model access and key budget on the routing test's classifier call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: tin <tin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(claude-code): make skill registration create-only with a PUT update route
POST /claude-code/plugins upserted by name, so re-registering an existing
name silently overwrote the stored skill's source and metadata. The "Add
New Skill" UI button posts here, so a name collision clobbered a different
skill with no signal to the user.
Make POST create-only: it returns 409 if the name already exists, with a
unique-violation guard mapping the find-then-create race to the same 409.
Add an explicit PUT /claude-code/plugins/{plugin_name} for updates (404 if
the name is missing). PUT is a full replace and documents that omitted
fields reset to their defaults, so UpdatePluginRequest defaults version to
None instead of fabricating the create-time 1.0.0.
The shared mutable fields move to a PluginSpec base; RegisterPluginRequest
keeps its name and its generated schema unchanged, UpdatePluginRequest
carries no name. Regenerated the dashboard types and the lazy openapi
snapshot for the new route.
Resolves LIT-4110
* fix(ui): surface the proxy error detail so the skill 409 conflict is legible
The add-skill form rendered the raw HTTPException envelope on failure
because deriveErrorMessage did not unwrap an object-shaped detail
({"detail": {"error": ...}}), so the new create-only 409 reached the user
as a JSON blob. Unwrap object-shaped detail at the client layer, which
covers every handler that returns detail={"error": ...}, and surface the
resulting message verbatim on the form instead of burying it under a
generic prefix.
* refactor(claude-code): replace blind excepts in plugin mutations with typed handling
Narrow register_plugin's create-conflict guard from a broad 'except Exception'
+ isinstance dance to a direct 'except UniqueViolationError', using an Exception
subclass sentinel (not None) as the prisma-absent fallback so the sentinel can be
caught directly. Drop update_plugin's outer 'except Exception -> 500' wrapper so
HTTPExceptions propagate on their own and unexpected DB errors surface as FastAPI's
default 500 rather than echoing str(e). Keeps the BLE001 strict-rule budget green.
* fix(claude-code): restore structured 500 handling on update_plugin via typed PrismaError catch
Flattening update_plugin to satisfy the no-blind-except rule dropped its error
wrapper entirely, so a data-layer failure (e.g. a dropped DB connection) would
skip the intentional verbose_proxy_logger.exception call and degrade the response
from the endpoint's structured {"error": ...} body to FastAPI's default
{"detail": "Internal Server Error"}, inconsistent with every sibling route.
Wrap update_plugin in 'except PrismaError' instead of the blind 'except Exception'
the other routes use: it logs and returns the structured 500 for real DB failures
while letting genuine code bugs surface rather than masking them as 'Update failed',
and stays off the BLE001 budget. Add a regression test that a PrismaError during
the update maps to a structured 500.
* fix(claude-code): import prisma error types at function level to satisfy LIT009
* refactor(claude-code): typed plugin mutation responses and lint gate fixes
Return RegisterPluginResponse models from POST and PUT instead of ad-hoc
dicts, declare them as response_model so the OpenAPI schema and dashboard
types carry the real response shape, build the stored manifest via
model_dump, and drop update_plugin's unused auth parameter (the route
dependency already enforces auth). Keeps the LIT002/B008/UP045 budgets at
their ratcheted ceilings after merging litellm_internal_staging
e2e_ui_testing and e2e_ui_testing_server_root_path run on
cimg/python:3.12-browsers, the one UI executor whose image supplies Node
rather than taking it from a cimg/node tag. That image ships Node 24.14.0,
which bundles npm 11.9.0, so both lanes have failed EBADENGINE against the
engines floor added in #35801. Every Node 24 release through 24.14.0 bundles
an npm below 11.10.0, so engines.node also rises to 24.14.1 (npm 11.11.0),
the first release where the two floors agree
The pinned install goes into /opt/node with /opt/node/bin prepended to PATH
instead of unpacking over /usr/local. On this image /usr/local already holds
npm 11.9.0, and extracting the tarball on top of it merges the two trees into
an npm that reports 11.17.0 and then exits 1 on npm ci printing no error text
at all, which is a worse failure than the one being fixed
The install moves into a reusable install_node command so the version and its
checksum have one home, shared with proxy_pass_through_endpoint_tests, and the
command refuses to run when it disagrees with ui/litellm-dashboard/.nvmrc. A
lane drifting off the version the rest of the toolchain uses is what produced
this failure, so that mismatch now stops the job instead of surfacing later as
an install error
The e2e node_modules cache key moves to v4 because the saved trees were built
by the old npm
* feat(ui): add template picker to the Add Auto Router flow
Add Auto Router now opens straight into name + an optional Template
dropdown (Anthropic/OpenAI model-family presets or Custom). A preset
prefills the full complexity-router config and collapses the Detailed
Configuration section to a one-line tier summary; choosing Custom (or
nothing yet) leaves it expanded, and a caller can toggle it manually
at any point. A preset option greys out with the specific missing
model(s) named when the caller lacks a model it needs, or while the
model list is loading or failed to load.
Prefill and submit-gating logic live in testable pure functions
(buildPresetPrefill, getReferencedModelsError) rather than inline in
the component, per the dashboard's own testing guidance.
* refactor(ui): memoize presetAvailability
Consistency with the other memoized derived values it closes over
(availableModelSet, presets). Negligible perf impact with two
presets today, but keeps the pattern uniform as more get added.
* refactor(ui): drop pointless useMemo around getAllPresets()
getAllPresets() already returns a stable module-level array
reference; wrapping it in useMemo added React machinery for
something that can't change.
* refactor(ui): hoist presets to module scope
getAllPresets() was still being called from inside the component
body on every render even after dropping the useMemo wrapper.
Resolving it once at module load, alongside PRESETS' own
module-level initialization in autorouter_presets.ts, is the
actually-clean version of the previous fix.
* fix(ui): collapse Detailed Configuration by default
It was defaulting to expanded before any template was chosen, so
the modal still opened onto the full tier/classifier form instead
of just Name + Template. Custom still auto-expands it, and a
preset still collapses it after prefilling.
* fix(ui): list Custom Configuration last in the Template dropdown
Custom is the escape hatch, not the headline choice, so the bundled
presets now come first with Custom listed after them.
Also lets the collapsed Detailed Configuration summary wrap onto
its own line(s) instead of sharing a line with the section label
and truncating mid-model-name.
* feat(ui): match preset models across "-"/"." version separators
Admins spell version numbers inconsistently (claude-sonnet-4-5 vs
claude-sonnet-4.5), so a preset's hardcoded name and a caller's
registered one can refer to the same model while differing only in
that punctuation. getMissingModels (and therefore presetAvailability
and the submit-blocking check) now treats the two as equivalent.
Applying a preset writes the caller's actual registered spelling
into the tiers, not the preset's literal string, since the caller
may only have the dotted (or hyphenated) form and never the other
one - buildPresetPrefill now takes the available-models set for
this rewrite. Two different model names never collide; only the
separator within one version number does.
* fix(ui): re-check referenced models inside submitRecommendedRouter
submitBlockedReason disables the button for a stale/missing model
reference, but Form's onFinish (wired to the same handler) fires on
a real form submission regardless of the button's own disabled
state. The other four blocking checks already re-validate inside
submitRecommendedRouter for this exact reason; this one was missing
it, so a router could still be created referencing a model no
longer in availableModelSet.
Found by Bugbot.
* Update autorouter_presets.json
Second slice of the same sweep, covering src/components. Same rule as the
first: every removal is an unused import, an unused interface or type alias,
or a local const whose only mention was its own declaration.
The modelGroupOptions computation in add_auto_router_tab goes whole rather
than losing only its binding, since a Set and two arrays allocated per render
and then discarded is no better than the dead const was.
ToolDetail is deliberately left alone. Its unread teamsData traces back to a
useQuery that still issues a /team/list request, so removing it drops a
network call; that is a behavior change and belongs in a slice that gets QA'd,
not this one.
Stacked on litellm_dead_locals_1_app_routes; review that one first.
Part of LIT-5162.
Dropping the binding but keeping the initializer left two statements that
compute a value and throw it away: a ternary in ChatUI returning rawSelected
from both branches under a comment about resolving server IDs, and an
isAdminRole call in the prompts panel that also kept its import alive.
Both computations were already unreachable in effect; remove them whole.
mcpTokenStore was the only OAuth path writing straight to window.sessionStorage;
useMcpOAuthFlow, useToolsOAuthFlow, the callback page and the edit-screen UI state
all already go through secureStorage. Align it so the OAuth surface has one storage
format instead of two.
The stored payload also carried a refresh_token that nothing ever read back. All
three read sites take access_token only, and nothing reads the mcp-session-token:
keys directly, so the field was write-only. Drop it from the store and from the four
callers that populated it. The client-forwarded modes (true_passthrough and
oauth_delegate) re-authorize rather than refresh, and authorization_code is
unaffected because it persists through storeMCPOAuthUserCredential on the backend,
which keeps its own refresh token.
Entries written before this change decode to null and are treated as absent, which
surfaces the normal Authorize prompt; they are session-scoped and expire in an hour.
Add two regression tests that decode the stored value before asserting, so neither
can pass merely because the payload is no longer plain text.
Backend parity check: a membership-granted org admin key gets 401 on
/v1/tool/list (route absent from org_admin_allowed_routes), so the
capability map denying the formatted Org Admin runtime value is the
intended behavior, now pinned by a test
@typescript-eslint/no-unused-vars is disabled in the dashboard eslint
config, so unused locals accumulated with nothing to catch them. This is
the first slice: symbols under src/app that no code reads.
Every removal is an unused import, an unused interface or type alias, or a
local const whose only mention was its own declaration. Nothing else on the
touched lines changes, so no behavior moves with it.
Part of LIT-5162.
api.ts owned the base-vs-origin precedence, the trailing-slash trim and the
base+path join inline. That logic belongs with the rest of base resolution and
was only reachable through a fetch client, so it could not be tested directly.
Extract resolveRequestUrl into resolveApiBase.ts with its own unit tests.
api.ts now only wires the shared resolver into openapi-fetch's Request option.
No behaviour change: same precedence, same trimming, same output.
* chore(build): move the Admin UI toolchain to Node 24
Node 18 and Node 20 both reached end of life (2025-04-30 and 2026-04-30), and
the release images along with every CI lane were still building on them. Node 24
is the current LTS through 2028-04-30, so this moves the four UI build images,
the CircleCI lanes, and the four GitHub Actions workflows onto it
Node 24 also ships npm 11.17, which is the first line that implements the
min-release-age setting this repo already carries in its .npmrc files. On npm 10
the key is parsed and discarded, so the release-age gate has had no effect
regardless of its value. Tightening the dashboard's engines range and turning on
engine-strict makes an unsupported npm fail loudly rather than skip the gate
quietly, and a new step in the UI build workflow probes an impossible cooldown
so an inert setting cannot pass unnoticed again
Node 24's bundled undici tightened its brand check on RequestInit.signal, which
rejects the AbortSignal jsdom installs and broke the two cases in
src/lib/http/api.test.ts that rebase a request onto a runtime base url. Under
jsdom the Request global comes from Node while AbortSignal comes from jsdom;
tests/jsdomFetchEnv.ts delegates to the jsdom environment and then restores
Node's native AbortController and AbortSignal so both come from one realm.
Upgrading jsdom does not address this, as jsdom still does not own Request
The workflows now read ui/litellm-dashboard/.nvmrc instead of repeating a
literal, so the Node version has a single source of truth, and ui/Dockerfile is
pinned by digest to match the other three build images. The lockfile changes are
npm 11 normalising the engines range and dropping optional peer entries it no
longer records
* fix(build): point every Admin UI build script at .nvmrc
The enterprise Docker path was left on Node 18. docker/build_admin_ui.sh runs
only when enterprise/enterprise_ui/enterprise_colors.json is present, which it
never is in the OSS tree, so neither CI nor a default image build reaches it;
it pinned nvm to v18.17.0 and then built the dashboard, which now requires Node
24, so a customized enterprise image would have failed EBADENGINE
All three UI build scripts now resolve the version from
ui/litellm-dashboard/.nvmrc rather than carrying their own pin, so the Node
version has a single home across Docker, CI, and local builds. build_ui.sh was
on v20 and build_ui_custom_path.sh on v18.17.0
Also drops the dependency-cooldown probe from the UI build workflow. The
engines floor plus engine-strict already fails an unsupported npm loudly at
install time, so the probe was redundant, and treating any nonzero exit from a
live registry call as proof of enforcement made it unsound besides
api.ts read globalThis.location when the module loaded, which froze the base
URL at import and pinned its test file to jsdom. The creation-time baseUrl and
the middleware's runtime rebase were also two mechanisms doing overlapping
work, and the rebase hand-copied eleven RequestInit fields on every call.
Pass openapi-fetch's Request option instead, so the constructor applies
whatever getRequestBaseUrl() returns at the moment the request is built.
registerBaseUrlGetter is now the single source of the base URL, rebaseUrl and
rebaseRequest are deleted, and the request is constructed once, so the init
openapi-fetch assembled reaches the platform Request untouched. The abort
signal is no longer copied by hand.
This preserves behaviour rather than approximating it: getProxyBaseUrl() falls
back to location.origin, so the runtime base was never empty in a browser and
the old middleware already rebased every request, discarding the creation-time
value each time.
setupTests.ts gates its DOM-only tail behind a window check; setup files run
for every environment, so that tail previously stopped any node-environment
test file from loading.
api.test.ts now runs under @vitest-environment node with its assertions intact
and no location stub, plus regressions for per-call base resolution and abort
forwarding. api.sameOrigin.test.ts covers the browser fallback to the page
origin, which needs a DOM environment.
Internal users saw the Tool Policies page but its /v1/tool/list call always
returned 401. This adds a single source of truth for which roles may trigger
which UI fetches (utils/capabilities.ts) plus a useCan hook, and wires the
Tool Policies route through it: the nav item, the page, and the query all
read the same capability, so the sidebar hides the entry, deep links render
an admin-only notice, and the query never fires. The tools list call also
moves onto a queryOptions factory
* feat(ui): add admin-configurable user banner
Proxy admins can publish a markdown announcement that renders as a
dismissible banner on every dashboard page for all authenticated users,
editable from Admin Settings > UI Settings without a redeploy. Backed by
new /get/user_banner and /update/user_banner endpoints persisting to the
existing LiteLLM_UISettings table
* fix(ui): re-surface dismissed banner on identical republish
Stamp a server-side revision on every banner update and fold it into
the client dismissal signature, so unpublishing and republishing the
same message reaches users who dismissed the earlier run
* fix(ui): stamp banner revision as an opaque uuid instead of a counter
Two overlapping admin updates could read the same prior revision and
both persist the same incremented value, letting an identical republish
collide with a previously dismissed signature. A server-generated uuid
per update makes every publication identity unique by construction with
no read-modify-write
* refactor(ui): drop the server-side banner cache
Reads go straight to the single-row table; the dashboard already
throttles fetches client-side, so the cache only added staleness
windows under concurrent updates and multiple workers
* refactor(ui): move banner storage behind a domain repository and drop the store_model_in_db gate
UserBannerRepository owns the row shape instead of the endpoint
reaching through the generic .table bridge, and publishing no longer
depends on the unrelated STORE_MODEL_IN_DB flag; a connected database
remains the only requirement
Adds the auto-router as a third optimization driver beside compression and prompt
caching: a summary card, a donut segment, and a series in the savings graph across
both the cumulative and per-day views.
The number is signed, because a switch that thrashes the prompt cache can cost more
than the cheaper rates save and an operator needs to see that. The donut plots only
drivers that saved, since a negative slice has no meaning, while the card and the
range total keep the sign. `usd()` sizes and signs off the magnitude so a small loss
renders as -$0.01 rather than "$-0.00".
The card's popover states the counterfactual and its two consequences: that a switch
pays to re-warm the cache, and that a first turn the router could not identify is
charged that write and therefore under-reported.
* feat(spend): add net auto-router savings to the cost-optimization dashboard
The dashboard credited compression and prompt caching but said nothing about the
optimization that picks the model, so the driver with the largest lever on a bill
was the one an operator could not see.
Savings are the counterfactual: without a router a deployment runs one model, and
it has to be one that can carry the hardest request, so the baseline is the
priciest model in the router's hardest configured tier. A cheap tier is a choice
the router made, not a ceiling it was bounded by. `auto_router_savings_baseline_model`
overrides it for operators who would genuinely have run something else. Both are
provider-qualified before pricing, because a bare name can resolve to a different
vendor's rates or to nothing at all, and a deployment is priced by its `base_model`
where it has one, which is how Azure deployments are priced everywhere else.
Both arms price the request's real usage through `generic_cost_per_token` rather
than re-deriving per-token arithmetic, so tiered rates, ephemeral cache-write tiers
and regional uplifts stay consistent with what was actually billed. `prompt_tokens`
already includes the cache buckets, so charging them again at the input rate would
price the same tokens twice.
Cache state is what makes this hard. The baseline serves every turn, so whether it
had the prompt cached is whether the conversation was already underway. On a
continuing conversation it wrote the prompt earlier and would only read it now, so
this request's write is what switching cost and counts against the saving. On a
first turn nothing was cached for any model, the baseline would have written the
same prompt, and both arms carry the write at their own rates. Charging the write
to both cases understates a first turn to a few percent of its value, and because
the write premium is fixed by prompt size while the saving grows with completion
length, it can render a profitable route as a loss.
That shape is read off the conversation rather than remembered: a second human ask
means an earlier turn was served. No cache, no session id, and no dependence on a
caller sending a session header. It cannot see a switch on a turn the router did
not classify, and it reads a few-shot prompt's synthetic turns as prior
conversation; both err toward charging the write, which under-claims.
The baseline and the shape ride on the existing `routing_decision` record, which is
already carried from the router to the spend log, already classified for redaction,
and already written-or-cleared per attempt. A fallback that re-enters the hook
therefore cannot leave either fact behind to be attributed to a deployment that
never routed, and no new metadata key crosses the trust boundary.
The result is signed. Whether a switch pays off is a race between the rate gap and
the cache-write cost, and a narrow gap loses; flooring at zero would hide exactly
the routing behaviour an operator needs to see. The donut plots only drivers that
saved, while the card and range total keep the sign.
Savings accrue into a new `autorouter_savings_spend` column on the six daily rollup
tables, declared `NotRequired` because rows queued by a pod on the previous release
carry no such key. It is summed by the rollup merge the cross-pod Redis drain also
runs, and carried through the aggregation query, the per-row accumulation and the
response model, so the dashboard reads a value the API actually sends. Tests
enumerate the drivers from the response model itself and assert each is summed,
accumulated, carried and totalled, so one added later cannot be half-wired.
* fix(spend): let the baseline pay for a continuing turn's own growth
`_baseline_usage` moved every cache-creation token into the baseline's read bucket
whenever the conversation was underway. That is right for a switch, where the
baseline never left the model it was on and really would only read, but wrong for a
turn that stayed put: the prompt grew, and the tokens written are that growth. They
are new to every model, so the baseline would have paid to write them too. Forgiving
it that write made the counterfactual cheaper than it was and shrank the reported
saving on ordinary steady-state traffic, by about 2% per turn.
The selected arm was never involved; it has always been priced on the real usage.
The error sat entirely on the baseline.
The condition is that the request read more than it wrote, not that it read anything.
A switch onto a model already holding a small prefix of this prompt still writes most
of it, and that write is the switch's own cost; keying off a nonzero read would have
handed such a request the full rate gap, turning +$0.0056 into +$0.1177. Comparing
the two buckets separates a warm continuation, which reads far more than it writes,
from a cold arrival, which does the reverse, and it leaves the existing invariant
intact: a request reading 0 and one reading 1 both still land in the same place.
* fix(spend): price each arm under the key litellm billed it, and see agent turns
Two ways the savings number read the wrong thing, both from identifying a model by
its name when the name is not what it costs.
The counterfactual was ranked and priced on the public rate for the model a
deployment names. A deployment may not be charged that rate: the router registers
its configured prices under the deployment's own id and deliberately keeps them off
the shared model-name key so deployments sharing a backend model do not pollute each
other. So a hardest-tier deployment configured above its public rate lost the
ranking to a cheaper candidate, and once chosen was priced at a rate nobody pays.
Which key prices a deployment is now `_select_model_name_for_cost_calc`'s decision,
the resolver the real request is billed through, rather than a second rule here that
would have to re-learn that per-second and tiered overrides count, that a partial
override still counts, and that a deployment configured at zero is priced at zero
rather than treated as unpriced.
The arm being subtracted had the same fault and a sharper edge. It priced the spend
log's `model`, which on Azure is the deployment name, absent from the cost map, so
the whole driver silently read zero for that traffic. It no longer re-derives
anything: `model_map_information.model_map_key` is what litellm actually billed the
request under, recorded at request time by that same resolver with `base_model` and
custom pricing already applied.
Separately, the conversation-shape discriminator counted human asks, and an agent
loop can run twenty turns on one of them. Its tool traffic rides `tool_result`
blocks on user turns that flatten to empty text, and `tool` roles that are never
read, so a long agentic conversation looked like its own first turn and was handed
the arithmetic that leaves the cache write on both arms. That is the one direction
this must never fail in, because it inflates. An assistant turn is the direct
evidence that something answered earlier, and it is blind to how the tool plumbing
is spelled on either surface.
* fix(spend): give the cost-key resolver both inputs the selected arm needs
The served model was resolved through one input at a time, and each choice broke the
half the other fixed.
`model_map_key` is the served model already resolved through `base_model`, which is
the only way an Azure deployment name reaches the cost map at all; without it the
selected arm priced a name absent from the map, returned nothing, and the whole
driver silently read zero for that traffic. But it is built without
`router_model_id`, so it never carries a deployment's own price overrides, and a
custom-priced deployment was compared at its public rate while the baseline used the
real override. On a deployment configured well above its public rate that inverted
the answer outright: a route that lost $21.88 reported saving $0.10.
`_select_model_name_for_cost_calc` takes both, so it gets both. Which key prices a
deployment stays its decision rather than a rule restated here.
* fix(spend): same model is only the same cost when it is the same deployment
The short-circuit compared resolved model identity, so two deployments of one model
collapsed to "no switch" and reported zero. They are not the same cost: a deployment
can carry a negotiated rate, and routing from the dear one to the list-price one is a
real saving the dashboard reported as $0.00 against a true $21.93.
Both arms now carry the key litellm prices them under, so the comparison is between
deployments rather than between names.
* refactor(spend): price from resolved rates, not from a name we keep re-resolving
Four review rounds landed on one mechanism: which identifier prices a deployment.
base_model, then the deployment id, then cache-only overrides. Each round added a
clause to a resolution rule that should not exist, and a wrong primitive fails once
per input shape, so each shape arrived as its own finding.
`Router.get_deployment_model_info` already owns this. It merges a deployment's
configured prices over the built-in map, folds in `base_model` defaults for
deployments whose name is not a model, and falls back to the model name when nothing
is overridden. Every shape hand-rolled here (cache-only, partial, per-second, Azure)
was that function re-implemented badly.
`generic_cost_per_token` now accepts already-resolved rates instead of demanding a
name it looks up itself, which is what forced the name-bending in the first place.
Both arms resolve through the owner and pass what they got: the counterfactual by the
deployment the router would have used, the served request by the deployment that
served it. The invented cost-key resolver is gone, and `Baseline` carries a
deployment id rather than a key we chose on litellm's behalf.
Net 64 insertions against 79 deletions.
* test(spend): follow _most_expensive onto the router that prices its candidates
Ranking moved through `Router.get_deployment_model_info`, since what a deployment
costs is the router's answer to give; these four cases were still calling the old
free-function signature.
* fix(spend): rank baseline candidates by what a request costs, not by two rates
"Most expensive" was decided by comparing output rate then input rate. That is a
property of a rate, not of a request: a deployment dearer per output token can be
cheaper per cached token, so the comparison ordered cache-heavy traffic backwards and
recorded the wrong counterfactual.
Candidates are now costed on one reference request through the same engine the
savings themselves use, which leaves cache read and write rates, tiered tables and
every other billing dimension to that engine rather than to another rule restated
here. The reference request is cache-heavy because auto-routed traffic is.
* fix(spend): pick the baseline against the request that ran, not a stand-in for one
Ranking happened in the pre-routing hook, where the request has not executed yet, so
candidates were costed against a hard-coded reference workload: 20k prompt, 19k of it
cached, 1k out. Which candidate is dearest depends on that mix, so a pooled hardest
tier holding a deployment with non-proportional configured rates could be ranked for
a request nothing like the one served.
The mix is known on the spend path, so the ranking belongs there. The routing
decision now carries the tier's candidates rather than a winner already chosen, and
the baseline is resolved against the usage that actually happened. The reference
workload is gone; nothing here assumes a traffic shape any more.
The router is passed in rather than imported from `proxy_server` inside the
computation, so the savings stay a pure function of their arguments and the caller
owns where the router comes from. That also makes the spend path testable without a
running proxy, which the previous shape was not.
* refactor(spend): measure savings against one configured model, not a derived one
The counterfactual was derived per request: enumerate the hardest tier's
deployments, resolve each one's effective pricing, price them all, take the dearest.
That machinery produced a review finding per input shape it had not anticipated,
and every answer it gave was one an operator could have stated in a line of config.
So they state it. `litellm_settings.autorouter_savings_baseline_model` names the
model the traffic would have run on without a router, for every auto-router on the
proxy, and unset means the driver is off rather than a model nobody named being
guessed at. `savings_baseline.py` and its tests are deleted outright, along with the
tier enumeration, the candidate list on the routing decision, and the per-deployment
override that shadowed it.
Cache-state handling is untouched: the baseline is still priced on this request's own
read and write split, so a switch still pays for re-warming the cache and a first
turn still charges the write to both arms.
45 insertions against 482 deletions.
* refactor(router): compute the conversation shape once and pass it down
`_classify_and_route` re-derived it from the messages the hook had already resolved,
so an ordinary routed request walked the turn list twice for one boolean. The hook
computes it and hands it over, which is also where the affinity-hit path already got
it from.
Also moves `_get_llm_router` below the imports it sat among.
* fix(router): drop the dead conversation_continuing parameter off the hook
It was added to `async_pre_routing_hook` by mistake and immediately overwritten by
the value the hook computes, so it never did anything. It also widened a signature
every pre-routing strategy shares with the protocol in `types/router.py`, leaving
this one router diverged from `AutoRouter` and the interface for no reason.
Also records why an unreadable request counts as continuing: no messages is no
evidence a turn was served, so it pays the cache write and under-claims rather than
being handed a first turn's larger saving on nothing.
* fix(spend): charge a baseline its input rate for cache buckets it cannot price
A model with no cache_creation_input_token_cost, which is every OpenAI, Azure and Gemini entry, resolved that rate to 0.0 and carried the whole written prompt for free, so a first turn routed onto a cheaper model reported a loss. Same hole on cache reads. Those tokens are plain input on such a model, so they move into the text bucket.
* refactor(spend): build the daily upsert payloads in one shot
`common_data` and `update_data` were constructed and then appended to: `request_id`
conditionally for tag rows, `endpoint` unconditionally a few lines later. A dict that
grows after its literal cannot be reasoned about by reading the literal, which is the
whole point of building it at once.
The conditional key resolves to a spreadable value before either payload, so both are
single expressions and the tag branch appears once instead of twice.
Not wrapped in MappingProxyType, though it was suggested: these go straight to
prisma, whose query builder branches on `isinstance(value, dict)` to tell a nested
node from a scalar. A mappingproxy is a Mapping but not a dict, so it falls through
to the serializer and raises `TypeError: Type <class 'mappingproxy'> not
serializable` inside the batch upsert, where the surrounding except would log it and
leave the rollups silently unwritten.
* fix(spend): keep the one-shot upsert payloads under the type-discipline budget
Building both payloads as single literals traded a mutation for two dict literals,
and LIT002 counts construction rather than mutation, so the change the review asked
for is the one the gate charges for.
The empty branch is the avoidable half: it is the same value every time, so it moves
to a module constant built once instead of a literal per transaction, and it is a
read-only mapping so none of the call sites that spread it can fill it in later.
"Add keyword rule" seeds a row with no keywords, and the only check that a
rule carried one lived inside getSemanticConfigError, which returns early
when semantic keyword matching is off. Off is the default, so an unfilled
row fell through to serializeKeywordTierRules and was discarded on the way
to the payload; the create reported success and the rule was gone.
The row now reports the gap itself and the submit is withheld while one is
outstanding, on the create form and the edit modal alike, both reading
emptyKeywordTierRuleIndexes so the row named and the row marked cannot
differ. Enter commits a typed keyword: the dropdown is kept closed, which
left antd nothing for Enter to select, and submitting was what used to
supply the blur that saved the word.
The backend already refused such a rule, but only when the router built the
deployment, so a caller that sent one anyway got the row written, dropped on
reload, and a 500. The management write paths now parse the incoming
complexity_router_config with the router's own ComplexityRouterConfig, judged
on the config alone so a patch that writes one without naming a model is
covered too, and reject it with a 400 having persisted nothing.
* feat(team): custom metadata validation hook for team create and update
Operators can point general_settings.custom_team_metadata_validate at an
async Python function that validates team metadata before /team/new,
POST /team/update, and PATCH /team/{team_id} commit their writes. The
hook receives the metadata that will actually be written (the merged
result on PATCH) plus the stored metadata and requester context, and
fails closed: a rejected value returns the function's own message as a
400 while any exception or timeout blocks the write with a configurable
generic message as a 503. Premium-gated like enforced_params.
* fix(team): validate metadata before model alias writes and strip system keys from validator input
Review follow-ups on the team metadata validation hook: run the validator
before the model_aliases table insert so a rejected create leaves no
orphaned model rows, strip system-managed keys from existing_metadata so
the validator sees symmetric input on both fields, and accept class
instances exposing an async __call__ as validators. Adds a three-way
validator implementation matrix (allowlist function, HTTP-service-backed
function, immutability-enforcing class instance) driven through the real
create, update, and patch endpoints, including an HTTP stub service and
outage coverage.
* test(team): run the metadata validation matrix against the DB-backed proxy in CI
Adds the validator matrix to the proxy_store_model_in_db_tests CircleCI
job so every scenario runs full e2e against a Postgres-backed proxy. The
proxy config registers a dispatching validator that routes each request
to one of the three implementations via a metadata key and accepts
anything that does not opt in, keeping the rest of the suite unaffected.
CI starts a stand-in cost center service on the host for the HTTP-backed
implementation, reached from the container via host.docker.internal, and
the outage path targets a closed port to prove the fail-closed 503
without stopping services.
* feat(ui): edit team metadata as key-value pairs in team create and edit forms
The team create and edit forms asked for metadata as a raw JSON blob in a
textarea buried under Additional Settings. Both forms now render a key-value
pair editor directly under the TPM/RPM limit fields, backed by a shared
MetadataKeyValueFields component. Values round-trip losslessly: non-string
values display as JSON and parse back to their typed form on save, and
JSON-ambiguous strings are quoted so their type survives the trip. The edit
form hides UI-managed keys (logging, guardrails, model rate limits, etc.)
that dedicated controls already own and re-add on save.
* fix(ui): explain typed JSON parsing in the team metadata help text
* feat(team): schema-driven metadata fields from team_metadata_schema config
* refactor(team): render schema metadata fields as locked key-value rows, drop allowed_values
* refactor(team): schema fields reduce to key and label, tag-rendered keys, clean rejection toasts
* refactor(ui): prepopulate declared metadata keys as ordinary key-value rows
* fix(team): let non-admin dashboard users read the team metadata schema
* test(proxy): pin timeout wiring, boundary, and error-message contracts for team metadata validation
* fix(proxy): use pooled async httpx client in the e2e team metadata validator example
* refactor(team): satisfy staging lint ratchets inherited by the merge
* fix(ui): hide guardrail review buttons from non-admin users
The team guardrail submissions list rendered Approve/Reject buttons for
non-admin users even though the backend correctly rejected the calls.
Thread userRole from the page through GuardrailsPanel into
TeamGuardrailsTab and gate the row-card and detail-panel review buttons
on isAdmin so the UI matches the backend authorization.
Defense in depth only — the backend remains the source of truth and is
double-gated at both the route admin check and the explicit endpoint
role check.
Refs LIT-2494
* refactor(ui): read userRole from useAuthorized hook instead of prop drilling
Drop the userRole prop chain through GuardrailsPage → GuardrailsPanel →
TeamGuardrailsTab. Each component reads userRole directly from the
useAuthorized hook, matching the pattern used elsewhere in the dashboard.
Tests now mock useAuthorized per case (the same pattern as
top_key_view.test.tsx) instead of passing userRole as a prop.
Refs LIT-2494
* fix(ui): drop userRole prop on GuardrailsPanel call site in src/app/page.tsx
Missed in the earlier refactor — GuardrailsPanel no longer accepts
userRole as a prop (reads from useAuthorized hook), so callers must
not pass it. The build was failing in production type-check.
Refs LIT-2494
* fix(ui): gate guardrail forward-key toggle and header editors on proxy admin
* refactor(ui): remove dead app_admin case from user role formatting
The Pretty view only parsed the Chat Completions shape (messages /
choices[0].message), so any spend log storing the Responses API shape
(input / output) rendered an empty Input card and the literal text
"No response data available" even though the row held the full request
and response. This also hit plain /v1/chat/completions callers, because
litellm may route those over the Responses bridge and then store the
upstream Responses-shaped body.
Parsing now branches on a tagged union covering both shapes, which also
replaces the any-typed key sniffing and the role guessing it relied on.
Adds 61 unit tests on the modules #35694 extracted: 46 on the payload
builder, 15 on the OAuth redirect snapshot. They run in 9ms against 240s
for the 77 full-render tests they partly replace. Nine of nine mutants
were killed when the extracted logic was deliberately broken, so the
speed does not come at the cost of signal.
Deletes six cases across four blocks that rendered the whole modal to
assert one payload key belonging to a field they never touched. Every
test that proves a form field reaches the right payload key stays; those
cover field to form value to payload, which a unit test cannot reach.
Replaces "should not render when user is not an admin", which asserted
the admin title was absent and so passed for the wrong reason: the modal
does render for a non-admin, retitled. registerMCPServer was mocked but
never asserted anywhere, leaving the whole non-admin submission path
uncovered. It now drives a real submit and asserts the call lands there
and never on createMCPServer.
Renames the slow file to CreateMCPServer.integration.test.tsx and
documents the three tiers in the dashboard CLAUDE.md. No production code
changes.
* feat(ui): expose an Auto-Router session affinity toggle
session_affinity on ComplexityRouterConfig defaults to True, and neither the
create form nor the edit modal ever emitted the key, so every auto-router built
in the UI silently pinned each session to its first turn's model for an hour
with no way to see or change that.
Adds an "Advanced: Session Affinity" switch to both surfaces, defaulted on to
match the backend field. Both paths now write the key explicitly instead of
falling through to the backend default, so a stored config states what the
router actually does. A stored config with the key absent hydrates as on, since
those routers are running with affinity enabled today; showing them as off would
report the opposite of reality and persist it on the next save.
* feat(complexity_router): default session affinity off and expose it in the UI
session_affinity defaulted to True and the Auto-Router UI never emitted the
key, so every router built there silently pinned each session to whatever model
its first turn classified into for an hour, refreshed on every hit. There was
no way to see that from the UI and no way to change it without hand-editing
config.yaml.
The default flips to False, so every turn is classified on its own merits and
lands on the cheapest adequate tier. Pinning is now opt-in.
The toggle added in the previous commit follows the field: it renders off, and
both the create tab and the edit modal keep writing the key explicitly, so a
stored config states what the router does instead of inheriting a default that
can move under it.
Behavior change for existing routers: those created before this have no
session_affinity key stored, so they pick up the new default and start
reclassifying every turn. That gives up the provider prompt cache the pin was
preserving, and a multi-turn session can now change model between turns. Set
session_affinity: true to keep the old behavior.
Google Cloud has renamed Vertex AI RAG Engine to "RAG Engine" and
Vertex AI Search to "Agent Search" in its console. Users following our
setup instructions hit a naming mismatch when they cross-reference the
GCP console. Keep "Vertex AI" as the primary term (the generic new
names would make our provider UI ambiguous) and surface the new names
as secondary asides only where users leave the UI for the console.
Resolves LIT-3081
Pulls four modules out of the 1398-line create component, which drops to
896 lines. No behavior changes: CreateMCPServer.test.tsx is untouched and
all 77 of its tests pass against the refactored component, which is the
review contract for this PR.
createServerPayload.ts is a pure form-values-to-payload function whose
failures are a tagged union instead of inline notification calls, so the
transformation is reachable without a DOM. createOAuthUiState.ts owns the
snapshot that survives the OAuth authorize redirect, keeping every
presence guard the inline version had. AwsSigV4Fields and
OpenApiByokFields are the two largest JSX blocks, moved verbatim so they
can be diffed as moves.
The create/edit setToken divergence, the mcpLogoImg export, and the
untyped form-values bag are left alone on purpose; each is a behavior or
cross-file change that does not belong in a move.
Pure rename, no behavior change. create_mcp_server.tsx and its test move
to CreateMCPServer, the two importers and one stale e2e comment follow,
and the local/filename-pascal-case suppression drops now that the file
passes the rule on its own.
The rename is scoped to this one component rather than the whole
directory because three PRs are currently open against its snake_case
siblings; the rest can follow once those land.
The dashboard pins both packages exactly in `overrides`, so the lockfile
stays on whatever those pins say. Move brace-expansion from 5.0.8 to 5.0.9
and postcss from 8.5.22 to 8.5.23, both upstream patch releases, and
regenerate the lockfile.
`npm ci`, `next build`, and the 5888-test vitest suite all pass on the
updated lockfile.
* feat(teams): apply default organization to new teams from default team settings
Adds organization_id to DefaultTeamSSOParams so proxy admins can pick a
default organization in Default Team Settings. new_team applies it before
org validation whenever a team is created without an explicit
organization_id, so API, Admin UI, SCIM, SSO, and team upsert creations
all inherit it and go through the same existence and org-limit checks.
Explicit organization selections win and existing teams are untouched.
The default is validated at save time (PATCH /update/default_team_settings
returns 400 for an unknown org) and at create time, where a missing org now
surfaces as a clean 400 instead of a 500 by routing OrganizationNotFoundError
into the previously dead org_table None guard.
The Admin UI Default Team Settings tab gets a Default Organization row
backed by the shared OrganizationDropdown.
* fix(teams): validate org limits against final team state including defaults
Applies default_team_params and the legacy max_budget fallback before the
organization validation block, so _check_org_team_limits sees the values the
team will actually be persisted with. Also loads the org's budget table in
the lookup; without include_budget_table every budget comparison in
_check_org_team_limits was skipped because litellm_budget_table was None.
* test(proxy_behavior): pin org team limits as enforced on /team/new
The dead-code pins existed to turn red when include_budget_table went
live; that happened, so the scenarios now assert the 400 rejections plus
within-cap acceptance, and the unknown-org pin asserts the handler's 400
instead of the surfaced 500.
Adds a Stream responses checkbox (default on) to the playground Model
Settings popover. When unchecked, chat completions and responses API
requests are sent with stream: false and the full reply renders at
once. The non-streamed result is replayed through the existing
streaming handlers as synthesized chunks/events so MCP events, vector
store results, usage and response ids behave identically in both
modes. TTFT is suppressed when not streaming; total latency now also
reported for the responses API. The toggle is scoped to the chat and
responses endpoints, persists via sessionStorage, and is isolated from
the simplified Agent Builder chat.
Resolves LIT-3251
disable_team_logging cleared only metadata["callback_settings"], but callbacks
registered through POST /team/{team_id}/callback and the Admin UI live in
metadata["logging"], and request-time resolution stops at that slot without
ever reading callback_settings. The endpoint reported success while the team
kept sending request and response data to its third-party destination.
Empty the logging slot alongside the existing callback_settings reset, and
refresh the cached team object so the change applies to keys that are already
in flight rather than at the next cache expiry. The same refresh is added to
add_team_callbacks, which has the symmetric problem of a newly registered
callback staying dormant until the entry expires.
Resolves LIT-5101
* fix(team-callbacks): report API-registered callbacks from GET /team/{team_id}/callback
POST /team/{team_id}/callback writes metadata["logging"] while the GET read
metadata["callback_settings"], so every team configured through the API or the
Admin UI got back an empty list. c620d76fe4 migrated the writer to the new key
and left this reader on the old one.
Resolve the read the same way request-time resolution does in
_get_dynamic_logging_metadata: a logging slot that is present wins outright and
callback_settings stays as the deprecated fallback, so the endpoint reports what
a request would really do rather than the union of both shapes. An empty logging
list therefore reports no callbacks, matching a request that fires none.
Decrypt callback_vars for the response and mask the credential keys. Ciphertext
would be unusable to the caller, and a value encrypted under a key that is no
longer classified as sensitive would otherwise come back as a raw blob.
Resolves LIT-5093
* Update litellm/proxy/management_endpoints/team_callback_endpoints.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(team-callbacks): mask callback vars that fail to decrypt
decrypt_callback_vars passes a value through untouched when it cannot be
decrypted, which happens to existing rows after a salt-key rotation. Under a
key that is not classified as sensitive that blob reached the caller as opaque
ciphertext it could not use or tell apart from a real value, so mask anything
still carrying the encrypted prefix.
Raised by Greptile on the first commit.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A keyless internal user signing in to the Admin UI was redirected off the
post-login landing to /ui/connect, which renders nothing but the MCP apps panel,
so a plain gateway sign-in ended on an MCP OAuth surface the user never asked
for. The landing now renders the keys dashboard for every role. The key lookup
that existed only to make that routing decision goes with it, along with the
useKeys enabled flag it was the sole caller of and the role-hydration hold that
guarded its one-frame dashboard flash
The gateway DCR consent flow moves the other way. Its /authorize handed the
browser to /ui/chat/integrations, whose layout hard-blocks when enable_chat_ui
is off, which is the default, and client-side redirects to /ui/ without the
query string; that destroys the connect_flow handle and strands the MCP client
until the 600s flow cookie expires. It now lands on /ui/connect, which reads
connect_flow and connect_client, mounts the consent banner and puts the apps
panel in connect mode. /ui/chat/integrations keeps its connect-mode handling
this release so flows sealed before the deploy still finish
Resolves LIT-5104
Resolves LIT-4911
About 35,000 fixes ruff marks safe across 32 rules (UP006/UP045/UP007
modern annotations, UP032 f-strings, SIM114/SIM118, RET501, and
friends), removal of the 1,296 typing imports the rewrite orphaned, and
hand fixes for what the fixers could not see: five star-import
freeloaders of typing names, two F823 late-import annotations, the
/get/config/list introspection crash on types.UnionType, redundant
function-local RoleMappings imports in ui_sso.py that shadowed the
module-level name once the annotation lost its quotes, and one FURB168
tautology.
B009/B010/PIE804/RUF019 are excluded on purpose: their safe fixes
rewrite getattr/setattr/**-splat/key-in-dict escape hatches into forms
basedpyright then rejects (283 new errors measured), so their budgets
stay at base values.
ruff-strict-budget.json drops by 39,579 this commit (39,968 across the
branch) with 28 rules at an actual 0 and 9 more sharply down.
type-discipline-budget.json ratchets LIT002/LIT006/LIT009 down; LIT001
moves to the now-honest total: the checker matches the spelling `set`
but not the alias `Set`, so the 160 typing.Set annotations rewritten to
set[...] were always mutable-set annotations and only now count.
PR #35471 added classifier_context_include_assistant_turns to ComplexityRouterConfig.
It worked through config.yaml and the model API but had no control on the Add Model or
Edit Auto-Router screens, so an operator working from the dashboard could not reach it.
Wires it into the create and edit forms, shown only when the LLM classifier is
selected, matching what #35315 did for the two context-window fields
The create and edit stacks share the rendered control but keep their own serializer,
their own hydration, and their own managed-key set, so the field is added in five
places rather than one. A field wired into only one stack fails in a way neither
serializer unit test can see, since those are handed a form value assembled by hand,
so the edit-modal test drives the real component through open, edit and save
The switch is emitted even when false, because there the operator turning it off is a
choice that has to overwrite a stored true rather than an absent value a truthiness
gate would drop
Live Cursor Ask-mode captures show the shape dialects mix PER LEVEL: the
tool envelope arrives chat-nested while the grammar format inside it is
still Responses-flat, so a normalizer that pattern-matches whole-tool
templates misses every hybrid. The cursor arm now normalizes the envelope
level and the format level independently and idempotently, making it
total over the envelope x format matrix; a parametrized 8-cell test pins
every combination. The reference BYOK bridge was checked and forwards
chat bodies verbatim, so there is no prior art for these hybrids
