dict(form) collapses duplicate multipart keys to the last one, so a request with
timestamp_granularities[]=segment and timestamp_granularities[]=word reached the
provider as a single-element list. Iterate form.multi_items() instead.
Fixes#35937
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(spend): derive a default auto-router savings baseline from the hardest tier
The savings driver shipped off by default: unless an operator names
litellm_settings.autorouter_savings_baseline_model, every auto-routed request
records $0.00 and the dashboard card never populates. Nobody discovers a knob
whose feature they have never seen work, so the default has to come from
somewhere the proxy already knows.
The router's own tier ladder is that place. Without a router a deployment runs
one model that can carry the hardest request it will see, so the derived
baseline is the priciest model in the hardest configured tier, REASONING when
present, otherwise the most severe tier the router actually defines. A cheap
tier is a choice the router made, not a ceiling it was bounded by.
An earlier draft of #35521 derived this per request and was deleted for it:
ranking candidates against the request that ran meant reading the request, and
every input shape it could take produced its own review finding. This
derivation is ranked against one fixed reference request instead, a cache-heavy
shape matching real auto-routed traffic, so it never reads the request at all.
Candidates still resolve through the router's deployments, so Azure base_model
and per-deployment pricing overrides rank correctly.
The deciding router records the result on its routing_decision, because one
model name can carry several tag-scoped routers with different tier ladders and
only the deciding instance knows which of them routed the request. The spend
writer's precedence is: configured baseline, then the recorded one, then off.
When the setting is present the router skips deriving entirely rather than
pricing candidates per decision only to be ignored.
Resolution never raises; an unresolvable baseline zeroes the driver instead of
failing a live request. Rows queued by a pod on the previous release carry no
recorded baseline and fall back to the configured setting, exactly as today.
The schema.d.ts regeneration also picks up the reminder_markers field that
UI-19232 (#35874) added without regenerating, so one hunk there is inherited
staleness rather than part of this change.
* fix(spend): cache the derived baseline, price it by deployment, keep it out of the routing preview
Three review findings on the derived baseline, addressed together because they
all sit on the same value's path from derivation to consumer.
Derivation walked and priced the hardest tier's whole pool inside a property
read on every routing decision, unbounded by pool size. The router now caches
the result per instance with a 30 second TTL, None results included, so the
hot path is a clock compare and a deployment edit still lands within a window
no operator watches closer than.
Ranking used each deployment's effective pricing but recorded only the model
name, so the spend writer priced the winning baseline at its public rate: a
hardest tier whose deployment carries a negotiated rate produced materially
wrong savings. The decision now also records savings_baseline_deployment_id
and the writer resolves it through Router.get_deployment_model_info, exactly
as the selected arm already does. The id is ignored whenever the configured
setting overrides the recorded baseline, since the setting names a model, not
a deployment.
/auto_router/test_routing returns the routing decision verbatim to team admins
while only authorizing the classifier and embedding models, so a derived
baseline would resolve another team's model-group alias into its backend
provider/model mapping and hand it to a caller never authorized for it. The
preview's throwaway router is built with derive_savings_baseline=False; its
decisions are never spend-tracked, so nothing is lost, and a source-pinning
test keeps the flag on the endpoint.
Also strips the explanatory comments this PR had added.
* refactor(spend): pin the derived baseline per router instance instead of a TTL
Creating or editing a router already rebuilds its ComplexityRouter instance,
through unregister and re-add on upsert and through the registry reset on a
full model_list load, so a value derived once per instance refreshes on
exactly the flows that can change it. That makes the TTL a solution to a
problem the rebuild lifecycle already solves, and it goes.
Derivation stays deferred to first use rather than running in __init__: during
a config load this router can be constructed before the deployments its tiers
name, and a baseline pinned at that moment would be empty for the process
lifetime.
The one behavior the TTL had that the pin does not: editing a tier deployment
without touching the router itself refreshed the baseline within a window.
That edit path rebuilds only the edited deployment's own strategies, so the
pin holds the old answer until the router is next saved or the config next
loads. A stale deployment id degrades to public-rate pricing rather than
failing, which is where every other unresolvable baseline already lands.
The dashboard pins engines node >=24.14.1 with engine-strict, so make
bootstrap dies with EBADENGINE on any shell whose default node is older.
Wrap the npm install in scripts/with_dashboard_node.sh: it execs the
command as-is when node already meets the floor, otherwise activates the
.nvmrc version via nvm or fnm, and fails fast with install instructions
when neither manager exists
* feat(auto-router): make reminder marker pair configurable
Some harnesses inject internal context using their own marker pair
instead of Claude Code's <system-reminder>/</system-reminder>
convention, and some send it as a separate follow-up user message
rather than inline with the ask. Both cases fall out of the same root
cause: the router's marker-matching is hardcoded, so foreign markers
never strip to empty and the reminder-only turn wins "newest human
ask" selection instead of being skipped.
Add an optional reminder_markers field to ComplexityRouterConfig so
operators can override the (open, close) pair via proxy config, with
the existing skip-when-empty selection logic handling both cases once
the markers match.
* test(auto-router): drop unsolicited comments from the reminder-markers regression test
Per Greptile review on #35874: no comments unless explicitly requested.
A three-segment token presented while `general_settings.enable_jwt_auth` is
unset is never treated as JWT-shaped, so it falls through to the virtual-key
path and is rejected for not starting with 'sk-'. That reads as a missing
key in the verification table and sends the operator off to inspect virtual
keys, when the real cause is one missing config line. The rejection now
names `enable_jwt_auth`, appended to the existing text so the Prometheus
invalid-key filter and the admin UI keep matching what they match today.
The hint claims only that the key is JWT-shaped. Segment count cannot tell a
JWT from any other dotted credential, so asserting the key IS a JWT would
swap one confident misdiagnosis for a narrower one.
The enterprise gate on that same path raised a bare `ValueError`, which the
terminal handler turns into a 401. Every sibling enterprise gate answers
403, and a 401 tells the client to retry with a better credential, which no
credential can satisfy while the install is unlicensed. It now raises a 403
`ProxyException` like the SSO gate does.
* feat(ui): add Test Routing to the auto router create form
Route a test prompt through the complexity-router config on screen before the router
is saved, showing the model it lands on and the same decision trace the Logs page renders.
Adds POST /auto_router/test_routing, which classifies with the live pre-routing hook and
sends nothing to the routed model.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): reset the routing test modal on reopen and expose /auto_router on the UI backend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): enforce caller model access and key budget on the routing test's classifier call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: tin <tin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This reverts commit dcb4e5033c.
The suites landed without the proof-of-fix and QA runbook the PR body
itself flagged as outstanding, so the coverage they claim is unverified
against a live proxy
* fix(zscaler_ai_guard): return 400 on guardrail block
* fix(zscaler_ai_guard): don't log error on intentional BLOCK
A BLOCK is expected guardrail behavior, not a failure. Before this
fix, raising HTTPException inside the try block caused the generic
except to log it as "Failed to apply guardrail", producing spurious
error-level noise for every normal block event.
Added except HTTPException: raise before the generic handler (matching
the existing pattern in make_zscaler_ai_guard_api_call), and a
regression test that asserts logger.error is not called on a BLOCK.
---------
Co-authored-by: yucheng-berri <yucheng@berri.ai>
* fix(claude-code): make skill registration create-only with a PUT update route
POST /claude-code/plugins upserted by name, so re-registering an existing
name silently overwrote the stored skill's source and metadata. The "Add
New Skill" UI button posts here, so a name collision clobbered a different
skill with no signal to the user.
Make POST create-only: it returns 409 if the name already exists, with a
unique-violation guard mapping the find-then-create race to the same 409.
Add an explicit PUT /claude-code/plugins/{plugin_name} for updates (404 if
the name is missing). PUT is a full replace and documents that omitted
fields reset to their defaults, so UpdatePluginRequest defaults version to
None instead of fabricating the create-time 1.0.0.
The shared mutable fields move to a PluginSpec base; RegisterPluginRequest
keeps its name and its generated schema unchanged, UpdatePluginRequest
carries no name. Regenerated the dashboard types and the lazy openapi
snapshot for the new route.
Resolves LIT-4110
* fix(ui): surface the proxy error detail so the skill 409 conflict is legible
The add-skill form rendered the raw HTTPException envelope on failure
because deriveErrorMessage did not unwrap an object-shaped detail
({"detail": {"error": ...}}), so the new create-only 409 reached the user
as a JSON blob. Unwrap object-shaped detail at the client layer, which
covers every handler that returns detail={"error": ...}, and surface the
resulting message verbatim on the form instead of burying it under a
generic prefix.
* refactor(claude-code): replace blind excepts in plugin mutations with typed handling
Narrow register_plugin's create-conflict guard from a broad 'except Exception'
+ isinstance dance to a direct 'except UniqueViolationError', using an Exception
subclass sentinel (not None) as the prisma-absent fallback so the sentinel can be
caught directly. Drop update_plugin's outer 'except Exception -> 500' wrapper so
HTTPExceptions propagate on their own and unexpected DB errors surface as FastAPI's
default 500 rather than echoing str(e). Keeps the BLE001 strict-rule budget green.
* fix(claude-code): restore structured 500 handling on update_plugin via typed PrismaError catch
Flattening update_plugin to satisfy the no-blind-except rule dropped its error
wrapper entirely, so a data-layer failure (e.g. a dropped DB connection) would
skip the intentional verbose_proxy_logger.exception call and degrade the response
from the endpoint's structured {"error": ...} body to FastAPI's default
{"detail": "Internal Server Error"}, inconsistent with every sibling route.
Wrap update_plugin in 'except PrismaError' instead of the blind 'except Exception'
the other routes use: it logs and returns the structured 500 for real DB failures
while letting genuine code bugs surface rather than masking them as 'Update failed',
and stays off the BLE001 budget. Add a regression test that a PrismaError during
the update maps to a structured 500.
* fix(claude-code): import prisma error types at function level to satisfy LIT009
* refactor(claude-code): typed plugin mutation responses and lint gate fixes
Return RegisterPluginResponse models from POST and PUT instead of ad-hoc
dicts, declare them as response_model so the OpenAPI schema and dashboard
types carry the real response shape, build the stored manifest via
model_dump, and drop update_plugin's unused auth parameter (the route
dependency already enforces auth). Keeps the LIT002/B008/UP045 budgets at
their ratcheted ceilings after merging litellm_internal_staging
Generic SigV4 double-encodes the canonical URI while S3 canonicalizes the wire path with single encoding, so any object key containing a character that percent-encodes (a team alias, key alias or s3_path with a space) was signed over %2520 while the request carried %20; S3 recomputed a different signature and answered 403.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
The 12 GB NODE_OPTIONS setting lived only in the Makefile export and the
CI env line, so any hand-run gate pipeline forgot it and node OOMed at
the ~4 GB default after 80 seconds, with || true feeding the gate empty
output. The gate now spawns basedpyright itself for both the head and
base passes, appends the heap flag last so it wins node's last-flag-wins
resolution while preserving other caller flags, and fails loudly on
crash exit codes instead of reading them as zero errors.
Since #35492 the evicted-client closer really closes litellm-owned
httpx clients once their cache entry is evicted and the grace window
passes. Objects that fetch get_async_httpx_client once in __init__ and
hold the handler forever (40 guardrail classes, pagerduty and email
callbacks, and more) then fail every request with 'RuntimeError: Cannot
send a request, as the client has been closed.'
AsyncHTTPHandler.client and HTTPHandler.client are now properties that
rebuild the inner client from the constructor's stored config when the
handler owns it and finds it closed. Caller-supplied or assigned
clients are never rebuilt, and close paths use the backing field so
closing a handler does not resurrect it.
This reverts commit 66bc70365f and the
follow-up 2-line type fix a6d4654261 (#35706), which only retyped a
signature #35492 introduced.
Closing evicted litellm-owned clients breaks every object that fetches
get_async_httpx_client once in __init__ and holds the handler for the
life of the process: 40 guardrail classes plus the pagerduty and email
callbacks. Once the cache entry is evicted (TTL 3600s or the 200-entry
size cap) and the 900s grace passes, the held client is closed and every
subsequent request through it fails with RuntimeError: Cannot send a
request, as the client has been closed. On a production deployment with
a default-on guardrail this surfaced as every request 500ing roughly 75
minutes after boot.
The connection-reclaim goal of #35492 can re-land once handlers survive
their inner client being closed.
The proxy serves POST /openai/v1/responses alongside /responses and
/v1/responses, but only the latter two were in API_ROUTE_TO_CALL_TYPES.
UnifiedLLMGuardrails.async_post_call_success_hook resolves the call type
from request_route, so on the alias it resolved to None and returned the
response unscanned; model output reached the client with post-call
guardrails never running. The key and team tool allowlist was unenforced
on the same alias for the same reason.
Register the alias family in API_ROUTE_TO_CALL_TYPES and in
LiteLLMRoutes.openai_routes, mirroring how the /openai/v1/realtime
aliases are registered, and log a warning at the two points where the
unified guardrail skips post-call scanning so a future unmapped route is
visible instead of silent.
The Responses block of API_ROUTE_TO_CALL_TYPES moves from list to tuple
literals because the LIT002 budget rejects net-new mutable-collection
construction; the map is read-only, so it is now typed as a Mapping of
Sequence and the budgets ratchet down accordingly.
* fix(proxy): apply key_alias/key_hash filters to all /key/list visibility branches
The filters previously lived only in the own-keys OR branch, so a team admin's admin-team branch matched every team key and the Key Alias filter in the Virtual Keys UI appeared broken. Both filters are now global AND conditions alongside team_id/project_id/access_group_id/agent_id, narrowing every visibility branch while leaving unfiltered visibility unchanged.
* chore: drop new explanatory comments flagged by review
* chore: restore schema.d.ts to base enum order
The management route-coverage guard fires because /team/metadata_schema landed
in #33353 without a behavior-suite scenario, so this adds one covering the nine
seeded actors plus the unauthenticated 401
The prometheus budget-metric assertions read the log call's first positional
arg, which #35703 turned into an unrendered "%s" format string when it moved
logging to lazy args. They now render the message from the call args, which
also pins the arg order and the exception text that the old substring check
never reached
GitHub Models was fully retired on 2026-07-30, so test_completion_github_api
can no longer pass: the endpoint the github provider targets returns 404 and
models.github.ai answers 410 "github_models_retirement_brownout". The dead live
test is removed rather than skipped
