Folds every successful auto-routed request into LiteLLM_AutoRouterSession with one
conditional upsert at spend-write time, classifying each turn (same model, first
visit, return to tier, out of order) against the row's own columns so nothing is
read before the write. The upsert's placeholders and argument tuple both derive
from the transaction dataclass's own field order, so the SQL and the call site
cannot drift apart. GET /auto_router/benchmarks aggregates the rollup, grouped
by the full (router, type) identity, and never scans LiteLLM_SpendLogs. A turn's
cache interaction is derived once from its usage record (savings.py owns the
extraction; compute_savings_spend derives cache reads from usage_object itself),
hits are counted order-independently so the overall hit rate matches its covered
denominator, caller-chosen session ids are bounded before entering the primary
key, and a poisoned statement drops only its own session's remaining turns.
Return misses inside the recorded TTL are named for what the telemetry shows
(within_ttl) rather than a presumed cause, since a provider can evict early.
Savings ride each router's derived baseline by default, so the response carries
no deployment-wide baseline label. Rollup retention has its own
maximum_autorouter_session_retention_period setting, pattern-identical to the
spend-logs knob and running in the same cleanup job on its own cutoff. Every
drain trigger sizes the queues through one owner and the enqueue honors
disable_spend_logs beside the tool-usage queue it mirrors.
* fix(autorouter): match CJK keyword_tier_rules that regex word boundaries miss
Single-word keywords were matched with a \b...\b regex. Every CJK character is a
regex word character and CJK is written without spaces, so \b never fires between
two of them and a rule like 发票 silently missed 我需要开发票, falling through to
complexity scoring instead of the configured tier.
Keywords containing CJK now match as plain substrings, the same way multi-word
phrases already did. The gate reads the keyword rather than the prompt, so a
keyword with no CJK in it keeps word boundary matching regardless of the script
the prompt is written in.
* fix(autorouter): cover Han extensions in planes 2 and 3, not just up to U+2FA1F
The supplementary range stopped at U+2FA1F, so Extension G and H ideographs kept
the word boundary path and stayed unmatchable. Both planes are dedicated to CJK
ideographs, so covering them whole also handles later extensions without chasing
each new block.
A price data reload replaced litellm.model_cost wholesale, discarding every
runtime registration: the deployment model_info the Router registers from
model_list, and pricing overrides passed to litellm.register_model. Custom
model groups lost max_input_tokens / max_output_tokens in /model_group/info,
and a deployment whose backend model is in the catalog silently reverted to
upstream values. Runtime registrations are now recorded and replayed on top of
the freshly fetched catalog.
Router._pre_call_checks resolved the per-deployment model name only after the
model-info lookup, so an unregistered model left it unset and the supported
params check ran against the bare model group name, raising "LLM Provider NOT
provided" out of deployment selection. The name is now resolved first, and an
unresolvable provider skips that check rather than failing the request.
Resolves LIT-4675
The enterprise gate on the OAuth2 auth path raised a bare `ValueError`,
which the terminal handler in auth_exception_handler.py converts to a 401.
Every sibling enterprise gate answers 403, including `_premium_user_check`
and the SSO gate. A 401 tells the client its credential was wrong and to
retry with a better one, and no credential can satisfy that while the
install is unlicensed, so it invites a retry loop that can never succeed.
It now raises a 403 `ProxyException` shaped like the SSO gate. Two response
fields move with it: the `Authentication Error, ` prefix goes away, since
the catch-all built that around `str(e)` and a `ProxyException` is re-raised
unmodified, and `param` becomes `premium_user`, naming the condition an
operator has to clear.
The gate's own text also gains the sentence break it was missing. The
message concatenated straight onto `CommonProxyErrors.not_premium_user`,
rendering as "premium usersYou must be a LiteLLM Enterprise user".
The build_from_pip image ran a bare `prisma generate`, so prisma recorded
absolute engine paths under $HOME/.cache, which is /root/.cache in that
build. /root is mode 0700 on python:3.13-slim, so any runtime uid other
than 0 gets EACCES just traversing it and the proxy dies during prisma
client initialisation. That is exactly the shape a securityContext with
runAsUser produces.
Generate under a fixed /opt/prisma and chmod it a+rX, matching what the
shipped images already do, and pin PRISMA_BINARY_CACHE_DIR at runtime so
the client resolves the baked engines instead of looking under $HOME. A
build-time assertion fails the build if any recorded engine path lands
outside the pinned prefix, since the original breakage was silent at
build time and only surfaced as a runtime crash for non-root users.
* feat(auto-router): let operators replace the LLM classifier's system prompt
The complexity router's LLM classifier has always sent one built-in rubric, so the
router could only ever grade difficulty. Operators can now supply their own system
prompt, which replaces the rubric outright and repurposes the same tier machinery for
whatever taxonomy the prompt defines, data sensitivity being the obvious case.
Replacement is total: neither the rubric nor its closing line is appended, since both
describe grading difficulty over a "current message" and a prompt grading something
else is entitled to contradict them. That closing paragraph is also the classifier's
prompt-injection defense, so the config field and the dashboard editor both warn that
a replacement omitting it lets a caller ask for a tier and get it.
The heuristic fallback still scores complexity, which is meaningless for a repurposed
taxonomy, so classifier_fallback now chooses between the heuristic scorer and routing
straight to default_model. The default_model path bypasses tier pools, the adaptive
bandit, and escalation, because no tier was decided and the point of that fallback is
a known destination. It reports itself as default_model_fallback in the spend logs.
The dashboard's prompt editor prefills from a new
/auto_router/classifier/default_prompt endpoint rather than a copy of the rubric in
the frontend, and stores no override when the draft matches the default, so later
rubric improvements still reach every router that never customized it.
Tier names stay SIMPLE/MEDIUM/COMPLEX/REASONING; a custom prompt redefines what they
mean, not what they are called.
* fix(complexity-router): don't let the default_model classifier fallback bypass routing plugins
* fix(complexity-router): don't pin a session to the default model after a classifier failure
* fix(complexity-router): omit the tier from a default-model-fallback routing decision
The classifier never answered, so no tier was decided. The record reported the
tier whose pool happens to hold default_model, which reads in the spend log and
the UI as if the request was classified. Matches how default_fallback already
records a route that no tier produced.
* fix(proxy): allowlist /auto_router/ on the UI backend component
The new GET /auto_router/classifier/default_prompt is a UI-consumed management
route, so it belongs on the control plane. Without the prefix it was exposed by
neither component and test_gateway_plus_backend_covers_full_app failed.
* docs(ui): reword the classifier prompt disclaimer
Frames the closing paragraph as a strong recommendation rather than a
description of what gets dropped, names prompt injection explicitly, and
notes the tier names stay fixed regardless of their display names.
* fix(complexity-router): stop logging a fabricated tier on the plugin fallback path
The classifier-failed fallback resolves a tier so the routing-plugin pipeline has a
pool to filter, but nothing about the request produced that tier. The non-plugin
short-circuit already dropped it from the logged decision; the plugin path still
reported it, so a spend log claimed a classification the request never received.
Record the pool as a plugin-filtered-pool signal instead.
Also name the real problem when the resolved tier has no models at all: that raised
"No candidate models left after routing-plugin filtering" and sent operators hunting
for a policy plugin that never narrowed anything.
SGR has had two independent definitions. The admin UI derived it from
SpendLogs, so it counted what litellm's logging callbacks observed and could
attribute and price. BillableRequestMetricsMiddleware counted what the proxy
actually answered at the ASGI edge, but only exported to OTLP for enterprise
metering. The two disagree by design in places, and the SpendLogs figure goes
quiet whenever spend logging is disabled or the callbacks are bypassed.
This adds LiteLLM_DailyGatewayRequests, written by the middleware, and points
the dashboard's Successful Requests tile at it.
Requests fold into an in-memory map at record time rather than going through a
queue like the spend path. A count is a pure aggregate, and every dimension of
the key is chosen by the proxy from a closed set: the date, the category, and a
route that the classifier maps to one of a fixed list of strings rather than
passing the raw path through. Nothing a caller sends can add a key, so the fold
and the table are bounded by (days x categories x routes) however much traffic
arrives; the spend queue blocks once full, which is not acceptable in the
response path. A scheduler job drains it on the existing batch interval, and a
failed flush merges its counts back so a database blip undercounts nothing.
The middleware previously returned early when no billing recorder was
injected, which is the unlicensed case. The new sink is not license-gated, so
that early return now requires both sinks to be absent. The billing recorder
keeps its 2xx-only gate; the sink takes every status so failed_requests is
real. The sink is not told which deployment served the request, unlike the
billing recorder. That id is a sha256 over litellm_params, credentials
included, so a caller who puts a credential in the request body mints a fresh
one per distinct value. No configuration is needed for that: api_base and
base_url are on _BANNED_REQUEST_BODY_PARAMS and need allow_client_side_
credentials, but api_key is not on that list, and both reach the same
_handle_clientside_credential branch. The read endpoint aggregates the
dimension away regardless, so the key is better off without it.
The new table carries no key, user or team dimension, so /gateway/daily/activity
is restricted to proxy admin roles and the per-key and per-model breakdowns
keep reading the daily spend tables. The old path is left running and marked
with TODOs.
A fetched result carries the range key it was fetched for, and the render
selects it only when that key matches the range on screen. Both the gateway
counts and the spend aggregate go through that rule: the request tiles read the
first and fall through to the second, so stamping only one of them would leave
the tile showing a superseded range by the other route.
The paginated pages behind that aggregate are reached through a failure flag,
so the flag is stamped too. A flag left over from the previous range would let
those pages through while a new range is in flight, which is the same defect
one fallback further down.
`_setup_prisma_client` ran `connect()`, then a `SELECT 1` health check, then
armed the DB health watchdog. Any failure fell into one handler that, with
`allow_requests_on_db_unavailable` set, swallowed the error and returned None,
which the caller assigns to the module-level `prisma_client`. A single
transient timeout on that health check therefore discarded a client that had
already connected, for the life of the process, and skipped the watchdog that
exists to reconnect it.
The watchdog now starts before the health check, and a swallowed post-connect
failure returns the connected client instead of None. A client whose
`connect()` failed is still discarded, and startup still hard-fails when
`allow_requests_on_db_unavailable` is not set.
The same check also misreported its own failure. `health_check()` labelled its
error `disconnect()`, a copy-paste from the real `disconnect()` below it, so
grepping the logs for the health check turned up nothing and read as "the check
never ran". Both it and the sibling `connect()` failure reported through
`print_verbose`, which reaches `verbose_proxy_logger.debug` and otherwise prints
only under the deprecated `litellm.set_verbose`, leaving a startup-blocking
database fault invisible at the verbosity operators actually run. Both now log
at warning under their own names. The proxy logger's handler carries the secret
redaction filter, so a connection string in the exception text is redacted
exactly as it was on the old print path.
Every ruff-strict rule that sat above its budget limit (FURB188, RUF022,
SIM118, UP007, UP032, UP037) is now at zero, LIT001 and LIT006 are back
under their ceilings, and the freed headroom is ratcheted out of
ruff-strict-budget.json, type-discipline-budget.json, and
basedpyright-code-budget.json so the gates take the fast path again
The gate only matched os.getenv(, litellm.get_secret( and
litellm.get_secret_str(, so a bare get_secret_bool("X") matched nothing and
the key bypassed the documentation requirement entirely. Add a fourth pattern
for get_secret_bool, with or without the litellm. prefix, and a negative
lookbehind so an unrelated receiver's .get_secret*( call is not mistaken for
an env var read.
Extraction and table parsing move into functions behind a __main__ guard so
the patterns can be unit tested; the script is still invoked exactly the same
way by CI.
This surfaces 13 keys the gate never checked, 8 of which have no reference
row yet.
PR #35929 zeroed the eslint budget headroom while #35893 added UI code in parallel, so staging went over budget by one complexity violation and two no-large-inline-object-arg violations, failing frontend-lint on every UI-touching PR until #35964 reverted the ratchet. This removes the three violations at the source so the budgets can ratchet back down: the submit-blocked-reason chain in add_auto_router_tab moves to a module-level helper, taking the component arrow from complexity 21 to 18, and the two four-property object literals in build_complexity_router_config.test.ts move into named variables. No behavior change; the touched suites pass (101 tests)
* fix(proxy): give proxy_admin_viewer read parity with proxy_admin
Route-level checks already default-allow management GETs for the viewer
role, but ~15 handlers compared user_role to PROXY_ADMIN only, dropping
viewers into regular-user scoping (/key/list, /user/info, /model/info,
guardrails, prompts, agents, memory, workflows, MCP catalog, coordination
redis settings, credential migration check, enterprise projects). Swap
those read paths to user_api_key_has_admin_view; write gates unchanged.
The dashboard now presents the viewer session as Admin for all gating
(effectiveSessionRole) so every page fetches with admin visibility, with
userRoleLabel/isViewOnly preserving the account-menu label and the
playground cost guard. The server remains the write authority.
* refactor(agents): remove side-effectful health_check param from GET /v1/agents
Addresses a security review finding on the admin viewer read parity change:
listing agents with health_check=true made the proxy issue a server-side GET
to every agent URL, so a read-scoped caller could trigger request fan-out
beyond their object permissions. The list endpoint is now a pure read for
every role.
Removes the query param, the URL probing helper and its timeouts, the
AgentHealthCheck httpx provider tag, and the dashboard's Health Check
toggle. Requests still passing health_check=true get the full list back
with the param ignored.
* fix(proxy): keep credential encryption check proxy_admin only
The residual scan behind GET /credentials/migrate-encryption/check loads
every model, credential, MCP, team, and verification-token row and runs a
decryption attempt on each stored value. Extending it to proxy_admin_viewer
let a read-only account repeatedly trigger deployment-wide scans, so the
route keeps its original full-admin gate.
* fix(agents): restore health_check, keep list fast path proxy_admin only
Restores the agent health_check feature exactly as before this PR: the
query param, the URL probing helper, the httpx provider tag, and the
dashboard toggle all return, so existing callers keep the filtering
contract. The viewer expansion is instead reverted at its source: the
GET /v1/agents admin fast path stays PROXY_ADMIN only, so a
proxy_admin_viewer goes through the object-permission scoped branch as
before and cannot fan out health checks beyond their allowlist. The
viewer read of a single agent stays viewer-inclusive since it has no
side effects.
Import-block conflicts against #35962's unused-import removals resolve
to this branch's side: its modernization already removed every typing
import staging trimmed, plus the usages
Resolve litellm/types/google_genai/main.py and litellm/types/utils.py by
keeping this branch's modernized annotations on top of staging's removal
of inert type: ignore comments. Rebuild ruff-strict-budget.json from
measured merged-tree counts where de-excluding litellm/types adds
violations, keeping the stricter of the two sides' limits everywhere
else so no rule gains headroom. Fix the four type-discipline additions
the merge surfaced: freeze GEMINI_1_5_ACCEPTED_FILE_TYPES, drop a
callback_args parameter rebind in guardrails, and give the two remaining
mutations reasoned suppressions
The Prisma CLI is a Node program that installs a private Node runtime on its
first invocation. That one-time install shared the 60s budget that bounds each
migration command, so on a slow or cold machine it was killed before it could
finish. Prisma then decides whether to reinstall by testing the cache
directory for existence alone, and a killed install leaves that directory
behind, so every later attempt skipped the install and failed on a node binary
that was never written. The existing four-attempt retry loop could not help:
each attempt hit the same missing binary, which turned a slow start into a
container that never migrated again.
Migrations now prepare the toolchain as its own step under its own budget, and
a cache directory that exists without a node binary is deleted first so an
interrupted install reinstalls instead of persisting. Both budgets are
overridable, LITELLM_PRISMA_BOOTSTRAP_TIMEOUT for the install and
LITELLM_PRISMA_COMMAND_TIMEOUT for each Prisma command, and every previously
hardcoded timeout now goes through one helper rather than thirteen literals.
The per-command default stays at 60s.
An override is only honoured when it parses as a finite positive number.
Infinity and NaN parse as floats and survive a plain positivity check, and
subprocess treats either as no deadline at all, so a value like `inf` or a
fat-fingered `1e400` would have silently disabled the timeout it was meant to
configure.
Three follow-ups surfaced while merging current staging into this branch.
`exc_info=True` at both fallback-failure log sites handed a live exception to
the logging machinery. SecretRedactionFilter rewrites `record.exc_text`, but
`record.exc_info` stays an exception object no filter can reach, so a handler
that renders it itself (Datadog and OTel log bridges do) received the
unredacted provider key. Both sites now pass `redact_string(traceback.format_exc())`
as a `%s` arg, keeping staging's lazy-logging form. The existing test only
asserted on `exc_text`, so it passed under the bug; it now renders `exc_info`
the way a bridge handler would and covers every record the call emits.
The eager deferred-stream fetch existed only on the async path. Vertex and
Bedrock build the same `completion_stream=None` plus `make_call` wrapper on
their sync branches, so `Router.completion(stream=True)` still surfaced the
provider error on first iteration, outside `_completion`'s except block, and
never reached the fallback chain. `_completion` now calls `fetch_sync_stream()`
under the same guard `_acompletion` uses.
The first of the three header-strip passes in the proxy error path was dead:
only the custom-header update and the response-headers hook run before the
second pass re-filters everything. Collapsed to one `safe_headers` binding.
* feat(complexity_router): let operators rename the four complexity tiers
Adds an optional tier_labels map to complexity_router_config so a deployment can
put its own vocabulary on the four tiers, e.g. Cheap / Standard / Premium / Deep,
instead of reading SIMPLE / MEDIUM / COMPLEX / REASONING in its dashboard, its
spend logs, and the rubric the LLM classifier reasons with.
Labels are display-only. Every config key stays canonical, so tiers,
keyword_tier_rules[].tier, and tier_boundaries are written exactly as they are
without labels, and partial maps are fine with unlisted tiers keeping their
default name. A validator rejects blank labels, two tiers sharing a label, and a
label that is another tier's canonical name, since any of those would make a log
row or a rubric line ambiguous. That validator runs on the /model/new and
/model/update write path already, so an ambiguous config gets a 400 rather than
being stored for the router to refuse later.
Under the default heuristic scorer the names are cosmetic: the scorer maps a
weighted score to a rung and never reads a tier name, verified by running the
eval corpus with and without a rename and getting identical tier and identical
score on all 29 cases. Under classifier_type: llm the labels are the names in the
rubric and the values the classifier must return, so the response format's enum
is now built from the configured labels and a reply is resolved back to its tier
against labels first, then canonical names, case-insensitively. An unresolvable
reply degrades to the heuristic on the existing fallback path. A test pins the
generated schema for an unrenamed deployment as equal to the shipped
TierClassification schema, so the wire shape can't drift.
Spend logs keep routing_decision.tier canonical so rows from before and after a
rename stay comparable, and gain routing_decision.tier_label on the tiers that
were renamed.
* refactor(complexity_router): drop added comments and the Counter construction
Review feedback: the repository guide bans new comments, so the explanatory
comments and the appended docstring paragraphs this branch added come back out.
One-line docstrings stay in complexity_router.py, matching that file's own
convention.
The duplicate-label check no longer builds a Counter, which the mutable-collection
budget counts, and the error text drops its list() reprs for joined strings. The
labels are stripped in tier_label() now rather than by rewriting the field in the
validator, so the stored config keeps exactly what the operator wrote.
schema.d.ts is regenerated: ComplexityRouterConfig is exposed in the OpenAPI spec,
so tier_labels surfaces there.
* fix(ui): carry tier_labels through the auto-router preset prefill
buildPresetPrefill maps every payload key onto form state, but the tier_labels
key added by this branch had no line, so a preset shipping labels would apply
its tiers and silently drop its names.
