* feat(auto-router): support classifier reasoning effort
* fix(auto-router): harden classifier reasoning effort
* fix(ui): satisfy classifier config lint limits
* refactor(auto-router): simplify classifier effort support
* fix(auto-router): clear frontend-lint and type-discipline gates, trim LOC
---------
Co-authored-by: Tin Chi Lo <tin@berri.ai>
A session pinned to a text-only model failed every image turn with a provider 400, because the modality gate exempts a kept pin by cause. Add modality_pin_override so that exemption is conditional: the image turn is re-placed on a capable model for that request only, reported as cause modality_pin_override, and the stored pin is left untouched so the next text turn replays it.
The pin write on the replay path already happens upstream of the gate and stores the session's own model, so pin survival is structural rather than bookkeeping. The new cause joins the non-pinnable set. Default off at every layer.
* fix(proxy): keep SpendLogs and callback session ids in sync when the request has none
Add general_settings.missing_session_id (generate | reject). In generate mode one id is
stamped into litellm_session_id, litellm_trace_id and metadata.session_id before callbacks
run, so LiteLLM_SpendLogs.session_id and the Langfuse session id match. In reject mode such
requests get a 400. Unset keeps the legacy behavior. MCP routes are not affected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): regenerate schema.d.ts and shorten mutable-ok comment for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark generated session ids so affinity consumers do not pin on them
Fireworks x-session-affinity, the router session_affinity pre-call check and the
complexity router session pin all read metadata.session_id as a caller-chosen
stable key. A missing_session_id: generate id is fresh per request, so it now
carries metadata.litellm_session_id_generated and those consumers skip it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(complexity_router): escalate oversized prompts to a tier that fits before dispatch
The classifier scores complexity and never prompt size, so a long agentic
session whose newest ask is trivial classifies SIMPLE onto a small-window
tier and the provider rejects it with a context-window 400 that nothing
retries. The gate runs after classification on every decision path
(classify tail and session-affinity pin), estimates prompt tokens
including the out-of-band carriers (top-level system, tools,
instructions), and when the decided tier provably cannot hold the prompt
moves the request to the lowest configured tier with a model whose
declared window fits, restricting the pick to fitting models when the
decided tier can keep it. Models with no resolvable window are never
escalated away from or onto, escalated decisions are never written as
session pins, and the decision records context_escalated plus the
original tier in spend logs.
Resolves LIT-6503
* fix(complexity_router): judge groups by smallest window, bound skips by bytes, filter adaptive picks
Review-round rework, one mechanism per finding. A group is judged by its
smallest resolvable deployment window, since the core router picks within
a group with no fit check. The counting skip is gated on UTF-8 byte
length, which BPE token counts can never exceed, so token-dense scripts
cannot slip past it; only a real tokenizer count ever moves a request and
a failed count leaves the placement alone. The fit facts now filter every
adaptive phase including cold start and the tier fallbacks. Window
questions adopt the declared provider and never resolve authenticating
providers, and a router instance without get_model_list degrades the gate
to a no-op. Tests rebuilt on real Router instances resolving deployment
model_info end to end, plus a full-path test through
async_get_available_deployment
An auto-router tier pin injects reasoning_effort into request_kwargs, but
provider translations give a caller-supplied thinking, output_config.effort,
or reasoning carrier precedence over the reasoning_effort alias, so the pin
never reached the wire whenever the client expressed effort natively. Drop
the client's other encodings of the setting at the tier-param merge; a
client output_config keeps its non-effort fields
Every complexity router now derives and records its savings baseline from
its hardest configured tier, and the spend writer always prices against the
decision-recorded baseline model and deployment id. A leftover
litellm_settings.autorouter_savings_baseline_model key is inert
* fix(router): drop a tier param the routed target cannot take
A complexity tier's litellm_params are an operator override applied to every request that tier
routes, and they were written into the request kwargs unconditionally. When the tier set a param
the target does not declare, get_optional_params raised UnsupportedParamsError before the request
left the proxy, so the whole tier answered 400. The bundled Lite preset sets reasoning_effort on
its complex tier, and four of the thirteen kimi-k3 map entries reject that param, so a router
built from a first-party template failed on every complex prompt
Filter the tier params at both store sites against what the group's deployments declare. The
candidate set is asked of the module that raises rather than derived from a second list, so
credentials, endpoint and transport controls are never at risk: base_url, timeout,
default_headers, organization and deployment_id are not chat completion params and never reach
that comparison. A param survives if any deployment could take it, since routing has not picked
one yet, and it survives an unresolvable provider or an empty group, since a best-effort filter
must not narrow what the request already did
The skip list _check_valid_arg applies before rejecting a param now has one owner both it and the
router read, so the two cannot drift
* fix(router): honor allowed_openai_params when gating tier params
* test(router): cover _declared_param_allowlist malformed declarations
* fix(router): never ask an authenticating provider whether it takes a tier param
Resolving github_copilot or chatgpt runs their OAuth device flow, so the
capability question _deployment_accepts_param asks would freeze the event
loop for minutes inside async_get_available_deployment. Promote
register_model's local skip set to
constants.PROVIDERS_THAT_AUTHENTICATE_ON_PROVIDER_INFO and fail open on
those providers before any lookup
* fix(utils): adopt a declared authenticating prefix instead of resolving it
The tier-param guard alone was not enough: the savings baseline and the
model-info funnels also resolve deployments during routing, and each
resolution of github_copilot or chatgpt runs their OAuth device flow.
declared_authenticating_provider gives every metadata funnel
(get_supported_openai_params, _get_potential_model_names,
_supports_factory, canonical_model) the resolver's answer by string, so
the whole routing path answers without authenticating. A through-test
drives async_get_available_deployment with a copilot deployment and
records that no copilot resolution happens
A coding agent names each conversation by quoting the whole session and asking
for a title. The classifier rated the quoted session rather than the request, so
the cheapest call the client makes routed to the most expensive tier: 11 of 17
title generations in one day of real traffic came back COMPLEX.
Recognize those prompts by literal sentinel on the newest ask and route them to
the cheapest configured tier without classifying them, so the call costs nothing
to route. The placement is scoped to the one request that carries the sentinel:
it never displaces an operator's classifier plugin, the bandit cannot reach above
the tier as raised, it never becomes the session pin, and the sentinel that
matched is recorded on the routing decision. Detection reads the newest ask
alone, so a title request quoted into a later turn cannot cheapen the work that
follows it, and a keyword rule, an escalation keyword or the plan-mode floor all
still decide over it.
Regenerates the lazy OpenAPI snapshot, which was already stale on the base for an
unrelated Presidio guardrail field and failed the schema check on every PR.
Resolves LIT-6349
A complexity-router setting placed beside complexity_router_config, or inside a
tier entry's litellm_params, is read by nobody: the router loads its settings only
from litellm_params.complexity_router_config. It does not stay inert. The
alias-marker forwarding and the per-tier param spread carry every unrecognized key
onto the outbound request, and all_litellm_params only knows the outer names, so
the key reaches the provider as an unknown body field and every call through that
model group fails with an error naming an internal config key.
Guard the whole set, derived from ComplexityRouterConfig.model_fields so a field
added later is covered, and scoped to complexity-router deployments because the
names only mean this there (embedding_model is a legitimate flat param on an
s3_vectors vector store). Scope is read from the same merged field view the naming
check is judged on, so a router named only by its default model is in scope and a
field added to the required-field table is covered without another edit. The write
endpoints reject with a 400 naming the keys and where they belong, config.yaml
refuses to start for the same reason max_agentic_loops does, and a tier entry is
judged by the config model itself.
An already-stored deployment keeps loading, so an upgrade cannot take a running
gateway down over a row that was written before the gate existed.
* feat(complexity_router): heuristic-first classifier chaining
Adds classifier_type 'heuristic_first', which scores locally on every request and
only calls the LLM classifier for traffic the scorer could not place at or below
heuristic_first_max_tier. A request short-circuits when the scorer landed at or
below the threshold and produced at least one signal; everything else escalates.
The signal requirement is load-bearing. A prompt where no dimension fires scores
exactly 0.0, which is under simple_medium, so the score-to-tier mapping calls it
SIMPLE by default rather than by evidence, and that is about half of general
traffic. Gating on the tier alone would route it to the cheapest model without
ever consulting the classifier.
Introduces uses_llm_classifier as the single owner of 'does this router call the
classifier model', replacing the classifier_type == 'llm' comparisons in the
config validator, the prompt prebuild, the health dependency graph, the
routing-test authorizer, and six dashboard sites.
* fix(complexity_router): reuse the heuristic verdict on classifier failure, load the threshold on edit
Three review findings, one push.
The heuristic-first fallback re-scored the prompt after a classifier failure,
which the README already documented as a reuse. The outcome computed before
escalation is now handed to the failure path, so the scorer runs once per request.
The edit modal never hydrated heuristic_first_max_tier, while save rebuilds every
managed key from form state, so opening a heuristic-first router and saving it
dropped a field the proxy requires. The dropdown's display fallback hid it. Both
are fixed, and the hydration is extracted into a pure function so a test can pin
the invariant: every managed key present in a stored config survives an untouched
open-and-save. That test also covers every field added later.
Classifier radio labels lost their em dashes, per the repo writing convention.
The LLM classifier capped every prior turn at 200 characters independently, so a
785 character turn was cut even when the whole block it belonged to was 353
characters. A character budget now bounds the block: turns are taken newest first
and quoted whole while they fit, older turns are dropped whole once it runs out,
and only the turn straddling the boundary is cut. The per-turn cap stays as an
optional clamp for operators who set it deliberately, defaulting to unset.
The LLM classifier's conversation context cut each prior turn head-only, so a turn
opening with an incident report and closing with the actual request reached the
classifier as the incident report alone. Keeping head and tail costs the same
budget and is what the truncation literature measures as best for classifying
long text.
* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
A test that asserts on the error inside its own except block passes when the
call stops raising, because nothing runs the handler. That is the exact case
the test exists to catch, so the regression lands green.
Rewrites all 111 such blocks into pytest.raises, which fails when the call
succeeds, and selects PT017 in ruff-tests.toml so no new one lands.
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* feat(complexity_router): add business classification rubric preset
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate api schema for business rubric
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): suppress preexisting antd import violations in touched files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: tin <tin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The forwarded-keys record moves off the shared metadata dict onto the
per-request kwargs, where _update_kwargs_with_deployment consumes it, so
sibling requests that share a metadata dict (abatch_completion) can no
longer clear it mid-routing. Per-tier litellm_params from the hook
response are never treated as forwarded marker params, and a deployment
only beats a forwarded value when it sets its own, not when it carries a
LiteLLM_Params default such as merge_reasoning_content_in_choices=False.
An `auto_router/<alias>` marker entry's litellm_params (for example
`aws_region_name: eu-west-3`) were forwarded onto every routed call with
setdefault and then won the `{**litellm_params, **kwargs}` merge against
the selected tier's own values, so a Bedrock tier pinned to us-east-1 was
called in eu-west-3 and failed with 400.
The hook now records which keys it actually forwarded on the request's
metadata bucket, and `_update_kwargs_with_deployment` drops every
forwarded key the selected deployment defines itself, so marker params
only fill gaps a tier leaves open. Request-supplied values still win over
both. The stamp is stripped from logged metadata like its siblings.
Fixes#37613
The reasoning override's floor was pinned to tier_boundaries.simple_medium,
so an operator could not restore the unconditional promotion nor raise the bar
independently of the SIMPLE/MEDIUM cut. Setting reasoning_override_min_score
was accepted and echoed back by /model/info, because the config model allows
extra keys, while routing ignored it.
Resolve the floor through one accessor that falls back to simple_medium when
the field is unset, so moving that boundary still moves the floor with it, and
an explicit 0 is a real floor rather than an absent one. Record the resolved
value on the routing decision so a logged row states the floor that applied,
which is also what lets the Admin UI stop hardcoding the copy PR #37500 added.
Two or more reasoning keyword matches promoted a request straight to the
REASONING tier no matter what the weighted score said, so "hi, step by step,
pros and cons" scored 0.100 and still bought the most expensive tier.
Require the score to clear the simple_medium boundary before the override
applies. Promotion from MEDIUM or COMPLEX is unchanged; only prompts the
scorer already placed in the cheapest band stay there.
* feat(complexity_router): custom classifier plugins via classifier_type 'plugin'
Adds a third classification mode where an operator-supplied hook decides the
tier instead of the heuristic scorer or the LLM classifier. The hook implements
an async classify(context) returning a tier name (built-in value, tier_labels
label, or tier_definitions name) or None to decline; failures, timeouts, and
unknown tiers fall back exactly like a failed LLM classifier. The context
carries the request messages and metadata, including caller identity, so a
plugin can route by team, spend, or any business rule.
The plugin resolves from a dotted path at proxy startup with a load-time check
that classify is a coroutine function, and is closed off over HTTP like the
routing plugins list. Routing decisions record the new classifier_plugin cause.
tier_definitions now accepts classifier_type 'plugin' alongside 'llm'.
* fix(proxy): resolve plugin dotted paths in _delete_deployment before hashing ids
The db-sync reconcile re-reads the raw config and hashes litellm_params to
compute which ids the config wants served, but the router's ids were hashed
from the resolved params where plugin dotted paths are live instances. The
mismatched ids made the reconcile evict every plugin-bearing auto-router one
sync after startup, on any proxy with a database connected. This also affected
the existing routing plugins list, not just the new classifier plugin.
Resolving the plugins in _delete_deployment the same way load_config does makes
both sides hash the same canonical form. A plugin module broken on disk at
reconcile time skips cleanup instead of evicting valid deployments, matching
how a get_config failure is handled
* fix(complexity_router): treat non-string plugin verdicts as declines, centralize the empty-mapping sentinel
A hook returning a non-string raised inside resolve_classified_tier outside the
plugin exception boundary, failing the request instead of falling back. Also
moves the read-only empty mapping to constants.py per repo convention and moves
the classifier plugin product docs out of the package README for the docs repo
* refactor(complexity_router): rename the plugin classifier mode to classifier_type 'custom'
The mode value now names the operator's intent while classifier_plugin keeps
naming the mechanism; routing decisions keep the classifier_plugin cause
* refactor(proxy): pin plugin-bearing deployment ids from the raw params instead of resolving in the reconcile
Replaces the previous approach of re-running plugin resolution inside
_delete_deployment, which imported operator modules on every reconcile cycle
and skipped the whole cleanup pass when any one module was broken on disk.
load_config now stamps model_info.id from the raw litellm_params before
resolution swaps dotted paths for live instances, so the reconcile's raw-config
hash matches by construction and needs no resolution at all: a broken module
cannot stall cleanup for unrelated models, and any future param-transforming
resolution is covered by the same pin. _generate_model_id becomes a staticmethod
so the pin can run before the Router exists; its statically dead non-string key
branches are removed. Also documents candidate_models as an informational
snapshot for classifier plugins, unlike the narrowing surface RoutingPlugin
filters
* fix(router): restore _generate_model_id key handling, align classifier context with the routing-plugin pattern
The staticmethod conversion accidentally dropped the non-string-key branches
from _generate_model_id, a silent hash change for any params with non-string
keys; they are restored verbatim. The classifier plugin context now follows
the Router-level routing-plugin recipe exactly: structured messages come from
resolve_structured_messages over the raw messages, and the metadata key comes
from the shared get_metadata_variable_name_from_kwargs helper, which also
replaces the duplicated inline sniff in _pick_model_for_tier. This removes the
raw-or-resolved fallback where a plugin could silently receive resolved
messages when a call site forgot to pass the raw ones
* refactor(router): make generate_model_id public, guard classifier context construction
Two modules legitimately hash deployment ids with the same helper now (Router
and the proxy's config-load pin), so the private name was lying about its
audience and the cross-module call needed a pyright suppression; renaming it
public restores the static safety net. The classifier plugin's RoutingContext
construction moves inside the failure boundary, matching the LLM path where
litellm-side prompt building also falls back rather than failing the request,
and a prompt-only call with no message list is now covered by a test
Keeps an explicit empty messages list on the pre-existing path (default
model, provider validation error) instead of dropping the routing
decision and surfacing a misleading tags 401.
Auto-router strategy hook returned None whenever the request carried
input instead of messages, so tagged /v1/responses requests (Codex CLI)
never picked a tier and tag filtering left nothing to route to. Resolve
input through the shared prompt-template helper before matching routes.
* feat(complexity_router): plan-mode tier floor for coding-agent clients
Claude Code and Copilot signal plan mode only through client-injected prompt
text, which the ask-extraction path deliberately strips, so the router could
never see it. Detect the sentinels on the raw wire body and route those
requests to at least plan_mode_min_tier.
The floor is raise-only and transient: classifier results above it still win,
it overrides a session-affinity pin only on turns carrying the sentinel
without rewriting the pin, and plan_mode decisions are not pinnable, so the
first turn after plan mode exits routes as if plan mode had never happened.
Classification is skipped when the floor is the top configured tier. On
adaptive routers the floor rides _soft_floor_pick as a hard_floor that
excludes below-floor candidates, closing the adaptive_eligible=all gap where
a request classified at or above the floor could still route below it.
Detection is staleness-aware: only leading system content and the newest-ask
tail count, so sentinels surviving in history after plan mode exits, built-in
or operator-supplied, never fire. Custom tier sets are supported with
severity from the tier_definitions list order, same as keyword_tier_rules.
Off by default; decisions are recorded with the new plan_mode cause and the
matched sentinel in matched_keyword
* fix(complexity_router): gate pin writes and the failure exit on sentinel presence, not the floor binding
A plan-mode turn classified at or above the floor keeps its ordinary cause,
but pinning it would carry a plan-mode-shaped choice past plan mode's exit
(on adaptive routers the hard floor constrained that pick), so no
sentinel-carrying turn writes the session pin. The default_model failure
exit is skipped for sentinel turns for the same reason: default_model's
placeholder tier can equal the floor while default_model itself sits in no
pool the floor can vouch for
* feat(complexity_router): calibrate the classifier rubric with worked examples
The built-in rubric stated its tier boundaries as prose alone, and prose
calibrated to consumer chat puts "non-trivial code, multi-step technical work"
at the top of the scale. That is the median request in developer and agent
traffic, so ordinary engineering read as top-tier and the router paid for the
most expensive model on it.
Adds calibration examples to the rubric, selected by a new
classifier_llm_config.rubric preset. The agentic preset (now the default)
anchors routine installs, builds, multi-file edits, and standard debugging at
MEDIUM; the chat preset omits those anchors for deployments serving only
conversational traffic. Both share the same tier criteria, the trust-boundary
paragraph, and the context-window closing line, so this moves where the
boundary sits without changing the taxonomy.
Both presets render byte-identical to the strings a prompt sweep scored, and a
test pins that, so the measured accuracy describes what a router sends.
* feat(ui): pick the classifier rubric preset on an auto-router
Adds a Rubric dropdown to the auto-router's classification panel, so the
agentic and chat presets are selectable rather than config-file only. The
prompt editor prefills from the selected preset, since prefilling agentic text
for a router on chat would show examples its classifier never receives.
The picker is disabled while a custom prompt is set, and the payload builder
drops the preset in that case: a custom prompt is the classifier's whole system
role, so the backend rejects the two together. The builder records the default
preset explicitly, so a later change to which preset is default cannot silently
move an existing router.
* fix(complexity_router): mark an unchosen rubric preset with None, not model_fields_set
The mutual-exclusion check read model_fields_set to tell an explicit preset
from the default. That flag does not survive serialization, and this config is
dumped and handed straight back to ComplexityRouter by /auto_router/test_routing,
where a dump re-states every field. So a custom-prompt classifier saved fine and
then failed validation on preview, rejecting on the second pass what it accepted
on the first.
The preset is now optional, with None meaning the default, matching how None
already means the built-in rubric for system_prompt on the same model. The
default lives in one place, DEFAULT_RUBRIC_PRESET, resolved where the prompt is
assembled. The dashboard stops sending a copy of the default it displays, so a
router nobody configured follows the default rather than pinning today's value,
and UI-built routers behave the same as hand-written config.
Regenerates schema.d.ts, which was left stale by an earlier description edit.
* feat(complexity_router): grandfather existing routers onto the uncalibrated rubric
An unset preset now means LEGACY, the rubric exactly as it shipped before
calibration examples existed, so upgrading cannot move the tier decisions or the
bill of a router that is already running. Config-file routers get this for free
since they name no preset, and a stored config that never had one reads the same
way.
New routers still get the calibrated rubric: switching a classifier to LLM
stamps the agentic preset, because a classifier being configured for the first
time has no prior tier behaviour to preserve. The picker offers legacy so an
existing router's state is representable and opening the form cannot silently
upgrade it.
Each preset is pinned byte-identical to the text the prompt sweep scored,
legacy included, which is what proves an existing router's prompt did not move.
Also collapses the preset data from a NamedTuple with group wrappers and
per-preset frozensets into plain text blocks in a MappingProxyType, matching how
the tier criteria next to it are already stored: 21 lines of prompt text no
longer cost 190 lines of constructors. Tiers are format placeholders so
tier_labels still reach the examples.
* refactor(complexity_router): name the field classification_rubric
`rubric` alone did not say what it selects, and the field sits beside
`system_prompt`, which genuinely is the whole classification prompt. The name
now says which of the two an operator is reaching for: the rubric the built-in
prompt is assembled from, not the prompt itself.
Renames the config field, the query param, the enum, and the dashboard label to
match, and moves the preset text to classification_rubrics.py.
* test(ui): set the preset the mutual-exclusion case is meant to drop
The rename left classification_classification_rubric in the custom-prompt case,
so its input never carried a preset and the assertion held for the wrong reason:
it proved an absent preset stays absent, not that a set one is dropped. A
normalizer that forwards the preset whenever one is set passed with the typo and
fails without it.
tsc reports the typo as TS2353; the earlier sweep grepped for the source file
and not the test, so it went unseen.
* test(ui): scope the role-gate assertions to each page's own endpoint
The memory, workflows, and guardrails-monitor page tests asserted that a denied
role fires no request at all. Their names, and the assertion on the very next
line, say the intent is narrower: the page must not fetch its own data.
Resolving whether a caller is an org admin goes through /organization/list for
every role, since deciding org-admin-for-any-org needs the list, and the route
scopes rows per caller. That legitimate request fails a blanket no-fetch
assertion, so all three files went red on staging for a reason unrelated to
what they test.
Drops the blanket assertion and keeps the scoped one. Bypassing the gate in
memory/page.tsx still fails five tests, so the narrower assertion continues to
catch a genuinely broken gate.
* fix(complexity_router): document that an unset rubric keeps the legacy prompt
The field said 'Leave unset for agentic' while an omitted rubric resolves to
LEGACY, so the OpenAPI schema an operator reads promised calibrated routing
where they got the uncalibrated one.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
System prompts (harnesses, tools, framework boilerplate) are session-wide
constants identical across all requests. Scoring them saturates keyword-match
signals and produces false-positive high-complexity classifications on
trivial utterances like 'hi', routing them to expensive models (sonnet/opus)
instead of tier-1 haiku. A real ~1.6KB CLI-agent harness alone supplied
5 codePresence + 2 technicalTerms matches, overshadowing user signal.
Rescope four scoring dimensions (codePresence, technicalTerms, simpleIndicators,
multiStepPatterns) from full_text (system + user) to user_text (user only).
reasoningMarkers was already scoped this way. This returns 0.63 of the weight
budget to text that actually varies per-request.
Now that every dimension scores user_text only, _score_keyword_match's
disclosable_text param is redundant -- it existed solely to let the signal
name terms matched in the caller's own message while withholding terms
matched only in the (invisible-to-the-caller) system prompt. With no more
system-prompt text in scope, text and disclosable_text were identical at
every call site, so the param is dropped and the function collapses to a
single text argument.
Add mutation-proven regression test: trivial 'hi' message with realistic
Claude Code agent system prompt now routes to haiku tier-1 (not sonnet).
- Unfixed: haiku -> sonnet (bug)
- Fixed: haiku -> haiku (correct)
Invert three pre-existing assertions in TestSignalsNeverQuoteTheSystemPrompt
to capture the corrected behavior: system-prompt-only terms produce no signal.
Co-authored-by: Claude <noreply@anthropic.com>
Pre-routing now reads the request's tags on every request with a registered
strategy, including the single-strategy case that used to short-circuit before
looking at tags. Metadata is request-controlled, so a caller that sends
`litellm_metadata` (or `tags`) as a string or any other non-dict shape crashed
tag lookup with an AttributeError instead of routing untagged.
* fix: never price a strategy-router alias
A strategy-router alias (auto_router/complexity_router/<name>) is never the
deployment that gets called or billed, but custom pricing configured on it was
being treated as real pricing in two places:
- registered in litellm.model_cost under the alias deployment id, so an
explicit zero made _is_cost_explicitly_configured() report the group as a
genuinely free model and every budget check was skipped, while the request
routed to a paid deployment and accrued real spend
- copied onto request_kwargs by the alias-params merge, so the routed
deployment got re-registered at the alias price and the request billed 0.0
Both are fixed at the writer, so config, /model/new and price-map reload all
take the same path
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: annotate filtered cost-map copy for the mutable-collection gate
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
/v1/messages and other litellm_metadata endpoints store proxy metadata,
including x-litellm-tags header tags, under litellm_metadata instead of
metadata. The pre-routing hook read request tags with a hardcoded
metadata bucket, so it never saw the tags that selected the marker and
cleared the consumed-tags stamp, and tag filtering then 401'd the routed
tier. Resolve the bucket from the request kwargs instead, matching how
the stamp write and the tag-filter read already resolve it.
