scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-09 22:31:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 133

feat(complexity_router): rebalance heuristic weights in the dashboard and grade custom dimensions by match count (#40205)

Browse source 
The Advanced scoring editor now lists built-in and custom dimensions together. Editing any weight holds it and rescales the others proportionally so the vector totals 1.00, and Save stores those explicit values. The backend scores exactly what is stored, with no runtime normalization, so routers nobody edits keep their weights.

CustomDimension gains an opt-in scoring_mode. match_count scores 0, 0.5 or 1 by distinct matcher hits; the default stays binary. The tuning fingerprint omits a binary scoring_mode, so routers written before this change keep their recorded baseline and the upgrade does not consume the free heuristic-v1 tuning slot.
This commit is contained in:
tin-berri 2026-09-08 13:28:57 -07:00 • committed by GitHub
parent 6a425a5cc5
commit 4a3a78c256
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
26 changed files with 1047 additions and 86 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
litellm/router_strategy/complexity_router/README.md
View file

@ -207,17 +207,27 @@ custom_dimensions:
- name: sqlMigration
weight: 0.7
patterns: ['\b(create|alter|drop)\s{1,4}table\b']
- name: dataPipeline
weight: 0.4
scoring_mode: match_count
keywords: [airflow, dbt, snowflake]
```
Each dimension contributes its weight once when any matcher hits the current ask. Repeated matches do not increase it. The built-in score and tier boundaries are unchanged, and the total score is not renormalized. Keywords use the existing case-insensitive word-boundary and CJK rules. Regexes search the first 2048 characters case-insensitively and compile during configuration validation and router initialization, never per request
`scoring_mode` is optional and defaults to `binary`, the behavior above. `match_count` grades the dimension by how many distinct matchers hit: none scores 0 and emits no signal, one scores half the weight, two or more score the full weight. Repeated occurrences of one matcher never raise the count, keywords are distinct case-insensitively, patterns are distinct by source, and a keyword and a pattern are always distinct from each other. Matching stops as soon as the selected mode's maximum is reached, so a binary dimension still stops at its first hit. Existing configurations without the field keep binary scoring and the same tuning fingerprint, so the field only counts as a tuning change when set to `match_count`
### Weights through the API versus the dashboard
The API and YAML store exactly the weights written. A `dimension_weights` map and inline custom weights are read literally, missing recognized built-in names score zero, and nothing renormalizes the vector, so a total other than 1 is legal and scores accordingly. The dashboard's heuristic scoring editor is the one place that rebalances: editing one weight there holds it and redistributes the remainder across the other active dimensions in the draft, then Save sends the resulting explicit values, which the backend stores and scores as written. Opening a router, applying a preset, editing matchers, changing `scoring_mode`, or saving unrelated fields never normalizes existing weights
Only `heuristic`, `heuristic_first` and `hybrid` accept custom dimensions. Each name must be a unique ASCII identifier starting with a letter, at most 64 characters, and cannot reuse a built-in dimension name or a key in `dimension_weights`. Set its weight inline, greater than zero and at most one
Patterns are checked at configuration time against a grammar whose worst case stays a few milliseconds on 2048 characters. Every quantifier needs an explicit upper bound of at most 64 and must repeat a single character or character class, so `\s{1,4}` is accepted while `\s+`, `(a|aa){0,12}` and `(?:ab){0,64}` are refused. Backreferences, lookarounds, atomic groups and possessive quantifiers are refused as well. Each pattern is then costed: alternation branches and repeat lengths multiply the ways the engine can retry, and every later piece of the pattern is charged once per path that can reach it, so `a?a?a?a?a?a?a?a?` followed by a long fixed tail is refused even though each quantifier is small. The budget is 2048 work units per pattern and 8192 across the router. An invalid or over-budget pattern fails the write with a message naming the pattern and the rule it broke
Limits are 16 dimensions, 32 combined keywords/patterns per dimension, 256 characters per matcher and 4096 matcher characters per dimension. Matching runs inline on the request path with no timeout and no worker thread, because the grammar is what bounds the cost. These are routing hints, not security enforcement rules
The existing heuristic-v1 tuning quota covers custom dimensions and their weights: one changed router without an auto-router license, unlimited with the entitlement. Omitting `custom_dimensions` preserves existing scoring. Routing decisions and spend logs include signals such as `custom (sqlMigration)` without recording the configured pattern or matched text. The field is configured through YAML or the model API; this change adds no dashboard editor
The existing heuristic-v1 tuning quota covers custom dimensions, their weights and their scoring mode: one changed router without an auto-router license, unlimited with the entitlement. Omitting `custom_dimensions` preserves existing scoring. Routing decisions and spend logs include signals such as `custom (sqlMigration)` without recording the configured pattern or matched text. The field is configured through YAML, the model API, or the dashboard's heuristic scoring editor
## Usage

38
litellm/router_strategy/complexity_router/complexity_router.py
View file

@ -20,7 +20,7 @@ import random
import re
import time
from collections.abc import Callable, Iterator, Mapping, Sequence
from itertools import accumulate, islice, takewhile
from itertools import accumulate, chain, islice, takewhile
from threading import Lock
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, Literal, NamedTuple, cast
@ -82,6 +82,7 @@ from .config import (
ClassificationRubric,
ComplexityRouterConfig,
ComplexityTier,
CustomDimension,
TierDefinition,
)
from .stall_detector import detect_stalled_task
@ -879,6 +880,15 @@ class DimensionScore:
self.signal = signal
class _CustomDimensionMatchers(NamedTuple):
"""One custom dimension's distinct matchers and the number of hits that saturates its score."""
dimension: CustomDimension
keywords: tuple[str, ...]
patterns: tuple[re.Pattern[str], ...]
saturation: int
class KeywordOverride(NamedTuple):
"""A keyword_tier_rules match: the winning tier and, on the lexical path, the keyword that fired."""
@ -1121,7 +1131,12 @@ class ComplexityRouter(CustomLogger):
)
self.simple_keywords = self.config.simple_keywords or DEFAULT_SIMPLE_KEYWORDS
self._custom_dimensions = tuple(
(dimension, tuple(re.compile(pattern, re.IGNORECASE) for pattern in dimension.patterns))
_CustomDimensionMatchers(
dimension,
tuple(dict.fromkeys(keyword.lower() for keyword in dimension.keywords)),
tuple(re.compile(pattern, re.IGNORECASE) for pattern in dict.fromkeys(dimension.patterns)),
2 if dimension.scoring_mode == "match_count" else 1,
)
for dimension in self.config.custom_dimensions
)
if self.config.has_custom_tiers:
@ -1325,15 +1340,26 @@ class ComplexityRouter(CustomLogger):
score: Final = score_high if match_count >= high_threshold else score_low
return DimensionScore(name, score, f"{signal_label} ({detail})"), match_count
def _count_custom_hits(self, matchers: _CustomDimensionMatchers, user_text: str, scanned: str) -> int:
hits: Final = chain(
(self._keyword_matches(user_text, keyword) for keyword in matchers.keywords),
(pattern.search(scanned) is not None for pattern in matchers.patterns),
)
return sum(islice((1 for hit in hits if hit), matchers.saturation))
def _score_custom_dimensions(self, prompt: str, user_text: str) -> tuple[tuple[DimensionScore, float], ...]:
if not self._custom_dimensions:
return ()
scanned: Final = prompt[:CUSTOM_PATTERN_SCAN_CHARS]
return tuple(
(DimensionScore(dimension.name, 1.0, f"custom ({dimension.name})"), dimension.weight)
for dimension, patterns in self._custom_dimensions
if any(self._keyword_matches(user_text, keyword) for keyword in dimension.keywords)
or any(pattern.search(scanned) is not None for pattern in patterns)
(
DimensionScore(
matchers.dimension.name, hits / matchers.saturation, f"custom ({matchers.dimension.name})"
),
matchers.dimension.weight,
)
for matchers in self._custom_dimensions
if (hits := self._count_custom_hits(matchers, user_text, scanned))
)
def _score_multi_step(self, text: str) -> DimensionScore:

13
litellm/router_strategy/complexity_router/config.py
View file

@ -667,6 +667,14 @@ class CustomDimension(BaseModel):
weight: float = Field(gt=0, le=1, allow_inf_nan=False)
keywords: tuple[Annotated[str, Field(min_length=1, max_length=256)], ...] = Field(default=(), max_length=32)
patterns: tuple[Annotated[str, Field(min_length=1, max_length=256)], ...] = Field(default=(), max_length=32)
scoring_mode: Literal["binary", "match_count"] = Field(
default="binary",
description=(
"'binary' scores 1 when any matcher hits. 'match_count' scores 0.5 when one distinct matcher hits and 1 "
"when two or more do; repeated occurrences of one matcher never raise it. Keywords are distinct "
"case-insensitively, patterns by source, and a keyword and a pattern are always distinct from each other."
),
)
@model_validator(mode="after")
def _validate_matchers(self) -> "CustomDimension":
@ -794,8 +802,9 @@ class ComplexityRouterConfig(BaseModel):
default=(),
max_length=16,
description=(
"Named binary dimensions added to the heuristic-v1 score. Each contributes its inline weight once "
"when any keyword matches the current ask or a case-insensitive regex matches its first 2048 characters. "
"Named dimensions added to the heuristic-v1 score. Each contributes its inline weight once "
"when any keyword matches the current ask or a case-insensitive regex matches its first 2048 characters; "
"scoring_mode 'match_count' instead grades half weight for one distinct matcher and full for two or more. "
"Regex quantifiers repeat one character or class at most 64 times. Unbounded quantifiers, repeated groups, "
"backreferences and lookarounds are rejected. Conservative work limits include alternation paths, "
"repeat lengths and subsequent matching: 2048 units per pattern, 8192 across the router. "

12
litellm/router_utils/auto_router_tuning_baseline.py
View file

@ -52,7 +52,17 @@ def tuning_fingerprint(complexity_router_config: object) -> str | None:
supplied: Final = ((_TUNING_FIELD_SET - frozenset(("tier_model_configs",))) & frozenset(raw)) | (
frozenset(("tier_model_configs",)) if validated.tier_model_configs else frozenset()
)
payload: Final = validated.model_dump(mode="json", include=supplied)
payload: Final = validated.model_dump(
mode="json",
include=supplied,
exclude={
"custom_dimensions": {
index: {"scoring_mode"}
for index, dimension in enumerate(validated.custom_dimensions)
if dimension.scoring_mode == "binary"
}
},
)
return hashlib.sha256(json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()).hexdigest()

148
tests/test_litellm/router_strategy/test_complexity_router.py
View file

@ -826,6 +826,8 @@ class TestCustomDimensions:
pytest.param({"keywords": ["x"] * 32, "patterns": ["y"]}, {}, id="combined-matcher-count"),
pytest.param({"keywords": ["x" * 256] * 17}, {}, id="matcher-character-budget"),
pytest.param({"unknown": True}, {}, id="extra-field"),
pytest.param({"scoring_mode": "graded"}, {}, id="unknown-scoring-mode"),
pytest.param({"scoring_mode": None}, {}, id="null-scoring-mode"),
],
)
def test_custom_dimension_invalid_configuration_rejected(
@ -878,43 +880,125 @@ class TestCustomDimensions:
)
@pytest.mark.asyncio
@pytest.mark.parametrize("current_ask", ("Hello!", "orbitmesh"))
@pytest.mark.parametrize("scoring_mode", ("binary", "match_count"))
@pytest.mark.parametrize("current_ask", ("Hello!", "orbitmesh", "orbitmesh fluxgate"))
async def test_custom_dimensions_public_hook_scores_only_current_ask(
self, mock_router_instance: MagicMock, current_ask: str
self, mock_router_instance: MagicMock, current_ask: str, scoring_mode: str
) -> None:
router: Final = ComplexityRouter(
"test-router",
mock_router_instance,
{
"tiers": {"SIMPLE": "cheap", "MEDIUM": "mid", "COMPLEX": "strong", "REASONING": "top"},
"custom_dimensions": [{"name": "internalFrameworks", "weight": 0.7, "keywords": ["orbitmesh"]}],
"dimension_weights": {},
"custom_dimensions": [
{
"name": "internalFrameworks",
"weight": 0.8,
"keywords": ["orbitmesh", "fluxgate"],
"scoring_mode": scoring_mode,
}
],
},
)
result: Final = await router.async_pre_routing_hook(
model="test-router",
request_kwargs={},
messages=[
{"role": "system", "content": "orbitmesh"},
{"role": "user", "content": "orbitmesh"},
{"role": "assistant", "content": "orbitmesh is ready"},
{"role": "system", "content": "orbitmesh fluxgate"},
{"role": "user", "content": "orbitmesh fluxgate"},
{"role": "assistant", "content": "orbitmesh fluxgate is ready"},
{"role": "user", "content": current_ask},
{"role": "tool", "tool_call_id": "previous", "content": "orbitmesh"},
{"role": "tool", "tool_call_id": "previous", "content": "orbitmesh fluxgate"},
],
)
assert result is not None
assert result.routing_decision is not None
assert ("custom (internalFrameworks)" in result.routing_decision["signals"]) is (current_ask == "orbitmesh")
assert result.model == ("top" if current_ask == "orbitmesh" else "cheap")
expected_score: Final = (
0.0
if current_ask == "Hello!"
else 0.4
if scoring_mode == "match_count" and current_ask == "orbitmesh"
else 0.8
)
assert result.routing_decision["score"] == expected_score
assert ("custom (internalFrameworks)" in result.routing_decision["signals"]) is (expected_score > 0)
assert result.model == ("cheap" if expected_score == 0 else "strong" if expected_score == 0.4 else "top")
assert "orbitmesh" not in " ".join(result.routing_decision["signals"])
def test_custom_patterns_scan_only_the_first_2048_characters(self, mock_router_instance: MagicMock) -> None:
@pytest.mark.parametrize("scoring_mode", ("binary", "match_count"))
def test_custom_patterns_scan_only_the_first_2048_characters(
self, mock_router_instance: MagicMock, scoring_mode: str
) -> None:
router: Final = ComplexityRouter(
"test-router",
mock_router_instance,
{"custom_dimensions": [{"name": "late", "weight": 0.7, "patterns": [r"zzz{1,3}"]}]},
{
"custom_dimensions": [
{
"name": "late",
"weight": 0.7,
"patterns": [r"zzz{1,3}", r"yyy{1,3}"],
"scoring_mode": scoring_mode,
}
]
},
)
baseline: Final = ComplexityRouter("test-router", mock_router_instance)
assert "custom (late)" in router.classify("a" * 2040 + " zzz")[2]
assert "custom (late)" not in router.classify("a" * 2048 + " zzz")[2]
second_hit_past_the_bound: Final = "yyy " + "a" * 2044 + " zzz"
contribution: Final = (
router.classify(second_hit_past_the_bound)[1] - baseline.classify(second_hit_past_the_bound)[1]
)
assert contribution == pytest.approx(0.7 if scoring_mode == "binary" else 0.35)
@pytest.mark.parametrize(
"prompt,expected_score",
[
pytest.param("Hello!", 0.0, id="no-hit"),
pytest.param("orbitmesh orbitmesh ORBITMESH again", 0.5, id="one-keyword-repeated"),
pytest.param("create table a; CREATE TABLE b; create table c", 0.5, id="one-pattern-repeated"),
pytest.param("orbitmesh and fluxgate", 1.0, id="two-keywords"),
pytest.param("orbitmesh then create table t", 1.0, id="keyword-plus-pattern"),
pytest.param("create table a; alter table b", 1.0, id="two-patterns"),
pytest.param("orbitmesh fluxgate create table a alter table b", 1.0, id="all-matchers"),
],
)
def test_match_count_grades_distinct_matchers(
self, mock_router_instance: MagicMock, prompt: str, expected_score: float
) -> None:
dimension: Final = {
"name": "graded",
"weight": 0.6,
"keywords": ["orbitmesh", "ORBITMESH", "fluxgate"],
"patterns": [r"\bcreate\s{1,4}table\b", r"\bcreate\s{1,4}table\b", r"\balter\s{1,4}table\b"],
}
baseline: Final = ComplexityRouter("test-router", mock_router_instance)
binary: Final = ComplexityRouter("test-router", mock_router_instance, {"custom_dimensions": [dimension]})
graded: Final = ComplexityRouter(
"test-router",
mock_router_instance,
{"custom_dimensions": [{**dimension, "scoring_mode": "match_count"}]},
)
_, baseline_score, baseline_signals = baseline.classify(prompt)
_, binary_score, binary_signals = binary.classify(prompt)
_, graded_score, graded_signals = graded.classify(prompt)
assert graded_score == pytest.approx(baseline_score + 0.6 * expected_score)
assert binary_score == pytest.approx(baseline_score + (0.6 if expected_score else 0.0))
expected_signals: Final = [*baseline_signals, *(["custom (graded)"] if expected_score else [])]
assert graded_signals == expected_signals
assert binary_signals == expected_signals
def test_scoring_mode_round_trips_and_defaults_to_binary(self) -> None:
dimension: Final = {"name": "graded", "weight": 0.6, "keywords": ["orbitmesh"]}
legacy: Final = ComplexityRouterConfig.model_validate({"custom_dimensions": [dimension]})
graded: Final = ComplexityRouterConfig.model_validate(
{"custom_dimensions": [{**dimension, "scoring_mode": "match_count"}]}
)
assert legacy.custom_dimensions[0].scoring_mode == "binary"
assert graded.model_dump(mode="json")["custom_dimensions"][0]["scoring_mode"] == "match_count"
assert ComplexityRouterConfig.model_validate(graded.model_dump(mode="json")) == graded
def test_custom_dimensions_router_wide_regex_work_is_capped(self) -> None:
heavy: Final = {"weight": 0.5, "patterns": ["a?" * 8 + "z"]}
@ -1511,6 +1595,7 @@ class TestRouterComplexityDeploymentMethods:
def test_the_shipped_rubric_and_default_prompt_stay_free(self) -> None:
"""Only an operator-written prompt is gated: picking a shipped rubric preset, or writing no
prompt at all, leaves a router unmetered, so several of them register under a ceiling of one."""
def rubric(model_name: str, model_id: str, preset: str | None) -> dict[str, object]:
llm_config: dict[str, object] = {"model": "gpt-4o-mini"}
if preset is not None:
@ -1647,6 +1732,7 @@ class TestRouterComplexityDeploymentMethods:
def test_renaming_built_in_tiers_is_not_a_custom_tier_set(self) -> None:
"""tier_labels renames the built-in ladder without defining one, so it stays ungated: two such
routers register under a ceiling of one."""
def labeled(model_name: str, model_id: str) -> dict[str, object]:
row = self._router_row(model_name, model_id, "heuristic")
row["litellm_params"]["complexity_router_config"]["tier_labels"] = {"SIMPLE": "Cheap", "MEDIUM": "Standard"}
@ -2520,9 +2606,7 @@ class TestLLMClassifier:
assert outcome.classifier_cost == pytest.approx(1.35e-05)
@pytest.mark.asyncio
async def test_aclassify_timeout_does_not_inherit_router_retries_or_fallbacks(
self, llm_classifier_config
):
async def test_aclassify_timeout_does_not_inherit_router_retries_or_fallbacks(self, llm_classifier_config):
real_router = Router(
model_list=[
{
@ -2565,9 +2649,7 @@ class TestLLMClassifier:
assert real_router.total_calls["openai/mock-backup-classifier"] == 0
@pytest.mark.asyncio
async def test_aclassify_enforces_total_classifier_deadline(
self, mock_router_instance, llm_classifier_config
):
async def test_aclassify_enforces_total_classifier_deadline(self, mock_router_instance, llm_classifier_config):
cancelled = asyncio.Event()
async def slow_classifier(**_kwargs: object) -> None:
@ -12414,9 +12496,7 @@ class TestTierHealthFailover:
llm_provider="",
)
filtered = (*cooling, *blocked, *excluded)
healthy = [
{"model_name": model, "model_info": {"id": i}} for i in ids_by_model[model] if i not in filtered
]
healthy = [{"model_name": model, "model_info": {"id": i}} for i in ids_by_model[model] if i not in filtered]
if not healthy:
raise RouterRateLimitError(
model=model, cooldown_time=60.0, enable_pre_call_checks=False, cooldown_list=[]
@ -12845,9 +12925,7 @@ class TestTierHealthFailover:
assert all(probed is not request_kwargs for probed in router.litellm_router_instance.probed_kwargs)
@pytest.mark.asyncio
async def test_a_peer_whose_every_deployment_is_over_its_rpm_is_not_a_failover_target(
self, mock_router_instance
):
async def test_a_peer_whose_every_deployment_is_over_its_rpm_is_not_a_failover_target(self, mock_router_instance):
"""RPM exhaustion is its own verdict from the owner (RouterRateLimitErrorBasic). A peer
in that state would be rejected downstream, so it cannot be the substitute."""
from litellm.types.router import RouterRateLimitErrorBasic
@ -12880,9 +12958,7 @@ class TestTierHealthFailover:
assert {r.model for r in results} == {"live-c"}
@pytest.mark.asyncio
async def test_the_probe_forwards_input_so_window_checks_run_on_input_only_surfaces(
self, mock_router_instance
):
async def test_the_probe_forwards_input_so_window_checks_run_on_input_only_surfaces(self, mock_router_instance):
"""The Responses API carries its prompt as `input`, never as messages. The owner only
runs its context-window pre-call check when one of them is present, so dropping `input`
would silently skip window filtering on that whole surface."""
@ -12908,9 +12984,7 @@ class TestTierHealthFailover:
), "the eligibility probe must forward `input` to the owner"
@pytest.mark.asyncio
async def test_a_group_the_router_has_no_deployment_for_is_not_a_failover_target(
self, mock_router_instance
):
async def test_a_group_the_router_has_no_deployment_for_is_not_a_failover_target(self, mock_router_instance):
"""The owner answers an unconfigured group with BadRequestError. Reading that as live
would both skip failover off it and let it be chosen as a substitute."""
router = self._router(
@ -13099,9 +13173,7 @@ class TestClassifierVision:
routed as default_fallback on text the request never contained.
"""
router = self._router(mock_router_instance, vision={"enabled": True})
response = await router.async_pre_routing_hook(
model="m", request_kwargs={}, messages=self._turn(IMG_PART)
)
response = await router.async_pre_routing_hook(model="m", request_kwargs={}, messages=self._turn(IMG_PART))
assert response.routing_decision["cause"] == "llm_classifier"
assert response.model == "t-complex"
assert [block["type"] for block in self._classifier_user_content(mock_router_instance)] == [
@ -13112,9 +13184,7 @@ class TestClassifierVision:
@pytest.mark.asyncio
async def test_image_only_turn_still_falls_back_when_vision_is_off(self, mock_router_instance):
router = self._router(mock_router_instance, vision={"enabled": False})
response = await router.async_pre_routing_hook(
model="m", request_kwargs={}, messages=self._turn(IMG_PART)
)
response = await router.async_pre_routing_hook(model="m", request_kwargs={}, messages=self._turn(IMG_PART))
assert response.routing_decision["cause"] == "default_fallback"
mock_router_instance.acompletion.assert_not_awaited()
@ -13184,9 +13254,7 @@ class TestClassifierVision:
makes the image the only variable; a margin loose enough to leave the score undecided
would pass whether or not the guard exists.
"""
router = self._router(
mock_router_instance, vision={"enabled": True}, classifier_type=classifier_type, **extra
)
router = self._router(mock_router_instance, vision={"enabled": True}, classifier_type=classifier_type, **extra)
response = await router.async_pre_routing_hook(
model="m", request_kwargs={}, messages=self._turn({"type": "text", "text": "what is this"}, IMG_PART)
)
@ -13200,9 +13268,7 @@ class TestClassifierVision:
self, mock_router_instance, classifier_type, extra, short_circuit_cause
):
"""The negative class: same router, same text, no image, and the scorer still decides."""
router = self._router(
mock_router_instance, vision={"enabled": True}, classifier_type=classifier_type, **extra
)
router = self._router(mock_router_instance, vision={"enabled": True}, classifier_type=classifier_type, **extra)
response = await router.async_pre_routing_hook(
model="m", request_kwargs={}, messages=[{"role": "user", "content": "what is this"}]
)

73
tests/test_litellm/router_utils/test_auto_router_tuning_baseline.py
View file

@ -7,6 +7,7 @@ from typing import Final
import pytest
from litellm.router_strategy.complexity_router.config import ComplexityRouterConfig
from litellm.router_utils.auto_router_tuning_baseline import (
DEFAULT_TUNING_FINGERPRINT,
HEURISTIC_V1_TUNING_FIELDS,
@ -21,6 +22,33 @@ from litellm.router_utils.auto_router_tuning_baseline import (
_TIERS = {"SIMPLE": "cheap", "MEDIUM": "mid", "COMPLEX": "strong", "REASONING": "top"}
_ALT_TIERS = {**_TIERS, "COMPLEX": "other-strong"}
_KEYWORD_DIMENSION: Final = {"name": "internalFrameworks", "weight": 0.7, "keywords": ["orbitmesh"]}
_HISTORICAL_FINGERPRINTS: Final = (
({}, "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"),
(
{"custom_dimensions": [_KEYWORD_DIMENSION]},
"b5c3c3f3be6341a8a16148d68d9067e03f94ed01a0bbfcde7955763042744372",
),
(
{"custom_dimensions": [{"name": "sqlDdl", "weight": 0.4, "patterns": [r"\bCREATE\s{1,4}TABLE\b"]}]},
"814ce0017fc7f60a160b262f658d910e9bdf784e6139a4ba4f1e2657aa203950",
),
(
{
"tiers": _TIERS,
"dimension_weights": {"codePresence": 0.3},
"custom_dimensions": [
{
"name": "internalFrameworks",
"weight": 0.2,
"keywords": ["orbitmesh", "fluxgate"],
"patterns": [r"\bALTER\s{1,4}TABLE\b"],
}
],
},
"38970dc9224e265ab38c89674563d8d0537822591f9239b45251db6f5ca6cc39",
),
)
def _router(
@ -77,6 +105,27 @@ class TestTuningFingerprint:
def test_explicit_empty_tier_model_configs_follow_omission(self) -> None:
assert tuning_fingerprint({"tier_model_configs": {}}) == DEFAULT_TUNING_FINGERPRINT
@pytest.mark.parametrize(("config", "fingerprint"), _HISTORICAL_FINGERPRINTS)
def test_fingerprints_recorded_before_scoring_mode_existed_are_preserved(
self, config: Mapping[str, object], fingerprint: str
) -> None:
"""Literal hashes captured from the merged implementation at 9bc9104102, before CustomDimension.scoring_mode."""
assert tuning_fingerprint(config) == fingerprint
def test_binary_scoring_mode_hashes_like_its_absence(self) -> None:
historical: Final = tuning_fingerprint({"custom_dimensions": [_KEYWORD_DIMENSION]})
explicit: Final = tuning_fingerprint({"custom_dimensions": [{**_KEYWORD_DIMENSION, "scoring_mode": "binary"}]})
reserialized: Final = ComplexityRouterConfig.model_validate(
{"custom_dimensions": [_KEYWORD_DIMENSION]}
).model_dump(mode="json", include={"custom_dimensions"})
assert reserialized["custom_dimensions"][0]["scoring_mode"] == "binary"
assert reserialized["custom_dimensions"][0]["patterns"] == []
assert historical == explicit == tuning_fingerprint(reserialized)
assert (
tuning_fingerprint({"custom_dimensions": [{**_KEYWORD_DIMENSION, "scoring_mode": "match_count"}]})
!= historical
)
def test_tier_model_overrides_change_the_fingerprint(self) -> None:
plain = tuning_fingerprint({"tiers": {"SIMPLE": "x"}})
with_override = tuning_fingerprint(
@ -218,15 +267,18 @@ class TestQuota:
is None
)
def test_custom_dimension_add_edit_and_revert_share_one_quota_slot(self) -> None:
@pytest.mark.parametrize(
"edit",
[
pytest.param({"weight": 0.9}, id="weight"),
pytest.param({"scoring_mode": "match_count"}, id="scoring-mode"),
],
)
def test_custom_dimension_add_edit_and_revert_share_one_quota_slot(self, edit: Mapping[str, object]) -> None:
baselines: Final = snapshot_tuning_baselines(())
original: Final = _router("a", {})
config: Final = {
"custom_dimensions": [{"name": "internalFrameworks", "weight": 0.7, "keywords": ["orbitmesh"]}]
}
edited_config: Final = {
"custom_dimensions": [{"name": "internalFrameworks", "weight": 0.9, "keywords": ["orbitmesh"]}]
}
config: Final = {"custom_dimensions": [_KEYWORD_DIMENSION]}
edited_config: Final = {"custom_dimensions": [{**_KEYWORD_DIMENSION, **edit}]}
added: Final = _router("a", config)
edited: Final = _router("a", edited_config)
second: Final = _router("b", config)
@ -240,6 +292,13 @@ class TestQuota:
assert mutable_tuned_identities((original,), baselines) == frozenset()
assert tuning_quota_violation(candidate=second, others=(original,), baselines=baselines, limit=1) is None
def test_graded_dimension_recorded_at_snapshot_is_its_own_baseline(self) -> None:
graded: Final = _router("a", {"custom_dimensions": [{**_KEYWORD_DIMENSION, "scoring_mode": "match_count"}]})
baselines: Final = snapshot_tuning_baselines((graded,))
assert mutable_tuned_identities((graded,), baselines) == frozenset()
reverted_to_binary: Final = _router("a", {"custom_dimensions": [_KEYWORD_DIMENSION]})
assert mutable_tuned_identities((reverted_to_binary,), baselines) == {router_identity(graded)}
def test_violation_message_names_the_limit_and_remedy(self) -> None:
message = tuning_limit_violation(held=2, limit=1)
assert message is not None

5
ui/litellm-dashboard/src/components/add_model/ClassificationMethodConfig.tsx
View file

@ -44,8 +44,9 @@ import {
} from "./ComplexityRouterConfig";
const DEFAULT_SCORING_EXPLANATION =
"The router scores each request across 7 dimensions: token count, code presence, reasoning markers, technical " +
"terms, simple indicators, multi-step patterns, and question complexity. The weighted score determines the tier:";
"The router scores each request across 7 built-in dimensions: token count, code presence, reasoning markers, technical " +
"terms, simple indicators, multi-step patterns, and question complexity, plus any custom dimensions you add. " +
"The weighted score determines the tier:";
const HEURISTIC_V2_EXPLANATION =
"The router estimates success probability for all four tiers with the bundled calibrated model, then selects " +

6
ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.test.tsx
View file

@ -1521,14 +1521,16 @@ describe("ComplexityRouterConfig tier editing", () => {
renderWithProviders(<ComplexityRouterConfig {...baseProps} value={customValue} onEditingTiersChange={vi.fn()} />);
fireEvent.click(screen.getByText("Advanced: Classification Method"));
expect(screen.queryByText("How Classification Works")).not.toBeInTheDocument();
expect(screen.queryByText("scores each request across 7 dimensions", { exact: false })).not.toBeInTheDocument();
expect(
screen.queryByText("scores each request across 7 built-in dimensions", { exact: false }),
).not.toBeInTheDocument();
});
it("keeps the scorer card on a built-in router, whose tiers the score still decides", () => {
renderWithProviders(<ComplexityRouterConfig {...baseProps} onEditingTiersChange={vi.fn()} />);
fireEvent.click(screen.getByText("Advanced: Classification Method"));
expect(screen.getByText("How Classification Works")).toBeInTheDocument();
expect(screen.getByText("scores each request across 7 dimensions", { exact: false })).toBeInTheDocument();
expect(screen.getByText("scores each request across 7 built-in dimensions", { exact: false })).toBeInTheDocument();
});
it("says why a custom row is blocked instead of only reddening its border", () => {

6
ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx
View file

@ -50,6 +50,7 @@ import EscalationKeywords from "./EscalationKeywords";
import KeywordTierRules, { KeywordTierRule } from "./KeywordTierRules";
import SemanticKeywordMatching from "./SemanticKeywordMatching";
import { type DimensionWeights, type TierBoundaries, type TokenThresholds } from "./heuristic_scoring_knobs";
import { type CustomDimensionRow } from "./custom_dimensions";
import CompressionControls from "./CompressionControls";
import { type AutoRouterCompressionState, DEFAULT_AUTO_ROUTER_COMPRESSION } from "./buildAutoRouterCompression";
@ -432,6 +433,11 @@ export interface ComplexityRouterConfigValue {
tier_boundaries?: TierBoundaries;
token_thresholds?: TokenThresholds;
dimension_weights?: DimensionWeights;
/**
* Operator-added scoring dimensions, each carrying its own inline weight. Undefined means the router has
* none and keeps the key out of the payload; an empty array is a real "the last row was removed" state.
*/
custom_dimensions?: CustomDimensionRow[];
/**
* Score floor the reasoning-marker override must clear. Undefined keeps the key out of the payload, so the
* floor tracks tier_boundaries.simple_medium; an explicit 0 is a real floor that promotes on the markers alone.

136
ui/litellm-dashboard/src/components/add_model/CustomDimensionRows.tsx Normal file
View file

@ -0,0 +1,136 @@
import { Trash2 } from "lucide-react";
import { useState } from "react";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Textarea } from "@/components/ui/textarea";
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
import { Slider } from "@/components/ui/slider";
import type { CustomDimensionRow } from "./custom_dimensions";
const SCORING_MODES = [
{ value: "binary", label: "Binary" },
{ value: "match_count", label: "Match count" },
] as const;
interface Props {
rows: CustomDimensionRow[];
disabled: boolean;
onChange: (rows: CustomDimensionRow[]) => void;
onWeight: (id: string, weight: number) => void;
onAdd: () => void;
onRemove: (id: string) => void;
}
export default function CustomDimensionRows({ rows, disabled, onChange, onWeight, onAdd, onRemove }: Props) {
const [draft, setDraft] = useState<{ id: string; raw: string } | null>(null);
const update = (id: string, patch: Partial<CustomDimensionRow>) =>
onChange(rows.map((row) => (row.id === id ? { ...row, ...patch } : row)));
const editWeight = (id: string, raw: string) => {
setDraft({ id, raw });
if (raw.trim() && Number.isFinite(Number(raw))) onWeight(id, Number(raw));
};
return (
<div className="space-y-4">
{rows.map((row, index) => (
<fieldset key={row.id} className="min-w-0 space-y-3 rounded-md border p-3">
<legend className="float-left text-sm font-semibold">Custom dimension {index + 1}</legend>
<div className="flex items-center gap-2">
<Button
type="button"
variant="ghost"
size="sm"
className="text-destructive hover:text-destructive/80"
aria-label={`Remove custom dimension ${index + 1}`}
disabled={disabled}
onClick={() => onRemove(row.id)}
>
<Trash2 />
Remove
</Button>
</div>
<div className="space-y-1">
<Label htmlFor={`${row.id}-name`}>Name</Label>
<Input
id={`${row.id}-name`}
value={row.name}
maxLength={64}
onChange={(event) => update(row.id, { name: event.target.value })}
/>
</div>
<div className="flex flex-wrap items-center gap-3">
<Label htmlFor={`${row.id}-weight`}>Weight</Label>
<Slider
min={0}
max={1}
step={0.01}
disabled={disabled}
value={[row.weight]}
className="min-w-24 flex-1"
aria-label={`${row.name || `Custom dimension ${index + 1}`} weight`}
onValueChange={(value) => onWeight(row.id, Array.isArray(value) ? value[0] : value)}
/>
<Input
id={`${row.id}-weight`}
className="w-24"
inputMode="decimal"
disabled={disabled}
value={draft?.id === row.id ? draft.raw : Number(row.weight.toPrecision(6)).toString()}
onBlur={() => setDraft(null)}
onChange={(event) => editWeight(row.id, event.target.value)}
/>
</div>
<div className="grid grid-cols-1 gap-3 sm:grid-cols-2">
{(["keywords", "patterns"] as const).map((field) => (
<div key={field} className="min-w-0 space-y-1">
<Label htmlFor={`${row.id}-${field}`}>
{field === "keywords" ? "Keywords" : "Regex patterns"} (one per line)
</Label>
<Textarea
id={`${row.id}-${field}`}
rows={2}
value={row[field]?.join("\n") ?? ""}
onChange={(event) =>
update(row.id, { [field]: event.target.value ? event.target.value.split("\n") : [] })
}
/>
</div>
))}
</div>
<div className="space-y-1">
<Label htmlFor={`${row.id}-scoring`}>Scoring</Label>
<Select
items={SCORING_MODES}
value={row.scoring_mode ?? "binary"}
onValueChange={(mode) => {
if (mode === "binary" || mode === "match_count") update(row.id, { scoring_mode: mode });
}}
>
<SelectTrigger id={`${row.id}-scoring`}>
<SelectValue />
</SelectTrigger>
<SelectContent>
{SCORING_MODES.map((mode) => (
<SelectItem key={mode.value} value={mode.value}>
{mode.label}
</SelectItem>
))}
</SelectContent>
</Select>
<p className="text-xs text-muted-foreground">
Binary uses the full weight for any hit. Match count uses half for one distinct matcher and full weight
for two or more.
</p>
</div>
</fieldset>
))}
<Button type="button" variant="outline" size="sm" disabled={disabled || rows.length >= 16} onClick={onAdd}>
Add custom dimension
</Button>
<p className="text-xs text-muted-foreground">
Keywords match the current ask. Regex scans its first 2,048 characters and permits bounded single-character
repeats up to 64. The proxy validates patterns on save.
</p>
</div>
);
}

90
ui/litellm-dashboard/src/components/add_model/HeuristicScoringConfig.integration.test.tsx Normal file
View file

@ -0,0 +1,90 @@
import { useState } from "react";
import { beforeEach, describe, expect, it, vi } from "vitest";
import userEvent from "@testing-library/user-event";
import { fireEvent, renderWithProviders, screen, testQueryClient, chooseSelectOption } from "../../../tests/test-utils";
import { SHIPPED_SCORER_DEFAULTS } from "../../../tests/mocks/complexityScorerDefaults";
import HeuristicScoringConfig from "./HeuristicScoringConfig";
import type { ComplexityRouterConfigValue } from "./ComplexityRouterConfig";
import { getComplexityScorerDefaults } from "@/components/networking";
vi.mock("@/components/networking", () => ({ getComplexityScorerDefaults: vi.fn() }));
const base: ComplexityRouterConfigValue = {
classifier_type: "heuristic",
tiers: { SIMPLE: [], MEDIUM: [], COMPLEX: [], REASONING: [] },
};
function Editor({ initial = base }: { initial?: ComplexityRouterConfigValue }) {
const [value, setValue] = useState(initial);
return (
<>
<HeuristicScoringConfig value={value} onChange={setValue} />
<output aria-label="Draft config">{JSON.stringify(value)}</output>
</>
);
}
const draft = (): ComplexityRouterConfigValue => JSON.parse(screen.getByLabelText("Draft config").textContent!);
beforeEach(() => {
testQueryClient.clear();
vi.mocked(getComplexityScorerDefaults).mockResolvedValue(SHIPPED_SCORER_DEFAULTS);
});
describe("combined heuristic editor", () => {
it("adds and edits a graded dimension, rebalances builtins, preserves matcher-only edits, and removes it", async () => {
const user = userEvent.setup();
renderWithProviders(<Editor />);
await user.click(screen.getByText("Advanced scoring"));
expect(await screen.findByRole("button", { name: "Restore default weights" })).toBeDisabled();
await user.click(screen.getByRole("button", { name: "Add custom dimension" }));
expect(screen.getByRole("button", { name: "Restore default weights" })).toBeEnabled();
expect(draft().dimension_weights?.codePresence).toBeCloseTo(0.27, 12);
expect(draft().custom_dimensions?.[0].scoring_mode).toBe("match_count");
expect(screen.getByRole("group", { name: "Custom dimension 1" })).toContainElement(
screen.getByLabelText("Name", { exact: true }),
);
fireEvent.change(screen.getByLabelText("Name", { exact: true }), { target: { value: "domain" } });
fireEvent.change(screen.getByLabelText("Keywords (one per line)"), { target: { value: "orbitmesh\nfluxgate" } });
fireEvent.change(screen.getByLabelText("Weight", { exact: true }), { target: { value: "0.2" } });
expect(screen.getByLabelText("Code presence", { exact: true })).toHaveValue("0.24");
expect(screen.getByTestId("dimension-weight-total")).toHaveTextContent("total 1.00");
const beforeMatchers = draft().dimension_weights;
fireEvent.change(screen.getByLabelText("Regex patterns (one per line)"), { target: { value: "abc" } });
await chooseSelectOption(user, screen.getByLabelText("Scoring"), "Binary");
expect(draft().dimension_weights).toEqual(beforeMatchers);
expect(draft().custom_dimensions?.[0].scoring_mode).toBe("binary");
await user.click(screen.getByRole("button", { name: "Remove custom dimension 1" }));
expect(draft().custom_dimensions).toBeUndefined();
expect(draft().dimension_weights?.codePresence).toBeCloseTo(0.3, 12);
});
it("preserves a legacy vector on load and resets both fields only when requested", async () => {
const legacy = {
...base,
dimension_weights: { codePresence: 0.4 },
custom_dimensions: [{ id: "stored-0", name: "domain", weight: 0.7, keywords: ["a"] }],
};
renderWithProviders(<Editor initial={legacy} />);
await userEvent.click(screen.getByText("Advanced scoring"));
expect(await screen.findByTestId("dimension-weight-total")).toHaveTextContent("total 1.10");
expect(draft()).toEqual(legacy);
expect(screen.getByLabelText("Token count", { exact: true })).toHaveValue("0");
await userEvent.click(screen.getByRole("button", { name: "Restore default weights" }));
expect(draft().custom_dimensions).toBeUndefined();
expect(draft().dimension_weights).toBeUndefined();
});
it("drops hidden custom drafts on a fallback-only weight edit so switching back cannot exceed the budget", async () => {
const initial: ComplexityRouterConfigValue = {
...base,
classifier_type: "llm",
classifier_fallback: "heuristic",
custom_dimensions: [{ id: "a", name: "domain", weight: 0.7, keywords: ["a"] }],
};
renderWithProviders(<Editor initial={initial} />);
await userEvent.click(screen.getByText("Advanced scoring"));
fireEvent.change(await screen.findByLabelText("Code presence", { exact: true }), { target: { value: "0.5" } });
expect(draft().custom_dimensions).toBeUndefined();
expect(Object.values(draft().dimension_weights!).reduce((sum, weight) => sum + weight, 0)).toBeCloseTo(1, 12);
expect(screen.queryByRole("button", { name: "Add custom dimension" })).not.toBeInTheDocument();
});
});

14
ui/litellm-dashboard/src/components/add_model/HeuristicScoringConfig.test.tsx
View file

@ -69,13 +69,12 @@ describe("HeuristicScoringConfig", () => {
expect(onChange).not.toHaveBeenCalled();
});
// min and max are inert attributes on a text input, so without the explicit clamp these would persist
// a weight of 999, or an infinite boundary, into the router config.
it.each([
["Code presence", "999", 1],
["Code presence", "-2", 0],
["Long above", "100000", 100000],
])("clamps %s = %s to %s", async (label, raw, expected) => {
it.each(["999", "-2"])("rejects an invalid weight %s instead of silently clamping", async (raw) => {
expect(await commit("Code presence", raw)).toBeUndefined();
expect(screen.getByRole("alert")).toHaveTextContent("Use a weight from 0 to 1");
});
it.each([["Long above", "100000", 100000]])("clamps %s = %s to %s", async (label, raw, expected) => {
const next = await commit(label, raw);
expect(
{ ...next?.dimension_weights, ...next?.token_thresholds }[label === "Long above" ? "complex" : "codePresence"],
@ -264,6 +263,7 @@ describe("HeuristicScoringConfig degraded states", () => {
await userEvent.click(screen.getByText("Advanced scoring"));
expect(screen.getByLabelText("Code presence")).toHaveValue("0.5");
expect(screen.getByLabelText("Code presence")).toBeDisabled();
expect(screen.queryByTestId("dimension-weight-total")).not.toBeInTheDocument();
});
});

85
ui/litellm-dashboard/src/components/add_model/HeuristicScoringConfig.tsx
View file

@ -8,7 +8,14 @@ import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Slider } from "@/components/ui/slider";
import { type ComplexityRouterConfigValue, heuristicScoringRole } from "./ComplexityRouterConfig";
import { dimensionLabel, weightTotal } from "./heuristic_scoring_knobs";
import {
dimensionLabel,
effectiveDimensionWeights,
rebalanceDimensionWeights,
type WeightEdit,
} from "./heuristic_scoring_knobs";
import CustomDimensionRows from "./CustomDimensionRows";
import { customDimensionsError } from "./custom_dimensions";
export type KnobGroup = "tier_boundaries" | "token_thresholds" | "dimension_weights";
@ -54,7 +61,8 @@ const GROUPS: GroupSpec[] = [
{
group: "dimension_weights",
title: "Dimension weights",
blurb: "How much each signal contributes to the score. Absolute multipliers, so the total need not be 1.00.",
blurb:
"Changing a weight rebalances the other built-in and custom weights to total 1.00. Save stores those values. Untouched routers keep their existing weights.",
min: 0,
max: 1,
step: 0.01,
@ -89,6 +97,19 @@ const HeuristicScoringConfig: React.FC<HeuristicScoringConfigProps> = ({ value,
// The panel owns its own visibility: the scorer does not run at all when an LLM classifier
// falls back to the default model, so there is nothing here to configure.
const scorerRuns = heuristicScoringRole(value) !== "never";
const customEnabled = heuristicScoringRole(value) === "decides";
const customRows = customEnabled ? value.custom_dimensions : undefined;
const [weightError, setWeightError] = useState<string | null>(null);
const rowError = customDimensionsError(customRows);
const changeWeights = (edit: WeightEdit) => {
const result = rebalanceDimensionWeights(defaults?.dimension_weights, value.dimension_weights, customRows, edit);
if (!result.ok) {
setWeightError(result.error);
return;
}
setWeightError(null);
onChange({ ...value, dimension_weights: result.dimension_weights, custom_dimensions: result.custom_dimensions });
};
// What an untouched override floor follows: the boundary in effect, override included, not the shipped one.
const trackedFloor: number | undefined = { ...defaults?.tier_boundaries, ...value.tier_boundaries }.simple_medium;
@ -102,6 +123,10 @@ const HeuristicScoringConfig: React.FC<HeuristicScoringConfigProps> = ({ value,
const commit = (spec: GroupSpec, effective: Record<string, number>, key: string, raw: string) => {
const parsed = Number(raw);
if (raw.trim() === "" || !Number.isFinite(parsed)) return;
if (spec.group === "dimension_weights") {
changeWeights({ type: "set", target: { kind: "builtin", id: key }, weight: parsed });
return;
}
const clamped = Math.min(spec.max ?? Infinity, Math.max(spec.min, parsed));
onChange({
...value,
@ -154,9 +179,19 @@ const HeuristicScoringConfig: React.FC<HeuristicScoringConfigProps> = ({ value,
</div>
)}
{GROUPS.map((spec) => {
const shipped = defaults?.[spec.group] ?? {};
const effective: Record<string, number> = { ...shipped, ...value[spec.group] };
const shipped = defaults?.[spec.group] ?? value[spec.group] ?? {};
const effective: Record<string, number> =
spec.group === "dimension_weights"
? effectiveDimensionWeights(shipped, value.dimension_weights)
: { ...shipped, ...value[spec.group] };
const total =
Object.values(effective).reduce((sum, weight) => sum + weight, 0) +
(customRows ?? []).reduce((sum, row) => sum + row.weight, 0);
const problem = warn(spec.group, effective);
const overridden =
value[spec.group] !== undefined || (spec.withSlider && value.custom_dimensions !== undefined);
const resettable = spec.withSlider || overridden;
const scoringError = spec.withSlider ? weightError || rowError : null;
return (
<section key={spec.group} className="space-y-2">
<div className="flex items-center justify-between">
@ -166,18 +201,26 @@ const HeuristicScoringConfig: React.FC<HeuristicScoringConfigProps> = ({ value,
alone would state a total that is not the router's. */}
{spec.withSlider && defaults !== undefined && (
<span className="text-xs text-muted-foreground" data-testid="dimension-weight-total">
total {weightTotal(effective).toFixed(2)}
total {total.toFixed(2)}
</span>
)}
</div>
{value[spec.group] !== undefined && (
{resettable && (
<Button
type="button"
variant="link"
size="xs"
onClick={() => onChange({ ...value, [spec.group]: undefined })}
disabled={!overridden}
onClick={() => {
setWeightError(null);
onChange({
...value,
[spec.group]: undefined,
...(spec.withSlider && { custom_dimensions: undefined }),
});
}}
>
Reset to defaults
{spec.withSlider ? "Restore default weights" : "Reset to defaults"}
</Button>
)}
</div>
@ -196,6 +239,7 @@ const HeuristicScoringConfig: React.FC<HeuristicScoringConfigProps> = ({ value,
min={spec.min}
max={spec.max}
step={spec.step}
disabled={defaults === undefined}
value={[effective[key]]}
onValueChange={(next) =>
commit(spec, effective, key, String(Array.isArray(next) ? next[0] : next))
@ -209,7 +253,8 @@ const HeuristicScoringConfig: React.FC<HeuristicScoringConfigProps> = ({ value,
type="text"
inputMode="decimal"
className={spec.withSlider ? "w-24" : "w-28"}
value={draft?.id === id ? draft.raw : String(effective[key])}
disabled={spec.withSlider && defaults === undefined}
value={draft?.id === id ? draft.raw : Number(effective[key].toPrecision(6)).toString()}
onChange={(event) => {
setDraft({ id, raw: event.target.value });
commit(spec, effective, key, event.target.value);
@ -220,6 +265,28 @@ const HeuristicScoringConfig: React.FC<HeuristicScoringConfigProps> = ({ value,
);
})}
{spec.withSlider && customEnabled && (
<CustomDimensionRows
rows={customRows ?? []}
disabled={defaults === undefined}
onChange={(rows) => onChange({ ...value, custom_dimensions: rows })}
onWeight={(id, weight) =>
changeWeights({ type: "set", target: { kind: "custom", id }, weight })
}
onAdd={() =>
changeWeights({
type: "add",
row: { id: crypto.randomUUID(), name: "", weight: 0.1, scoring_mode: "match_count" },
})
}
onRemove={(id) => changeWeights({ type: "remove", id })}
/>
)}
{scoringError && (
<p className="text-xs text-destructive" role="alert">
{scoringError}
</p>
)}
{problem && (
<p className="text-xs font-medium text-destructive" role="alert">
{problem}

4
ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx
View file

@ -25,12 +25,14 @@ import ComplexityRouterConfig, {
ComplexityRouterConfigValue,
effectiveClassifierType,
usesLlmClassifier,
heuristicScoringRole,
DEFAULT_ADAPTIVE_WEIGHTS,
DEFAULT_SESSION_AFFINITY,
DEFAULT_DEPLOYMENT_AFFINITY,
DEFAULT_TIER_DISTANCE_PENALTY,
} from "./ComplexityRouterConfig";
import { KeywordTierRule } from "./KeywordTierRules";
import { customDimensionsError } from "./custom_dimensions";
import { DEFAULT_ESCALATION_KEYWORDS } from "./EscalationKeywords";
import {
type AutoRouterCompressionState,
@ -139,6 +141,7 @@ export const getSubmitBlockedReason = (
getPlanModeTierError(config.plan_mode_min_tier, activeTierRows(config)) ??
getKeywordTierRulesError(keywordTierRules, activeTierRows(config)) ??
getClassifierModelError(config) ??
(heuristicScoringRole(config) === "decides" ? customDimensionsError(config.custom_dimensions) : null) ??
getClassifierReasoningEffortError(config, modelInfo) ??
getReferencedModelsError(referencedModelsParams, availability)
);
@ -411,6 +414,7 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
tierBoundaries: complexityRouterConfig.tier_boundaries,
tokenThresholds: complexityRouterConfig.token_thresholds,
dimensionWeights: complexityRouterConfig.dimension_weights,
customDimensions: complexityRouterConfig.custom_dimensions,
reasoningOverrideMinScore: complexityRouterConfig.reasoning_override_min_score,
enableContextWindowEscalation: complexityRouterConfig.enable_context_window_escalation,
contextWindowEscalationBuffer: complexityRouterConfig.context_window_escalation_buffer,

47
ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.test.ts
View file

@ -704,6 +704,49 @@ describe("buildComplexityRouterConfig scorer knobs", () => {
expect(buildComplexityRouterConfig(tuned).tier_boundaries).toEqual(BOUNDARIES);
});
it("serializes custom rows without changing weights, matcher order, or optional-field absence", () => {
const weights = { codePresence: 0.12345678901234568, unknownStoredWeight: 9 };
const dimension = { name: "internalFrameworks", weight: 0.3765432109876543, keywords: ["ORBITMESH", "fluxgate"] };
const graded = { name: "sqlDdl", weight: 0.5, patterns: ["create table"], scoring_mode: "match_count" as const };
const payload = buildComplexityRouterConfig({
...baseParams,
dimensionWeights: weights,
customDimensions: [
{ id: "row-1", ...dimension },
{ id: "row-2", ...graded },
],
});
expect(payload.dimension_weights).toEqual(weights);
expect(payload.custom_dimensions).toEqual([dimension, graded]);
expect(buildComplexityRouterConfig(baseParams)).not.toHaveProperty("custom_dimensions");
expect(buildComplexityRouterConfig({ ...baseParams, customDimensions: [] }).custom_dimensions).toEqual([]);
});
it.each([
["heuristic", undefined, true],
["heuristic_first", "heuristic", true],
["hybrid", "heuristic", true],
["llm", "heuristic", false],
["custom", "heuristic", false],
["llm", "default_model", false],
["custom", "default_model", false],
["heuristic_v2", undefined, false],
] as const)(
"%s with fallback %s only emits custom dimensions when its scorer decides",
(classifierType, classifierFallback, emits) => {
const dimension = { name: "d", weight: 0.4, keywords: ["orbitmesh"] };
const params = {
...baseParams,
classifierType,
classifierFallback,
customDimensions: [{ id: "row", ...dimension }],
};
const payload = buildComplexityRouterConfig(params);
if (emits) expect(payload.custom_dimensions).toEqual([dimension]);
else expect(payload).not.toHaveProperty("custom_dimensions");
},
);
it("drops them when the classifier falls back to the default model and nothing is scored", () => {
expect(buildComplexityRouterConfig(llmWithDefaultFallback)).not.toHaveProperty("tier_boundaries");
});
@ -1060,6 +1103,7 @@ describe("buildComplexityRouterConfig with an edited tier set", () => {
tierBoundaries: { simple_medium: 0.1, medium_complex: 0.25, complex_reasoning: 0.5 },
tokenThresholds: { short: 1, long: 2 },
dimensionWeights: { length: 1 },
customDimensions: [{ id: "row-1", name: "sqlDdl", weight: 0.4, keywords: ["orbitmesh"] }],
reasoningOverrideMinScore: 0.5,
heuristicFirstMaxTier: "SIMPLE",
hybridBoundaryMargin: 0.03,
@ -1068,7 +1112,8 @@ describe("buildComplexityRouterConfig with an edited tier set", () => {
stallEscalationWindow: 6,
stallEscalationRepeatThreshold: 3,
};
const emittingType = key === "heuristic_first_max_tier" ? "heuristic_first" : "llm";
// custom_dimensions only ever ship when the scorer decides, so "llm" cannot prove it emits.
const emittingType = key === "heuristic_first_max_tier" || key === "custom_dimensions" ? "heuristic_first" : "llm";
const typeForKey = key === "hybrid_boundary_margin" ? "hybrid" : emittingType;
expect(buildComplexityRouterConfig({ ...baseParams, ...loaded, classifierType: typeForKey })).toHaveProperty(key);
expect(build(loaded)).not.toHaveProperty(key);

17
ui/litellm-dashboard/src/components/add_model/build_complexity_router_config.ts
View file

@ -11,6 +11,7 @@ import {
tierRowByName,
} from "./tier_rows";
import { emptyKeywordTierRuleIndexes, serializeKeywordTierRules } from "./complexity_router_keywords";
import { type CustomDimension, type CustomDimensionRow, serializeCustomDimensions } from "./custom_dimensions";
import {
TierModelParams,
TierModelParamsByTier,
@ -92,6 +93,7 @@ interface ScorerKnobInputs {
tierBoundaries: TierBoundaries | undefined;
tokenThresholds: TokenThresholds | undefined;
dimensionWeights: DimensionWeights | undefined;
customDimensions: CustomDimensionRow[] | undefined;
reasoningOverrideMinScore: number | undefined;
}
@ -106,16 +108,23 @@ const scorerKnobPayload = ({
tierBoundaries,
tokenThresholds,
dimensionWeights,
customDimensions,
reasoningOverrideMinScore,
}: ScorerKnobInputs) =>
heuristicScoringRoleFor(classifierType, classifierFallback) === "never"
}: ScorerKnobInputs) => {
const role = heuristicScoringRoleFor(classifierType, classifierFallback);
return role === "never"
? {}
: {
...(tierBoundaries && { tier_boundaries: tierBoundaries }),
...(tokenThresholds && { token_thresholds: tokenThresholds }),
...(dimensionWeights && { dimension_weights: dimensionWeights }),
// Only a scorer that decides accepts these; the backend rejects them on every other
// classifier, so a fallback-only router must not carry rows a switch left behind.
...(role === "decides" &&
customDimensions !== undefined && { custom_dimensions: serializeCustomDimensions(customDimensions) }),
...(reasoningOverrideMinScore !== undefined && { reasoning_override_min_score: reasoningOverrideMinScore }),
};
};
export interface BuildComplexityRouterConfigParams {
tiers: ComplexityTiers;
@ -155,6 +164,7 @@ export interface BuildComplexityRouterConfigParams {
tierBoundaries?: TierBoundaries;
tokenThresholds?: TokenThresholds;
dimensionWeights?: DimensionWeights;
customDimensions?: CustomDimensionRow[];
reasoningOverrideMinScore?: number;
tierModelParams?: TierModelParamsByTier;
enableContextWindowEscalation?: boolean;
@ -219,6 +229,7 @@ export interface ComplexityRouterConfigPayload {
tier_boundaries?: TierBoundaries;
token_thresholds?: TokenThresholds;
dimension_weights?: DimensionWeights;
custom_dimensions?: CustomDimension[];
reasoning_override_min_score?: number;
enable_context_window_escalation?: boolean;
context_window_escalation_buffer?: number;
@ -496,6 +507,7 @@ export const buildComplexityRouterConfig = ({
tierBoundaries,
tokenThresholds,
dimensionWeights,
customDimensions,
reasoningOverrideMinScore,
tierModelParams,
enableContextWindowEscalation,
@ -517,6 +529,7 @@ export const buildComplexityRouterConfig = ({
tierBoundaries,
tokenThresholds,
dimensionWeights,
customDimensions,
reasoningOverrideMinScore,
};
const scorerKnobs = scorerKnobPayload(scorerInputs);

41
ui/litellm-dashboard/src/components/add_model/custom_dimensions.test.ts Normal file
View file

@ -0,0 +1,41 @@
import { describe, expect, it } from "vitest";
import { customDimensionsError, hydrateCustomDimensions, serializeCustomDimensions } from "./custom_dimensions";
describe("custom dimension drafts", () => {
it.each([
[[{ name: "domain", weight: 0.2, keywords: [" orbitmesh "] }]],
[[{ name: "domain", weight: 0.2, patterns: ["abc"], keywords: [], scoring_mode: "binary" }]],
[[{ name: "domain", weight: 0.2, keywords: ["a", "b"], scoring_mode: "match_count" }]],
[[]],
])("round-trips stored optional fields and matcher text without normalization: %j", (raw) => {
const rows = hydrateCustomDimensions(raw);
expect(rows).toBeDefined();
expect(serializeCustomDimensions(rows!)).toEqual(raw);
});
it("does not insert a default mode or custom list on load", () => {
expect(hydrateCustomDimensions(undefined)).toBeUndefined();
const rows = hydrateCustomDimensions([{ name: "domain", weight: 0.2, keywords: ["a"] }])!;
expect(rows[0].scoring_mode).toBeUndefined();
expect(customDimensionsError(rows)).toBeNull();
});
it.each([
{ name: "", weight: 0.2, keywords: ["a"] },
{ name: "codePresence", weight: 0.2, keywords: ["a"] },
{ name: "bad name", weight: 0.2, keywords: ["a"] },
{ name: "domain", weight: 0, keywords: ["a"] },
{ name: "domain", weight: 0.2, keywords: [] },
{ name: "domain", weight: 0.2, keywords: [" "] },
{ name: "domain", weight: 0.2, keywords: ["a".repeat(257)] },
])("rejects an invalid draft %j", (row) => {
expect(customDimensionsError([{ ...row, id: "draft" }])).not.toBeNull();
});
it("rejects duplicate names and aggregate matcher limits", () => {
const row = { id: "a", name: "domain", weight: 0.2, keywords: ["a"] };
expect(customDimensionsError([row, { ...row, id: "b", name: "DOMAIN" }])).not.toBeNull();
expect(customDimensionsError([{ ...row, keywords: Array(33).fill("a") }])).not.toBeNull();
expect(customDimensionsError([{ ...row, keywords: Array(32).fill("a".repeat(256)) }])).not.toBeNull();
});
});

54
ui/litellm-dashboard/src/components/add_model/custom_dimensions.ts Normal file
View file

@ -0,0 +1,54 @@
import { z } from "zod";
import { DIMENSION_LABELS } from "./heuristic_scoring_knobs";
const customDimensionShape = {
name: z.string(),
weight: z.number(),
keywords: z.array(z.string()).optional(),
patterns: z.array(z.string()).optional(),
scoring_mode: z.enum(["binary", "match_count"]).optional(),
};
const customDimensionSchema = z.object(customDimensionShape);
export type CustomDimension = z.infer<typeof customDimensionSchema>;
export type CustomDimensionRow = CustomDimension & { id: string };
export const hydrateCustomDimensions = (raw: unknown): CustomDimensionRow[] | undefined => {
if (raw === undefined) return undefined;
const parsed = z.array(customDimensionSchema).safeParse(raw);
return parsed.success ? parsed.data.map((row, index) => ({ ...row, id: `stored-${index}` })) : undefined;
};
export const serializeCustomDimensions = (rows: CustomDimensionRow[]): CustomDimension[] =>
rows.map(({ id: _id, ...dimension }) => dimension);
export const customDimensionsError = (
rows: CustomDimensionRow[] | undefined,
builtinNames: string[] = Object.keys(DIMENSION_LABELS),
): string | null => {
if (!rows) return null;
if (rows.length > 16) return "A router can have at most 16 custom dimensions";
const names = rows.map((row) => row.name.toLowerCase());
for (const [index, row] of rows.entries()) {
const prefix = `Custom dimension ${index + 1}: `;
if (!/^[A-Za-z][A-Za-z0-9_]{0,63}$/.test(row.name))
return (
prefix + "use a name starting with a letter, followed by letters, numbers or underscores (64 characters max)"
);
if (builtinNames.some((name) => name.toLowerCase() === row.name.toLowerCase()))
return prefix + "choose a name that is not already a built-in weight";
if (names.indexOf(row.name.toLowerCase()) !== index) return prefix + "names must be unique";
if (!Number.isFinite(row.weight) || row.weight <= 0 || row.weight > 1)
return prefix + "weight must be greater than 0 and at most 1";
const matchers = [...(row.keywords ?? []), ...(row.patterns ?? [])];
if (!matchers.length || matchers.some((matcher) => !matcher.trim()))
return prefix + "add at least one nonblank keyword or pattern";
if (
matchers.length > 32 ||
matchers.some((matcher) => [...matcher].length > 256) ||
matchers.reduce((total, matcher) => total + [...matcher].length, 0) > 4096
)
return prefix + "use at most 32 matchers, 256 characters each and 4096 characters combined";
}
return null;
};

119
ui/litellm-dashboard/src/components/add_model/heuristic_scoring_knobs.test.ts
View file

@ -3,6 +3,8 @@ import { describe, expect, it } from "vitest";
import { heuristicScoringRoleFor } from "./ComplexityRouterConfig";
import {
dimensionLabel,
effectiveDimensionWeights,
rebalanceDimensionWeights,
hydrateDimensionWeights,
hydrateReasoningOverrideMinScore,
hydrateTierBoundaries,
@ -68,6 +70,123 @@ describe("hydrating the scorer knobs", () => {
});
});
describe("rebalancing the complete weight vector", () => {
const defaults = {
codePresence: 0.3,
reasoningMarkers: 0.25,
technicalTerms: 0.25,
tokenCount: 0.1,
simpleIndicators: 0.05,
multiStepPatterns: 0.03,
questionComplexity: 0.02,
};
const row = { id: "domain", name: "domainMarkers", weight: 0.2, keywords: ["orbitmesh"] };
const success = (result: ReturnType<typeof rebalanceDimensionWeights>) => {
if (!result.ok) throw new Error(result.error);
const total =
Object.keys(defaults).reduce((sum, name) => sum + result.dimension_weights[name], 0) +
(result.custom_dimensions ?? []).reduce((sum, dimension) => sum + dimension.weight, 0);
expect(total).toBeCloseTo(1, 12);
return result;
};
it("adds a dimension, edits either kind, and redistributes its share on removal", () => {
const added = success(rebalanceDimensionWeights(defaults, undefined, undefined, { type: "add", row }));
expect(added.dimension_weights.codePresence).toBeCloseTo(0.24, 12);
expect(added.custom_dimensions?.[0].weight).toBe(0.2);
const edited = success(
rebalanceDimensionWeights(defaults, added.dimension_weights, added.custom_dimensions, {
type: "set",
target: { kind: "custom", id: row.id },
weight: 0.4,
}),
);
expect(edited.dimension_weights.codePresence).toBeCloseTo(0.18, 12);
const builtin = success(
rebalanceDimensionWeights(defaults, edited.dimension_weights, edited.custom_dimensions, {
type: "set",
target: { kind: "builtin", id: "codePresence" },
weight: 0.5,
}),
);
expect(builtin.dimension_weights.codePresence).toBe(0.5);
expect(builtin.custom_dimensions?.[0].weight).toBeCloseTo((0.4 * 0.5) / 0.82, 12);
const removed = success(
rebalanceDimensionWeights(defaults, added.dimension_weights, added.custom_dimensions, {
type: "remove",
id: row.id,
}),
);
expect(removed.custom_dimensions).toBeUndefined();
expect(removed.dimension_weights.codePresence).toBeCloseTo(defaults.codePresence, 12);
});
it("uses zero for omitted keys of an explicit map and preserves ignored keys", () => {
expect(effectiveDimensionWeights(defaults, { codePresence: 0.2 }).technicalTerms).toBe(0);
expect(effectiveDimensionWeights(defaults, undefined)).toEqual(defaults);
const result = success(
rebalanceDimensionWeights(defaults, { codePresence: 0.2, unknown: 7 }, undefined, { type: "add", row }),
);
expect(result.dimension_weights.unknown).toBe(7);
expect(result.dimension_weights.codePresence).toBeCloseTo(0.8, 12);
expect(result.dimension_weights.technicalTerms).toBe(0);
});
it("distributes an all-zero vector without inventing custom matchers", () => {
const result = success(rebalanceDimensionWeights(defaults, {}, undefined, { type: "add", row }));
expect(result.dimension_weights.codePresence).toBeCloseTo(0.8 / 7, 12);
expect(result.custom_dimensions).toEqual([row]);
});
it("keeps small positive custom weights and full precision", () => {
const tiny = { ...row, weight: 1e-8 };
const result = success(
rebalanceDimensionWeights(defaults, defaults, [tiny], {
type: "set",
target: { kind: "builtin", id: "codePresence" },
weight: 0.123456789,
}),
);
expect(result.dimension_weights.codePresence).toBe(0.123456789);
expect(result.custom_dimensions?.[0].weight).toBeGreaterThan(0);
expect(result.custom_dimensions?.[0].weight).toBeLessThan(0.01);
});
it.each([0, 1, Number.NaN, -0.1, 1.1])(
"rejects invalid or impossible custom weight %s without changing the input",
(weight) => {
const original = structuredClone(row);
const result = rebalanceDimensionWeights(defaults, defaults, [row, { ...row, id: "other" }], {
type: "set",
target: { kind: "custom", id: row.id },
weight,
});
expect(result.ok).toBe(false);
expect(row).toEqual(original);
},
);
it("allows one dimension to take the whole budget when no custom sibling needs a share", () => {
const result = success(
rebalanceDimensionWeights(defaults, defaults, [row], {
type: "set",
target: { kind: "custom", id: row.id },
weight: 1,
}),
);
expect(Object.values(result.dimension_weights).every((weight) => weight === 0)).toBe(true);
expect(result.custom_dimensions?.[0].weight).toBe(1);
});
it("refuses missing defaults and invalid stored weights, preserving absence on built-in edits", () => {
const edit = { type: "set", target: { kind: "builtin", id: "codePresence" }, weight: 0.2 } as const;
expect(rebalanceDimensionWeights(undefined, defaults, undefined, edit).ok).toBe(false);
expect(rebalanceDimensionWeights(defaults, { codePresence: -1 }, undefined, edit).ok).toBe(false);
expect(success(rebalanceDimensionWeights(defaults, undefined, undefined, edit)).custom_dimensions).toBeUndefined();
expect(success(rebalanceDimensionWeights(defaults, undefined, [], edit)).custom_dimensions).toEqual([]);
});
});
describe("heuristicScoringRoleFor", () => {
it.each([
["heuristic", undefined, "decides"],

97
ui/litellm-dashboard/src/components/add_model/heuristic_scoring_knobs.ts
View file

@ -1,3 +1,5 @@
import type { CustomDimensionRow } from "./custom_dimensions";
export type TierBoundaries = Record<string, number>;
export type TokenThresholds = Record<string, number>;
@ -27,8 +29,8 @@ const asRecord = (raw: unknown): Record<string, unknown> | undefined =>
/**
* Absent means the router is tracking the shipped defaults, so it must hydrate to undefined rather than to
* a copy of them: hydrating defaults would make an untouched save write them out and pin the router to
* whatever they were the day the modal was opened. A stored dict is kept exactly as stored, since the
* backend fills in any key it omits at scoring time.
* whatever they were the day the modal was opened. A stored weight map replaces the defaults;
* missing dimension names score zero.
*/
const hydrateNumericMap = (raw: unknown): Record<string, number> | undefined => {
const stored = asRecord(raw);
@ -53,3 +55,94 @@ export const hydrateReasoningOverrideMinScore = (raw: unknown): number | undefin
export const weightTotal = (weights: DimensionWeights): number =>
Math.round(Object.values(weights).reduce((total, weight) => total + weight, 0) * 100) / 100;
export const effectiveDimensionWeights = (
defaults: DimensionWeights,
stored: DimensionWeights | undefined,
): DimensionWeights =>
Object.fromEntries(
Object.keys(defaults).map((name) => [name, stored === undefined ? defaults[name] : stored[name] ?? 0]),
);
type WeightTarget = { kind: "builtin" | "custom"; id: string };
const weightValid = ({ kind, weight }: { kind: WeightTarget["kind"]; weight: number }): boolean => {
const inRange = Number.isFinite(weight) && weight >= 0 && weight <= 1;
return inRange && (kind === "builtin" || weight > 0);
};
export type WeightEdit =
| { type: "set"; target: WeightTarget; weight: number }
| { type: "add"; row: CustomDimensionRow }
| { type: "remove"; id: string };
type WeightResult =
| { ok: true; dimension_weights: DimensionWeights; custom_dimensions: CustomDimensionRow[] | undefined }
| { ok: false; error: string };
export const rebalanceDimensionWeights = (
defaults: DimensionWeights | undefined,
stored: DimensionWeights | undefined,
custom: CustomDimensionRow[] | undefined,
edit: WeightEdit,
): WeightResult => {
if (!defaults || !Object.keys(defaults).length)
return { ok: false, error: "Load the shipped defaults before changing weights" };
const builtin = effectiveDimensionWeights(defaults, stored);
const rows =
edit.type === "add"
? [...(custom ?? []), edit.row]
: (custom ?? []).filter((row) => edit.type !== "remove" || row.id !== edit.id);
const vector = [
...Object.entries(builtin).map(([id, weight]) => ({ kind: "builtin" as const, id, weight })),
...rows.map(({ id, weight }) => ({ kind: "custom" as const, id, weight })),
];
if (!vector.every(weightValid))
return {
ok: false,
error: "Existing weights must be finite and nonnegative; custom weights must be greater than 0 and at most 1",
};
const target = edit.type === "set" ? edit.target : undefined;
const pinned = (entry: WeightTarget) =>
edit.type === "add"
? entry.kind === "custom" && entry.id === edit.row.id
: entry.kind === target?.kind && entry.id === target.id;
if (edit.type === "set" && !vector.some(pinned)) return { ok: false, error: "The dimension is no longer available" };
const requestedWeight = (): number => {
if (edit.type === "set") return edit.weight;
return edit.type === "add" ? edit.row.weight : 0;
};
const weight = requestedWeight();
if (!weightValid({ kind: target?.kind ?? "builtin", weight }))
return { ok: false, error: "Use a weight from 0 to 1; custom dimensions must stay greater than 0" };
const others = vector.filter((entry) => !pinned(entry));
const total = others.reduce((sum, entry) => sum + entry.weight, 0);
if (!Number.isFinite(total)) return { ok: false, error: "Existing weights are too large to rebalance" };
const remainder = 1 - weight;
const builtinCount = others.filter((entry) => entry.kind === "builtin").length;
const redistributed = (entry: WeightTarget & { weight: number }): number => {
if (pinned(entry)) return weight;
if (total > 0) return remainder * (entry.weight / total);
return entry.kind === "builtin" ? remainder / builtinCount : 0;
};
const balanced = vector.map((entry) => ({ ...entry, weight: redistributed(entry) }));
const residual = 1 - balanced.reduce((sum, entry) => sum + entry.weight, 0);
const receiver = balanced
.filter((entry) => entry.kind === "builtin" && !pinned(entry) && entry.weight > 0)
.sort((a, b) => b.weight - a.weight)[0];
const corrected = balanced.map((entry) =>
entry === receiver ? { ...entry, weight: entry.weight + residual } : entry,
);
const offBudget = Math.abs(corrected.reduce((sum, entry) => sum + entry.weight, 0) - 1) > 1e-12;
if (!corrected.every(weightValid) || offBudget)
return { ok: false, error: "Leave a positive share for every custom dimension, or remove it first" };
const weights = Object.fromEntries(
corrected.filter((entry) => entry.kind === "builtin").map(({ id, weight }) => [id, weight]),
);
const customWeights = new Map(
corrected.filter((entry) => entry.kind === "custom").map(({ id, weight }) => [id, weight]),
);
const emptyRows = edit.type === "remove" ? undefined : custom;
return {
ok: true,
dimension_weights: { ...stored, ...weights },
custom_dimensions: rows.length ? rows.map((row) => ({ ...row, weight: customWeights.get(row.id)! })) : emptyRows,
};
};

1
ui/litellm-dashboard/src/components/add_model/tier_rows.ts
View file

@ -136,6 +136,7 @@ export const CUSTOM_TIER_RESTRICTIONS = {
"tier_boundaries",
"token_thresholds",
"dimension_weights",
"custom_dimensions",
"reasoning_override_min_score",
"custom_technical_keywords",
],

49
ui/litellm-dashboard/src/components/edit_auto_router/build_updated_complexity_router_config.test.ts
View file

@ -63,6 +63,54 @@ describe("buildUpdatedComplexityRouterConfig keyword matching", () => {
expect(result.some_future_backend_key).toEqual({ nested: true });
});
describe("custom dimensions", () => {
const stored = [
{ name: "internalFrameworks", weight: 0.7, keywords: ["orbitmesh"] },
{ name: "sqlDdl", weight: 0.3, patterns: ["create table"], scoring_mode: "match_count" },
];
const withDimensions = { ...STORED, custom_dimensions: stored };
it("round-trips stored rows through hydration and save without dropping or reshaping one", () => {
const hydrated = hydrateComplexityRouterConfig(withDimensions, null);
expect(hydrated.custom_dimensions).toEqual([
{ id: "stored-0", ...stored[0] },
{ id: "stored-1", ...stored[1] },
]);
const saved = buildUpdatedComplexityRouterConfig(withDimensions, {
...FORM_VALUE,
custom_dimensions: hydrated.custom_dimensions,
});
expect(saved.custom_dimensions).toEqual(stored);
});
it("is a managed key, so removing the last row does not resurrect the stored dimensions", () => {
expect(MANAGED_COMPLEXITY_ROUTER_KEYS.has("custom_dimensions")).toBe(true);
const saved = buildUpdatedComplexityRouterConfig(withDimensions, { ...FORM_VALUE, custom_dimensions: [] });
expect(saved.custom_dimensions).toEqual([]);
});
it("omits the key entirely when the editor never held rows, so an untouched router gains nothing", () => {
expect(hydrateComplexityRouterConfig(STORED, null).custom_dimensions).toBeUndefined();
expect(buildUpdatedComplexityRouterConfig(STORED, FORM_VALUE)).not.toHaveProperty("custom_dimensions");
});
it("drops rows a custom tier set forbids rather than sending them to a scorer that never runs", () => {
const customTierStored = storedCustomConfig({ custom_dimensions: stored });
const saved = buildUpdatedComplexityRouterConfig(customTierStored, {
tiers: { SIMPLE: [], MEDIUM: [], COMPLEX: [], REASONING: [] },
classifier_type: "llm" as const,
custom_tier_set: {
tiers: [
{ id: "a", name: "CASUAL", definition: "small talk", models: ["gpt-4o-mini"] },
{ id: "b", name: "AUDIT", definition: "security review", models: ["o1"] },
],
fallback_tier_id: "a",
},
});
expect(saved).not.toHaveProperty("custom_dimensions");
});
});
it("persists an edited keyword rule", () => {
const result = buildUpdatedComplexityRouterConfig(STORED, FORM_VALUE, undefined, {
...hydratedState,
@ -580,6 +628,7 @@ describe("managed keys survive an untouched open-and-save", () => {
tier_boundaries: { simple_medium: 0.2, medium_complex: 0.4, complex_reasoning: 0.7 },
token_thresholds: { simple: 20, complex: 500 },
dimension_weights: { tokenCount: 0.1 },
custom_dimensions: [{ name: "domain", weight: 0.9, keywords: ["orbitmesh"] }],
reasoning_override_min_score: 0.3,
enable_context_window_escalation: false,
context_window_escalation_buffer: 0.9,

17
ui/litellm-dashboard/src/components/edit_auto_router/edit_auto_router_modal.tsx
View file

@ -48,6 +48,7 @@ import {
hydrateAutoRouterCompression,
} from "../add_model/buildAutoRouterCompression";
import { hydrateKeywordTierRules } from "../add_model/complexity_router_keywords";
import { customDimensionsError, hydrateCustomDimensions } from "../add_model/custom_dimensions";
import {
hydrateDimensionWeights,
hydrateReasoningOverrideMinScore,
@ -61,6 +62,7 @@ import ComplexityRouterConfig, {
ClassifierType,
ComplexityRouterConfigValue,
ComplexityTiers,
heuristicScoringRole,
DEFAULT_ADAPTIVE_WEIGHTS,
DEFAULT_SESSION_AFFINITY,
DEFAULT_DEPLOYMENT_AFFINITY,
@ -109,6 +111,7 @@ export interface StoredComplexityRouterConfig {
tier_boundaries?: unknown;
token_thresholds?: unknown;
dimension_weights?: unknown;
custom_dimensions?: unknown;
reasoning_override_min_score?: unknown;
session_affinity?: unknown;
session_affinity_ttl_seconds?: unknown;
@ -194,6 +197,7 @@ export const hydrateComplexityRouterConfig = (
tier_boundaries: hydrateTierBoundaries(parsedConfig.tier_boundaries),
token_thresholds: hydrateTokenThresholds(parsedConfig.token_thresholds),
dimension_weights: hydrateDimensionWeights(parsedConfig.dimension_weights),
custom_dimensions: hydrateCustomDimensions(parsedConfig.custom_dimensions),
reasoning_override_min_score: hydrateReasoningOverrideMinScore(parsedConfig.reasoning_override_min_score),
session_affinity:
typeof parsedConfig.session_affinity === "boolean" ? parsedConfig.session_affinity : DEFAULT_SESSION_AFFINITY,
@ -264,6 +268,7 @@ export const MANAGED_COMPLEXITY_ROUTER_KEYS = new Set([
"tier_boundaries",
"token_thresholds",
"dimension_weights",
"custom_dimensions",
"reasoning_override_min_score",
"enable_context_window_escalation",
"context_window_escalation_buffer",
@ -373,6 +378,7 @@ export const buildUpdatedComplexityRouterConfig = (
tierBoundaries: value.tier_boundaries,
tokenThresholds: value.token_thresholds,
dimensionWeights: value.dimension_weights,
customDimensions: value.custom_dimensions,
reasoningOverrideMinScore: value.reasoning_override_min_score,
tierModelParams: value.tier_model_params,
enableContextWindowEscalation: value.enable_context_window_escalation,
@ -481,7 +487,10 @@ const EditAutoRouterModal: React.FC<EditAutoRouterModalProps> = ({
: null) ?? getTierLabelsError(complexityRouterConfig.tier_labels)) ??
getPlanModeTierError(complexityRouterConfig.plan_mode_min_tier, activeTierRows(complexityRouterConfig)) ??
getKeywordTierRulesError(keywordTierRules, activeTierRows(complexityRouterConfig)) ??
getClassifierModelError(complexityRouterConfig);
getClassifierModelError(complexityRouterConfig) ??
(heuristicScoringRole(complexityRouterConfig) === "decides"
? customDimensionsError(complexityRouterConfig.custom_dimensions)
: null);
useEffect(() => {
if (isVisible && modelData) {
@ -601,7 +610,11 @@ const EditAutoRouterModal: React.FC<EditAutoRouterModalProps> = ({
toast.fromError(tierSetError);
return;
}
const classifierError = getClassifierModelError(complexityRouterConfig);
const classifierError =
getClassifierModelError(complexityRouterConfig) ??
(heuristicScoringRole(complexityRouterConfig) === "decides"
? customDimensionsError(complexityRouterConfig.custom_dimensions)
: null);
if (classifierError) {
setShowValidationErrors(true);
toast.fromError(classifierError);

28
ui/litellm-dashboard/src/lib/autorouter_presets.test.ts
View file

@ -87,6 +87,19 @@ describe("autorouter_presets", () => {
}
});
it("keeps every preset free of custom dimensions, so applying one never adds scoring rows", () => {
for (const { complexity_router_config: config } of getAllPresets()) {
expect(config.custom_dimensions).toBeUndefined();
}
const config = getPresetByKey("anthropic_family")!.complexity_router_config;
expect(buildPresetPrefill(config, groupsOnly([])).complexityRouterConfig.custom_dimensions).toBeUndefined();
});
it("resets both scoring overrides when the form falls back to an empty prefill", () => {
expect(buildEmptyPrefill().complexityRouterConfig.custom_dimensions).toBeUndefined();
expect(buildEmptyPrefill().complexityRouterConfig.dimension_weights).toBeUndefined();
});
it("carries a preset's session affinity idle window into the prefilled form state", () => {
const config = getPresetByKey("anthropic_family")!.complexity_router_config;
const prefill = buildPresetPrefill({ ...config, session_affinity_ttl_seconds: 300 }, groupsOnly([]));
@ -96,6 +109,21 @@ describe("autorouter_presets", () => {
).toBeUndefined();
});
it("carries a preset's stored weights and custom dimensions into the prefill without rebalancing them", () => {
const config = getPresetByKey("anthropic_family")!.complexity_router_config;
const weights = { codePresence: 0.4 };
const dimension = { name: "domain", weight: 0.9, keywords: ["orbitmesh"] };
const prefill = buildPresetPrefill(
{ ...config, dimension_weights: weights, custom_dimensions: [dimension] },
groupsOnly([]),
).complexityRouterConfig;
expect(prefill.dimension_weights).toEqual(weights);
expect(prefill.custom_dimensions).toEqual([{ ...dimension, id: "stored-0" }]);
const plain = buildPresetPrefill(config, groupsOnly([])).complexityRouterConfig;
expect(plain.dimension_weights).toBeUndefined();
expect(plain.custom_dimensions).toBeUndefined();
});
it("keeps the model-family presets on the heuristic classifier", () => {
for (const key of ["anthropic_family", "gemini_family", "openai_family"]) {
expect(getPresetByKey(key)!.complexity_router_config.classifier_type).toBe("heuristic");

12
ui/litellm-dashboard/src/lib/autorouter_presets.ts
View file

@ -13,6 +13,13 @@ import {
} from "@/components/add_model/ComplexityRouterConfig";
import { KeywordTierRule } from "@/components/add_model/KeywordTierRules";
import { hydrateKeywordTierRules } from "@/components/add_model/complexity_router_keywords";
import { hydrateCustomDimensions } from "@/components/add_model/custom_dimensions";
import {
hydrateDimensionWeights,
hydrateTierBoundaries,
hydrateTokenThresholds,
hydrateReasoningOverrideMinScore,
} from "@/components/add_model/heuristic_scoring_knobs";
import {
TierModelParams,
TierModelParamsByTier,
@ -293,6 +300,11 @@ export const buildPresetPrefill = (
tier_distance_penalty: config.tier_distance_penalty,
adaptive_eligible: config.adaptive_eligible,
return_raw_model_name: config.return_raw_model_name,
dimension_weights: hydrateDimensionWeights(config.dimension_weights),
custom_dimensions: hydrateCustomDimensions(config.custom_dimensions),
tier_boundaries: hydrateTierBoundaries(config.tier_boundaries),
token_thresholds: hydrateTokenThresholds(config.token_thresholds),
reasoning_override_min_score: hydrateReasoningOverrideMinScore(config.reasoning_override_min_score),
enable_context_window_escalation: config.enable_context_window_escalation,
context_window_escalation_buffer: config.context_window_escalation_buffer,
},

9
ui/litellm-dashboard/src/lib/http/schema.d.ts generated vendored
View file

@ -26591,6 +26591,13 @@ export interface components {
* @default []
*/
patterns: string[];
/**
* Scoring Mode
* @description 'binary' scores 1 when any matcher hits. 'match_count' scores 0.5 when one distinct matcher hits and 1 when two or more do; repeated occurrences of one matcher never raise it. Keywords are distinct case-insensitively, patterns by source, and a keyword and a pattern are always distinct from each other.
* @default binary
* @enum {string}
*/
scoring_mode: "binary" | "match_count";
/** Weight */
weight: number;
};
@ -34909,7 +34916,7 @@ export interface components {
context_window_escalation_buffer: number;
/**
* Custom Dimensions
* @description Named binary dimensions added to the heuristic-v1 score. Each contributes its inline weight once when any keyword matches the current ask or a case-insensitive regex matches its first 2048 characters. Regex quantifiers repeat one character or class at most 64 times. Unbounded quantifiers, repeated groups, backreferences and lookarounds are rejected. Conservative work limits include alternation paths, repeat lengths and subsequent matching: 2048 units per pattern, 8192 across the router. Only heuristic, heuristic_first and hybrid accept this field. Uses the existing heuristic tuning quota.
* @description Named dimensions added to the heuristic-v1 score. Each contributes its inline weight once when any keyword matches the current ask or a case-insensitive regex matches its first 2048 characters; scoring_mode 'match_count' instead grades half weight for one distinct matcher and full for two or more. Regex quantifiers repeat one character or class at most 64 times. Unbounded quantifiers, repeated groups, backreferences and lookarounds are rejected. Conservative work limits include alternation paths, repeat lengths and subsequent matching: 2048 units per pattern, 8192 across the router. Only heuristic, heuristic_first and hybrid accept this field. Uses the existing heuristic tuning quota.
* @default []
*/
custom_dimensions: components["schemas"]["CustomDimension"][];