mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
feat(complexity_router): add opt-in scorer_version 2 to the heuristic classifier
Version 2 replaces the scoring basis: difficulty evidence comes from the operations a request demands (reasoningDemand, domainDepth, constraintDensity, deliverableCount, outputScope, contextOperation, multiHop) instead of software vocabulary, a zero-weight taskType category rides the signals for evaluation, and SIMPLE requires positive triviality evidence. Below the simple_medium boundary everything else defaults to MEDIUM with the new routing decision cause insufficient_evidence, which also abstains to the LLM classifier under heuristic_first. A v2 abstention on a continuation turn inherits the conversation's tier instead, derived statelessly from the request's own history (cause insufficient_evidence_inherit), so agentic turns carrying terminal or tool output keep the tier the real ask earned, and a three-turn consecutive failure streak raises the inherited tier one step (progress-stall). Weights are provisional hand-set defaults pending calibration. Existing routers stay on version 1 so routing and spend do not move on upgrade.
This commit is contained in:
Tin Chi Lo
parent
8f56dbe7a3
commit
635de7fe47
9 changed files with 1439 additions and 49 deletions
|
|
@ -36,6 +36,37 @@ The weighted sum is mapped to tiers using configurable boundaries:
|
|||
|
||||
Tier names are defaults you can rename with [`tier_labels`](#renaming-the-tiers). The three `tier_boundaries` keys are named after those defaults but they are scorer knobs, not tiers: each one names the gap between two rungs and is persisted by name on every routing decision, so they stay `simple_medium` / `medium_complex` / `complex_reasoning` no matter what you call the tiers. The column above tells a renamed deployment which knob it is turning.
|
||||
|
||||
### Scorer versions
|
||||
|
||||
`scorer_version: 1` (the default) scores the seven dimensions above and maps any score below `simple_medium` to SIMPLE. Since most of that weight budget keys on software vocabulary, a prompt matching no keyword scores 0.0 and routes to the cheapest tier by default: a graduate chemistry question, a legal analysis, or a math proof all land SIMPLE with no evidence either way.
|
||||
|
||||
`scorer_version: 2` is opt-in and replaces the feature basis. Length and question-mark counting are gone (context-window fit is already its own gate), code vocabulary no longer means difficulty, and SIMPLE requires positive evidence of triviality. The v2 dimensions:
|
||||
|
||||
| Dimension | Detects | Weight |
|
||||
|-----------|---------|--------|
|
||||
| `reasoningDemand` | Operations required: proof, derivation, diagnosis, optimization, causal inference | 0.22 |
|
||||
| `domainDepth` | Terminology depth in the single best-matching domain (math, physical_science, medicine, law, finance, data, engineering); needs 2+ hits | 0.18 |
|
||||
| `contextOperation` | What must be done with supplied material; reconcile/synthesize/diagnose/forecast score, copy/extract/translate/summarize do not | 0.15 |
|
||||
| `constraintDensity` | Independent requirements (must, at least, exactly, without using); needs 2+ hits | 0.13 |
|
||||
| `multiHop` | Dependent inference steps (given that, based on the result, if..then) | 0.10 |
|
||||
| `deliverableCount` | Independent asks (and also, as well as, coordinated imperatives); no option-list regexes | 0.08 |
|
||||
| `outputScope` | Artifacts named for delivery next to an authoring verb | 0.07 |
|
||||
| `trivialityEvidence` | Greetings, acknowledgements, bounded transformations; the only dimension that can produce SIMPLE | 0.07 (negative) |
|
||||
| `taskType` | Descriptive category (code/math/writing/...) for logs and evaluation | 0.00, never a tier prior |
|
||||
|
||||
The v2 weights are provisional hand-set defaults; fitting them against public model-outcome data is staged follow-up work. Override them per key in `dimension_weights` using the v2 names (v1 names are ignored under v2). Per-domain vocabulary is extendable via `domain_keywords`. The mapping: score at or above `simple_medium` maps through the shared `tier_boundaries` exactly as v1; below it, SIMPLE requires a `trivialityEvidence` match with no dimension scoring positive, and everything else defaults to MEDIUM with cause `insufficient_evidence` (empty `signals` means nothing fired at all). The reasoning override keeps working, keyed on `reasoningDemand` matches.
|
||||
|
||||
Operator-supplied keyword lists override the defaults under both versions (`reasoning_keywords` feeds `reasoningDemand`, `simple_keywords` feeds `trivialityEvidence`). Under `heuristic_first`, a v2 request whose tier came from `insufficient_evidence` abstains to the LLM classifier rather than short-circuiting. Existing routers stay on v1, so routing and spend do not move on upgrade; opting a router in raises spend on traffic that previously defaulted to the cheapest tier.
|
||||
|
||||
On agentic traffic most turns carry terminal or tool output as the newest message, which holds no evidence, so defaulting every such continuation to MEDIUM would override the tier the real ask earned. A v2 abstention on a continuation therefore inherits the conversation's tier: the scorer walks the request's own history newest first and takes the tier of the most recent prior ask that carried real evidence above SIMPLE (trivial interjections like "thanks" never reset it), reported with cause `insufficient_evidence_inherit`. This is stateless and unrelated to `session_affinity`: nothing is cached, no session id is needed, and it is on for every v2 router. When the last three consecutive non-assistant turns carry failure markers (tracebacks, nonzero exits, `make: ***`) the inherited tier is raised one configured step, reported in signals as `progress-stall`, so a stalled run reaches a stronger model; one clean run resets the streak. A continuation with real evidence still classifies for itself.
|
||||
|
||||
```yaml
|
||||
complexity_router_config:
|
||||
scorer_version: 2
|
||||
domain_keywords: # optional: replace or extend a domain's vocabulary
|
||||
gastronomy: ["sourdough starter", "lamination"]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Basic Configuration
|
||||
|
|
|
|||
|
|
@ -45,15 +45,26 @@ from .classification_rubrics import BUSINESS_TIER_CRITERIA, calibration_examples
|
|||
from .config import (
|
||||
DEFAULT_CLASSIFICATION_RUBRIC,
|
||||
DEFAULT_CODE_KEYWORDS,
|
||||
DEFAULT_CODE_KEYWORDS_V2,
|
||||
DEFAULT_CONSTRAINT_KEYWORDS,
|
||||
DEFAULT_DIMENSION_WEIGHTS_V2,
|
||||
DEFAULT_DOMAIN_KEYWORDS,
|
||||
DEFAULT_ESCALATION_KEYWORDS,
|
||||
DEFAULT_REASONING_DEMAND_KEYWORDS,
|
||||
DEFAULT_REASONING_KEYWORDS,
|
||||
DEFAULT_SIMPLE_KEYWORDS,
|
||||
DEFAULT_SIMPLE_KEYWORDS_V2,
|
||||
DEFAULT_TECHNICAL_KEYWORDS,
|
||||
HOUSEKEEPING_ASK_SENTINELS,
|
||||
PLAN_MODE_SYSTEM_SENTINELS,
|
||||
PLAN_MODE_TAIL_SENTINELS,
|
||||
PLAN_MODE_TOOL_NAME,
|
||||
TIER_SEVERITY_ORDER,
|
||||
V2_ARTIFACT_NOUNS,
|
||||
V2_AUTHORING_VERBS,
|
||||
V2_CONTEXT_MARKERS,
|
||||
V2_CONTEXT_OPS_HIGH,
|
||||
V2_CONTEXT_OPS_MEDIUM,
|
||||
ClassificationRubric,
|
||||
ComplexityRouterConfig,
|
||||
ComplexityTier,
|
||||
|
|
@ -387,6 +398,44 @@ def _iter_human_asks_newest_first(
|
|||
)
|
||||
|
||||
|
||||
# Failure evidence in recent non-assistant turns, scanned only when a v2 abstention inherits a
|
||||
# tier. Assistant turns are excluded because a model DESCRIBING an error is not a failing run.
|
||||
_FAILURE_MARKERS: Final[tuple[re.Pattern[str], ...]] = (
|
||||
re.compile(r"traceback \(most recent call last\)"),
|
||||
re.compile(r"\berror(?::|\b code)"),
|
||||
re.compile(r"\bfailed\b"),
|
||||
re.compile(r"\bexit code [1-9]"),
|
||||
re.compile(r"\bcommand not found\b"),
|
||||
re.compile(r"no such file or directory"),
|
||||
re.compile(r"segmentation fault"),
|
||||
re.compile(r"\bpanic:"),
|
||||
re.compile(r"make: \*\*\*"),
|
||||
re.compile(r"assertionerror"),
|
||||
re.compile(r"\bexception\b"),
|
||||
)
|
||||
_FAILURE_STREAK_TO_ESCALATE: Final = 3
|
||||
|
||||
|
||||
def _failure_streak(messages: Sequence[Mapping[str, object]]) -> int:
|
||||
"""Consecutive newest-first non-assistant turns whose text carries a failure marker.
|
||||
|
||||
A turn with no readable text (a tool_result-only carrier) breaks the streak rather than
|
||||
extending it: escalation needs visible consecutive failures, never a guess."""
|
||||
recent: Final = tuple(
|
||||
_message_text(msg.get("content")).lower()
|
||||
for msg in reversed(messages[-12:])
|
||||
if msg.get("role") not in ("assistant", "system")
|
||||
)
|
||||
return next(
|
||||
(
|
||||
index
|
||||
for index, text in enumerate(recent)
|
||||
if not text or not any(pattern.search(text) for pattern in _FAILURE_MARKERS)
|
||||
),
|
||||
len(recent),
|
||||
)
|
||||
|
||||
|
||||
def _conversation_is_continuing(messages: Sequence[Mapping[str, object]] | None) -> bool:
|
||||
"""Whether this request continues a conversation that was already underway.
|
||||
|
||||
|
|
@ -751,6 +800,8 @@ def _decision_is_pinnable(decision: StandardLoggingRoutingDecision | None) -> bo
|
|||
"plan_mode",
|
||||
"housekeeping",
|
||||
"modality_escalation",
|
||||
"insufficient_evidence",
|
||||
"insufficient_evidence_inherit",
|
||||
)
|
||||
and not decision.get("context_escalated")
|
||||
)
|
||||
|
|
@ -791,6 +842,7 @@ class ClassificationOutcome(NamedTuple):
|
|||
cause: Literal[
|
||||
"heuristic_scorer",
|
||||
"reasoning_override",
|
||||
"insufficient_evidence",
|
||||
"llm_classifier",
|
||||
"heuristic_first_short_circuit",
|
||||
"housekeeping",
|
||||
|
|
@ -882,7 +934,7 @@ class ComplexityRouter(CustomLogger):
|
|||
self,
|
||||
model_name: str,
|
||||
litellm_router_instance: Router,
|
||||
complexity_router_config: dict[str, Any] | None = None,
|
||||
complexity_router_config: Mapping[str, object] | None = None,
|
||||
default_model: str | None = None,
|
||||
derive_savings_baseline: bool = True,
|
||||
):
|
||||
|
|
@ -924,13 +976,29 @@ class ComplexityRouter(CustomLogger):
|
|||
)
|
||||
|
||||
# Build effective keyword lists (use config overrides or defaults)
|
||||
self.code_keywords = self.config.code_keywords or DEFAULT_CODE_KEYWORDS
|
||||
self.reasoning_keywords = self.config.reasoning_keywords or DEFAULT_REASONING_KEYWORDS
|
||||
scorer_v2: Final = self.config.scorer_version == 2
|
||||
self.code_keywords: Sequence[str] = self.config.code_keywords or (
|
||||
DEFAULT_CODE_KEYWORDS_V2 if scorer_v2 else DEFAULT_CODE_KEYWORDS
|
||||
)
|
||||
self.reasoning_keywords: Sequence[str] = self.config.reasoning_keywords or (
|
||||
DEFAULT_REASONING_DEMAND_KEYWORDS if scorer_v2 else DEFAULT_REASONING_KEYWORDS
|
||||
)
|
||||
self.technical_keywords = _append_custom_keywords(
|
||||
self.config.technical_keywords or DEFAULT_TECHNICAL_KEYWORDS,
|
||||
self.config.custom_technical_keywords,
|
||||
)
|
||||
self.simple_keywords = self.config.simple_keywords or DEFAULT_SIMPLE_KEYWORDS
|
||||
self.simple_keywords: Sequence[str] = self.config.simple_keywords or (
|
||||
DEFAULT_SIMPLE_KEYWORDS_V2 if scorer_v2 else DEFAULT_SIMPLE_KEYWORDS
|
||||
)
|
||||
# The engineering domain tracks the technical list (custom_technical_keywords included) so
|
||||
# that knob keeps working under v2; an operator domain_keywords entry replaces per domain.
|
||||
self._domain_keywords: Mapping[str, Sequence[str]] = MappingProxyType(
|
||||
{
|
||||
**DEFAULT_DOMAIN_KEYWORDS,
|
||||
"engineering": tuple(self.technical_keywords),
|
||||
**(self.config.domain_keywords or EMPTY_MAPPING),
|
||||
}
|
||||
)
|
||||
if self.config.has_custom_tiers:
|
||||
self.escalation_keywords: tuple[str, ...] = ()
|
||||
elif self.config.escalation_keywords is not None:
|
||||
|
|
@ -952,12 +1020,35 @@ class ComplexityRouter(CustomLogger):
|
|||
|
||||
# Pre-compile regex patterns for efficiency
|
||||
# Use non-greedy .*? to prevent ReDoS on pathological inputs
|
||||
# v1 only; v2's deliverableCount deliberately has no option-list regexes ("1. ", "a) "),
|
||||
# which match ordinary enumerated prose rather than multi-step work.
|
||||
self._multi_step_patterns = [
|
||||
re.compile(r"first.*?then", re.IGNORECASE),
|
||||
re.compile(r"step\s*\d", re.IGNORECASE),
|
||||
re.compile(r"\d+\.\s"),
|
||||
re.compile(r"[a-z]\)\s", re.IGNORECASE),
|
||||
]
|
||||
# The two-anchor patterns cap the gap at 80 chars ({0,80}?) rather than .*?: a lazy
|
||||
# wildcard between anchors still scans to end of input from every start position, which is
|
||||
# quadratic on an adversarial prompt full of "if" with no "then". A first/if..then clause
|
||||
# is short, so the bound loses nothing real.
|
||||
self._deliverable_patterns: tuple[re.Pattern[str], ...] = (
|
||||
re.compile(r"\band also\b"),
|
||||
re.compile(r"\bas well as\b"),
|
||||
re.compile(r"\bin addition\b"),
|
||||
re.compile(r"\balong with\b"),
|
||||
re.compile(r"\bfirst\b.{0,80}?\bthen\b"),
|
||||
re.compile(r"\bstep\s*\d"),
|
||||
)
|
||||
self._multi_hop_patterns: tuple[re.Pattern[str], ...] = (
|
||||
re.compile(r"\bgiven that\b"),
|
||||
re.compile(r"\bbased on (?:the|that)\b"),
|
||||
re.compile(r"\buse that to\b"),
|
||||
re.compile(r"\busing the result\b"),
|
||||
re.compile(r"\bif\b.{0,80}?\bthen\b"),
|
||||
re.compile(r"\bwhich would then\b"),
|
||||
re.compile(r"\band from that\b"),
|
||||
)
|
||||
|
||||
self.adaptive_router: AdaptiveRouter | None = None
|
||||
self._model_tiers: dict[str, tuple[ComplexityTier, ...]] = {}
|
||||
|
|
@ -1080,7 +1171,7 @@ class ComplexityRouter(CustomLogger):
|
|||
def _score_keyword_match(
|
||||
self,
|
||||
text: str,
|
||||
keywords: list[str],
|
||||
keywords: Sequence[str],
|
||||
name: str,
|
||||
signal_label: str,
|
||||
thresholds: tuple[int, int], # (low, high)
|
||||
|
|
@ -1134,7 +1225,12 @@ class ComplexityRouter(CustomLogger):
|
|||
|
||||
def _score_and_classify(
|
||||
self, prompt: str, system_prompt: str | None = None
|
||||
) -> tuple[ComplexityTier, float, tuple[str, ...], Literal["heuristic_scorer", "reasoning_override"]]:
|
||||
) -> tuple[
|
||||
ComplexityTier,
|
||||
float,
|
||||
tuple[str, ...],
|
||||
Literal["heuristic_scorer", "reasoning_override", "insufficient_evidence"],
|
||||
]:
|
||||
"""
|
||||
Classify a prompt by complexity, reporting whether the score chose the tier.
|
||||
|
||||
|
|
@ -1148,6 +1244,9 @@ class ComplexityRouter(CustomLogger):
|
|||
- score: The raw weighted score
|
||||
- signals: List of triggered signals for debugging
|
||||
"""
|
||||
if self.config.scorer_version == 2:
|
||||
return self._score_and_classify_v2(prompt)
|
||||
|
||||
# Score the caller's ask only. The system prompt is a per-session constant, so it
|
||||
# carries no information about how requests within a session differ, yet it
|
||||
# saturates the keyword thresholds (codePresence trips at 2 matches, which any
|
||||
|
|
@ -1230,6 +1329,162 @@ class ComplexityRouter(CustomLogger):
|
|||
|
||||
return tier, weighted_score, tuple(signals), "heuristic_scorer"
|
||||
|
||||
def _effective_dimension_weights_v2(self) -> Mapping[str, float]:
|
||||
"""The v2 weight per dimension: shipped provisional defaults, overridden per key by any
|
||||
v2-named entries in dimension_weights. The config field is default-filled with the v1
|
||||
names, so a plain lookup there would zero every v2 dimension."""
|
||||
configured: Final = self.config.dimension_weights
|
||||
return MappingProxyType(
|
||||
{name: configured.get(name, default) for name, default in DEFAULT_DIMENSION_WEIGHTS_V2.items()}
|
||||
)
|
||||
|
||||
def _score_pattern_count(
|
||||
self,
|
||||
name: str,
|
||||
text: str,
|
||||
patterns: Sequence[re.Pattern[str]],
|
||||
signal_label: str,
|
||||
thresholds: tuple[int, int],
|
||||
scores: tuple[float, float, float],
|
||||
) -> DimensionScore:
|
||||
low_threshold, high_threshold = thresholds
|
||||
score_none, score_low, score_high = scores
|
||||
hits: Final = sum(1 for p in patterns if p.search(text))
|
||||
if hits < low_threshold:
|
||||
return DimensionScore(name, score_none, None)
|
||||
score: Final = score_high if hits >= high_threshold else score_low
|
||||
return DimensionScore(name, score, f"{signal_label} ({hits} markers)")
|
||||
|
||||
def _score_domain_depth(self, text: str) -> tuple[DimensionScore, str | None]:
|
||||
"""Depth of the best-matching knowledge domain, plus which domain it was.
|
||||
|
||||
Terminology is supporting evidence only: one stray term scores nothing, and only the
|
||||
single best domain counts so cross-domain word salad cannot stack."""
|
||||
best_domain: str | None = None
|
||||
best_matches: tuple[str, ...] = ()
|
||||
for domain, keywords in self._domain_keywords.items():
|
||||
matches = tuple(kw for kw in keywords if self._keyword_matches(text, kw))
|
||||
if len(matches) > len(best_matches):
|
||||
best_domain = domain # rebind-ok: running max over the domain loop
|
||||
best_matches = matches # rebind-ok: running max over the domain loop
|
||||
if len(best_matches) < 2:
|
||||
return DimensionScore("domainDepth", 0, None), None
|
||||
detail: Final = ", ".join(best_matches[:3])
|
||||
score: Final = 1.0 if len(best_matches) >= 4 else 0.5
|
||||
return DimensionScore("domainDepth", score, f"domain ({best_domain}: {detail})"), best_domain
|
||||
|
||||
def _score_output_scope(self, text: str) -> DimensionScore:
|
||||
"""Artifacts requested for delivery; counted only next to an authoring verb so naming a
|
||||
report in passing is not an ask to produce one."""
|
||||
if not any(self._keyword_matches(text, verb) for verb in V2_AUTHORING_VERBS):
|
||||
return DimensionScore("outputScope", 0, None)
|
||||
artifacts: Final = tuple(noun for noun in V2_ARTIFACT_NOUNS if self._keyword_matches(text, noun))
|
||||
if not artifacts:
|
||||
return DimensionScore("outputScope", 0, None)
|
||||
score: Final = 1.0 if len(artifacts) >= 2 else 0.4
|
||||
return DimensionScore("outputScope", score, f"artifacts ({', '.join(artifacts[:3])})")
|
||||
|
||||
def _score_context_operation(self, text: str) -> DimensionScore:
|
||||
"""The operation demanded over supplied material. The material's presence scores nothing,
|
||||
and neither do bounded operations (copy, extract, translate, summarize), so a mechanical
|
||||
transformation of a large input stays SIMPLE-eligible."""
|
||||
if not any(marker in text for marker in V2_CONTEXT_MARKERS):
|
||||
return DimensionScore("contextOperation", 0, None)
|
||||
high_op: Final = next((op for op in V2_CONTEXT_OPS_HIGH if self._keyword_matches(text, op)), None)
|
||||
if high_op is not None:
|
||||
return DimensionScore("contextOperation", 1.0, f"context op ({high_op})")
|
||||
medium_op: Final = next((op for op in V2_CONTEXT_OPS_MEDIUM if self._keyword_matches(text, op)), None)
|
||||
if medium_op is not None:
|
||||
return DimensionScore("contextOperation", 0.6, f"context op ({medium_op})")
|
||||
return DimensionScore("contextOperation", 0, None)
|
||||
|
||||
def _task_type_dimension(
|
||||
self, text: str, domain: str | None, has_triviality: bool, has_artifacts: bool
|
||||
) -> DimensionScore:
|
||||
"""Descriptive task category. Always scores 0: the design doc forbids task type as a tier
|
||||
prior, so it rides the signals for evaluation and cohort analysis only."""
|
||||
code_matches: Final = sum(1 for kw in self.code_keywords if self._keyword_matches(text, kw))
|
||||
if code_matches >= 2:
|
||||
return DimensionScore("taskType", 0, "task (code)")
|
||||
if domain is not None:
|
||||
return DimensionScore("taskType", 0, f"task ({domain})")
|
||||
if has_artifacts:
|
||||
return DimensionScore("taskType", 0, "task (writing)")
|
||||
if has_triviality:
|
||||
return DimensionScore("taskType", 0, "task (conversation)")
|
||||
return DimensionScore("taskType", 0, None)
|
||||
|
||||
def _score_and_classify_v2(
|
||||
self, prompt: str
|
||||
) -> tuple[
|
||||
ComplexityTier,
|
||||
float,
|
||||
tuple[str, ...],
|
||||
Literal["heuristic_scorer", "reasoning_override", "insufficient_evidence"],
|
||||
]:
|
||||
"""The scorer_version 2 basis: difficulty evidence from the operations a request demands
|
||||
rather than from software vocabulary, and SIMPLE only on positive evidence of triviality.
|
||||
|
||||
Weights are provisional hand-set defaults pending calibration against public
|
||||
model-outcome data; the mapping semantics are the contract. Like v1, only the caller's
|
||||
ask is scored, never the system prompt."""
|
||||
user_text: Final = prompt.lower()
|
||||
|
||||
triviality, _ = self._score_keyword_match(
|
||||
user_text, self.simple_keywords, "trivialityEvidence", "trivial", (1, 2), (0, -1.0, -1.0)
|
||||
)
|
||||
reasoning, reasoning_match_count = self._score_keyword_match(
|
||||
user_text, self.reasoning_keywords, "reasoningDemand", "reasoning", (1, 2), (0, 0.6, 1.0)
|
||||
)
|
||||
constraints, _ = self._score_keyword_match(
|
||||
user_text, DEFAULT_CONSTRAINT_KEYWORDS, "constraintDensity", "constraints", (2, 4), (0, 0.5, 1.0)
|
||||
)
|
||||
domain_depth, best_domain = self._score_domain_depth(user_text)
|
||||
output_scope: Final = self._score_output_scope(user_text)
|
||||
dimensions: Final[tuple[DimensionScore, ...]] = (
|
||||
triviality,
|
||||
reasoning,
|
||||
constraints,
|
||||
domain_depth,
|
||||
output_scope,
|
||||
self._score_context_operation(user_text),
|
||||
self._score_pattern_count(
|
||||
"deliverableCount", user_text, self._deliverable_patterns, "deliverables", (1, 3), (0, 0.5, 1.0)
|
||||
),
|
||||
self._score_pattern_count(
|
||||
"multiHop", user_text, self._multi_hop_patterns, "multi-hop", (1, 2), (0, 0.5, 1.0)
|
||||
),
|
||||
self._task_type_dimension(
|
||||
user_text,
|
||||
domain=best_domain,
|
||||
has_triviality=triviality.signal is not None,
|
||||
has_artifacts=output_scope.signal is not None,
|
||||
),
|
||||
)
|
||||
|
||||
signals: Final = tuple(d.signal for d in dimensions if d.signal is not None)
|
||||
weights: Final = self._effective_dimension_weights_v2()
|
||||
weighted_score: Final = sum(d.score * weights.get(d.name, 0) for d in dimensions)
|
||||
boundaries: Final = self._effective_tier_boundaries()
|
||||
|
||||
if reasoning_match_count >= 2 and weighted_score >= self._effective_reasoning_override_min_score():
|
||||
return ComplexityTier.REASONING, weighted_score, signals, "reasoning_override"
|
||||
|
||||
if weighted_score >= boundaries["complex_reasoning"]:
|
||||
return ComplexityTier.REASONING, weighted_score, signals, "heuristic_scorer"
|
||||
if weighted_score >= boundaries["medium_complex"]:
|
||||
return ComplexityTier.COMPLEX, weighted_score, signals, "heuristic_scorer"
|
||||
if weighted_score >= boundaries["simple_medium"]:
|
||||
return ComplexityTier.MEDIUM, weighted_score, signals, "heuristic_scorer"
|
||||
|
||||
# Below the boundary, SIMPLE requires positive evidence of triviality with nothing
|
||||
# pointing the other way. A no-match prompt scores 0.0, which is absence of evidence,
|
||||
# not evidence of simplicity, so it and any mixed weak evidence default to MEDIUM.
|
||||
triviality_only: Final = triviality.signal is not None and all(d.score <= 0 for d in dimensions)
|
||||
if triviality_only:
|
||||
return ComplexityTier.SIMPLE, weighted_score, signals, "heuristic_scorer"
|
||||
return ComplexityTier.MEDIUM, weighted_score, signals, "insufficient_evidence"
|
||||
|
||||
def _effective_reasoning_override_min_score(self) -> float:
|
||||
"""The score a request must reach before the reasoning-marker override may promote it.
|
||||
|
||||
|
|
@ -1378,9 +1633,15 @@ class ComplexityRouter(CustomLogger):
|
|||
tier, score, signals, cause = self._score_and_classify(prompt, system_prompt)
|
||||
scored: Final = ClassificationOutcome(tier=tier, score=score, signals=signals, cause=cause)
|
||||
threshold: Final = self.config.heuristic_first_max_tier
|
||||
# Under v2 the scorer names its own abstention: insufficient_evidence means the tier is a
|
||||
# default, not a decision, so it never short-circuits even though a zero-weight taskType
|
||||
# signal may be present. v1 has no such cause and keeps the signals gate.
|
||||
decided_by_evidence: Final = (
|
||||
cause != "insufficient_evidence" if self.config.scorer_version == 2 else bool(signals)
|
||||
)
|
||||
decided_cheaply: Final = (
|
||||
threshold is not None
|
||||
and bool(signals)
|
||||
and decided_by_evidence
|
||||
and self._active_tier_severity(tier) <= self._active_tier_severity(threshold)
|
||||
)
|
||||
if decided_cheaply:
|
||||
|
|
@ -2669,6 +2930,65 @@ class ComplexityRouter(CustomLogger):
|
|||
caller_scope: Final = self._get_user_api_key_hash_from_request_kwargs(request_kwargs) or "unscoped"
|
||||
return f"complexity_router_session_affinity:v1:{self.model_name}:{caller_scope}:{session_id}"
|
||||
|
||||
def _inherited_conversation_tier(self, resolved_messages: Sequence[Mapping[str, object]]) -> ComplexityTier | None:
|
||||
"""The tier the conversation's most recent informative ask earns under v2, or None.
|
||||
|
||||
Walks prior human asks newest first (the newest is the one that just abstained) and
|
||||
rescores each with the same v2 scorer; the first whose outcome is evidence-based and
|
||||
above SIMPLE names the tier a low-information continuation inherits. Trivial
|
||||
interjections ("thanks", "ok") never reset the working tier, and a conversation with no
|
||||
informative ask inherits nothing. Stateless by design: everything is derived from the
|
||||
request's own history, so this needs no session id, no cache and no opt-in."""
|
||||
asks: Final = _iter_human_asks_newest_first(resolved_messages, self._reminder_markers)
|
||||
next(asks, None)
|
||||
return next(
|
||||
(
|
||||
prior_tier
|
||||
for text in islice(asks, 25)
|
||||
for prior_tier, _, _, prior_cause in (self._score_and_classify_v2(text),)
|
||||
if prior_cause in ("heuristic_scorer", "reasoning_override") and prior_tier != ComplexityTier.SIMPLE
|
||||
),
|
||||
None,
|
||||
)
|
||||
|
||||
def _apply_continuation_inheritance(
|
||||
self,
|
||||
tier: ComplexityTier | str,
|
||||
cause: RoutingDecisionCause,
|
||||
resolved_messages: Sequence[Mapping[str, object]] | None,
|
||||
signals: tuple[str, ...],
|
||||
) -> tuple[ComplexityTier | str, RoutingDecisionCause, tuple[str, ...]]:
|
||||
"""A low-information continuation inherits the conversation's tier instead of a default.
|
||||
|
||||
On agentic traffic the newest message is usually terminal or tool output, which carries
|
||||
no evidence; routing every such turn on its own weak score overrides the tier the real
|
||||
ask earned. Only the v2 scorer's own two low-information outcomes qualify: a MEDIUM
|
||||
abstention (insufficient_evidence), and a SIMPLE, which v2 emits solely on triviality
|
||||
evidence, so terminal noise like "okay" no longer drops the turn to the cheapest model
|
||||
mid-task. Every other outcome is a real decision and is left alone: v1 routers never
|
||||
inherit (spend must not move on a router that did not opt in), and a v2 housekeeping,
|
||||
LLM-classifier, plugin, or heuristic_first placement already decided the turn is cheap.
|
||||
Inheritance only fires when a prior informative ask exists, so a genuine standalone
|
||||
greeting still routes SIMPLE, and the inherited tier is always at least as strong as the
|
||||
scored one. A consecutive failure streak raises it one step, so a stalled run reaches a
|
||||
stronger model without any external state."""
|
||||
low_information: Final = self.config.scorer_version == 2 and (
|
||||
cause == "insufficient_evidence" or (cause == "heuristic_scorer" and tier == ComplexityTier.SIMPLE)
|
||||
)
|
||||
if not low_information or resolved_messages is None:
|
||||
return tier, cause, signals
|
||||
inherited: Final = self._inherited_conversation_tier(resolved_messages)
|
||||
if inherited is None:
|
||||
return tier, cause, signals
|
||||
streak: Final = _failure_streak(resolved_messages)
|
||||
if streak < _FAILURE_STREAK_TO_ESCALATE:
|
||||
return inherited, "insufficient_evidence_inherit", (*signals, "inherited-tier")
|
||||
return (
|
||||
self._escalate_tier(inherited),
|
||||
"insufficient_evidence_inherit",
|
||||
(*signals, "inherited-tier", f"progress-stall ({streak} failing turns)"),
|
||||
)
|
||||
|
||||
@property
|
||||
def _uses_tier_pin(self) -> bool:
|
||||
"""classification_mode 'user_turn' implies the tier pin machinery: the pin write after each
|
||||
|
|
@ -3036,6 +3356,11 @@ class ComplexityRouter(CustomLogger):
|
|||
)
|
||||
)
|
||||
tier, score, signals = outcome.tier, outcome.score, outcome.signals
|
||||
tier, decided_cause, signals = (
|
||||
self._apply_continuation_inheritance( # rebind-ok: staged tier resolution, matching the escalation and floor rebinding below
|
||||
tier, outcome.cause, resolved_messages, signals
|
||||
)
|
||||
)
|
||||
classified_tier: Final = tier
|
||||
if escalation_keyword is not None:
|
||||
tier = self._escalate_tier(tier)
|
||||
|
|
@ -3159,7 +3484,7 @@ class ComplexityRouter(CustomLogger):
|
|||
if outcome.cause == "default_model_fallback" and self.config.plugins
|
||||
else signals
|
||||
)
|
||||
decision_cause: Final[RoutingDecisionCause] = "plan_mode" if plan_floored else outcome.cause
|
||||
decision_cause: Final[RoutingDecisionCause] = "plan_mode" if plan_floored else decided_cause
|
||||
decision_keyword: Final = (
|
||||
plan_mode_sentinel if plan_floored else (housekeeping_sentinel if outcome.cause == "housekeeping" else None)
|
||||
)
|
||||
|
|
|
|||
|
|
@ -383,6 +383,281 @@ DEFAULT_SIMPLE_KEYWORDS: Final[list[str]] = [
|
|||
# Note: "ok" removed due to false positives (matches "token", "book", etc.)
|
||||
]
|
||||
|
||||
# scorer_version 2 drops keywords that fire on plain English outside their intended meaning:
|
||||
# "let"/"class"/"return" match math and everyday prose. Under v2 the code list only feeds the
|
||||
# zero-weight taskType category, never a tier.
|
||||
V2_PRUNED_CODE_KEYWORDS: Final[frozenset[str]] = frozenset(
|
||||
("let", "class", "return", "request", "query", "error", "git", "var")
|
||||
)
|
||||
DEFAULT_CODE_KEYWORDS_V2: Final[tuple[str, ...]] = tuple(
|
||||
k for k in DEFAULT_CODE_KEYWORDS if k not in V2_PRUNED_CODE_KEYWORDS
|
||||
)
|
||||
|
||||
# trivialityEvidence, the only v2 dimension that can produce SIMPLE: greetings, acknowledgements,
|
||||
# and bounded transformations. Question openers ("what is", "how many") mark phrasing, not task
|
||||
# difficulty (GSM8K word problems open with them), so none of them survive from the v1 list.
|
||||
DEFAULT_SIMPLE_KEYWORDS_V2: Final[tuple[str, ...]] = (
|
||||
"hello",
|
||||
"hi",
|
||||
"hey",
|
||||
"thanks",
|
||||
"thank you",
|
||||
"goodbye",
|
||||
"bye",
|
||||
"okay",
|
||||
"translate",
|
||||
"rewrite",
|
||||
"reword",
|
||||
"rephrase",
|
||||
"reformat",
|
||||
"proofread",
|
||||
"fix the typo",
|
||||
"fix this typo",
|
||||
"fix the grammar",
|
||||
"convert this to",
|
||||
"uppercase",
|
||||
"lowercase",
|
||||
)
|
||||
|
||||
# reasoningDemand: the operations a request requires (derivation, proof, diagnosis, optimization,
|
||||
# causal inference), not just chain-of-thought stock phrases. Extends the v1 phrasebook.
|
||||
DEFAULT_REASONING_DEMAND_KEYWORDS: Final[tuple[str, ...]] = (
|
||||
*DEFAULT_REASONING_KEYWORDS,
|
||||
"prove",
|
||||
"derive",
|
||||
"diagnose",
|
||||
"justify",
|
||||
"reconcile",
|
||||
"show that",
|
||||
"walk me through",
|
||||
"explain why",
|
||||
"determine whether",
|
||||
"what follows from",
|
||||
)
|
||||
|
||||
# domainDepth: terminology per knowledge domain, widened past software. Supporting evidence only:
|
||||
# the score comes from the single best-matching domain and needs multiple hits to fire.
|
||||
DEFAULT_DOMAIN_KEYWORDS: Final[Mapping[str, tuple[str, ...]]] = MappingProxyType(
|
||||
{
|
||||
"math": (
|
||||
"theorem",
|
||||
"lemma",
|
||||
"proof",
|
||||
"prime",
|
||||
"integer",
|
||||
"congruent",
|
||||
"modulo",
|
||||
"mod",
|
||||
"polynomial",
|
||||
"derivative",
|
||||
"integral",
|
||||
"matrix",
|
||||
"eigenvalue",
|
||||
"probability",
|
||||
"geometry",
|
||||
"algebra",
|
||||
"calculus",
|
||||
"equation",
|
||||
"inequality",
|
||||
"real roots",
|
||||
),
|
||||
"physical_science": (
|
||||
"molecule",
|
||||
"atom",
|
||||
"electron",
|
||||
"reaction",
|
||||
"compound",
|
||||
"chemical",
|
||||
"thermodynamics",
|
||||
"entropy",
|
||||
"quantum",
|
||||
"velocity",
|
||||
"acceleration",
|
||||
"magnetic",
|
||||
"spectrum",
|
||||
"resonance",
|
||||
"isotope",
|
||||
"catalyst",
|
||||
"oxidation",
|
||||
"chemical shift",
|
||||
"periodic system",
|
||||
"periodic table",
|
||||
),
|
||||
"medicine": (
|
||||
"diagnosis",
|
||||
"differential",
|
||||
"symptom",
|
||||
"patient",
|
||||
"dosage",
|
||||
"contraindication",
|
||||
"prognosis",
|
||||
"pathology",
|
||||
"syndrome",
|
||||
"lesion",
|
||||
"tachycardia",
|
||||
"hypotension",
|
||||
"infarction",
|
||||
"etiology",
|
||||
"clinical",
|
||||
"workup",
|
||||
"presents with",
|
||||
"treatment pathway",
|
||||
),
|
||||
"law": (
|
||||
"statute",
|
||||
"statutory",
|
||||
"plaintiff",
|
||||
"defendant",
|
||||
"liability",
|
||||
"tort",
|
||||
"remedy",
|
||||
"remedies",
|
||||
"jurisdiction",
|
||||
"precedent",
|
||||
"negligence",
|
||||
"damages",
|
||||
"lease",
|
||||
"landlord",
|
||||
"tenant",
|
||||
"injunction",
|
||||
"indemnify",
|
||||
),
|
||||
"finance": (
|
||||
"portfolio",
|
||||
"equity",
|
||||
"liquidity",
|
||||
"amortization",
|
||||
"arbitrage",
|
||||
"hedge",
|
||||
"valuation",
|
||||
"discounted cash flow",
|
||||
"balance sheet",
|
||||
"ebitda",
|
||||
"yield curve",
|
||||
"volatility",
|
||||
),
|
||||
"data": (
|
||||
"dataset",
|
||||
"regression",
|
||||
"correlation",
|
||||
"seasonality",
|
||||
"forecast",
|
||||
"variance",
|
||||
"median",
|
||||
"percentile",
|
||||
"cohort",
|
||||
"time series",
|
||||
"distribution",
|
||||
"outlier",
|
||||
"sample size",
|
||||
"confidence interval",
|
||||
"decelerating",
|
||||
),
|
||||
"engineering": tuple(DEFAULT_TECHNICAL_KEYWORDS),
|
||||
}
|
||||
)
|
||||
|
||||
# constraintDensity: markers of independent requirements. Raw counts are a provisional detector
|
||||
# per the design doc; the two-hit floor keeps casual "must"/"ensure" prose from firing.
|
||||
DEFAULT_CONSTRAINT_KEYWORDS: Final[tuple[str, ...]] = (
|
||||
"must",
|
||||
"at least",
|
||||
"at most",
|
||||
"exactly",
|
||||
"no more than",
|
||||
"no fewer than",
|
||||
"without using",
|
||||
"ensure",
|
||||
"requirement",
|
||||
"requirements",
|
||||
"constraint",
|
||||
"adhering to",
|
||||
"within the limit",
|
||||
"addressing at least",
|
||||
"spell",
|
||||
"lands exactly",
|
||||
)
|
||||
|
||||
# outputScope: artifacts the caller names for delivery. Only counted next to an authoring verb.
|
||||
V2_AUTHORING_VERBS: Final[tuple[str, ...]] = (
|
||||
"write",
|
||||
"compose",
|
||||
"draft",
|
||||
"create",
|
||||
"generate",
|
||||
"produce",
|
||||
"build",
|
||||
"prepare",
|
||||
)
|
||||
V2_ARTIFACT_NOUNS: Final[tuple[str, ...]] = (
|
||||
"essay",
|
||||
"report",
|
||||
"table",
|
||||
"poem",
|
||||
"sonnet",
|
||||
"plan",
|
||||
"outline",
|
||||
"presentation",
|
||||
"slide",
|
||||
"email",
|
||||
"letter",
|
||||
"memo",
|
||||
"script",
|
||||
"function",
|
||||
"module",
|
||||
"test suite",
|
||||
"diagram",
|
||||
"schema",
|
||||
"spec",
|
||||
"article",
|
||||
"blog post",
|
||||
)
|
||||
|
||||
# contextOperation: what must be done WITH supplied material. The context marker alone scores
|
||||
# nothing; copy/extract/translate/summarize score nothing either, so bounded transformations of
|
||||
# big inputs stay SIMPLE-eligible per the design doc.
|
||||
V2_CONTEXT_MARKERS: Final[tuple[str, ...]] = (
|
||||
"here is",
|
||||
"here's",
|
||||
"the following",
|
||||
"below",
|
||||
"given the",
|
||||
"this table",
|
||||
"this data",
|
||||
"attached",
|
||||
"```",
|
||||
)
|
||||
V2_CONTEXT_OPS_HIGH: Final[tuple[str, ...]] = (
|
||||
"reconcile",
|
||||
"synthesize",
|
||||
"diagnose",
|
||||
"forecast",
|
||||
"identify which",
|
||||
"compare",
|
||||
)
|
||||
V2_CONTEXT_OPS_MEDIUM: Final[tuple[str, ...]] = (
|
||||
"analyze",
|
||||
"interpret",
|
||||
"explain",
|
||||
)
|
||||
|
||||
# Provisional hand-set weights for the v2 dimension basis; fitting them against public
|
||||
# model-outcome data is the staged follow-up. taskType deliberately carries no weight: task
|
||||
# type is descriptive context, never a tier prior.
|
||||
DEFAULT_DIMENSION_WEIGHTS_V2: Final[Mapping[str, float]] = MappingProxyType(
|
||||
{
|
||||
"reasoningDemand": 0.22,
|
||||
"domainDepth": 0.18,
|
||||
"contextOperation": 0.15,
|
||||
"constraintDensity": 0.13,
|
||||
"multiHop": 0.10,
|
||||
"deliverableCount": 0.08,
|
||||
"outputScope": 0.07,
|
||||
"trivialityEvidence": 0.07,
|
||||
"taskType": 0.0,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ─── Default Dimension Weights ───
|
||||
|
||||
|
|
@ -607,7 +882,20 @@ class ComplexityRouterConfig(BaseModel):
|
|||
)
|
||||
simple_keywords: list[str] | None = Field(
|
||||
default=None,
|
||||
description="Keywords indicating simple/basic queries",
|
||||
description=(
|
||||
"Keywords indicating simple/basic queries. Under scorer_version 2 this list feeds "
|
||||
"trivialityEvidence, the only dimension that can produce SIMPLE"
|
||||
),
|
||||
)
|
||||
|
||||
domain_keywords: Mapping[str, tuple[str, ...]] | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Per-domain terminology for scorer_version 2's domainDepth dimension, replacing or "
|
||||
"extending the shipped domains (math, physical_science, medicine, law, finance, data, "
|
||||
"engineering). A domain named here replaces the shipped list for that domain; other "
|
||||
"shipped domains stay active. Requires scorer_version 2, rejected otherwise"
|
||||
),
|
||||
)
|
||||
|
||||
# Default model if scoring fails
|
||||
|
|
@ -624,6 +912,23 @@ class ComplexityRouterConfig(BaseModel):
|
|||
),
|
||||
)
|
||||
|
||||
scorer_version: Literal[1, 2] = Field(
|
||||
default=1,
|
||||
description=(
|
||||
"Heuristic scorer semantics, wherever the scorer runs (classifier_type 'heuristic', the "
|
||||
"local half of 'heuristic_first', and the heuristic classifier_fallback). Version 1 maps "
|
||||
"any score below simple_medium to SIMPLE, so a prompt matching no keyword scores 0.0 and "
|
||||
"routes to the cheapest tier by default. Version 2 scores a different dimension basis "
|
||||
"(reasoningDemand, domainDepth, constraintDensity, deliverableCount, outputScope, "
|
||||
"contextOperation, multiHop, trivialityEvidence, plus a zero-weight taskType category), "
|
||||
"requires positive evidence of triviality for SIMPLE, and defaults no-evidence and "
|
||||
"below-boundary traffic to MEDIUM (cause 'insufficient_evidence'). v2 weights are "
|
||||
"provisional hand-set defaults pending calibration; override them by v2 dimension name "
|
||||
"in dimension_weights. Opt-in: existing routers stay on version 1, so routing and spend "
|
||||
"do not change on upgrade"
|
||||
),
|
||||
)
|
||||
|
||||
# Classifier strategy
|
||||
classifier_type: Literal["heuristic", "llm", "custom", "heuristic_first"] = Field(
|
||||
default="heuristic",
|
||||
|
|
@ -1092,6 +1397,15 @@ class ComplexityRouterConfig(BaseModel):
|
|||
return value.strip()
|
||||
return value
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_domain_keywords_need_v2(self) -> "ComplexityRouterConfig":
|
||||
if self.domain_keywords is not None and self.scorer_version != 2:
|
||||
raise ValueError(
|
||||
"domain_keywords is set but scorer_version is 1; the v1 scorer has no domainDepth "
|
||||
"dimension so the lists would never be read. Set scorer_version 2 or remove domain_keywords"
|
||||
)
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_heuristic_first_max_tier(self) -> "ComplexityRouterConfig":
|
||||
if self.classifier_type != "heuristic_first":
|
||||
|
|
|
|||
|
|
@ -11,7 +11,9 @@ import os
|
|||
import sys
|
||||
|
||||
# ruff: noqa: T201
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../../..")))
|
||||
|
|
@ -31,7 +33,7 @@ class EvalCase:
|
|||
description: str
|
||||
system_prompt: str | None = None
|
||||
# Allow some flexibility - if actual tier is in acceptable_tiers, still passes
|
||||
acceptable_tiers: list[ComplexityTier] | None = None
|
||||
acceptable_tiers: Sequence[ComplexityTier] | None = None
|
||||
|
||||
|
||||
# ─── Evaluation Dataset ───
|
||||
|
|
@ -230,31 +232,150 @@ EVAL_CASES: Final[list[EvalCase]] = [
|
|||
),
|
||||
]
|
||||
|
||||
# scorer_version 2 cases: hard prompts with no software vocabulary must not land SIMPLE,
|
||||
# triviality evidence still does, and the pruned keywords no longer fire.
|
||||
V2_ROUTER_CONFIG: Final[Mapping[str, object]] = MappingProxyType({"scorer_version": 2})
|
||||
|
||||
def run_eval() -> tuple[int, int, list[dict]]:
|
||||
V2_EVAL_CASES: Final[tuple[EvalCase, ...]] = (
|
||||
EvalCase(
|
||||
prompt="hi",
|
||||
expected_tier=ComplexityTier.SIMPLE,
|
||||
description="Greeting stays SIMPLE on triviality evidence",
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Thanks, that fixed it!",
|
||||
expected_tier=ComplexityTier.SIMPLE,
|
||||
description="Acknowledgement stays SIMPLE on triviality evidence",
|
||||
),
|
||||
EvalCase(
|
||||
prompt="A reaction of a liquid organic compound, which molecules consist of carbon and "
|
||||
"hydrogen atoms, is performed at 80 centigrade and 20 bar for 24 hours. In the proton "
|
||||
"nuclear magnetic resonance spectrum, the signals with the highest chemical shift of the "
|
||||
"reactant are replaced by a signal of the product that is observed about three to four "
|
||||
"units downfield. Compounds from which position in the periodic system of the elements "
|
||||
"have most likely been initially added in small amounts?",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Graduate chemistry (no SWE vocabulary) must not default SIMPLE",
|
||||
acceptable_tiers=(ComplexityTier.COMPLEX, ComplexityTier.REASONING),
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Prove that there are infinitely many primes p such that p is congruent to 3 mod 4.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Number-theory proof must not default SIMPLE",
|
||||
acceptable_tiers=(ComplexityTier.COMPLEX, ComplexityTier.REASONING),
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Our landlord entered the apartment without notice three times last month and "
|
||||
"changed the locks once. We are in California. Walk me through every remedy available "
|
||||
"to us, the statutory notice requirements the landlord violated, and whether we can "
|
||||
"terminate the lease early without penalty.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Legal analysis must not default SIMPLE",
|
||||
acceptable_tiers=(ComplexityTier.COMPLEX, ComplexityTier.REASONING),
|
||||
),
|
||||
EvalCase(
|
||||
prompt="A 54 year old man presents with crushing substernal chest pain radiating to the "
|
||||
"left arm, diaphoresis, and nausea for 45 minutes. Blood pressure 88/60, heart rate 118. "
|
||||
"Describe the differential diagnosis, the immediate workup, and the treatment pathway if "
|
||||
"the ECG shows ST elevation in leads II, III and aVF.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Clinical case must not default SIMPLE",
|
||||
acceptable_tiers=(ComplexityTier.COMPLEX, ComplexityTier.REASONING),
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Compose a 1500 word persuasive essay arguing that urban car bans improve public "
|
||||
"health, addressing at least three counterarguments and citing the strongest empirical "
|
||||
"evidence on each side.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Long-form constrained writing must not default SIMPLE",
|
||||
acceptable_tiers=(ComplexityTier.COMPLEX, ComplexityTier.REASONING),
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Compose a Shakespearean sonnet about the ocean in iambic pentameter where the "
|
||||
"first letters of each line spell OCEANWAVESRIDE, and the volta lands exactly at line 9.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Constrained verse must not default SIMPLE",
|
||||
acceptable_tiers=(ComplexityTier.COMPLEX, ComplexityTier.REASONING),
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Five houses in a row are painted different colors, and their owners have "
|
||||
"different pets, drinks, and jobs. Given these twelve clues, determine who owns the fish.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Logic puzzle must not default SIMPLE",
|
||||
acceptable_tiers=(ComplexityTier.COMPLEX, ComplexityTier.REASONING),
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Here is a table of monthly revenue by region for the last three years. Identify "
|
||||
"which regions are decelerating, whether seasonality explains the Q4 spikes, and "
|
||||
"forecast next quarter with your reasoning.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Data analysis must not default SIMPLE",
|
||||
acceptable_tiers=(ComplexityTier.COMPLEX, ComplexityTier.REASONING),
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Natalia sold clips to 48 of her friends in April, and then she sold half as many "
|
||||
"clips in May. How many clips did Natalia sell altogether in April and May?",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Math word problem: pruned 'how many' no longer marks it trivial",
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Let f(x) = x^3 - 3x + 1. Find all real roots.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Pruned 'let' no longer reads math as code",
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Our payment service loses events when the broker partitions; design an "
|
||||
"exactly-once delivery approach across three regions given we cannot use distributed "
|
||||
"transactions.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Hard distributed-systems prompt must not default SIMPLE",
|
||||
acceptable_tiers=(ComplexityTier.COMPLEX, ComplexityTier.REASONING),
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Write a python function that returns the max of a list.",
|
||||
expected_tier=ComplexityTier.MEDIUM,
|
||||
description="Code detection still works after the prune",
|
||||
),
|
||||
EvalCase(
|
||||
prompt="Step by step, explain your reasoning as you evaluate whether we should use "
|
||||
"PostgreSQL or MongoDB for our new project. Consider our requirements: complex queries, "
|
||||
"high write volume, and eventual consistency is acceptable.",
|
||||
expected_tier=ComplexityTier.REASONING,
|
||||
description="Reasoning override still fires under v2",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def run_eval(
|
||||
cases: Sequence[EvalCase] | None = None,
|
||||
complexity_router_config: Mapping[str, object] | None = None,
|
||||
title: str = "COMPLEXITY ROUTER EVALUATION",
|
||||
) -> tuple[int, int, list[dict]]:
|
||||
"""
|
||||
Run the evaluation suite.
|
||||
|
||||
Returns:
|
||||
Tuple of (passed, total, failures)
|
||||
"""
|
||||
eval_cases: Final = cases if cases is not None else EVAL_CASES
|
||||
# Create router with default config
|
||||
mock_router: Final = MagicMock()
|
||||
router: Final = ComplexityRouter(
|
||||
model_name="eval-router",
|
||||
litellm_router_instance=mock_router,
|
||||
complexity_router_config=complexity_router_config,
|
||||
)
|
||||
|
||||
passed = 0
|
||||
total: Final = len(EVAL_CASES)
|
||||
total: Final = len(eval_cases)
|
||||
failures: Final = []
|
||||
|
||||
print("=" * 70)
|
||||
print("COMPLEXITY ROUTER EVALUATION")
|
||||
print(title)
|
||||
print("=" * 70)
|
||||
print()
|
||||
|
||||
for i, case in enumerate(EVAL_CASES, 1):
|
||||
for i, case in enumerate(eval_cases, 1):
|
||||
tier, score, signals = router.classify(case.prompt, case.system_prompt)
|
||||
|
||||
# Check if pass
|
||||
|
|
@ -308,21 +429,29 @@ def run_eval() -> tuple[int, int, list[dict]]:
|
|||
return passed, total, failures
|
||||
|
||||
|
||||
def _gate(pass_rate: float, name: str) -> bool:
|
||||
"""Report one suite's verdict; True when it is a hard failure."""
|
||||
if pass_rate < 0.80:
|
||||
print(f"\n{name} FAILED: Pass rate {pass_rate:.1%} is below 80% threshold")
|
||||
return True
|
||||
if pass_rate < 0.90:
|
||||
print(f"\n{name} WARNING: Pass rate {pass_rate:.1%} is below 90%")
|
||||
return False
|
||||
print(f"\n{name} PASSED: Pass rate {pass_rate:.1%}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point."""
|
||||
passed, total, failures = run_eval()
|
||||
v1_passed, v1_total, _ = run_eval(title="COMPLEXITY ROUTER EVALUATION (scorer_version 1, default)")
|
||||
v2_passed, v2_total, _ = run_eval(
|
||||
cases=V2_EVAL_CASES,
|
||||
complexity_router_config=V2_ROUTER_CONFIG,
|
||||
title="COMPLEXITY ROUTER EVALUATION (scorer_version 2)",
|
||||
)
|
||||
|
||||
# Exit with error code if too many failures
|
||||
pass_rate: Final = passed / total
|
||||
if pass_rate < 0.80:
|
||||
print(f"\n❌ EVAL FAILED: Pass rate {pass_rate:.1%} is below 80% threshold")
|
||||
sys.exit(1)
|
||||
elif pass_rate < 0.90:
|
||||
print(f"\n⚠️ EVAL WARNING: Pass rate {pass_rate:.1%} is below 90%")
|
||||
sys.exit(0)
|
||||
else:
|
||||
print(f"\n✅ EVAL PASSED: Pass rate {pass_rate:.1%}")
|
||||
sys.exit(0)
|
||||
failed: Final = _gate(v1_passed / v1_total, "V1 EVAL") | _gate(v2_passed / v2_total, "V2 EVAL")
|
||||
sys.exit(1 if failed else 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -2844,6 +2844,16 @@ RoutingDecisionCause = Literal[
|
|||
# that tells a reader the score did NOT choose the tier; encoding it as free text
|
||||
# meant anything that filtered `signals` silently changed what the row claimed.
|
||||
"reasoning_override",
|
||||
# scorer_version 2 only: the score sat below simple_medium without positive evidence of
|
||||
# triviality, so the scorer defaulted to MEDIUM instead of mapping the score. Empty
|
||||
# `signals` means no dimension fired at all; non-empty means mixed or weak evidence.
|
||||
"insufficient_evidence",
|
||||
# scorer_version 2 only: the scorer abstained (see insufficient_evidence) on a continuation
|
||||
# turn, so the request inherited the tier of the conversation's most recent informative ask
|
||||
# instead of the MEDIUM default, derived statelessly from the request's own history. A
|
||||
# consecutive-failure streak in recent turns raises the inherited tier one step, reported in
|
||||
# signals as progress-stall.
|
||||
"insufficient_evidence_inherit",
|
||||
"llm_classifier",
|
||||
# classifier_type 'heuristic_first': the local scorer produced at least one signal and landed at
|
||||
# or below heuristic_first_max_tier, so it decided the tier and the LLM classifier was never
|
||||
|
|
|
|||
|
|
@ -653,6 +653,536 @@ class TestConfigOverrides:
|
|||
assert any("long" in s.lower() if s else False for s in signals), f"Expected 'long' signal, got {signals}"
|
||||
|
||||
|
||||
# Hard prompt with no keyword matches under either scorer version: every dimension stays
|
||||
# silent, so the score is exactly 0.0 and the tier is whatever the mapping defaults to.
|
||||
V2_NO_EVIDENCE_PROMPT = (
|
||||
"Our payment service loses events when the broker partitions; design an exactly-once "
|
||||
"delivery approach across three regions given we cannot rely on coordinated commits."
|
||||
)
|
||||
|
||||
|
||||
def _versioned_router(mock_router_instance, scorer_version: int, **config_overrides):
|
||||
config = {
|
||||
"tiers": dict(HEURISTIC_FIRST_TIERS),
|
||||
"tier_boundaries": dict(HEURISTIC_FIRST_BOUNDARIES),
|
||||
"scorer_version": scorer_version,
|
||||
**config_overrides,
|
||||
}
|
||||
return ComplexityRouter(
|
||||
model_name="test-complexity-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config=config,
|
||||
)
|
||||
|
||||
|
||||
class TestScorerVersion2:
|
||||
"""scorer_version 2 requires positive evidence of triviality for SIMPLE and defaults
|
||||
below-boundary traffic to MEDIUM; version 1 (the default) keeps mapping score 0.0 to
|
||||
SIMPLE, so existing routers do not move."""
|
||||
|
||||
def test_no_evidence_defaults_medium_under_v2_and_simple_under_v1(self, mock_router_instance):
|
||||
v1_tier, v1_score, v1_signals = _versioned_router(mock_router_instance, 1).classify(V2_NO_EVIDENCE_PROMPT)
|
||||
assert (v1_tier, v1_score, v1_signals) == (ComplexityTier.SIMPLE, 0.0, [])
|
||||
|
||||
tier, score, signals, cause = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
V2_NO_EVIDENCE_PROMPT
|
||||
)
|
||||
assert (tier, score, signals) == (ComplexityTier.MEDIUM, 0.0, ())
|
||||
assert cause == "insufficient_evidence"
|
||||
|
||||
def test_absent_scorer_version_defaults_to_v1(self, mock_router_instance):
|
||||
router = ComplexityRouter(
|
||||
model_name="test-complexity-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={"tiers": dict(HEURISTIC_FIRST_TIERS)},
|
||||
)
|
||||
assert router.config.scorer_version == 1
|
||||
tier, _, _ = router.classify(V2_NO_EVIDENCE_PROMPT)
|
||||
assert tier == ComplexityTier.SIMPLE
|
||||
|
||||
def test_v2_mixed_evidence_below_boundary_defaults_medium(self, mock_router_instance):
|
||||
"""A weak positive signal is not triviality evidence: the prompt must not land SIMPLE
|
||||
even though its score sits below simple_medium."""
|
||||
prompt = (
|
||||
"first water the plants on the balcony and then take the recycling bins out "
|
||||
"to the curb before the truck arrives tomorrow morning"
|
||||
)
|
||||
tier, score, signals, cause = _versioned_router(mock_router_instance, 2)._score_and_classify(prompt)
|
||||
assert any(s.startswith("deliverables") for s in signals)
|
||||
assert score < HEURISTIC_FIRST_BOUNDARIES["simple_medium"]
|
||||
assert tier == ComplexityTier.MEDIUM
|
||||
assert cause == "insufficient_evidence"
|
||||
v1_tier, _, _ = _versioned_router(mock_router_instance, 1).classify(prompt)
|
||||
assert v1_tier == ComplexityTier.SIMPLE
|
||||
|
||||
def test_v2_triviality_evidence_still_lands_simple(self, mock_router_instance):
|
||||
tier, _, signals, cause = _versioned_router(mock_router_instance, 2)._score_and_classify("hi")
|
||||
assert tier == ComplexityTier.SIMPLE
|
||||
assert cause == "heuristic_scorer"
|
||||
assert any(s.startswith("trivial") for s in signals)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"prompt, pruned_signal_label",
|
||||
[
|
||||
("Let f(x) = x^3 - 3x + 1. Find all real roots of the cubic on the real line.", "code"),
|
||||
(
|
||||
"Natalia sold clips to 48 of her friends in April, and then she sold half as many "
|
||||
"clips in May. How many clips did Natalia sell altogether in April and May?",
|
||||
"simple",
|
||||
),
|
||||
("a) buy milk from the store b) walk the dog around the block twice", "multi-step"),
|
||||
],
|
||||
)
|
||||
def test_v2_prunes_false_positive_detectors(self, mock_router_instance, prompt, pruned_signal_label):
|
||||
"""Keywords and patterns that fire on plain English are gone from the v2 defaults but
|
||||
keep firing under v1."""
|
||||
_, _, v1_signals = _versioned_router(mock_router_instance, 1).classify(prompt)
|
||||
assert any(pruned_signal_label in s for s in v1_signals)
|
||||
_, _, v2_signals = _versioned_router(mock_router_instance, 2).classify(prompt)
|
||||
assert not any(pruned_signal_label in s for s in v2_signals)
|
||||
|
||||
def test_v2_operator_keyword_overrides_still_win(self, mock_router_instance):
|
||||
"""An operator-supplied list replaces the defaults under both versions, so opting into
|
||||
v2 never silently discards configured keywords."""
|
||||
router = _versioned_router(mock_router_instance, 2, simple_keywords=["never mind that"])
|
||||
tier, _, signals, _ = router._score_and_classify("never mind that")
|
||||
assert tier == ComplexityTier.SIMPLE
|
||||
assert any(s.startswith("trivial") for s in signals)
|
||||
default_tier, _, _, cause = _versioned_router(mock_router_instance, 2)._score_and_classify("never mind that")
|
||||
assert (default_tier, cause) == (ComplexityTier.MEDIUM, "insufficient_evidence")
|
||||
|
||||
def test_v2_code_detection_survives_the_prune(self, mock_router_instance):
|
||||
tier, _, signals = _versioned_router(mock_router_instance, 2).classify(
|
||||
"Write a python function that returns the max of a list."
|
||||
)
|
||||
assert tier == ComplexityTier.MEDIUM
|
||||
assert "task (code)" in signals
|
||||
|
||||
def test_v2_domain_depth_needs_multiple_hits(self, mock_router_instance):
|
||||
"""Terminology is supporting evidence only: a clinical case with several medicine terms
|
||||
fires, a lookup with one technical term does not (the doc's easy-lookup hard negative)."""
|
||||
_, _, signals, _ = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
"A patient presents with tachycardia; describe the differential diagnosis and workup."
|
||||
)
|
||||
assert any(s.startswith("domain (medicine") for s in signals)
|
||||
_, _, lookup_signals, cause = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
"what port does https use"
|
||||
)
|
||||
assert not any(s.startswith("domain") for s in lookup_signals)
|
||||
assert cause == "insufficient_evidence"
|
||||
|
||||
def test_v2_constraint_words_quoted_do_not_fire(self, mock_router_instance):
|
||||
"""The doc's hard negative: naming a constraint word once is not an imposed constraint."""
|
||||
_, _, signals, _ = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
"How often does the word must appear in the US constitution?"
|
||||
)
|
||||
assert not any(s.startswith("constraints") for s in signals)
|
||||
_, _, constrained_signals, _ = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
"The summary must cover every chapter, use at most 200 words, and end with exactly one question."
|
||||
)
|
||||
assert any(s.startswith("constraints") for s in constrained_signals)
|
||||
|
||||
def test_v2_context_operation_scores_the_operation_not_the_material(self, mock_router_instance):
|
||||
"""Supplied material scores nothing by itself, and a bounded extraction over it stays
|
||||
SIMPLE-eligible; only a judgment-heavy operation counts (the doc's mechanical-extraction
|
||||
hard negative)."""
|
||||
v2 = _versioned_router(mock_router_instance, 2)
|
||||
_, _, signals, _ = v2._score_and_classify(
|
||||
"Here is a table of quarterly ledger entries. Reconcile the two accounts and flag gaps."
|
||||
)
|
||||
assert any(s.startswith("context op") for s in signals)
|
||||
_, _, bounded_signals, _ = v2._score_and_classify(
|
||||
"Here is a table of quarterly ledger entries. List the vendor names that appear in it."
|
||||
)
|
||||
assert not any(s.startswith("context op") for s in bounded_signals)
|
||||
|
||||
def test_v2_bounded_translation_of_supplied_text_stays_simple(self, mock_router_instance):
|
||||
tier, _, signals, cause = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
"Translate the following paragraph into French: We hereby agree to the terms."
|
||||
)
|
||||
assert tier == ComplexityTier.SIMPLE
|
||||
assert cause == "heuristic_scorer"
|
||||
assert any(s.startswith("trivial") for s in signals)
|
||||
|
||||
def test_v2_multi_hop_and_deliverables_detect_structure(self, mock_router_instance):
|
||||
_, _, signals, _ = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
"Given that the cache misses, compute the fallback latency, and if it exceeds budget "
|
||||
"then use that to size the pool."
|
||||
)
|
||||
assert any(s.startswith("multi-hop") for s in signals)
|
||||
_, _, deliverable_signals, _ = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
"Draft the launch announcement and also prepare the rollback checklist for the release."
|
||||
)
|
||||
assert any(s.startswith("deliverables") for s in deliverable_signals)
|
||||
|
||||
def test_v2_task_type_never_moves_the_tier(self, mock_router_instance):
|
||||
"""taskType is descriptive context: it may be the only signal and the score stays 0.0, so
|
||||
the request still defaults instead of short-circuiting on a category label."""
|
||||
tier, score, signals, cause = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
"the pipeline is emitting duplicates"
|
||||
)
|
||||
assert score == 0.0
|
||||
assert (tier, cause) == (ComplexityTier.MEDIUM, "insufficient_evidence")
|
||||
|
||||
def test_v2_weights_overridable_by_v2_name_and_default_sum_is_one(self, mock_router_instance):
|
||||
from litellm.router_strategy.complexity_router.config import DEFAULT_DIMENSION_WEIGHTS_V2
|
||||
|
||||
assert sum(DEFAULT_DIMENSION_WEIGHTS_V2.values()) == pytest.approx(1.0)
|
||||
baseline, _, _, _ = _versioned_router(mock_router_instance, 2)._score_and_classify(
|
||||
"Prove that there are infinitely many primes p such that p is congruent to 3 mod 4."
|
||||
)
|
||||
zeroed = _versioned_router(
|
||||
mock_router_instance, 2, dimension_weights={"reasoningDemand": 0.0, "domainDepth": 0.0}
|
||||
)
|
||||
_, zeroed_score, _, zeroed_cause = zeroed._score_and_classify(
|
||||
"Prove that there are infinitely many primes p such that p is congruent to 3 mod 4."
|
||||
)
|
||||
assert baseline == ComplexityTier.MEDIUM
|
||||
assert zeroed_score == 0.0
|
||||
assert zeroed_cause == "insufficient_evidence"
|
||||
|
||||
def test_v2_domain_keywords_override_replaces_one_domain(self, mock_router_instance):
|
||||
router = _versioned_router(
|
||||
mock_router_instance, 2, domain_keywords={"gastronomy": ["sourdough starter", "lamination"]}
|
||||
)
|
||||
_, _, signals, _ = router._score_and_classify(
|
||||
"My sourdough starter collapses during lamination; walk me through why."
|
||||
)
|
||||
assert any(s.startswith("domain (gastronomy") for s in signals)
|
||||
|
||||
def test_domain_keywords_rejected_under_v1(self):
|
||||
with pytest.raises(ValidationError, match="domain_keywords is set but scorer_version is 1"):
|
||||
ComplexityRouterConfig(domain_keywords={"math": ["theorem"]})
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_medium_default_resolves_like_v1_when_medium_tier_unconfigured(self, mock_router_instance):
|
||||
"""A tier set without MEDIUM serves the v2 insufficient-evidence default through
|
||||
default_model, exactly as a v1 mid-band MEDIUM score does (get_model_for_tier owns both),
|
||||
and the deployment path guarantees default_model at registration (router.py derives it
|
||||
from fallback_tier, MEDIUM, then SIMPLE, and raises without one)."""
|
||||
v2 = ComplexityRouter(
|
||||
model_name="t",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"scorer_version": 2,
|
||||
"tiers": {"SIMPLE": "cheap-model", "REASONING": "big-model"},
|
||||
},
|
||||
default_model="cheap-model",
|
||||
)
|
||||
response = await v2.async_pre_routing_hook(
|
||||
model="t", request_kwargs={}, messages=[{"role": "user", "content": V2_NO_EVIDENCE_PROMPT}]
|
||||
)
|
||||
assert response is not None
|
||||
assert response.model == "cheap-model"
|
||||
assert response.routing_decision is not None
|
||||
assert response.routing_decision["tier"] == "MEDIUM"
|
||||
assert response.routing_decision["cause"] == "insufficient_evidence"
|
||||
|
||||
v1 = ComplexityRouter(
|
||||
model_name="t",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={"tiers": {"SIMPLE": "cheap-model", "REASONING": "big-model"}},
|
||||
default_model="cheap-model",
|
||||
)
|
||||
v1_response = await v1.async_pre_routing_hook(
|
||||
model="t",
|
||||
request_kwargs={},
|
||||
messages=[{"role": "user", "content": "Write a python function that returns the max of a list."}],
|
||||
)
|
||||
assert v1_response is not None
|
||||
assert v1_response.model == "cheap-model"
|
||||
assert v1_response.routing_decision is not None
|
||||
assert v1_response.routing_decision["tier"] == "MEDIUM"
|
||||
|
||||
@pytest.mark.parametrize("bad_version", [0, 3])
|
||||
def test_rejects_unknown_scorer_version(self, bad_version):
|
||||
with pytest.raises(ValidationError):
|
||||
ComplexityRouterConfig(scorer_version=bad_version)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_decision_record_carries_insufficient_evidence_cause(self, mock_router_instance):
|
||||
"""PreRoutingHookResponse validates routing_decision against RoutingDecisionCause, so
|
||||
this pins that the new cause survives the per-request path end to end."""
|
||||
router = _versioned_router(mock_router_instance, 2)
|
||||
response = await router.async_pre_routing_hook(
|
||||
model="test-complexity-router",
|
||||
request_kwargs={},
|
||||
messages=[{"role": "user", "content": V2_NO_EVIDENCE_PROMPT}],
|
||||
)
|
||||
assert response is not None
|
||||
decision = response.routing_decision
|
||||
assert decision is not None
|
||||
assert decision["cause"] == "insufficient_evidence"
|
||||
assert decision["tier"] == "MEDIUM"
|
||||
assert decision["routed_model"] == response.model == "gpt-4o"
|
||||
assert decision["score"] == 0.0
|
||||
assert "signals" not in decision
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_heuristic_first_no_signal_still_abstains_to_classifier(self, mock_router_instance):
|
||||
mock_router_instance.acompletion = AsyncMock(return_value=_llm_response('{"tier": "REASONING"}'))
|
||||
router = _heuristic_first_router(mock_router_instance, scorer_version=2)
|
||||
outcome = await router.aclassify(V2_NO_EVIDENCE_PROMPT)
|
||||
assert outcome.cause == "llm_classifier"
|
||||
assert outcome.tier == ComplexityTier.REASONING
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v2_heuristic_first_trivial_prompt_still_short_circuits(self, mock_router_instance):
|
||||
mock_router_instance.acompletion = AsyncMock(return_value=_llm_response('{"tier": "REASONING"}'))
|
||||
router = _heuristic_first_router(mock_router_instance, scorer_version=2)
|
||||
outcome = await router.aclassify("thanks!")
|
||||
assert outcome.cause == "heuristic_first_short_circuit"
|
||||
assert outcome.tier == ComplexityTier.SIMPLE
|
||||
mock_router_instance.acompletion.assert_not_called()
|
||||
|
||||
|
||||
TERMINAL_DUMP = (
|
||||
"total 64\ndrwxr-xr-x 2 root root 4096 Sep 2 01:01 .\n-rw-r--r-- 1 root root 812 main.go\n"
|
||||
"go: downloading github.com/hashicorp/raft v1.3.0\ngo build ./...\nall checks green"
|
||||
)
|
||||
FAILING_DUMP = "go build ./...\nmain.go:14:2: undefined: raft.Config\nmake: *** [build] Error 2"
|
||||
EVIDENCE_TASK = (
|
||||
"Diagnose why the raft consensus service loses linearizability during partitions and prove "
|
||||
"the fix preserves it. Reconcile the quorum math with the deployment topology and justify "
|
||||
"each step."
|
||||
)
|
||||
MEDIUM_EVIDENCE_TASK = (
|
||||
"A patient presents with tachycardia and hypotension; describe the differential diagnosis and the immediate workup."
|
||||
)
|
||||
|
||||
|
||||
def _continuation_router(mock_router_instance):
|
||||
return ComplexityRouter(
|
||||
model_name="smart",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"scorer_version": 2,
|
||||
"tiers": {"SIMPLE": "haiku", "MEDIUM": "sonnet", "COMPLEX": "opus", "REASONING": "opus-high"},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
async def _turn(router, messages):
|
||||
response = await router.async_pre_routing_hook(model="smart", request_kwargs={}, messages=messages)
|
||||
assert response is not None
|
||||
assert response.routing_decision is not None
|
||||
return response
|
||||
|
||||
|
||||
class TestInsufficientEvidenceInheritance:
|
||||
"""A v2 abstention on a continuation turn inherits the conversation's tier instead of the
|
||||
MEDIUM default, derived statelessly from the request's own history (no session id, no cache,
|
||||
no opt-in, unlike session_affinity). A consecutive-failure streak in recent turns raises the
|
||||
inherited tier one step."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_low_information_continuation_inherits_the_conversation_tier(self, mock_router_instance):
|
||||
router = _continuation_router(mock_router_instance)
|
||||
follow_up = await _turn(
|
||||
router,
|
||||
[
|
||||
{"role": "user", "content": EVIDENCE_TASK},
|
||||
{"role": "assistant", "content": "Running the build to see the failure."},
|
||||
{"role": "user", "content": TERMINAL_DUMP},
|
||||
],
|
||||
)
|
||||
assert follow_up.model == "opus-high"
|
||||
assert follow_up.routing_decision["tier"] == "REASONING"
|
||||
assert follow_up.routing_decision["cause"] == "insufficient_evidence_inherit"
|
||||
assert "inherited-tier" in follow_up.routing_decision["signals"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trivial_interjections_do_not_reset_the_working_tier(self, mock_router_instance):
|
||||
"""The walk skips prior asks whose only evidence is triviality, so a mid-session
|
||||
"thanks!" cannot down-route the continuation that follows it."""
|
||||
router = _continuation_router(mock_router_instance)
|
||||
follow_up = await _turn(
|
||||
router,
|
||||
[
|
||||
{"role": "user", "content": EVIDENCE_TASK},
|
||||
{"role": "assistant", "content": "done"},
|
||||
{"role": "user", "content": "thanks!"},
|
||||
{"role": "assistant", "content": "anything else?"},
|
||||
{"role": "user", "content": TERMINAL_DUMP},
|
||||
],
|
||||
)
|
||||
assert follow_up.model == "opus-high"
|
||||
assert follow_up.routing_decision["cause"] == "insufficient_evidence_inherit"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_triviality_word_in_continuation_output_still_inherits(self, mock_router_instance):
|
||||
"""A continuation that scores SIMPLE only because its output contains a triviality word
|
||||
(terminal noise like "okay") is still a low-information turn, so it inherits rather than
|
||||
dropping to the cheapest model mid-task."""
|
||||
router = _continuation_router(mock_router_instance)
|
||||
follow_up = await _turn(
|
||||
router,
|
||||
[
|
||||
{"role": "user", "content": EVIDENCE_TASK},
|
||||
{"role": "assistant", "content": "building"},
|
||||
{"role": "user", "content": "okay"},
|
||||
],
|
||||
)
|
||||
assert follow_up.model == "opus-high"
|
||||
assert follow_up.routing_decision["tier"] == "REASONING"
|
||||
assert follow_up.routing_decision["cause"] == "insufficient_evidence_inherit"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_v1_router_never_inherits(self, mock_router_instance):
|
||||
"""Inheritance is a v2-only behavior: a v1 router that lands SIMPLE on a continuation
|
||||
keeps SIMPLE, so opting nobody in leaves v1 routing and spend exactly as before."""
|
||||
v1 = ComplexityRouter(
|
||||
model_name="smart",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"tiers": {"SIMPLE": "haiku", "MEDIUM": "sonnet", "COMPLEX": "opus", "REASONING": "opus-high"}
|
||||
},
|
||||
)
|
||||
follow_up = await _turn(
|
||||
v1,
|
||||
[
|
||||
{"role": "user", "content": EVIDENCE_TASK},
|
||||
{"role": "assistant", "content": "building"},
|
||||
{"role": "user", "content": "okay"},
|
||||
],
|
||||
)
|
||||
assert follow_up.model == "haiku"
|
||||
assert follow_up.routing_decision["tier"] == "SIMPLE"
|
||||
assert follow_up.routing_decision["cause"] == "heuristic_scorer"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_housekeeping_placement_is_never_overridden_by_inheritance(self, mock_router_instance):
|
||||
"""A housekeeping title turn after real work is a deliberate cheap placement, so
|
||||
inheritance must leave it on the cheapest tier rather than billing the prior tier."""
|
||||
router = ComplexityRouter(
|
||||
model_name="smart",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"scorer_version": 2,
|
||||
"route_housekeeping_to_cheapest_tier": True,
|
||||
"tiers": {"SIMPLE": "haiku", "MEDIUM": "sonnet", "COMPLEX": "opus", "REASONING": "opus-high"},
|
||||
},
|
||||
)
|
||||
follow_up = await _turn(
|
||||
router,
|
||||
[
|
||||
{"role": "user", "content": EVIDENCE_TASK},
|
||||
{"role": "assistant", "content": "done"},
|
||||
{"role": "user", "content": "Write the title in the predominant language of the session"},
|
||||
],
|
||||
)
|
||||
assert follow_up.model == "haiku"
|
||||
assert follow_up.routing_decision["cause"] == "housekeeping"
|
||||
|
||||
def test_two_anchor_patterns_are_bounded_against_adversarial_input(self, mock_router_instance):
|
||||
"""The if/first..then patterns cap the gap so a long prompt with no closing anchor cannot
|
||||
force a quadratic scan."""
|
||||
import time
|
||||
|
||||
router = _continuation_router(mock_router_instance)
|
||||
adversarial = "if " + "x " * 60000 + "banana"
|
||||
start = time.monotonic()
|
||||
router._score_and_classify(adversarial)
|
||||
assert time.monotonic() - start < 2.0
|
||||
# the bound still matches a real short clause
|
||||
_, _, signals, _ = router._score_and_classify("compute the latency and if it exceeds budget then size the pool")
|
||||
assert any(s.startswith("multi-hop") for s in signals)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_standalone_greeting_still_routes_simple(self, mock_router_instance):
|
||||
"""Inheritance needs a prior informative ask, so a genuine one-off greeting or bounded
|
||||
transformation with no history still routes SIMPLE rather than inheriting anything."""
|
||||
router = _continuation_router(mock_router_instance)
|
||||
greeting = await _turn(router, [{"role": "user", "content": "hi there"}])
|
||||
assert greeting.model == "haiku"
|
||||
assert greeting.routing_decision["cause"] == "heuristic_scorer"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_conversation_with_no_informative_ask_keeps_the_medium_default(self, mock_router_instance):
|
||||
router = _continuation_router(mock_router_instance)
|
||||
first = await _turn(router, [{"role": "user", "content": TERMINAL_DUMP}])
|
||||
assert first.model == "sonnet"
|
||||
assert first.routing_decision["cause"] == "insufficient_evidence"
|
||||
second = await _turn(
|
||||
router,
|
||||
[
|
||||
{"role": "user", "content": TERMINAL_DUMP},
|
||||
{"role": "assistant", "content": "ok"},
|
||||
{"role": "user", "content": "drwxr-xr-x 2 root root 4096 Sep 2 lib"},
|
||||
],
|
||||
)
|
||||
assert second.model == "sonnet"
|
||||
assert second.routing_decision["cause"] == "insufficient_evidence"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failure_streak_escalates_the_inherited_tier_one_step(self, mock_router_instance):
|
||||
router = _continuation_router(mock_router_instance)
|
||||
history = [{"role": "user", "content": MEDIUM_EVIDENCE_TASK}]
|
||||
for _ in range(3):
|
||||
history = [
|
||||
*history,
|
||||
{"role": "assistant", "content": "retrying"},
|
||||
{"role": "user", "content": FAILING_DUMP},
|
||||
]
|
||||
stalled = await _turn(router, history)
|
||||
assert stalled.model == "opus"
|
||||
assert stalled.routing_decision["tier"] == "COMPLEX"
|
||||
assert stalled.routing_decision["cause"] == "insufficient_evidence_inherit"
|
||||
assert any(s.startswith("progress-stall") for s in stalled.routing_decision["signals"])
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_short_failure_streak_inherits_without_escalating(self, mock_router_instance):
|
||||
router = _continuation_router(mock_router_instance)
|
||||
two_failures = [
|
||||
{"role": "user", "content": MEDIUM_EVIDENCE_TASK},
|
||||
{"role": "assistant", "content": "retrying"},
|
||||
{"role": "user", "content": FAILING_DUMP},
|
||||
{"role": "assistant", "content": "retrying"},
|
||||
{"role": "user", "content": FAILING_DUMP},
|
||||
]
|
||||
inherited = await _turn(router, two_failures)
|
||||
assert inherited.model == "sonnet"
|
||||
assert inherited.routing_decision["tier"] == "MEDIUM"
|
||||
assert inherited.routing_decision["cause"] == "insufficient_evidence_inherit"
|
||||
assert not any(s.startswith("progress-stall") for s in inherited.routing_decision["signals"])
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_successful_output_breaks_the_failure_streak(self, mock_router_instance):
|
||||
"""A clean run between failures resets the count: escalation needs CONSECUTIVE failures,
|
||||
so flaky-but-progressing sessions stay on the inherited tier."""
|
||||
router = _continuation_router(mock_router_instance)
|
||||
history = [
|
||||
{"role": "user", "content": MEDIUM_EVIDENCE_TASK},
|
||||
{"role": "assistant", "content": "retrying"},
|
||||
{"role": "user", "content": FAILING_DUMP},
|
||||
{"role": "assistant", "content": "retrying"},
|
||||
{"role": "user", "content": FAILING_DUMP},
|
||||
{"role": "assistant", "content": "fixed the import"},
|
||||
{"role": "user", "content": TERMINAL_DUMP},
|
||||
{"role": "assistant", "content": "one more check"},
|
||||
{"role": "user", "content": FAILING_DUMP},
|
||||
]
|
||||
response = await _turn(router, history)
|
||||
assert response.routing_decision["tier"] == "MEDIUM"
|
||||
assert not any(s.startswith("progress-stall") for s in response.routing_decision["signals"])
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_informative_continuation_still_decides_for_itself(self, mock_router_instance):
|
||||
"""Inheritance only replaces the abstention default: a continuation carrying real
|
||||
evidence classifies normally."""
|
||||
router = _continuation_router(mock_router_instance)
|
||||
response = await _turn(
|
||||
router,
|
||||
[
|
||||
{"role": "user", "content": MEDIUM_EVIDENCE_TASK},
|
||||
{"role": "assistant", "content": "ok"},
|
||||
{"role": "user", "content": EVIDENCE_TASK},
|
||||
],
|
||||
)
|
||||
assert response.model == "opus-high"
|
||||
assert response.routing_decision["cause"] == "reasoning_override"
|
||||
|
||||
|
||||
class TestCustomTechnicalKeywords:
|
||||
"""Test the custom_technical_keywords config option."""
|
||||
|
||||
|
|
@ -2410,9 +2940,7 @@ class TestRouterPreRoutingAliasOverrides:
|
|||
import time
|
||||
|
||||
monkeypatch.setenv("GITHUB_COPILOT_TOKEN_DIR", str(tmp_path))
|
||||
(tmp_path / "api-key.json").write_text(
|
||||
json.dumps({"token": "tid=test", "expires_at": int(time.time()) + 3600})
|
||||
)
|
||||
(tmp_path / "api-key.json").write_text(json.dumps({"token": "tid=test", "expires_at": int(time.time()) + 3600}))
|
||||
router = Router(
|
||||
model_list=[
|
||||
{
|
||||
|
|
@ -2437,7 +2965,9 @@ class TestRouterPreRoutingAliasOverrides:
|
|||
copilot_resolutions: List = []
|
||||
|
||||
def _guarded(*args, **kwargs):
|
||||
target = str(kwargs.get("model") or (args[0] if args else "")) + str(kwargs.get("custom_llm_provider") or "")
|
||||
target = str(kwargs.get("model") or (args[0] if args else "")) + str(
|
||||
kwargs.get("custom_llm_provider") or ""
|
||||
)
|
||||
if "github_copilot" in target:
|
||||
copilot_resolutions.append(target)
|
||||
raise RuntimeError("routing must not resolve an authenticating provider")
|
||||
|
|
@ -7467,7 +7997,6 @@ class TestClientHousekeepingCalls:
|
|||
assert result is not None
|
||||
assert result.model == "claude-sonnet-4-20250514"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_classifier_plugin_still_decides_its_own_routers(self, mock_router_instance):
|
||||
"""A plugin is where an operator encodes policy the tier ladder cannot express.
|
||||
|
|
@ -7502,9 +8031,7 @@ class TestClientHousekeepingCalls:
|
|||
assert result.model == "o1-preview"
|
||||
assert result.routing_decision["cause"] == "classifier_plugin"
|
||||
|
||||
def _adaptive_router(
|
||||
self, tier_distance_penalty: float, plan_mode_min_tier: str | None = None
|
||||
) -> ComplexityRouter:
|
||||
def _adaptive_router(self, tier_distance_penalty: float, plan_mode_min_tier: str | None = None) -> ComplexityRouter:
|
||||
adaptive_instance = MagicMock()
|
||||
adaptive_instance.model_list = [
|
||||
{
|
||||
|
|
@ -7541,9 +8068,7 @@ class TestClientHousekeepingCalls:
|
|||
return router
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_bandit_cannot_route_a_housekeeping_call_above_the_cheapest_tier(
|
||||
self, mock_router_instance
|
||||
):
|
||||
async def test_the_bandit_cannot_route_a_housekeeping_call_above_the_cheapest_tier(self, mock_router_instance):
|
||||
"""The tier here is what the request IS, not how hard it is, so the bandit has nothing to win.
|
||||
|
||||
Without a ceiling the tier distance penalty is the only thing holding the tier, so a
|
||||
|
|
@ -7576,7 +8101,6 @@ class TestClientHousekeepingCalls:
|
|||
assert result is not None
|
||||
assert result.model == "premium"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_housekeeping_call_never_becomes_the_session_pin(self, mock_router_instance):
|
||||
"""Pinning this is the most expensive mistake of the transient causes.
|
||||
|
|
@ -7618,9 +8142,7 @@ class TestClientHousekeepingCalls:
|
|||
assert work_turn.routing_decision["cause"] == "llm_classifier"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_decision_records_which_sentinel_matched(
|
||||
self, mock_router_instance, llm_classifier_config
|
||||
):
|
||||
async def test_the_decision_records_which_sentinel_matched(self, mock_router_instance, llm_classifier_config):
|
||||
"""The cause's contract says the sentinel rides in matched_keyword, so it has to be there.
|
||||
|
||||
Without it an operator reading the logs can see that a call was treated as housekeeping but
|
||||
|
|
@ -7641,7 +8163,6 @@ class TestClientHousekeepingCalls:
|
|||
"Write the title in the predominant language of the session"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_plan_mode_floor_raises_a_housekeeping_call_under_adaptive(self, mock_router_instance):
|
||||
"""Floor and ceiling must not contradict each other on the same request.
|
||||
|
|
@ -10367,7 +10888,9 @@ class TestContextWindowEscalation:
|
|||
copilot_resolutions: List = []
|
||||
|
||||
def _guarded(*args, **kwargs):
|
||||
target = str(kwargs.get("model") or (args[0] if args else "")) + str(kwargs.get("custom_llm_provider") or "")
|
||||
target = str(kwargs.get("model") or (args[0] if args else "")) + str(
|
||||
kwargs.get("custom_llm_provider") or ""
|
||||
)
|
||||
if "github_copilot" in target:
|
||||
copilot_resolutions.append(target)
|
||||
raise RuntimeError("the gate must not resolve an authenticating provider")
|
||||
|
|
|
|||
|
|
@ -71,6 +71,42 @@ describe("RoutingDecisionCard", () => {
|
|||
expect(screen.queryByText(/SIMPLE|MEDIUM|COMPLEX|at or above/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("labels a scorer v2 defaulted row and does not claim the score met a boundary", () => {
|
||||
render(
|
||||
<RoutingDecisionCard
|
||||
decision={{
|
||||
...heuristic,
|
||||
cause: "insufficient_evidence",
|
||||
tier: "MEDIUM",
|
||||
score: 0.0,
|
||||
signals: [],
|
||||
}}
|
||||
/>,
|
||||
);
|
||||
expect(screen.getByText("Heuristic scorer, defaulted on insufficient evidence")).toBeInTheDocument();
|
||||
expect(screen.getByText("0.00")).toBeInTheDocument();
|
||||
// The score sits below simple_medium, so the band copy would claim SIMPLE while the
|
||||
// badge says MEDIUM: it must not render on a defaulted row.
|
||||
expect(screen.queryByText(/below 0.15/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("labels a conversation-inherited abstention and does not claim the score met a boundary", () => {
|
||||
render(
|
||||
<RoutingDecisionCard
|
||||
decision={{
|
||||
...heuristic,
|
||||
cause: "insufficient_evidence_inherit",
|
||||
tier: "REASONING",
|
||||
score: 0.0,
|
||||
signals: [],
|
||||
}}
|
||||
/>,
|
||||
);
|
||||
expect(screen.getByText("Heuristic scorer abstained, inherited the conversation's tier")).toBeInTheDocument();
|
||||
expect(screen.getByText("0.00")).toBeInTheDocument();
|
||||
expect(screen.queryByText(/below 0.15/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("names the judge model on the LLM classifier path and shows no score", () => {
|
||||
render(
|
||||
<RoutingDecisionCard
|
||||
|
|
|
|||
|
|
@ -84,6 +84,8 @@ function describeReasoningOverride(tierLabel: string | undefined, floor: number
|
|||
|
||||
const CONSTANT_CAUSE_LABELS: Record<string, string> = {
|
||||
heuristic_scorer: "Heuristic scorer",
|
||||
insufficient_evidence: "Heuristic scorer, defaulted on insufficient evidence",
|
||||
insufficient_evidence_inherit: "Heuristic scorer abstained, inherited the conversation's tier",
|
||||
heuristic_first_short_circuit: "Heuristic scorer, classifier skipped",
|
||||
classifier_plugin: "Custom classifier plugin",
|
||||
semantic_keyword_match: "Semantic keyword match",
|
||||
|
|
@ -170,11 +172,17 @@ export function RoutingDecisionCard({
|
|||
tier_boundaries: tierBoundaries,
|
||||
} = decision;
|
||||
|
||||
// On an override row the score did not decide the tier, so showing it against a
|
||||
// boundary would claim something untrue. Keyed off the cause rather than a marker
|
||||
// inside `signals`, which redaction can remove.
|
||||
// On these rows the score did not decide the tier (an override forced it, or scorer v2
|
||||
// defaulted below the boundary), so showing it against a boundary would claim something
|
||||
// untrue. Keyed off the cause rather than a marker inside `signals`, which redaction can
|
||||
// remove.
|
||||
const scoreDidNotDecide =
|
||||
decision.cause === "reasoning_override" ||
|
||||
decision.cause === "plan_mode" ||
|
||||
decision.cause === "insufficient_evidence" ||
|
||||
decision.cause === "insufficient_evidence_inherit";
|
||||
const scoreExplanation =
|
||||
score !== undefined && decision.cause !== "reasoning_override" && decision.cause !== "plan_mode"
|
||||
score !== undefined && !scoreDidNotDecide
|
||||
? describeScoreAgainstBoundaries(score, tierBoundaries, tierLabel !== undefined)
|
||||
: null;
|
||||
|
||||
|
|
|
|||
18
ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
18
ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
|
|
@ -34431,6 +34431,13 @@ export interface components {
|
|||
dimension_weights?: {
|
||||
[key: string]: number;
|
||||
};
|
||||
/**
|
||||
* Domain Keywords
|
||||
* @description Per-domain terminology for scorer_version 2's domainDepth dimension, replacing or extending the shipped domains (math, physical_science, medicine, law, finance, data, engineering). A domain named here replaces the shipped list for that domain; other shipped domains stay active. Requires scorer_version 2, rejected otherwise
|
||||
*/
|
||||
domain_keywords?: {
|
||||
[key: string]: string[];
|
||||
} | null;
|
||||
/**
|
||||
* Embedding Model
|
||||
* @description Embedding model (LiteLLM model name) used when semantic_keyword_matching is enabled
|
||||
|
|
@ -34521,6 +34528,13 @@ export interface components {
|
|||
* @default true
|
||||
*/
|
||||
route_housekeeping_to_cheapest_tier: boolean;
|
||||
/**
|
||||
* Scorer Version
|
||||
* @description Heuristic scorer semantics, wherever the scorer runs (classifier_type 'heuristic', the local half of 'heuristic_first', and the heuristic classifier_fallback). Version 1 maps any score below simple_medium to SIMPLE, so a prompt matching no keyword scores 0.0 and routes to the cheapest tier by default. Version 2 scores a different dimension basis (reasoningDemand, domainDepth, constraintDensity, deliverableCount, outputScope, contextOperation, multiHop, trivialityEvidence, plus a zero-weight taskType category), requires positive evidence of triviality for SIMPLE, and defaults no-evidence and below-boundary traffic to MEDIUM (cause 'insufficient_evidence'). v2 weights are provisional hand-set defaults pending calibration; override them by v2 dimension name in dimension_weights. Opt-in: existing routers stay on version 1, so routing and spend do not change on upgrade
|
||||
* @default 1
|
||||
* @enum {integer}
|
||||
*/
|
||||
scorer_version: 1 | 2;
|
||||
/**
|
||||
* Semantic Keyword Matching
|
||||
* @description Match keyword_tier_rules by embedding similarity instead of literal text
|
||||
|
|
@ -34541,7 +34555,7 @@ export interface components {
|
|||
session_affinity_ttl_seconds: number;
|
||||
/**
|
||||
* Simple Keywords
|
||||
* @description Keywords indicating simple/basic queries
|
||||
* @description Keywords indicating simple/basic queries. Under scorer_version 2 this list feeds trivialityEvidence, the only dimension that can produce SIMPLE
|
||||
*/
|
||||
simple_keywords?: string[] | null;
|
||||
/**
|
||||
|
|
@ -35651,7 +35665,7 @@ export interface components {
|
|||
* Cause
|
||||
* @enum {string}
|
||||
*/
|
||||
cause?: "heuristic_scorer" | "reasoning_override" | "llm_classifier" | "heuristic_first_short_circuit" | "classifier_plugin" | "classifier_fallback" | "default_model_fallback" | "literal_keyword_match" | "semantic_keyword_match" | "plan_mode" | "housekeeping" | "modality_escalation" | "session_affinity_pin" | "session_affinity_escalation" | "user_turn_continuation" | "default_fallback" | "keyword" | "quality_tier" | "bandit";
|
||||
cause?: "heuristic_scorer" | "reasoning_override" | "insufficient_evidence" | "insufficient_evidence_inherit" | "llm_classifier" | "heuristic_first_short_circuit" | "classifier_plugin" | "classifier_fallback" | "default_model_fallback" | "literal_keyword_match" | "semantic_keyword_match" | "plan_mode" | "housekeeping" | "modality_escalation" | "session_affinity_pin" | "session_affinity_escalation" | "user_turn_continuation" | "default_fallback" | "keyword" | "quality_tier" | "bandit";
|
||||
/** Classifier Cost */
|
||||
classifier_cost?: number;
|
||||
/** Classifier Model */
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue