mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
fix(complexity_router): route no-signal prompts to default_tier, not SIMPLE
The heuristic scorer has no way to say "I don't know". Its seven dimensions are a roughly 100-word software-vocabulary whitelist, so when a prompt matches none of them every dimension contributes 0, the weighted sum is 0.0, and 0.0 sits below simple_medium (0.15). Absence of evidence was being scored as evidence of simplicity, and unclassifiable traffic went to the cheapest tier. On a graded 809-question benchmark that no-signal mass is 50.2% of prompts; on a 257-session agent transcript corpus it is 36.6% of turns _score_and_classify now returns config.default_tier before the band mapping when nothing was recognised, under its own cause=no_signal_default with signals=['no-signal'], so a spend log row says the score did not choose the tier, the way reasoning_override already does. default_tier is new on ComplexityRouterConfig and defaults to MEDIUM; setting it to SIMPLE restores the previous behavior exactly. The LLM classifier falls back to this scorer on timeout or error, so the same setting decides where unmatched traffic lands during a classifier outage The branch tests the signals rather than the score, and rather than the individual dimension scores. A weighted score of zero does not mean nothing was recognised, since contributions cancel: "hi, quick python question" comes to zero with three dimensions firing and is real evidence of a simple request. A dimension scoring zero does not mean it stayed silent either, because _score_keyword_match takes the no-match score as a parameter, so a future dimension with a nonzero baseline would silently kill the branch. A signal is the one thing a dimension emits only when it recognised something, and a test pins that invariant across every scorer An explicitly configured default_tier with no model behind it is rejected at load rather than surfacing as a routing error on the first unmatched request. The check is skipped when the field is left implicit, so a partial tiers map keeps loading as it does today instead of turning an upgrade into a startup failure The router e2e config pins default_tier: SIMPLE. That test tells a live LLM classifier apart from a silent fallback by which backend served the request, and its prompt has no scoring signal, so leaving the new default in place would have made both arms land on the same backend and turned the test into a false green
This commit is contained in:
Tin Chi Lo
parent
3c2264cfac
commit
2d25d23566
7 changed files with 456 additions and 4 deletions
|
|
@ -34,6 +34,8 @@ The weighted sum is mapped to tiers using configurable boundaries:
|
|||
| COMPLEX | 0.35 - 0.60 | Technical, multi-part requests |
|
||||
| REASONING | > 0.60 | Chain-of-thought, analysis |
|
||||
|
||||
A prompt that matches nothing at all never reaches that table. When no dimension fires, the scorer has no evidence in either direction, so the request goes to `default_tier` (MEDIUM unless you change it) rather than falling through a score of 0.0 into SIMPLE. See [No-Signal Default](#no-signal-default) below.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Basic Configuration
|
||||
|
|
@ -71,6 +73,9 @@ model_list:
|
|||
simple_medium: 0.15
|
||||
medium_complex: 0.35
|
||||
complex_reasoning: 0.60
|
||||
|
||||
# Tier for prompts that match nothing at all (default: MEDIUM)
|
||||
default_tier: MEDIUM
|
||||
|
||||
# Token count thresholds
|
||||
token_thresholds:
|
||||
|
|
@ -126,6 +131,23 @@ response = litellm.completion(
|
|||
|
||||
## Special Behaviors
|
||||
|
||||
### No-Signal Default
|
||||
|
||||
The scorer only recognizes what is on its keyword lists, so a prompt can be genuinely hard and still match nothing: a logic puzzle, a business tradeoff, a piece of prose to edit. Every dimension then scores 0, and treating that silence as evidence of simplicity is how unmatched traffic used to end up on the cheapest model.
|
||||
|
||||
When no dimension fires, the router returns `default_tier` instead of consulting the boundaries, and logs the classification with `signals=['no-signal']`:
|
||||
|
||||
```yaml
|
||||
complexity_router_config:
|
||||
default_tier: MEDIUM # SIMPLE | MEDIUM | COMPLEX | REASONING
|
||||
```
|
||||
|
||||
Set `default_tier: SIMPLE` to keep unmatched traffic on the cheapest tier, which is how the router behaved before this setting existed. A `default_tier` you set explicitly has to have a model behind it, either its own non-empty entry in `tiers` or a `default_model`; the config is rejected at load time otherwise, rather than failing on the first unmatched request.
|
||||
|
||||
The check is on the individual dimensions, not on the weighted score, because contributions cancel. `"hi, quick python question"` scores zero with three dimensions firing (short prompt, a simple indicator, a code keyword); it has real evidence of being simple and stays SIMPLE. Prompts shorter than the `simple` token threshold or longer than the `complex` one also fire `tokenCount`, so they score normally and are outside this path.
|
||||
|
||||
The LLM classifier falls back to this scorer when it times out or errors, so `default_tier` also decides where unmatched traffic lands during a classifier outage.
|
||||
|
||||
### Reasoning Override
|
||||
|
||||
If 2+ reasoning markers are detected in the user message, the request is automatically routed to the REASONING tier regardless of the weighted score. This ensures complex reasoning tasks get the appropriate model.
|
||||
|
|
|
|||
|
|
@ -42,6 +42,7 @@ from .config import (
|
|||
DEFAULT_REASONING_KEYWORDS,
|
||||
DEFAULT_SIMPLE_KEYWORDS,
|
||||
DEFAULT_TECHNICAL_KEYWORDS,
|
||||
NO_SIGNAL_MARKER,
|
||||
TIER_SEVERITY_ORDER,
|
||||
ComplexityRouterConfig,
|
||||
ComplexityTier,
|
||||
|
|
@ -473,10 +474,27 @@ class ComplexityRouter(CustomLogger):
|
|||
|
||||
def _score_and_classify(
|
||||
self, prompt: str, system_prompt: str | None = None
|
||||
) -> tuple[ComplexityTier, float, tuple[str, ...], Literal["heuristic_scorer", "reasoning_override"]]:
|
||||
) -> tuple[
|
||||
ComplexityTier,
|
||||
float,
|
||||
tuple[str, ...],
|
||||
Literal["heuristic_scorer", "reasoning_override", "no_signal_default"],
|
||||
]:
|
||||
"""
|
||||
Classify a prompt by complexity, reporting whether the score chose the tier.
|
||||
|
||||
When no dimension fires at all, the prompt carries no evidence either way, so
|
||||
`default_tier` (MEDIUM unless configured otherwise) is returned instead of letting
|
||||
the score of 0.0 fall through the band mapping into SIMPLE.
|
||||
|
||||
That check reads the signals, not the score, and not the individual dimension
|
||||
scores either. A weighted score of zero does not mean nothing was recognised,
|
||||
because contributions cancel: "hi, quick python question" comes to zero with three
|
||||
dimensions firing and is genuinely simple. A dimension scoring zero does not mean
|
||||
it stayed silent either, since a scorer is free to treat "no match" as a nonzero
|
||||
baseline or to report a neutral finding; a signal is the one thing a dimension
|
||||
emits only when it recognised something.
|
||||
|
||||
Args:
|
||||
prompt: The user's prompt/message.
|
||||
system_prompt: Optional system prompt for context.
|
||||
|
|
@ -558,6 +576,9 @@ class ComplexityRouter(CustomLogger):
|
|||
if reasoning_match_count >= 2:
|
||||
return ComplexityTier.REASONING, weighted_score, tuple(signals), "reasoning_override"
|
||||
|
||||
if not signals:
|
||||
return self.config.default_tier, weighted_score, (NO_SIGNAL_MARKER,), "no_signal_default"
|
||||
|
||||
# Map score to tier
|
||||
boundaries = self._effective_tier_boundaries()
|
||||
if weighted_score < boundaries["simple_medium"]:
|
||||
|
|
|
|||
|
|
@ -34,6 +34,8 @@ DEFAULT_TIER_DISTANCE_PENALTY: float = 0.5
|
|||
DEFAULT_CLASSIFIER_CONTEXT_WINDOW_SIZE: int = 3
|
||||
DEFAULT_CLASSIFIER_CONTEXT_PER_TURN_CHARS: int = 200
|
||||
|
||||
NO_SIGNAL_MARKER: str = "no-signal"
|
||||
|
||||
|
||||
class KeywordTierRule(BaseModel):
|
||||
"""A deterministic override: if any keyword matches, route to this tier."""
|
||||
|
|
@ -269,6 +271,17 @@ class ComplexityRouterConfig(BaseModel):
|
|||
description="Score boundaries between tiers",
|
||||
)
|
||||
|
||||
default_tier: ComplexityTier = Field(
|
||||
default=ComplexityTier.MEDIUM,
|
||||
description=(
|
||||
"Tier used when no scoring dimension fires, i.e. the prompt matched no keyword, "
|
||||
"pattern or token-count threshold and the scorer has no evidence either way. "
|
||||
"When set explicitly it must have a model behind it: a non-empty entry in `tiers`, "
|
||||
"or `default_model`. "
|
||||
"Set to SIMPLE to restore the previous behavior of treating unmatched traffic as simple"
|
||||
),
|
||||
)
|
||||
|
||||
# Token count thresholds
|
||||
token_thresholds: dict[str, int] = Field(
|
||||
default_factory=lambda: DEFAULT_TOKEN_THRESHOLDS.copy(),
|
||||
|
|
@ -459,6 +472,20 @@ class ComplexityRouterConfig(BaseModel):
|
|||
raise ValueError("classifier_llm_config is required when classifier_type is 'llm'")
|
||||
return self
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_default_tier_is_servable(self) -> "ComplexityRouterConfig":
|
||||
if "default_tier" not in self.model_fields_set:
|
||||
return self
|
||||
if self.tiers.get(self.default_tier.value) or self.default_model:
|
||||
return self
|
||||
raise ValueError(
|
||||
f"default_tier {self.default_tier.value} is not a non-empty entry in tiers "
|
||||
f"({sorted(self.tiers)}). Add it to tiers, name a tier that is configured, or set "
|
||||
f"default_model in complexity_router_config; the deployment-level "
|
||||
f"complexity_router_default_model does not count here, because falling through to it "
|
||||
f"would serve every no-signal request from a model this tier never names"
|
||||
)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_adaptive_pools(self) -> "ComplexityRouterConfig":
|
||||
if not self.adaptive:
|
||||
|
|
|
|||
|
|
@ -2699,6 +2699,7 @@ RoutingDecisionCause = Literal[
|
|||
# that tells a reader the score did NOT choose the tier; encoding it as free text
|
||||
# meant anything that filtered `signals` silently changed what the row claimed.
|
||||
"reasoning_override",
|
||||
"no_signal_default",
|
||||
"llm_classifier",
|
||||
"literal_keyword_match",
|
||||
"semantic_keyword_match",
|
||||
|
|
|
|||
|
|
@ -33,6 +33,7 @@ ROUTER_PARAMS = LiteLLMParamsBody(
|
|||
complexity_router_config={
|
||||
"classifier_type": "llm",
|
||||
"classifier_llm_config": {"model": "gpt-5.5"},
|
||||
"default_tier": "SIMPLE",
|
||||
"tiers": {
|
||||
"SIMPLE": "gpt-5.5",
|
||||
"MEDIUM": "claude-haiku-4-5",
|
||||
|
|
|
|||
|
|
@ -14,9 +14,13 @@ to the openai backend and every higher tier to the anthropic backend. The prompt
|
|||
below carries none of the heuristic scorer's reasoning/technical/code keywords and
|
||||
stays short, so heuristic scoring lands it in SIMPLE (openai), but an LLM classifier
|
||||
reads it as a decision that has to weigh tradeoffs and lands it above SIMPLE
|
||||
(anthropic). The served deployment is read back from the spend log's `model`, so
|
||||
anthropic proves the classifier ran and openai proves it silently fell back - the
|
||||
exact failure before the fix.
|
||||
(anthropic). The router config pins `default_tier: SIMPLE` to keep that split intact:
|
||||
a prompt with no scoring signal otherwise abstains to MEDIUM, the same anthropic
|
||||
backend the LLM arm picks, and the fallback would stop being visible.
|
||||
|
||||
The served deployment is read back from the spend log's `model`, so anthropic proves
|
||||
the classifier ran and openai proves it silently fell back - the exact failure before
|
||||
the fix.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
|
|
|||
|
|
@ -28,7 +28,9 @@ from litellm.router_strategy.complexity_router.complexity_router import (
|
|||
)
|
||||
from litellm.router_strategy.complexity_router.config import (
|
||||
DEFAULT_COMPLEXITY_CONFIG,
|
||||
DEFAULT_DIMENSION_WEIGHTS,
|
||||
DEFAULT_TECHNICAL_KEYWORDS,
|
||||
NO_SIGNAL_MARKER,
|
||||
ComplexityRouterConfig,
|
||||
ComplexityTier,
|
||||
)
|
||||
|
|
@ -4738,3 +4740,377 @@ class TestClassifierTrustBoundary:
|
|||
assert system_message["content"] == _CLASSIFICATION_SYSTEM_RUBRIC
|
||||
assert hostile not in system_message["content"]
|
||||
assert hostile in user_message["content"]
|
||||
|
||||
|
||||
DEFAULT_TIER_MODELS_FOR_TEST = {
|
||||
"SIMPLE": "simple-model",
|
||||
"MEDIUM": "medium-model",
|
||||
"COMPLEX": "complex-model",
|
||||
"REASONING": "reasoning-model",
|
||||
}
|
||||
|
||||
RIVER_CROSSING = (
|
||||
"A farmer must ferry a wolf, a goat, and a cabbage across a river. The boat holds the farmer "
|
||||
"and one other item. Left alone, the wolf eats the goat and the goat eats the cabbage. Give "
|
||||
"the shortest sequence of crossings."
|
||||
)
|
||||
|
||||
ESCALATION_TRIAGE = (
|
||||
"Three of our largest accounts have all asked for the same thing this quarter, but each one "
|
||||
"wants it on a different timeline and two of them have contracts that expire before the "
|
||||
"earliest date we could deliver. Draft the note to the leadership team laying out what we "
|
||||
"should promise, to whom, and in what order."
|
||||
)
|
||||
|
||||
BOARD_SUMMARY = (
|
||||
"Summarize the attached quarterly report for a board audience and highlight anything that "
|
||||
"changed materially from last quarter, keeping it to roughly one page and avoiding jargon "
|
||||
"that a non-specialist reader would stumble over."
|
||||
)
|
||||
|
||||
NO_SIGNAL_PROMPTS = (RIVER_CROSSING, ESCALATION_TRIAGE, BOARD_SUMMARY)
|
||||
|
||||
CANCELLING_PROMPT = "hi, quick python question"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def default_boundaries_router(mock_router_instance):
|
||||
"""Router on the shipped defaults: only the tier -> model map is overridden."""
|
||||
return ComplexityRouter(
|
||||
model_name="test-default-tier-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={"tiers": dict(DEFAULT_TIER_MODELS_FOR_TEST)},
|
||||
)
|
||||
|
||||
|
||||
def _router_with_default_tier(mock_router_instance, default_tier):
|
||||
return ComplexityRouter(
|
||||
model_name="test-default-tier-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"tiers": dict(DEFAULT_TIER_MODELS_FOR_TEST),
|
||||
"default_tier": default_tier,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
class TestNoSignalDefaultTier:
|
||||
"""LIT-4898: a prompt that fires no dimension is unclassifiable, not simple."""
|
||||
|
||||
@pytest.mark.parametrize("prompt", NO_SIGNAL_PROMPTS)
|
||||
def test_no_signal_prompt_abstains_to_medium(self, default_boundaries_router, prompt):
|
||||
tier, score, signals = default_boundaries_router.classify(prompt)
|
||||
assert tier == ComplexityTier.MEDIUM
|
||||
assert score == 0.0
|
||||
assert signals == [NO_SIGNAL_MARKER]
|
||||
|
||||
def test_cancelling_dimensions_score_zero_but_do_not_abstain(self, default_boundaries_router):
|
||||
"""Guards the predicate itself: this prompt cancels to a score of zero with three
|
||||
dimensions firing (short prompt, simple indicator, code keyword), so an abstain branch
|
||||
written against the weighted score would wrongly promote it to MEDIUM."""
|
||||
tier, score, signals = default_boundaries_router.classify(CANCELLING_PROMPT)
|
||||
assert score == pytest.approx(0.0, abs=1e-9)
|
||||
assert tier == ComplexityTier.SIMPLE
|
||||
assert NO_SIGNAL_MARKER not in signals
|
||||
assert signals == ["short (6 tokens)", "code (python)", "simple (quick, hi)"]
|
||||
|
||||
def test_exactly_zero_weighted_score_with_signal_does_not_abstain(self, mock_router_instance):
|
||||
"""Same guard as above, but with weights that cancel to a bit-exact 0.0 rather than to
|
||||
float dust, so `weighted_score == 0.0` is a live substitution the suite has to reject."""
|
||||
router = ComplexityRouter(
|
||||
model_name="test-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"tiers": dict(DEFAULT_TIER_MODELS_FOR_TEST),
|
||||
"dimension_weights": {**DEFAULT_DIMENSION_WEIGHTS, "tokenCount": 0.25, "codePresence": 0.5},
|
||||
},
|
||||
)
|
||||
tier, score, signals = router.classify("please add a def here")
|
||||
assert score == 0.0
|
||||
assert signals == ["short (5 tokens)", "code (def)"]
|
||||
assert tier == ComplexityTier.SIMPLE
|
||||
|
||||
def test_only_simple_indicator_fires_stays_simple(self, default_boundaries_router):
|
||||
tier, _, signals = default_boundaries_router.classify("What is machine learning?")
|
||||
assert tier == ComplexityTier.SIMPLE
|
||||
assert NO_SIGNAL_MARKER not in signals
|
||||
|
||||
def test_only_token_count_fires_stays_simple(self, default_boundaries_router):
|
||||
tier, _, signals = default_boundaries_router.classify("Where did we land on the offer letter")
|
||||
assert signals == ["short (9 tokens)"]
|
||||
assert tier == ComplexityTier.SIMPLE
|
||||
|
||||
def test_default_tier_simple_restores_previous_behavior(self, mock_router_instance):
|
||||
router = _router_with_default_tier(mock_router_instance, "SIMPLE")
|
||||
for prompt in NO_SIGNAL_PROMPTS:
|
||||
tier, score, _ = router.classify(prompt)
|
||||
assert tier == ComplexityTier.SIMPLE, prompt
|
||||
assert score == 0.0
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"configured, expected",
|
||||
[
|
||||
("SIMPLE", ComplexityTier.SIMPLE),
|
||||
("MEDIUM", ComplexityTier.MEDIUM),
|
||||
("COMPLEX", ComplexityTier.COMPLEX),
|
||||
("REASONING", ComplexityTier.REASONING),
|
||||
],
|
||||
)
|
||||
def test_default_tier_is_configurable(self, mock_router_instance, configured, expected):
|
||||
router = _router_with_default_tier(mock_router_instance, configured)
|
||||
tier, _, _ = router.classify(RIVER_CROSSING)
|
||||
assert tier == expected
|
||||
|
||||
def test_default_tier_defaults_to_medium(self):
|
||||
assert ComplexityRouterConfig().default_tier == ComplexityTier.MEDIUM
|
||||
assert DEFAULT_COMPLEXITY_CONFIG.default_tier == ComplexityTier.MEDIUM
|
||||
|
||||
def test_unknown_default_tier_is_rejected(self, mock_router_instance):
|
||||
with pytest.raises(ValidationError):
|
||||
_router_with_default_tier(mock_router_instance, "CHEAPEST")
|
||||
|
||||
def test_default_tier_without_a_model_is_rejected_at_config_time(self, mock_router_instance):
|
||||
"""An explicit default_tier naming a tier with no model would only surface as a routing
|
||||
failure on the first no-signal request, so reject the config instead."""
|
||||
with pytest.raises(ValidationError, match="default_tier COMPLEX is not a non-empty entry in tiers"):
|
||||
ComplexityRouter(
|
||||
model_name="test-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"tiers": {"SIMPLE": "simple-model", "MEDIUM": "medium-model"},
|
||||
"default_tier": "COMPLEX",
|
||||
},
|
||||
)
|
||||
|
||||
def test_default_tier_with_an_empty_pool_is_rejected(self, mock_router_instance):
|
||||
with pytest.raises(ValidationError, match="default_tier MEDIUM is not a non-empty entry in tiers"):
|
||||
ComplexityRouter(
|
||||
model_name="test-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"tiers": {"SIMPLE": "simple-model", "MEDIUM": []},
|
||||
"default_tier": "MEDIUM",
|
||||
},
|
||||
)
|
||||
|
||||
def test_default_tier_outside_tiers_is_allowed_with_default_model(self, mock_router_instance):
|
||||
router = ComplexityRouter(
|
||||
model_name="test-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"tiers": {"SIMPLE": "simple-model"},
|
||||
"default_tier": "MEDIUM",
|
||||
"default_model": "fallback-model",
|
||||
},
|
||||
)
|
||||
assert router.get_model_for_tier(router.classify(RIVER_CROSSING)[0]) == "fallback-model"
|
||||
|
||||
def test_partial_tiers_map_still_loads_without_an_explicit_default_tier(self, mock_router_instance):
|
||||
"""The implicit MEDIUM default must not turn an existing partial tiers map into a
|
||||
startup failure; it keeps the same resolution chain every other tier already has."""
|
||||
router = ComplexityRouter(
|
||||
model_name="test-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={"tiers": {"SIMPLE": "simple-model", "REASONING": "reasoning-model"}},
|
||||
)
|
||||
assert router.config.default_tier == ComplexityTier.MEDIUM
|
||||
|
||||
def test_abstain_does_not_consult_tier_boundaries(self, mock_router_instance):
|
||||
"""Boundaries that would map 0.0 to COMPLEX must not reach the no-signal path."""
|
||||
router = ComplexityRouter(
|
||||
model_name="test-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={
|
||||
"tiers": dict(DEFAULT_TIER_MODELS_FOR_TEST),
|
||||
"tier_boundaries": {
|
||||
"simple_medium": -1.0,
|
||||
"medium_complex": -0.5,
|
||||
"complex_reasoning": 1.0,
|
||||
},
|
||||
},
|
||||
)
|
||||
tier, _, _ = router.classify(RIVER_CROSSING)
|
||||
assert tier == ComplexityTier.MEDIUM
|
||||
|
||||
def test_reasoning_override_wins_over_abstain(self, default_boundaries_router):
|
||||
tier, _, signals = default_boundaries_router.classify(
|
||||
"Let's think step by step and reason through this carefully"
|
||||
)
|
||||
assert tier == ComplexityTier.REASONING
|
||||
assert NO_SIGNAL_MARKER not in signals
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_signal_request_routes_to_the_medium_model(self, default_boundaries_router):
|
||||
result = await default_boundaries_router.async_pre_routing_hook(
|
||||
model="test-model",
|
||||
request_kwargs={},
|
||||
messages=[{"role": "user", "content": RIVER_CROSSING}],
|
||||
)
|
||||
assert result is not None
|
||||
assert result.model == "medium-model"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_signal_decision_names_its_own_cause(self, default_boundaries_router):
|
||||
"""Same argument as reasoning_override: the fact worth logging is that the score did
|
||||
not choose the tier, so it is a cause rather than a marker only inside signals."""
|
||||
response = await default_boundaries_router.async_pre_routing_hook(
|
||||
model="test-model",
|
||||
request_kwargs={},
|
||||
messages=[{"role": "user", "content": RIVER_CROSSING}],
|
||||
)
|
||||
decision = response.routing_decision
|
||||
assert decision["cause"] == "no_signal_default"
|
||||
assert decision["tier"] == "MEDIUM"
|
||||
assert decision["signals"] == [NO_SIGNAL_MARKER]
|
||||
assert decision["routed_model"] == "medium-model"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_signal_request_honors_default_tier_simple(self, mock_router_instance):
|
||||
router = _router_with_default_tier(mock_router_instance, "SIMPLE")
|
||||
result = await router.async_pre_routing_hook(
|
||||
model="test-model",
|
||||
request_kwargs={},
|
||||
messages=[{"role": "user", "content": RIVER_CROSSING}],
|
||||
)
|
||||
assert result is not None
|
||||
assert result.model == "simple-model"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_abstain_still_escalates(self, mock_router_instance):
|
||||
router = ComplexityRouter(
|
||||
model_name="test-router",
|
||||
litellm_router_instance=mock_router_instance,
|
||||
complexity_router_config={"tiers": dict(DEFAULT_TIER_MODELS_FOR_TEST)},
|
||||
)
|
||||
result = await router.async_pre_routing_hook(
|
||||
model="test-model",
|
||||
request_kwargs={},
|
||||
messages=[{"role": "user", "content": f"LITELLM ESCALATE {RIVER_CROSSING}"}],
|
||||
)
|
||||
assert result is not None
|
||||
assert result.model == "complex-model"
|
||||
|
||||
|
||||
LONG_UNMATCHED_PROMPT = " ".join(
|
||||
[
|
||||
"The tenant moved out on the fifteenth and left the place in a state that neither of us",
|
||||
"expected. We had walked through it together in the spring and everything looked ordinary",
|
||||
"then. Now there are marks along the hallway that were not in the move-in photos, the back",
|
||||
"door does not sit flush in its frame, and the garden that was part of the agreement has",
|
||||
"clearly been left to itself for most of the summer. I have the deposit, the signed",
|
||||
"inventory from the start of the tenancy, and about forty photographs taken on the day they",
|
||||
"handed the keys back. The tenant says the hallway marks were there when they arrived and",
|
||||
"that the garden was never their responsibility because we had someone come round for the",
|
||||
"first two months. I would like a letter to send them that sets out what I intend to hold",
|
||||
"back from the deposit and why, that reads as reasonable rather than aggressive, and that",
|
||||
"leaves room for them to come back with their side before I file anything formally. Keep it",
|
||||
"to a page. I want the tone to be the sort a longtime neighbour would use, firm about the",
|
||||
"money but not spoiling for a fight, because they may well end up renting the flat upstairs",
|
||||
"from my sister and I would rather this ended quietly than turned into a matter that follows",
|
||||
"everyone around for a year afterwards. Mention the walkthrough, mention the photographs,",
|
||||
"and leave the specific figures blank for me to fill in myself once I have the quotes back",
|
||||
"from the two people I asked to look at the door and the hallway this coming week, and say",
|
||||
"plainly that I would rather settle the difference between us than have either of us spend",
|
||||
"another weekend on it.",
|
||||
]
|
||||
)
|
||||
|
||||
TIER_CORPUS = (
|
||||
(RIVER_CROSSING, ComplexityTier.MEDIUM),
|
||||
(ESCALATION_TRIAGE, ComplexityTier.MEDIUM),
|
||||
(BOARD_SUMMARY, ComplexityTier.MEDIUM),
|
||||
("The quarterly numbers came in under what we told the board in April. Walk me through how "
|
||||
"you would open that conversation with them, what you would concede up front, and where you "
|
||||
"would push back if they ask whether the forecast was ever realistic.", ComplexityTier.MEDIUM),
|
||||
(CANCELLING_PROMPT, ComplexityTier.SIMPLE),
|
||||
("Hi there!", ComplexityTier.SIMPLE),
|
||||
("Thanks!", ComplexityTier.SIMPLE),
|
||||
("What is machine learning?", ComplexityTier.SIMPLE),
|
||||
("Explain how REST APIs work with HTTP methods", ComplexityTier.SIMPLE),
|
||||
(LONG_UNMATCHED_PROMPT, ComplexityTier.SIMPLE),
|
||||
("Write a function that reverses a linked list in python", ComplexityTier.MEDIUM),
|
||||
("Design a distributed microservice architecture for a high-throughput real-time data "
|
||||
"processing pipeline with Kubernetes orchestration, implementing proper authentication "
|
||||
"and encryption protocols", ComplexityTier.COMPLEX),
|
||||
("Think step by step and reason through the pros and cons of different database "
|
||||
"architectures", ComplexityTier.REASONING),
|
||||
)
|
||||
|
||||
|
||||
class TestDefaultConfigTierCorpus:
|
||||
"""Pins the shipped-default tier mix so a weight or keyword edit cannot silently walk
|
||||
unmatched traffic back onto the cheapest tier."""
|
||||
|
||||
@pytest.mark.parametrize("prompt, expected", TIER_CORPUS)
|
||||
def test_corpus_tier(self, default_boundaries_router, prompt, expected):
|
||||
tier, score, signals = default_boundaries_router.classify(prompt)
|
||||
assert tier == expected, f"{prompt[:60]!r} -> {tier} (score={score:.4f}, signals={signals})"
|
||||
|
||||
def test_no_signal_share_of_corpus_never_lands_on_simple(self, default_boundaries_router):
|
||||
abstained = [
|
||||
prompt
|
||||
for prompt, _ in TIER_CORPUS
|
||||
if default_boundaries_router.classify(prompt)[2] == [NO_SIGNAL_MARKER]
|
||||
]
|
||||
assert len(abstained) == 4
|
||||
for prompt in abstained:
|
||||
assert default_boundaries_router.classify(prompt)[0] != ComplexityTier.SIMPLE
|
||||
|
||||
def test_long_unmatched_prompt_fires_token_count_and_is_not_abstain(self, default_boundaries_router):
|
||||
"""The >400-token band is outside the abstain set by design: tokenCount fires, so the
|
||||
prompt scores +0.10 and the boundary (not default_tier) decides. Moving that boundary
|
||||
is the separate pricing decision tracked as a follow-up."""
|
||||
_, score, signals = default_boundaries_router.classify(LONG_UNMATCHED_PROMPT)
|
||||
assert len(signals) == 1 and signals[0].startswith("long (")
|
||||
assert score == pytest.approx(0.10)
|
||||
|
||||
|
||||
class TestDimensionSignalInvariant:
|
||||
"""The abstain branch reads the signals, so a dimension has to say when it recognised
|
||||
something. A scorer returning a nonzero score with no signal would be evidence the
|
||||
abstain check cannot see; one returning zero with a signal would be silence it mistakes
|
||||
for evidence. Either way the no-signal default would fire on the wrong requests."""
|
||||
|
||||
def _dimension_scores(self, router):
|
||||
keyword_args = (
|
||||
(router.code_keywords, "codePresence", "code", (1, 2), (0, 0.5, 1.0)),
|
||||
(router.reasoning_keywords, "reasoningMarkers", "reasoning", (1, 2), (0, 0.7, 1.0)),
|
||||
(router.technical_keywords, "technicalTerms", "technical", (2, 4), (0, 0.5, 1.0)),
|
||||
(router.simple_keywords, "simpleIndicators", "simple", (1, 2), (0, -1.0, -1.0)),
|
||||
)
|
||||
texts = (
|
||||
"",
|
||||
"the meeting moved again",
|
||||
"def foo",
|
||||
"import a function to refactor the database schema",
|
||||
"step by step, think through and analyze this",
|
||||
"distributed architecture with encryption and authentication and latency budgets",
|
||||
"hi, thanks, quick question",
|
||||
)
|
||||
keyword_dimensions = [
|
||||
router._score_keyword_match(text, text, keywords, name, label, thresholds, scores)[0]
|
||||
for text in texts
|
||||
for keywords, name, label, thresholds, scores in keyword_args
|
||||
]
|
||||
other_dimensions = [
|
||||
*(router._score_token_count(count) for count in (0, 3, 14, 15, 100, 400, 401, 5000)),
|
||||
*(router._score_multi_step(text) for text in (*texts, "first do this, then do that")),
|
||||
*(router._score_question_complexity(text) for text in (*texts, "a? b? c? d? e?")),
|
||||
]
|
||||
return keyword_dimensions + other_dimensions
|
||||
|
||||
def test_a_signal_is_emitted_exactly_when_a_dimension_scores(self, default_boundaries_router):
|
||||
for dimension in self._dimension_scores(default_boundaries_router):
|
||||
assert (dimension.score != 0) == (dimension.signal is not None), (
|
||||
f"{dimension.name} broke the invariant: score={dimension.score!r} signal={dimension.signal!r}"
|
||||
)
|
||||
|
||||
def test_a_system_prompt_only_match_still_signals(self, default_boundaries_router):
|
||||
"""The signal withholds the matched term when only the system prompt supplied it,
|
||||
reporting a count instead. It must still be a signal, or a prompt whose evidence
|
||||
came entirely from the system prompt would read as no signal at all."""
|
||||
_, _, signals = default_boundaries_router.classify(
|
||||
"please help with that", system_prompt="You are a kubernetes and docker expert"
|
||||
)
|
||||
assert signals == ["short (5 tokens)", "code (2 matches)"]
|
||||
assert NO_SIGNAL_MARKER not in signals
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue