scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-05 08:07:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 136

chore: remove preset feature, keep simple tier config

Browse source
This commit is contained in:
OpenClaw Assistant 2026-02-21 19:27:48 +00:00
parent f096d9dce6
commit 95c6bfaa06
3 changed files with 0 additions and 358 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

186
PR_DESCRIPTION.md
View file

@ -1,186 +0,0 @@
# feat(router): Add complexity-based auto routing strategy
## Summary
This PR adds a new `complexity_router` - a rule-based routing strategy that uses weighted scoring to classify requests by complexity and route them to appropriate models. Unlike the existing `auto_router` which uses semantic/embedding matching (requiring API calls), the complexity router operates entirely locally in <1ms with zero cost.
**Also included:** UI updates to make complexity routing accessible via a simple 4-dropdown interface.
## Motivation
Many users want intelligent model routing based on query complexity without:
- The latency of embedding API calls
- The cost of embedding API calls
- The complexity of configuring semantic routes with utterances
The complexity router provides a simple, fast alternative that handles 70-80% of routing decisions well.
## How It Works
### Weighted Scoring Across 7 Dimensions
| Dimension | What It Detects | Score Range |
|-----------|-----------------|-------------|
| `tokenCount` | Short prompts → simple, long → complex | -1.0 to 1.0 |
| `codePresence` | Code keywords (function, class, python, etc.) | 0 to 1.0 |
| `reasoningMarkers` | "step by step", "think through", etc. | 0 to 1.0 |
| `technicalTerms` | Architecture, distributed, ML terms | 0 to 1.0 |
| `simpleIndicators` | "what is", "define", greetings | -1.0 to 0 |
| `multiStepPatterns` | "first...then", numbered steps | 0 to 0.5 |
| `questionComplexity` | Multiple questions | 0 to 0.5 |
### Tier Assignment
The weighted score maps to 4 tiers:
- **SIMPLE** (score < 0.25): Quick factual questions, greetings
- **MEDIUM** (0.25 ≤ score < 0.50): Moderate complexity
- **COMPLEX** (0.50 ≤ score < 0.75): Technical, code-heavy requests
- **REASONING** (score ≥ 0.75): Multi-step reasoning required
**Special Override:** If 2+ reasoning markers are detected in the user message, the request is automatically routed to REASONING tier regardless of overall score.
## Configuration
### Via proxy config.yaml
```yaml
model_list:
- model_name: complexity_router_1
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gemini-2.0-flash
MEDIUM: gpt-4o-mini
COMPLEX: claude-sonnet-4
REASONING: claude-opus-4
# Optional: adjust tier boundaries (defaults shown)
tier_boundaries:
simple_medium: 0.25
medium_complex: 0.50
complex_reasoning: 0.75
# Optional: adjust token count thresholds
token_thresholds:
simple: 15 # Below = "short"
complex: 400 # Above = "long"
```
### Via UI (New!)
The UI now has a "Router Type" selector with two options:
1. **Complexity Router (Recommended)** - Simple 4-dropdown interface:
- Simple Tasks: [model dropdown]
- Medium Tasks: [model dropdown]
- Complex Tasks: [model dropdown]
- Reasoning Tasks: [model dropdown]
2. **Semantic Router** - Existing utterance-based configuration
Users can now set up smart routing in ~30 seconds by just picking 4 models.
### Programmatic Usage
```python
from litellm import Router
router = Router(model_list=[{
"model_name": "smart-router",
"litellm_params": {
"model": "auto_router/complexity_router",
"complexity_router_config": {
"tiers": {
"SIMPLE": "gpt-4o-mini",
"MEDIUM": "gpt-4o",
"COMPLEX": "claude-sonnet-4-20250514",
"REASONING": "claude-sonnet-4-20250514",
}
}
}
}])
# Routes automatically based on complexity
response = await router.acompletion(
model="smart-router",
messages=[{"role": "user", "content": "What is 2+2?"}]
) # → Routes to gpt-4o-mini (SIMPLE)
response = await router.acompletion(
model="smart-router",
messages=[{"role": "user", "content": "Think step by step about this distributed systems architecture problem..."}]
) # → Routes to claude-sonnet-4-20250514 (REASONING)
```
## Files Changed
### New Files - Backend
- `litellm/router_strategy/complexity_router/__init__.py`
- `litellm/router_strategy/complexity_router/complexity_router.py` - Main router implementation
- `litellm/router_strategy/complexity_router/config.py` - Configuration and defaults
- `litellm/router_strategy/complexity_router/README.md` - Documentation
- `tests/test_litellm/router_strategy/test_complexity_router.py` - 37 tests
### New Files - UI
- `ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx` - 4-dropdown tier configuration component
### Modified Files - Backend
- `litellm/router.py` - Added complexity router initialization and pre-routing hook
- `litellm/types/router.py` - Added `complexity_router_config` and `complexity_router_default_model` params
### Modified Files - UI
- `ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx` - Added router type selector, integrated ComplexityRouterConfig
- `ui/litellm-dashboard/src/components/add_model/handle_add_auto_router_submit.tsx` - Handle complexity_router model type in submit
## UI Changes
### Router Type Selector
![Router Type Selector](docs/images/router-type-selector.png)
The "Add Auto Router" page now shows:
- **Complexity Router (Recommended)** with a green badge - default selected
- **Semantic Router** for the existing utterance-based approach
### Complexity Router Configuration
![Complexity Router Config](docs/images/complexity-router-config.png)
Simple 4-dropdown interface:
- Each dropdown shows available models
- Tooltips explain what each tier handles
- Recommendation card with suggested model types
## Testing
```bash
pytest tests/test_litellm/router_strategy/test_complexity_router.py -v
# 37 passed in 0.21s
```
Tests cover:
- Scoring logic for each dimension
- Tier assignment at boundaries
- Reasoning marker override
- Model selection
- Pre-routing hook integration
- Edge cases (empty prompts, unicode, very long prompts)
- Configuration overrides
## Performance
- **Latency:** <1ms per classification (all local regex/string matching)
- **Cost:** $0 (no API calls)
- **Memory:** Minimal (pre-compiled regex patterns)
## Inspiration
This implementation is inspired by [ClawRouter](https://github.com/BlockRunAI/ClawRouter), which uses similar weighted scoring for complexity classification.
## Checklist
- [x] Tests added (37 tests)
- [x] Backend implementation complete
- [x] UI implementation complete
- [x] Documentation in PR description
- [x] No breaking changes
- [x] Follows existing patterns (like `auto_router`)
cc @ishaan-jaff for review

53
litellm/router_strategy/complexity_router/config.py
View file

@ -109,7 +109,6 @@ DEFAULT_TOKEN_THRESHOLDS: Dict[str, int] = {
# ─── Default Tier to Model Mapping ───
# Standard defaults - best cost/performance for most users
DEFAULT_TIER_MODELS: Dict[str, str] = {
"SIMPLE": "gpt-4o-mini",
"MEDIUM": "gpt-4o",
@ -117,68 +116,16 @@ DEFAULT_TIER_MODELS: Dict[str, str] = {
"REASONING": "claude-sonnet-4-20250514",
}
# Enterprise presets - for teams using specific cloud providers
ENTERPRISE_TIER_PRESETS: Dict[str, Dict[str, str]] = {
# AWS Bedrock - for enterprises on AWS
"bedrock": {
"SIMPLE": "bedrock/anthropic.claude-3-haiku-20240307-v1:0",
"MEDIUM": "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
"COMPLEX": "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
"REASONING": "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
},
# Google Vertex AI - for enterprises on GCP
"vertex": {
"SIMPLE": "vertex_ai/gemini-2.0-flash",
"MEDIUM": "vertex_ai/gemini-2.0-flash",
"COMPLEX": "vertex_ai/gemini-2.5-pro",
"REASONING": "vertex_ai/gemini-2.5-pro",
},
# Azure OpenAI - for enterprises on Azure
"azure": {
"SIMPLE": "azure/gpt-4o-mini",
"MEDIUM": "azure/gpt-4o",
"COMPLEX": "azure/gpt-4o",
"REASONING": "azure/o1",
},
# Direct API (OpenAI + Anthropic) - best quality, recommended for startups
"standard": {
"SIMPLE": "gpt-4o-mini",
"MEDIUM": "gpt-4o",
"COMPLEX": "claude-sonnet-4-20250514",
"REASONING": "claude-sonnet-4-20250514",
},
# Cost-optimized - maximum savings
"cost_optimized": {
"SIMPLE": "gemini/gemini-2.0-flash",
"MEDIUM": "gpt-4o-mini",
"COMPLEX": "claude-3-5-sonnet-20241022",
"REASONING": "claude-sonnet-4-20250514",
},
}
class ComplexityRouterConfig(BaseModel):
"""Configuration for the ComplexityRouter."""
# Preset name (bedrock, vertex, azure, standard, cost_optimized)
# If set, overrides 'tiers' with the preset values
preset: Optional[str] = Field(
default=None,
description="Preset name: bedrock, vertex, azure, standard, cost_optimized",
)
# Tier to model mapping
tiers: Dict[str, str] = Field(
default_factory=lambda: DEFAULT_TIER_MODELS.copy(),
description="Mapping of complexity tiers to model names",
)
def model_post_init(self, __context) -> None:
"""Apply preset if specified."""
if self.preset and self.preset in ENTERPRISE_TIER_PRESETS:
# Override tiers with preset values
self.tiers = ENTERPRISE_TIER_PRESETS[self.preset].copy()
# Tier boundaries (normalized scores)
tier_boundaries: Dict[str, float] = Field(
default_factory=lambda: DEFAULT_TIER_BOUNDARIES.copy(),

119
ui/litellm-dashboard/src/components/add_model/ComplexityRouterConfig.tsx
View file

@ -1,119 +0,0 @@
import React from "react";
import { Card, Select as AntdSelect, Typography, Tooltip } from "antd";
import { InfoCircleOutlined } from "@ant-design/icons";
import { ModelGroup } from "../playground/llm_calls/fetch_models";
const { Text } = Typography;
interface ComplexityTiers {
SIMPLE: string;
MEDIUM: string;
COMPLEX: string;
REASONING: string;
}
interface ComplexityRouterConfigProps {
modelInfo: ModelGroup[];
value?: ComplexityTiers;
onChange?: (tiers: ComplexityTiers) => void;
}
const tierDescriptions = {
SIMPLE: "Quick questions, greetings, simple lookups (e.g., \"What is the capital of France?\")",
MEDIUM: "Moderate complexity, explanations, summaries",
COMPLEX: "Code generation, technical analysis, detailed research",
REASONING: "Multi-step reasoning, complex problem solving, chain-of-thought tasks",
};
const tierLabels = {
SIMPLE: "Simple Tasks",
MEDIUM: "Medium Tasks",
COMPLEX: "Complex Tasks",
REASONING: "Reasoning Tasks",
};
const ComplexityRouterConfig: React.FC<ComplexityRouterConfigProps> = ({
modelInfo,
value,
onChange
}) => {
const tiers: ComplexityTiers = value || {
SIMPLE: "",
MEDIUM: "",
COMPLEX: "",
REASONING: "",
};
const handleTierChange = (tier: keyof ComplexityTiers, model: string) => {
const updatedTiers = { ...tiers, [tier]: model };
onChange?.(updatedTiers);
};
// Prepare model options for dropdowns
const modelOptions = Array.from(
new Set(modelInfo.map((model) => model.model_group))
).map((model_group) => ({
value: model_group,
label: model_group,
}));
return (
<div className="w-full">
<div className="mb-4">
<Text className="text-gray-600">
Configure which model handles each complexity tier. Requests are automatically classified and routed — no training data needed.
</Text>
</div>
<Card className="w-full">
<div className="space-y-6">
{(["SIMPLE", "MEDIUM", "COMPLEX", "REASONING"] as const).map((tier) => (
<div key={tier} className="w-full">
<div className="flex items-center gap-2 mb-2">
<Text className="text-sm font-medium">{tierLabels[tier]}</Text>
<Tooltip title={tierDescriptions[tier]}>
<InfoCircleOutlined className="text-gray-400" />
</Tooltip>
</div>
<AntdSelect
value={tiers[tier] || undefined}
onChange={(value) => handleTierChange(tier, value)}
placeholder={`Select model for ${tierLabels[tier].toLowerCase()}`}
showSearch
style={{ width: "100%" }}
options={modelOptions}
allowClear
/>
<Text className="text-xs text-gray-400 mt-1 block">
{tierDescriptions[tier]}
</Text>
</div>
))}
</div>
</Card>
{/* Recommendations */}
<Card className="mt-4 bg-blue-50 border-blue-200">
<div className="flex items-start gap-2">
<InfoCircleOutlined className="text-blue-500 mt-1" />
<div>
<Text className="text-sm font-medium text-blue-800 block mb-1">
Recommendations
</Text>
<Text className="text-xs text-blue-700">
• <strong>Simple:</strong> Use fast, cheap models (e.g., GPT-4o-mini, Gemini Flash)
<br />
• <strong>Medium:</strong> Balanced models (e.g., GPT-4o, Claude Sonnet)
<br />
• <strong>Complex:</strong> Capable models (e.g., Claude Sonnet, GPT-4o)
<br />
• <strong>Reasoning:</strong> Best reasoning models (e.g., Claude Opus, o1-preview)
</Text>
</div>
</div>
</Card>
</div>
);
};
export default ComplexityRouterConfig;