# Adaptive Router (v0)

A request-type-aware routing strategy. For each incoming request, classify the
prompt into one of seven `RequestType` buckets (code generation, writing,
analytical reasoning, …), then Thompson-sample a Beta(α, β) bandit posterior
per `(request_type, model)` cell to pick the best model. Quality estimates are
combined with a normalized cost score via a weighted linear sum.

A post-call hook reads the response and runs lightweight regex + tool-call
detectors (see `signals.py`) to award per-turn credit/blame to the model that
served the turn. Updates are batched in-memory and flushed to Postgres every
~10s by a background task in `proxy_server.py`.

## Config example

```yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
    model_info:
      input_cost_per_token: 0.0000025
      adaptive_router_preferences:
        quality_tier: 3
        strengths: ["code_generation", "analytical_reasoning"]

  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
    model_info:
      input_cost_per_token: 0.00000015
      adaptive_router_preferences:
        quality_tier: 2
        strengths: ["general", "factual_lookup"]

  - model_name: smart-router
    litellm_params:
      model: auto_router/adaptive_router
      adaptive_router_default_model: gpt-4o-mini
      adaptive_router_config:
        available_models: ["gpt-4o", "gpt-4o-mini"]
        weights:
          quality: 0.7
          cost: 0.3
```

Callers may pass header `x-litellm-min-quality-tier: 3` (or metadata key
`min_quality_tier: 3`) to force selection from tier-3-or-higher models only.

## Behavior summary

- **Cold start.** Each `(request_type, model)` cell starts with a
  Beta prior whose mean = `BASE_TIER_WEIGHT[tier] (+ STRENGTH_BONUS if declared)`
  and total mass = `COLD_START_MASS` (10). About ten real observations move it
  meaningfully.
- **Per-request decision.** Sample once per eligible model, score with
  `quality_weight·sample + cost_weight·normalized_cost`, pick the argmax.
  Routing is stateless per-turn — no sticky lookup. Each call resamples.
- **Previous-response attribution.** Post-call, feedback from the current user
  message is attributed to the model that produced the previous response, while
  response signals are attributed to the current model. Contexts expire after
  24 hours and the in-memory cache is capped at 1,024 sessions. Conversation identity is the
  client-supplied `litellm_session_id` if present, otherwise a sha256 over
  caller identity (api key hash, team, user, end-user) + the first message.
- **Per-turn updates.** `satisfaction → +α`. `misalignment, stagnation,
  disengagement, failure → +β` (each). `loop → +0.5β`. `exhaustion → 0`
  (uptime, not quality). Skipped if conversation has fewer than
  `SIGNAL_GATE_MIN_MESSAGES` messages.
- **Persistence.** Bandit cells: aggregated deltas, eventually consistent.
  Session rows: last-write-wins snapshots.

## Known v0 limitations

- **Latency is not in the score.** Quality + cost only. A pathologically slow
  model can still be picked.
- **Hard sample cap at 200.** Once `α + β > 200`, deltas are silently dropped.
  No rescaling — drift is a v1 concern.
- **Signals are regex + tool-call only.** No LLM-judge, no embedding similarity,
  no exemplar storage. Signals are best-effort and biased toward English.
- **One AdaptiveRouter per `Router`.** Multiple `adaptive_router/*` deployments
  on the same `litellm.Router` raise at init.
- **Bandit-delta mapping is unvalidated.** `_compute_bandit_delta` is a v0
  guess; expect to retune after the first ~1000 sessions of real traffic.
- **`request_type` is classified per turn from the latest user message.** For
  non-GENERAL turns, the current-turn type is used for bandit attribution (so
  genuine mid-session topic shifts update the correct cell). For GENERAL turns
  ("thanks!", "ok", "sounds good"), attribution falls back to the session's
  original type to avoid misattributing closing pleasantries.
