From 86e7541cd5b7398df10bd3141dd077d7c0aa3da8 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 20:57:11 +0000 Subject: [PATCH] docs(router): draft a contract for model-driven routing strategies Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Aw1xktpUnix5tWhvSzey8t --- .../crates/router/POC-strategy-contract.md | 219 ++++++++++++++++++ litellm-rust/crates/router/POC.md | 4 +- 2 files changed, 222 insertions(+), 1 deletion(-) create mode 100644 litellm-rust/crates/router/POC-strategy-contract.md diff --git a/litellm-rust/crates/router/POC-strategy-contract.md b/litellm-rust/crates/router/POC-strategy-contract.md new file mode 100644 index 00000000000..32802cdc25c --- /dev/null +++ b/litellm-rust/crates/router/POC-strategy-contract.md @@ -0,0 +1,219 @@ +# Routing strategy contract (draft, 2026-10-07) + +A contract the model-driven routing strategies (auto, complexity, adaptive, quality) implement, so whoever drives the retry and fallback loop, Python or Rust, can call them the same way. Draft only: adopting it means refactoring the Python strategies first, and the Rust side waits on that + +## Why + +Today a strategy is anything with `async_pre_routing_hook(model, request_kwargs, messages, input)` returning `PreRoutingHookResponse`, but that signature hides most of what they depend on: +- they read loosely typed keys out of `request_kwargs` (session id, key hash, user agent, tools, instructions, min quality tier, proxy headers) +- they write their results back into `request_kwargs` metadata for other code to find later (the adaptive chosen model for the response header and the learning hook, `quality_router_decision`, the raw model name flag) +- they hold the whole router and call into it: `cache` for session pins, `get_model_list` and `get_router_model_info` for context windows and vision support, `async_get_healthy_deployments` to ask whether a group can serve the request, `acompletion` / `aresponses` / `aembedding` for classifier calls +- context compaction and routing compression are armed through contextvars that the inference call reads later +- the adaptive router learns through a `CustomLogger` in `litellm.callbacks` that finds its decision again by reading a metadata key back off the logged kwargs + +None of that crosses a process or language boundary cleanly, and most of it can't be unit tested without a full router + +## Shape + +A strategy is built once from its definition and a set of ports, then asked for a decision on every attempt and told the outcome after it. It never sees the router, never mutates its input, and never reaches a contextvar + +```python +class RoutingStrategy(Protocol): + async def route(self, request: RoutingRequest) -> RoutingDecision: ... + async def observe(self, feedback: Feedback) -> None: ... + + +class StrategyFactory(Protocol): + def build(self, definition: StrategyDefinition, ports: StrategyPorts) -> RoutingStrategy: ... +``` + +`route` is called at the start of every attempt, retries included, the same place the pre-routing hook runs today (spec 2.2). `observe` is called once per attempt after it settles. Neither may raise for an expected condition: a strategy that can't decide returns `Reject`, and a failed classifier call is the strategy's to turn into a default pick, the way the auto router already does + +## Input + +```python +@dataclass(frozen=True, slots=True) +class RoutingRequest: + requested_model: str + surface: Literal["chat", "responses", "anthropic_messages"] + conversation: Conversation + caller: Caller + session: Session | None + hints: RoutingHints + attempt: AttemptInfo + + +@dataclass(frozen=True, slots=True) +class Conversation: + messages: tuple[ChatMessage, ...] + system_prompt: str | None + tools: tuple[ToolDefinition, ...] + native: NativePayload + + +@dataclass(frozen=True, slots=True) +class Caller: + key_hash: str | None + team_id: str | None + user_id: str | None + tags: tuple[str, ...] + log_message_content: bool + + +@dataclass(frozen=True, slots=True) +class Session: + id: str + agent_id: str | None + user_agent: str | None + + +@dataclass(frozen=True, slots=True) +class RoutingHints: + min_quality_tier: int | None + + +@dataclass(frozen=True, slots=True) +class AttemptInfo: + number: int + fallback_depth: int +``` + +`messages` is the chat-completions shape every surface already normalizes to (`resolve_structured_messages`), so strategies stop resolving it themselves. `native` keeps the original surface payload (Responses `instructions` and `input`, encrypted reasoning) for the few cases that need it, such as the complexity router's encrypted classifier task. `Session` is only set for a client-supplied id; a generated one is not a session to pin. The host builds the whole request once, from the request kwargs, with Pydantic at the boundary + +## Output + +```python +RoutingDecision = Route | Decline | Reject + + +@dataclass(frozen=True, slots=True) +class Route: + target_group: str + tier: str | None + params_overlay: Mapping[str, object] + deployment_affinity_ttl: int | None + compaction: CompactionPlan | None + response_headers: Mapping[str, str] + provenance: StandardLoggingRoutingDecision + feedback_key: str | None + + +@dataclass(frozen=True, slots=True) +class Decline: + pass + + +@dataclass(frozen=True, slots=True) +class Reject: + reason: str +``` + +`Route` covers everything the strategies hand back today through the response and the metadata side channel. `params_overlay` is the tier's `litellm_params`. `compaction` replaces arming contextvars: the host arms compaction right before the call it makes. `response_headers` replaces stashing the adaptive chosen model in metadata for a post-call hook to turn into a header. `feedback_key` is opaque to the host and handed back in `observe`. `Decline` means the strategy does not apply to this request and normal routing proceeds, which is what returning `None` means today. Strategies never rewrite the conversation: none does today apart from routing-only compression, which the host owns + +## Feedback + +```python +@dataclass(frozen=True, slots=True) +class Feedback: + key: str + target_group: str + request: RoutingRequest + outcome: Served | Failed + + +@dataclass(frozen=True, slots=True) +class Served: + assistant_text: str | None + tool_calls: tuple[ToolCall, ...] + latency_ms: int + + +@dataclass(frozen=True, slots=True) +class Failed: + status: int + error_type: str +``` + +This replaces `AdaptiveRouterPostCallHook` finding its decision again in the logged kwargs. The host calls `observe` for every attempt that `route` decided, so a failed attempt that falls back still teaches the strategy. `observe` runs after the response is returned and must never delay or fail the request + +## Ports + +```python +@dataclass(frozen=True, slots=True) +class StrategyPorts: + sessions: SessionStore + catalog: Catalog + eligibility: Eligibility + models: ModelClient + state: StrategyStateStore + clock: Callable[[], float] + rng: Random + + +class SessionStore(Protocol): + async def get(self, key: str) -> str | None: ... + async def put(self, key: str, value: str, ttl_seconds: int) -> None: ... + async def claim(self, key: str, value: str, ttl_seconds: int, eligible: frozenset[str]) -> str: ... + async def delete(self, key: str) -> None: ... + + +class Catalog(Protocol): + def groups(self) -> frozenset[str]: ... + def facts(self, group: str) -> tuple[DeploymentFacts, ...]: ... + + +@dataclass(frozen=True, slots=True) +class DeploymentFacts: + deployment_id: str + provider: str + context_window: int | None + max_output_tokens: int | None + supports_vision: bool | None + supports_prompt_caching: bool + input_cost_per_token: float | None + supported_params: frozenset[str] | None + + +class Eligibility(Protocol): + async def can_serve(self, group: str, request: RoutingRequest) -> bool: ... + + +class ModelClient(Protocol): + async def classify(self, model: str, call: ClassifierCall, on_behalf_of: RoutingRequest) -> ClassifierResult: ... + async def embed(self, model: str, texts: tuple[str, ...], on_behalf_of: RoutingRequest) -> tuple[Vector, ...]: ... + + +class StrategyStateStore(Protocol): + async def load(self, strategy: str) -> StrategyState: ... + async def save(self, strategy: str, delta: StrategyState) -> None: ... +``` + +Each port replaces one way the strategies reach into the router today: +- `SessionStore` replaces `router.cache` for the complexity router's tier and deployment pins. Keys are built by the strategy and already namespaced by caller and router name; `claim` is the existing `claim_affinity_pin` compare-and-set +- `Catalog` replaces `get_model_list` and `get_router_model_info`. It is a read-only view of one snapshot, so a model list change mid-request can't hand a strategy two different views. It also answers the question `_tier_params_the_target_accepts` asks today +- `Eligibility` replaces calling `async_get_healthy_deployments` on a copy of the request kwargs, which the health gate, the modality gate and cache-aware routing all do. It also covers the caller's access to the group (`can_key_call_resolved_model`). The host answers it with the same filters and cooldowns its own pick will use, which is the point: a Rust host answers from Rust's cooldowns, the Python host from Python's +- `ModelClient` replaces calling `router.acompletion` / `aresponses` / `aembedding` with hand-built metadata. The client attaches spend attribution, message logging and session linking from `on_behalf_of` itself, so the strategy never forwards caller metadata +- `StrategyStateStore` replaces the adaptive router's direct Prisma reads and its update queue the proxy flusher drains. The host decides where state lives and when it flushes +- `clock` and `rng` make the bandit sampling, random tier picks and TTL math deterministic in tests + +## What stays with the host + +These run once per attempt around `route`, today inside `PythonRouter.async_pre_routing_hook`, and are host code, not strategy code, so they're written once per host: +- resolving aliases and picking which registered strategy owns the requested name, by tags (`_select_pre_routing_strategy`) +- the Claude Code session router binding (`_resolve_claude_code_session_router`) +- routing plugins (`Router(plugins=...)`) and member authorization (`authorize_member_auto_router_inference`) +- routing-only compression of the conversation before `route` sees it +- applying a `Route`: dropping overlay params no deployment accepts, forwarding the marker's params, stamping `routing_decision`, consumed tags and the affinity TTL for the filters that read them, arming compaction, adding the response headers + +## Where each strategy needs work to fit + +- adaptive: closest already. `pick_model` is pure apart from sampling; it needs `rng`, `feedback_key` and `observe` in place of the CustomLogger, and `state` in place of Prisma and the update queue +- quality: small. Drop the metadata write for `quality_router_decision` in favor of `provenance`, and build the scorer from the complexity router's pure classifier instead of a whole `ComplexityRouter` holding the router +- auto: small. The encoder takes `ModelClient.embed` instead of the router +- complexity: the bulk of the work. Session and deployment pins move to `SessionStore`, the context, modality and health gates move to `Catalog` and `Eligibility`, the LLM and JEV classifiers move to `ModelClient`, plan mode, escalation and the adaptive mode stop writing metadata, and compaction becomes a `CompactionPlan` on the decision. Splitting `complexity_router.py` along those lines would fall out of the same change + +## Open questions + +- whether `Eligibility.can_serve` should answer per group or return the eligible deployment ids, since cache-aware routing wants to know whether one specific deployment is live +- whether custom classifier plugins (`ClassifierPlugin`) and routing plugins keep their current `RoutingContext` contract or move onto `RoutingRequest` +- how far `NativePayload` should be typed: enough for the encrypted classifier task, or the full surface payload diff --git a/litellm-rust/crates/router/POC.md b/litellm-rust/crates/router/POC.md index 2c5bee232e1..5f31fa4fcbb 100644 --- a/litellm-rust/crates/router/POC.md +++ b/litellm-rust/crates/router/POC.md @@ -138,6 +138,8 @@ Decision (2026-10-07): Option C. Rust drives the loop, and the strategy routers - context compaction and routing compression are armed through contextvars in the hook and read by the inference call, so the hook and `litellm.` have to run in the same Python task, which `Select` does and per-hook ops would not - under `Select`, the health and prompt-cache gates still read the Python-side cooldown cache, which is empty, so the Python-side router needs Rust's cooldowns as its cooldown view, not only as `_excluded_deployment_ids` for the routed group +Known sore spot (2026-10-07): the strategies have no clear interface, which is what makes calling them from Rust awkward. A contract they should move to is drafted in `POC-strategy-contract.md`. Implementing it needs refactors on the Python side first, so it waits until later + ### Increment 4: what the proxy needs to switch over Today the proxy always gets `PythonRouter`, because it builds `Router(...)` with arguments the Rust backend declines (`router_general_settings`, `search_tools`, `ignore_invalid_deployments`, `fallback_access_check`, `fallback_budget_check`, `auto_router_capability_limit`). Switching over needs three things: @@ -214,6 +216,6 @@ Increment 2 (runtime model list changes): `RustRouter.upsert_deployment`, `add_d Suite survey (2026-10-07, every test run on both backends with Rust pinned to `RUST_REQUIRED`): no Rust-only failure was a wrong routing decision. In `tests/local_testing/test_router_{fallbacks,retries,timeout}.py`, 23 tests pass on Python and fail on Rust: 11 use configs the backend declines (rpm/tpm, wildcards, routing strategies, dict fallback entries, `allowed_fails_policy`) and 12 patch a `PythonRouter` internal (`should_retry_this_error`, `_time_to_sleep_before_retry`, `make_call`, `log_retry`, `_get_stream_timeout`); those modules also hold live tests that need provider keys, so they stay Python-only. In `tests/unit/test_router/test_router.py` 777 of 992 fail on Rust: 248 declined configs, and the rest call `PythonRouter` internals (`async_function_with_fallbacks_common_utils`, the Anthropic stream helpers, ...) or read views increment 4 has to provide (`get_model_group_info`, `get_deployment_by_model_group_name`, `get_deployment_model_info`, `get_model_list`, `get_model_access_groups`, `get_deployment_credentials_with_provider`, ...). Suites that test Python helpers directly (`router_utils/test_fallback_event_handlers.py`, `test_cooldown_handlers.py`, `litellm_core_utils/test_fallback_generalizations.py`) never call the router's own methods, so running them twice adds nothing. Opted in: `tests/unit/test_router_streaming_fallback_metadata.py`, and `tests/unit/test_router_retry_policy_update.py` with increment 2. The parity suites under `tests/test_litellm_rust/router` carry the behavior coverage instead -Next: prototype increment 3 with Option C (`Select` target first, then `AllowFallback`), then increment 4 +Next: increment 3 waits on the Python-side strategy refactor toward `POC-strategy-contract.md`; Option C (`Select`, then `AllowFallback`) is still the plan once it lands Known gaps to close or decline: CustomLogger fallback-event hooks and `router_cooldown_event_callback` (increment 3), the 200-key `InMemoryCache` eviction, fallback keys containing `/` (provider-prefixed matching needs the cost map)