From a607d9657cf6747f9c949d8bfaed306d801ddf7d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 21:58:45 +0000 Subject: [PATCH] docs(router): record the proxy switchover in the poc notes Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Aw1xktpUnix5tWhvSzey8t --- litellm-rust/crates/router/POC.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/litellm-rust/crates/router/POC.md b/litellm-rust/crates/router/POC.md index 5f31fa4fcbb..acc0f77f725 100644 --- a/litellm-rust/crates/router/POC.md +++ b/litellm-rust/crates/router/POC.md @@ -126,7 +126,7 @@ Option B, hand selection to Python for configs that use these features. The atte - good: still one crossing per attempt, exact Python behavior for every selection feature at once, and the same path could later carry tags, `order`, team models, aliases and wildcards, which is most of what the proxy needs - costs: for these configs Rust no longer makes the pick, so the POC's claim that Rust drives filter and pick only holds for plain configs. Python's "no deployments" error reads Python's own cooldown cache for its message and cooldown time, so a rejection from Python selection needs mapping onto Rust's state -Option C, hybrid (recommended). Option B for everything on the selection side, and Option A only for the hooks that sit inside the loop Rust owns: an `AllowFallback` op before each fallback target, and notifications folded into the ops Python already applies at the end of a call. That gets every selection feature to parity cheaply, keeps the fallback checks where the proxy expects them, and leaves room to port simple filters (tags, `order`, rate limits) to native Rust later, one at a time +Option C, hybrid (recommended). Option B for everything on the selection side, and Option A only for the hooks that sit inside the loop Rust owns: an `AllowFallback` op before each fallback target (built in increment 4 as `FallbackCheck`), and notifications folded into the ops Python already applies at the end of a call. That gets every selection feature to parity cheaply, keeps the fallback checks where the proxy expects them, and leaves room to port simple filters (tags, `order`, rate limits) to native Rust later, one at a time Option D, keep declining and port features natively. Pure-config filters could move to Rust as native code, but the model-driven routers (auto, complexity, adaptive, quality) and custom strategies or plugins are user or ML code that has to stay in Python, so they need A or B eventually anyway @@ -142,7 +142,7 @@ Known sore spot (2026-10-07): the strategies have no clear interface, which is w ### Increment 4: what the proxy needs to switch over -Today the proxy always gets `PythonRouter`, because it builds `Router(...)` with arguments the Rust backend declines (`router_general_settings`, `search_tools`, `ignore_invalid_deployments`, `fallback_access_check`, `fallback_budget_check`, `auto_router_capability_limit`). Switching over needs three things: +Done 2026-10-07 (see the checkpoint notes). Before it, the proxy always got `PythonRouter`, because it builds `Router(...)` with arguments the Rust backend declines (`router_general_settings`, `search_tools`, `ignore_invalid_deployments`, `fallback_access_check`, `fallback_budget_check`, `auto_router_capability_limit`). Switching over needs three things: - accept the proxy's constructor arguments. The two fallback checks come from increment 3. The others are settings the Rust backend has to honor or can safely pass to the Python side - serve the proxy's reads. 116 proxy files use about 109 distinct router members, about 386 times. Most uses read the model list (`get_model_list`, `get_deployment`, `model_list`, `get_model_names`, `get_model_group_info`, `get_model_access_groups`, `get_deployment_credentials_with_provider`, ...), and those become views of the snapshot. The rest reach into live internals (`cache`, `pattern_router`, `adaptive_routers`, `deployment_latency_map`, `health_state_cache`, `model_name_to_deployment_indices`, ...). Each of those gets a stand-in, or becomes a typed method on the facade, which is where the provisional full-surface decision gets settled - serve the request path. `route_llm_request` calls `getattr(llm_router, route_type)(**data)` for every operation, so one instance serves chat, embeddings, files, batches, passthrough and the rest @@ -216,6 +216,21 @@ Increment 2 (runtime model list changes): `RustRouter.upsert_deployment`, `add_d Suite survey (2026-10-07, every test run on both backends with Rust pinned to `RUST_REQUIRED`): no Rust-only failure was a wrong routing decision. In `tests/local_testing/test_router_{fallbacks,retries,timeout}.py`, 23 tests pass on Python and fail on Rust: 11 use configs the backend declines (rpm/tpm, wildcards, routing strategies, dict fallback entries, `allowed_fails_policy`) and 12 patch a `PythonRouter` internal (`should_retry_this_error`, `_time_to_sleep_before_retry`, `make_call`, `log_retry`, `_get_stream_timeout`); those modules also hold live tests that need provider keys, so they stay Python-only. In `tests/unit/test_router/test_router.py` 777 of 992 fail on Rust: 248 declined configs, and the rest call `PythonRouter` internals (`async_function_with_fallbacks_common_utils`, the Anthropic stream helpers, ...) or read views increment 4 has to provide (`get_model_group_info`, `get_deployment_by_model_group_name`, `get_deployment_model_info`, `get_model_list`, `get_model_access_groups`, `get_deployment_credentials_with_provider`, ...). Suites that test Python helpers directly (`router_utils/test_fallback_event_handlers.py`, `test_cooldown_handlers.py`, `litellm_core_utils/test_fallback_generalizations.py`) never call the router's own methods, so running them twice adds nothing. Opted in: `tests/unit/test_router_streaming_fallback_metadata.py`, and `tests/unit/test_router_retry_policy_update.py` with increment 2. The parity suites under `tests/test_litellm_rust/router` carry the behavior coverage instead -Next: increment 3 waits on the Python-side strategy refactor toward `POC-strategy-contract.md`; Option C (`Select`, then `AllowFallback`) is still the plan once it lands +Increment 4 (proxy switchover): a proxy started with `LITELLM_RUST=1` now routes on the Rust router from startup through its request path. What it took: +- the proxy's constructor arguments: `router_general_settings`, `search_tools`, `assistants_config`, `auto_router_capability_limit` and `cache_responses` go to the Python side as they are; `ignore_invalid_deployments` lets the support check skip entries it cannot read, as Python drops them; `cache_responses` is declined only when it would build `litellm.cache` from the router's own Redis arguments +- the fallback checks: `fallback_access_check` and `fallback_budget_check` are a `FallbackCheck` host op the engine sends before each fallback target, only when one is configured (`settings.fallback_checks`), with the pending ops applied first so the checks read the request as the failed hop left it. A rejected target is skipped and not recorded as attempted. Python's sync calls run the async fallback loop, so the sync driver runs the checks too (through `run_async_function`), which the `fallback_access_check` docstring gets wrong +- the proxy's always-registered `_PROXY_VirtualKeyModelMaxBudgetLimiter` overrides `async_filter_deployments` only to return every deployment, so the support check treats it as a pass-through instead of declining every proxy +- the proxy's reads (`rust_surface.VIEWS`, about 70 members: model list lookups, model group info, access groups, credentials, pattern and team lookups, the strategy registries, `cache`, health state, `_zero_cost_cache`, ...) are answered by the Python-side router, which increment 2 keeps in step with the snapshot. `cache_responses`, `search_tools` and `guardrail_list` can be assigned there (`ASSIGNABLE`). `arefresh_model_info` refreshes the Python side's model list and re-projects the snapshot, and `arm_routing_read_prefetch` is a no-op because the native router batches its own reads. `anthropic_messages` is the same entry point as `aanthropic_messages`, as in Python +- the proxy's Redis: `_update_redis_cache` attaches it to the Python side and rebuilds the native router on the same URL (dropping its in-memory state, which is empty at startup), so cooldowns land in the shared Redis and Python-side reads see them. A cluster, sentinel or TLS connection hands over instead +- everything else hands over for good, per the 2026-10-07 decision: reading or setting any other `Router` member, a request option the engine does not serve (`specific_deployment`, which the proxy sends for a litellm model name, client-side credentials, per-key or per-team `routing_strategy`, `enable_tag_filtering`, `model_group_retry_policy`, `_router_weights`, `priority`, `include_fallback_errors`), or a runtime change it declines. The `PythonRouter` is built from the current model list, settings updates, assigned attributes and Redis, the facade swaps to it, and the `RustRouter` forwards anything still holding it. Under `RUST_REQUIRED` each of these raises `RustRouterUnsupportedError` instead. Selection logs "Routing with the Rust router" so an operator can tell which backend runs -Known gaps to close or decline: CustomLogger fallback-event hooks and `router_cooldown_event_callback` (increment 3), the 200-key `InMemoryCache` eviction, fallback keys containing `/` (provider-prefixed matching needs the cost map) +Verified against a live proxy (mock deployments, a curl sweep run once per backend): chat (plain and streamed), Responses and `/v1/messages` with fallbacks, `/v1/models`, `/model/info`, `/model_group/info` and readiness give the same statuses, routing headers and bodies on both backends, apart from which of two equal-weight deployments the unseeded shuffle picks. An embeddings request hands over with one log line and later chat calls keep routing. With `cache: true` on Redis, both backends leave the same cooldown and `allowed_fails` keys with the same TTLs. Not exercised live: the database-backed model sync (no database in this environment), which goes through the increment 2 path that the parity tests cover + +Gaps left by increment 4: +- without a shared Redis, the Python-side reads that consult cooldowns (`async_get_healthy_deployments`, `async_get_available_deployment` for the token counter) see none, since the cooldowns live in the native router's memory +- the native router does not write the per-minute tpm/rpm usage keys Python's success callback writes, so `get_model_group_usage` hands over; `get_remaining_model_group_usage` stays a view because it only reads usage for groups with tpm or rpm limits, which the Rust router declines +- any read of a member outside `VIEWS` hands over, `hasattr` probes included, so a proxy feature that reads a new member moves the instance to Python until it is added + +Next: iterate on the shape of the facade over both backends (`litellm/router.py` and `rust_surface.py` are where the current answer lives). Increment 3 waits on the Python-side strategy refactor toward `POC-strategy-contract.md`; Option C (`Select`, then `FallbackCheck`) is still the plan once it lands + +Known gaps to close or decline: the increment 4 gaps above, CustomLogger fallback-event hooks and `router_cooldown_event_callback` (increment 3), the 200-key `InMemoryCache` eviction, fallback keys containing `/` (provider-prefixed matching needs the cost map)