diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f457720c58d..765c8c2db77 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -124,7 +124,7 @@ graph TD - `proxy/proxy_server.py` - Main API endpoints - `proxy/auth/` - Authentication (API keys, JWT, OAuth2) - `proxy/hooks/` - Proxy-level callbacks -- `router.py` - Load balancing, fallbacks +- `router.py` - Load balancing, fallbacks (see [docs/architecture/router](docs/architecture/router/README.md)) - `router_strategy/` - Routing algorithms (`lowest_latency.py`, `simple_shuffle.py`, etc.) **LLM-specific proxy endpoints:** diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 00000000000..3652f7f18dc --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,25 @@ +# Architecture + +This folder explains how the parts of LiteLLM work inside, for the people who +change them. It covers one component so far, the router. More will follow as the +Rust migration moves each module behind a public interface and into Rust. + +Each component's docs follow the [Diataxis](https://diataxis.fr/) layout, so you +can choose the kind of page that fits what you need right now. + +![A two-by-two grid of the four kinds of pages: tutorials, how-to guides, +explanation and reference, placed on a studying-to-working axis and a +theory-to-practice axis](img/diataxis-quadrants.svg) + +| Kind | Read it when | Router | +| --- | --- | --- | +| Tutorial | you are new to the component and want to see it work | [Route your first requests](router/tutorial/route-your-first-requests.md) | +| How-to guide | you have a specific task to get done | [router/how-to](router/how-to/) | +| Reference | you need an exact name, default, key or signature | [router/reference](router/reference/) | +| Explanation | you want to know why it works the way it does | [router/explanation](router/explanation/) | + +Decisions that are hard to reverse are recorded separately as ADRs in +[`docs/adrs/`](../adrs/AGENTS.md). These pages describe the code as it is on the +branch they ship with, and they link to the source so you can check them. When +the code and a page disagree, the code is right, and the page needs a fix in the +same PR that changed the behavior. diff --git a/docs/architecture/img/README.md b/docs/architecture/img/README.md new file mode 100644 index 00000000000..dd250af7746 --- /dev/null +++ b/docs/architecture/img/README.md @@ -0,0 +1,17 @@ +# Diagrams + +The architecture docs use hand-laid-out SVGs instead of Mermaid, so each diagram +fits a docs column without wasted space. + +- Every SVG embeds [diagram-style.css](diagram-style.css) verbatim in one + ` + + + + +Four kinds of pages + +Tutorials + +How-to guides + +Explanation + +Reference +Practice + +Theory +Studying + +Working + diff --git a/docs/architecture/router/README.md b/docs/architecture/router/README.md new file mode 100644 index 00000000000..e1404a0358c --- /dev/null +++ b/docs/architecture/router/README.md @@ -0,0 +1,70 @@ +# Router + +`litellm.Router` decides which deployment serves a request, makes the call, and +handles what happens when the call fails. It retries in the same model group, +falls back to other model groups, and puts failing deployments into cooldown so +that later requests avoid them. The proxy builds one Router from its config and +sends every model request through it. SDK users can build their own. + +![A caller calls litellm.Router, which picks a deployment, calls the provider, +and keeps cooldowns, usage and fail counts in the routing +cache](img/router-overview.svg) + +## Concepts + +Start with [Router concepts](explanation/concepts.md), which names the ideas and +shows how they fit together. Each idea then has its own page. + +- [Deployments and model groups](explanation/deployments-and-model-groups.md), + what is configured and how a deployment gets its identity +- [How a name finds its deployments](explanation/names.md), aliases, wildcards, + routing groups, team names, deployment ids and defaults +- [Teams and access](explanation/teams-and-access.md), team deployments, access + groups, access windows and PTU shares +- [How a deployment is picked](explanation/selection.md), the filter chain and + priority with `order` +- [Strategies, routing groups and sub-routers](explanation/strategies.md), how + one deployment is chosen and how a request is sent to the right group +- [Retries, fallbacks and cooldowns](explanation/reliability.md), including + allowed fails, order fallbacks, weighted failover and streams +- [Shared state across proxy workers](explanation/shared-state.md) + +Two pages cover how the router is built. + +- [The core and the host](explanation/core-and-hosts.md), what the Rust core + owns and what it asks its host for +- [The life of a request](explanation/request-lifecycle.md), from `acompletion` + through the core to the provider + +## Tutorial + +New to the router? Work through [Route your first +requests](tutorial/route-your-first-requests.md). It builds a Router with mock +deployments, so it needs no API keys. It shows load balancing, retries, +cooldowns and fallbacks in about fifteen minutes. + +## How-to guides + +- [Read router state from outside the router](how-to/read-router-state.md) +- [Change deployments or settings at + runtime](how-to/change-the-router-at-runtime.md) +- [Move a caller off a deprecated Router + member](how-to/move-off-deprecated-members.md) +- [Add a custom routing strategy](how-to/add-a-routing-strategy.md) +- [Find out why a request went where it did](how-to/debug-a-routing-decision.md) + +## Reference + +- [Entry points](reference/entry-points.md), every method on the public surface + and what it replaced +- [Core, config and host interfaces](reference/core-and-host.md), the step + protocol, the Rust `Router` and the `Host` trait +- [Configuration and defaults](reference/configuration.md), constructor + arguments, strategies and constants +- [Routing state and cache keys](reference/state-and-cache-keys.md), what the + router stores and for how long +- [Code map](reference/code-map.md), which file holds what + +## Status + +These pages describe the branch `nate_router_step6_rust_core_v2` (#49513). diff --git a/docs/architecture/router/explanation/concepts.md b/docs/architecture/router/explanation/concepts.md new file mode 100644 index 00000000000..182873abb7c --- /dev/null +++ b/docs/architecture/router/explanation/concepts.md @@ -0,0 +1,74 @@ +# Router concepts + +The router is responsible for answering one question for every request: which +deployment should serve it. It then has to deal with what happens when that +deployment fails, retrying the request, and falling back to other deployments. + +There are a few main ideas that are referenced through the rest of these docs. +This page gives a brief overview of them. + +![The router concepts in order: a name resolves to deployments, access rules, +filters and a strategy pick one for an attempt, and a failure goes to retries, +fallbacks and cooldowns, which write the shared state the filters +read](../img/concepts-1.svg) + +## What is configured + +A **deployment** is one place that can serve a model, such as an Azure OpenAI +resource in one region or a vLLM cluster behind a URL. It is one entry of +`model_list`, with the provider call's arguments in `litellm_params` and +metadata in `model_info`. Its `model_info.id` is the key for all of its routing +state. See [Deployments and model groups](deployments-and-model-groups.md). + +A **model group** is every deployment that shares a `model_name`. Callers ask +for a model group, never for a deployment. Load balancing, retries and cooldowns +happen inside one group, and fallbacks move a request between groups. + +A **name** is what a request asks for. It is usually a model group, but it can +also be an alias, a wildcard such as `openai/*`, a routing group, a team's +public name for a model, or a deployment id. **Resolution** turns a name into +the list of deployments that may serve the request. See [How a name finds its +deployments](names.md). + +## Who may use what + +A deployment can belong to one **team**, be reserved for some teams during an +**access window**, be split across teams by **PTU shares**, or sit behind an +**access group** that only some keys reach. These rules run right after +resolution and remove the deployments the caller may not use. See [Teams and +access](teams-and-access.md). + +## How one deployment is picked + +A **filter** removes deployments that must not serve this request right now: +ones that are failing health checks, cooling down, blocked by an admin, too +small for the prompt, missing a tag, or at a higher **order** (a lower priority) +than another healthy one. See [How a deployment is picked](selection.md). + +A **strategy** picks one deployment from the remaining eligible ones. +`simple-shuffle`, the default, picks a weighted random one. Others pick by load, +latency, usage or cost. A **routing group** gives a set of model groups its own +strategy, and **sub-routers** such as the auto router choose which model group a +request goes to before any of this runs. See [Strategies, routing groups and +sub-routers](strategies.md). + +## What happens on failure + +A **retry** runs the pick again in the same model group, so it can land on a +different deployment. A **fallback** moves the request to another model group +once retries are spent. A **cooldown** takes one failing deployment out of every +pick for a few seconds, and **allowed fails** decide how many failures that +takes. See [Retries, fallbacks and cooldowns](reliability.md). + +## State + +Cooldowns, failure counters and usage live in a **routing cache** with an +in-memory tier and an optional Redis tier, so a fleet of proxy workers shares +one view of which deployments are healthy. See [Shared state across proxy +workers](shared-state.md). + +## How it is built + +You do not need this section to use the router. For how the router is built +inside and how it runs a request, see [The core and the host](core-and-hosts.md) +and [The life of a request](request-lifecycle.md). diff --git a/docs/architecture/router/explanation/core-and-hosts.md b/docs/architecture/router/explanation/core-and-hosts.md new file mode 100644 index 00000000000..b8668ecdda2 --- /dev/null +++ b/docs/architecture/router/explanation/core-and-hosts.md @@ -0,0 +1,136 @@ +# The core and the host + +The router has three layers. `litellm.Router` is a thin Python facade that +callers use. Behind it, the Rust crate `litellm-router-core` (the core) makes +every routing decision and owns the state those decisions need. The core reaches +outside itself only through a host, which runs the provider calls and anything +else that needs live objects from the process the core runs in. This page +explains that split and why the line is drawn where it is. + +![litellm.Router and the RustCore driver over the Rust crate, which hands steps +to PythonHost, which calls the provider](../img/core-and-hosts-1.svg) + +## What the core owns + +The core owns the routing decisions and their state. + +- the registry of deployments, its indexes and the deployment build +- name resolution, the filter chain and `simple-shuffle` +- teams, access groups, access windows and PTU shares +- routing groups, pre-call check decisions, health-check routing and the order + filter +- the call loop: retries, retry policies, fallbacks, order fallbacks, weighted + failover and stream recovery +- cooldowns, allowed fails counters and usage, written through the routing cache +- model info and model group info, merged from what the host reports + +Everything the core decides is a function of its config, the request, the +routing cache, and the answers the host gave. It never reads process globals, +never calls a provider and never awaits Python. That is what makes the core +testable with a scripted host and portable to a host that is not Python. + +## What the host answers + +The core asks for what it needs as a step, and the host answers it. Every +attempt is a step: the core picks the deployment, the host runs the call, and +the core records the outcome. There is no second attempt path inside the core. +The `Host` trait (`litellm-rust/crates/router-core/src/host.rs`) groups the +hostcalls. + +| Hostcall | What the core needs | +| --- | --- | +| `invoke` | run one attempt on one deployment | +| `select` | a strategy implemented in Python, or the routing plugin, tag or text-only stage | +| `pre_route` | the sub-routers and session binding before a pick resolves its name, or a whole pick by a custom strategy | +| `pre_call_facts`, `callback_filters`, `routing_plugin_models` | the facts and filters that need live objects, such as token counts and `CustomLogger` filters | +| `allow_fallback` | the access and budget checks before a cross-group fallback | +| `secrets`, `validate`, `provider`, `added` | the deployment build, one deployment at a time | +| `model_info` | cost map and capability lookups during a model info query | +| `sleep`, `observe`, `cache` | backoff, routing events, and the shared store all routing state lives in | + +The rule for where a piece of logic goes is simple. If a decision needs only +config, the request, routing state and plain facts, it belongs in the core. If +it needs a live object, such as a stateful strategy, a logging callback or the +provider SDK, the host runs that part and hands the core a plain answer. The +core still makes the decision. Pre-call checks show the pattern: the host counts +the prompt's tokens and looks up context windows, and the core decides which +deployments to drop. + +## The host today + +Today the core runs inside the Python process, under `PythonHost` +(`litellm/router_python_host.py`) and the modules under `litellm/router_rust/`. + +- `invoke/` runs one attempt. `litellm/router_attempt.py` builds the keyword + arguments for the deployment the core picked, and `PythonHost.invoke(op)` + calls `litellm.` with them. The endpoint wrappers, stream wrappers, + `max_parallel_requests` slots and retry and fallback metadata live here. + Nothing here picks, retries, falls back or records cooldowns +- `steps/` answers the hostcalls that need live Python objects: the stateful + strategies, the sub-routers, pre-call facts, `Router(plugins=[...])` objects, + `CustomLogger` filters and caller access facts +- `surface/` keeps the Python `Router` API over the core's state. `registry()`, + `model_list`, the indexes and `names()` read the core, and every write to them + becomes a change set + +A host that answers these steps natively, inside a Rust gateway, would need none +of `surface/`. Through `Host::supports` and `Host::can_select`, it would refuse +a config that needs Python objects when the router is built, rather than failing +partway through a request. + +## Driving the core + +`RustCore` holds one `RouterCoreHandle`. A call starts with +`RouterCoreHandle.start(op, request)` and then loops: each +`RouterCall.next(answer)` runs the core until it needs something only the host +can give, and returns that need as a step. `select()` and the model info queries +are driven the same way. + +![The step loop for one call: the driver reads cooldowns, tries deployment a and +gets a 429, records a cooldown, tries deployment b, and returns its response +with routing headers](../img/core-and-hosts-2.svg) + +Every step also carries the routing events since the last one (`Selected`, +`AttemptFailed`, `CooldownSet`, `Retry`, `Fallback` and others). The driver +hands them to the host's observers, which feed alerting and metrics, and uses +them to stamp the request's metadata and to raise the exception a caller +expects. + +## Outcomes are recorded by the core + +The core records each attempt's outcome itself, through cache steps, right after +the attempt, so nothing depends on a logging callback firing. A streamed attempt +is deferred: the core counts the request when the stream opens, and when the +stream ends the host's stream wrapper reports either the success, with the final +usage, or the failure. + +## The public surface + +Code outside the router uses the endpoint methods and eight entry points, and +nothing else: `registry()`, `settings()`, `health()`, `select()`, `apply()`, +`observe()`, `register_plugin()` and `retire()`. Routing plugins get a +`RoutingContext` with the same views instead of the Router. +`tests/code_coverage_tests/router_surface_check.py` keeps an allowlist of the +members that outside code touches, and the list may only shrink. See [Entry +points](../reference/entry-points.md). + +## Settings are static + +`Router(**arguments)` freezes its arguments into a `RouterConfig`, and the core +is built from it. After that, `apply(ChangeSet(...))` is the only way to change +deployments or settings. The process settings the router reads while serving, +such as `litellm.num_retries`, the fallback lists, the model cost map and the +callback lists, arrive as a versioned `RouterDynamicConfig` snapshot, so the +core never reads a global. + +![How the router's settings reach the core: constructor arguments frozen into +RouterConfig, process settings snapshotted into RouterDynamicConfig, and +apply(ChangeSet) for later changes](../img/core-and-hosts-3.svg) + +## The application host + +Separately, the router needs a few answers that only the proxy has, such as +vertex pass-through credentials and whether a caller may use a team's auto +router. It asks a `RouterHost` (`litellm/router_host.py`). That is the proxy's +`ProxyRouterHost` when the proxy can be imported, and otherwise a default that +refuses team auto routers. diff --git a/docs/architecture/router/explanation/deployments-and-model-groups.md b/docs/architecture/router/explanation/deployments-and-model-groups.md new file mode 100644 index 00000000000..06030a53b72 --- /dev/null +++ b/docs/architecture/router/explanation/deployments-and-model-groups.md @@ -0,0 +1,102 @@ +# Deployments and model groups + +Everything the router routes to is a deployment, and every request asks for a +name, which is usually a model group. This page explains those, lists the +deployment fields the router itself reads, and shows how a deployment gets its +identity. + +## Deployment + +A deployment is one place that can serve a model, such as an Azure OpenAI +resource in `eastus`, a Bedrock model in one AWS account, or a vLLM cluster +behind a URL. In config, it is one entry of `model_list`. + +```yaml +model_list: + - model_name: gpt-4o # the model group callers ask for + litellm_params: # the provider call's arguments + model: azure/gpt-4o-eastus + api_base: https://eastus.example.openai.azure.com + api_key: os.environ/AZURE_EASTUS_KEY + rpm: 600 + order: 1 + model_info: # metadata about the deployment + id: gpt-4o-eastus +``` + +![A deployment and its two parts: litellm_params, the provider call's arguments, +and model_info, its metadata](../img/deployments-and-model-groups-1.svg) + +`litellm_params` holds mostly the arguments for the provider call, which the +router passes on unchanged when the deployment is picked. The router reads a few +of them itself. + +| Field | What the router does with it | +| --- | --- | +| `weight`, `rpm`, `tpm` | `simple-shuffle` weights, in that order of preference. `rpm` and `tpm` are also limits for usage-aware strategies and pre-call checks | +| `order` | in-group priority, lowest first. See [How a deployment is picked](selection.md) | +| `tags` | which tagged requests it serves, when tag filtering is on | +| `use_in_pass_through` | whether [pass-through routes](../../../../litellm/proxy/pass_through_endpoints/architecture.md) may use it | +| `timeout`, `stream_timeout` | the attempt's timeout, which wins over the router's | +| `max_parallel_requests` | how many attempts may run on it at once | + +`model_info` is metadata. The router reads who may use the deployment +(`team_id`, `access_groups`, `access_windows`, `ptu_shares`, see [Teams and +access](teams-and-access.md)), whether an admin paused it (`blocked`), and how +it cools down (`cooldown_time`, `allowed_fails`, `allowed_fails_policy`, see +[Retries, fallbacks and cooldowns](reliability.md)). Everything else is kept as +written and handed back to callers that read the registry. + +## Identity + +`model_info.id` is the deployment's identity. Cooldowns, failure counters, usage +and health results are all keyed by it, and a change set replaces or deletes a +deployment by it. When the config leaves it out, the router derives one with +`deployment_id()`: a SHA-256 over the model group name and every +`litellm_params` key and value, in order. The same config gets the same id on +every worker, which is what lets workers share routing state. Two entries with +the same id both stay in the group and share that state, and a lookup by id +returns the later one. + +## How a deployment is built + +The router builds deployments from config in order. For each entry it resolves +`os.environ/` secret references, validates the entry, resolves the provider that +serves it, and records it. An entry with an `organization` list becomes one +deployment per organization. A deployment joins the registry as soon as it is +accepted, so a later entry sees the earlier ones, and an error points at the +entry that caused it. An entry that fails to build is skipped, and the rest +still build. + +![For each model_list entry the router resolves secrets, validates the entry, +finds the provider, records it, then gives it an id and adds it to the +registry](../img/deployments-and-model-groups-2.svg) + +## Model group + +A model group is every deployment that shares a `model_name`. Callers ask for a +group, never for a deployment, and the router picks a deployment inside it. Load +balancing, retries and cooldowns all happen inside one group, and fallbacks move +a request from one group to another. + +![A request for gpt-4o reaches the gpt-4o model group of three deployments at +two orders, which falls back to the claude-sonnet model +group](../img/deployments-and-model-groups-3.svg) + +Group size changes behavior in a few places. In a group of one: a 429 does not +cool the only deployment down, an auth error is not retried, and the default +error-rate rule leaves the lone deployment alone unless it saw 1,000 requests +that all failed. Those rules exist because cooling down the only deployment +would fail every request for the group. [Retries, fallbacks and +cooldowns](reliability.md) has the details. + +## The registry + +All deployments live in the registry, an immutable, indexed snapshot held by the +router. It is indexed by id, model group, alias, wildcard pattern and team, and +it carries a version that goes up on every change. A change replaces the +snapshot, so a request that already read the registry keeps a consistent view +while the change lands. Code outside the router reads it through +`router.registry()` and changes it through `router.apply()`, see [Read router +state](../how-to/read-router-state.md) and [Change deployments or settings at +runtime](../how-to/change-the-router-at-runtime.md). diff --git a/docs/architecture/router/explanation/names.md b/docs/architecture/router/explanation/names.md new file mode 100644 index 00000000000..18f2b2a25c9 --- /dev/null +++ b/docs/architecture/router/explanation/names.md @@ -0,0 +1,76 @@ +# How a name finds its deployments + +A request names a model, and the name does not have to be a model group. It can +be an alias, a wildcard match, a routing group, a team's own name for a model, +or a deployment id. Resolution turns the name into the list of deployments that +may serve this caller, which is used for later steps. Resolution runs at the +start of every pick, so a retry or a fallback resolves the name again. + +## The kinds of name + +| Name | Example | Request names | Resolves to | +| --- | --- | --- | --- | +| Model group | `gpt-4o` | `gpt-4o` | the deployments with that `model_name` | +| Alias | `model_group_alias={"gpt-4": "gpt-4o"}` | `gpt-4` | the target group. `{"model": "gpt-4o", "hidden": true}` also hides the alias from model listings | +| Routing group | `routing_groups=[{"group_name": "fast", "models": [...]}]` | `fast` | the deployments of every member group, see [Strategies, routing groups and sub-routers](strategies.md) | +| Wildcard | a deployment with `model_name: "openai/*"` | `openai/gpt-4o-mini` | that deployment, with the requested model filled into its params | +| Team public name | `model_info.team_public_model_name: "my-gpt"` | `my-gpt` | the team's deployment, for members of that team | +| Deployment id | `gpt-4o-eastus` | `gpt-4o-eastus` | that one deployment, even while it cools down | +| Specific deployment | a deployment's `litellm_params.model` | `azure/gpt-4o`, with `specific_deployment=True` | the deployments whose litellm model is exactly that name | +| Anything else | | `unknown-model` | the `default_deployment` if one is set, else the first `default_fallbacks` group, else an error | + +## The order + +The checks run in a fixed order, and the first match wins. + +![How a name resolves, in order: specific_deployment, deployment id, alias, +routing group, configured group, then the litellm model name, the first +default_fallbacks group, or an error](../img/names-1.svg) + +Two rules apply on every path. Deployments that are sub-router markers, such as +`auto_router/...`, are never returned, since they are not meant to be called. +Deployments that an access window reserves for other teams right now are +withheld, see [Teams and access](teams-and-access.md). + +## Aliases + +An alias maps one name to a model group. Aliases come from the router's +`model_group_alias` and from the process's model alias map, which the router +reads through its dynamic config. Resolution swaps the alias for its target +before it looks anything else up, so an alias can also point at a routing group. +The attempt still reports the requested name to the caller. + +## Wildcards + +A deployment whose `model_name` contains `*`, such as `openai/*` or +`anthropic/claude-*`, serves any name it matches that no model group claims. The +registry keeps the patterns ordered from most to least specific, so +`anthropic/claude-3-*` wins over `anthropic/*`. The matched part is substituted +into the deployment's `litellm_params.model`: a deployment with +`model_name: openai/*` and `model: openai/*` serves a request for +`openai/gpt-4o-mini` by calling `openai/gpt-4o-mini`. Wildcards are checked only +for a name that is not a configured model group, so a configured group always +wins over a pattern that also matches it. + +## Deployment ids + +A name that is not a model group but is a deployment id resolves to that one +deployment. The router serves it even while it cools down, and skips the +strategy, since there is nothing to choose. This is how a caller pins a request +to one deployment. + +## Default deployment and default fallbacks + +`default_deployment` is a deployment that serves any name nothing else matched, +with the requested model filled in. Without one, a name that resolves to nothing +tries `default_fallbacks`, the catch-all fallback list, and resolves to its +first model group. If that list is empty too, the pick fails with an error that +says no deployments are configured for the model. + +## The result + +Resolution returns three things: the deployments, which kind of name matched, +and the model group the attempt reports. For an alias or a routing group, that +model group is the requested name, so headers and logs show what the caller +asked for while the deployments come from the target. [How a deployment is +picked](selection.md) takes it from there. diff --git a/docs/architecture/router/explanation/reliability.md b/docs/architecture/router/explanation/reliability.md new file mode 100644 index 00000000000..c7e9284308d --- /dev/null +++ b/docs/architecture/router/explanation/reliability.md @@ -0,0 +1,200 @@ +# Retries, fallbacks and cooldowns + +The router has three tools for failures, and each works at a different scope. A +retry tries the same model group again, and the new pick can land on a different +deployment. A fallback moves the request to a different model group. A cooldown +takes one deployment out of every pick, for every request, for a few seconds. + +![Inside each model group the strategy picks one deployment. The strategy picks +D1 in gpt-4o, which returns 429 and cools down, the retry picks D3 in the same +group, which also returns 429, then the request falls back to opus, where the +strategy picks D5, which returns 200 to the router](../img/reliability-1.svg) + +Every decision reads three facts about the failure: its kind (rate limit, +context window, content policy, auth, not found, timeout, ...), its status code +and its `Retry-After`. The router takes these from the provider's exception and +decides from them alone. + +The router is the only retry loop. It sets `max_retries` to 0 on the provider +call, so neither the provider SDK nor the HTTP client retries on its own, and +each retry can pick a different deployment. + +## Retries + +The retry loop runs the first attempt. When that attempt fails, the router works +out how many retries this request gets, decides whether this failure is worth +retrying, waits, and loops. + +![The retry loop: a guardrail intervention gives up at once, otherwise the +router works out the retries, checks the failure, sleeps, selects again skipping +failed deployments, and returns, loops or hands the failure to the fallback +walk](../img/reliability-2.svg) + +A failure is not worth retrying in these cases. + +- a context window error when `context_window_fallbacks` is set, or a content + policy error when `content_policy_fallbacks` is set, because a fallback will + do better than another deployment of the same model +- a 4xx status other than 401, 403, 408, 409 and 429 +- a not found error +- a rate limit when no healthy deployment is left and fallbacks exist +- an auth or permission error in a group with a single deployment +- any failure when no healthy deployment is left + +The wait is zero while the group has more than one deployment and some are still +healthy, since the next attempt can go somewhere else. Otherwise the router +honors the provider's `Retry-After`, up to one minute. Without a `Retry-After`, +it backs off exponentially from 0.5 seconds up to 8 seconds, never less than the +router's `retry_after`, plus up to 0.75 seconds of jitter. + +### Retry policies + +`retry_policy` sets the number of retries per failure kind. For example, more +retries for rate limits than for timeouts. `model_group_retry_policy` does the +same per model group. + +```python +Router( + model_list=..., + retry_policy={"RateLimitErrorRetries": 3, "TimeoutErrorRetries": 1, "ContentPolicyViolationErrorRetries": 0}, + model_group_retry_policy={"gpt-4o": {"RateLimitErrorRetries": 5}}, +) +``` + +A policy's count for the failure's kind wins over `num_retries`, unless the +request itself turned retries off with `num_retries=0`. + +## Fallbacks + +When the retry loop gives up, the router chooses where to go next. + +![Where the router goes once retries are exhausted: raise on disable_fallbacks +or a guardrail, else the next orders in the group, weighted failover, then the +context window, content policy or generic fallbacks by failure +kind](../img/reliability-3.svg) + +Each fallback list is a list of single-key dicts, each mapping a model group to +an ordered list of model groups to try, for example +`fallbacks=[{"gpt-4o": ["claude-sonnet", "gpt-4o-mini"]}]`. A target can also be +a dict such as `{"model": "gpt-4o-mini", "temperature": 0}`, whose other keys +override the request's params for that hop. `default_fallbacks=["gpt-4o-mini"]` +becomes the catch-all entry `{"*": ["gpt-4o-mini"]}`. The router looks up the +group being served first and then the group the request asked for, so a fallback +target's own fallbacks apply on the next hop. The lists come from the router's +arguments or, when the router sets none, from the process settings in the +dynamic snapshot. A request can also carry its own `fallbacks`, which win for +that request. + +The fallback walk takes each target in order. Before a target in another model +group, it runs `fallback_access_check` (may this caller use that group) and +`fallback_budget_check`, which use the proxy's key and team data. It skips +targets this request already tried, then runs the whole retry loop again against +the target, which can fall back further. A depth counter stops the walk at +`max_fallbacks` (default 5, or the `ROUTER_MAX_FALLBACKS` environment variable). +Each hop is reported as a `Fallback` routing event with its source, target, +order, depth and level. + +![A fallback in sequence: model A fails with a 503 after three attempts, the +fallback checks allow group B, and B answers with a +200](../img/reliability-4.svg) + +With `fallbacks=[{"A": ["B"]}]`, `context_window_fallbacks=[{"A": ["A-long"]}]` +and `num_retries=2`, a 503 from A is retried twice in A, then the request goes +to B with two retries of its own. A context window error from A is not retried +at all, since a context window fallback exists. The request goes straight to +A-long. + +### Order fallbacks + +When a model group's deployments use more than one `order`, a failure first +walks down the orders inside the same group before it leaves the group. With +`order: 1` PTU deployments and an `order: 2` pay-as-you-go deployment, a 429 on +the PTU tier sends the request to pay-as-you-go in the same group, and only then +to the group's configured fallbacks. Each order level gets its own retry loop. A +context window or content policy error skips the order walk, since a lower +priority deployment of the same model would fail in the same way. + +### Weighted failover + +With `enable_weighted_failover` and `simple-shuffle`, a failed deployment is +excluded and the router picks again among the group's remaining deployments that +are not cooling down, before any configured fallback runs. This spreads a +failing deployment's traffic over its healthy siblings, by their weights, +instead of sending it to another group. Context window and content policy errors +skip weighted failover for the same reason as above. + +## Streams + +A streamed attempt succeeds as soon as the provider opens the stream, but it can +still fail while the caller reads it. The router wraps every stream it returns. +When the stream breaks before the caller has received any content, the router +takes over with a retry in the same group or the next fallback, as if the +attempt had failed up front. Once content has reached the caller, a retry would +repeat output, so the stream fails, unless +`enable_mid_stream_fallback_continuation` is set. With that setting, a chat +completions stream that broke after plain assistant text continues on a fallback +deployment that supports assistant prefill: the router sends the text so far as +a prefill, so the new deployment carries on instead of starting over, and the +caller sees one response. A stream that has already emitted tool calls, thinking +blocks, audio or images, or constrained output such as JSON mode, is never +continued. Without an eligible deployment, the original error is raised. + +![What happens when a stream breaks: before any content the router retries or +falls back, after text with continuation on it continues on a fallback +deployment with a prefill, otherwise it raises to the +caller](../img/reliability-5.svg) + +## Cooldowns + +A cooldown is a key `deployment:{id}:cooldown` in the routing cache, with a TTL +equal to the cooldown time. While the key exists, the cooldown filter drops the +deployment from every pick, on every process that shares the Redis. When the TTL +expires, the deployment is simply eligible again. There is no probe request and +no early reset on success. + +The failure handler decides whether a failure cools the deployment down. It runs +two checks: first on the failure itself, then on the deployment's recent +history. + +![How a failure on a deployment leads to a cooldown: the skip and counting +checks, then either the allowed_fails counter or the last 60 seconds error-rate +rules](../img/reliability-6.svg) + +The cooldown time comes from the first of these that is set: the deployment's +own `cooldown_time`, then the provider's `Retry-After`, then the router's +`cooldown_time`, which defaults to 5 seconds. Cooldown alerts and metrics fire +from the `CooldownSet` routing event when a deployment is cooled down. + +### Allowed fails + +`allowed_fails` lets a deployment fail a few times before it cools down. Each +counted failure increments `deployment:{id}:allowed_fails`, which expires with +the cooldown time, and the deployment cools down once the count goes over the +limit. `allowed_fails_policy` sets the limit per failure kind, for example +tolerating a few timeouts but cooling down on the first auth error. + +```python +Router( + model_list=..., + allowed_fails_policy={"TimeoutErrorAllowedFails": 5, "AuthenticationErrorAllowedFails": 0, "RateLimitErrorAllowedFails": 2}, +) +``` + +A deployment can set its own `allowed_fails` or `allowed_fails_policy` in +`model_info`, and that setting wins over the router's. The router-wide +`allowed_fails` takes part only when it was set explicitly or changed from its +default, so a router that never mentions it uses the error-rate rule below. + +Two details are easy to miss. First, the success and failure counts for the +error-rate rule are kept in memory only, per process, with a 60 second TTL, so +on a fleet each worker judges a deployment by the traffic it saw itself. The +`allowed_fails` counter and the cooldown key live in Redis when it is +configured, so they apply to the whole fleet. Second, by default a lone +deployment is protected from the error-rate rule, since cooling down the only +deployment of a group would fail every request for that group. + +When every candidate is cooled down, the pick raises `RouterRateLimitError` with +the time until the soonest cooldown ends, and the fallback walk handles it like +any rate limit. `enable_health_check_routing` together with an allowed fails +policy changes this: when every candidate is cooled down, the router picks one +of them anyway instead of raising. diff --git a/docs/architecture/router/explanation/request-lifecycle.md b/docs/architecture/router/explanation/request-lifecycle.md new file mode 100644 index 00000000000..366b40b8856 --- /dev/null +++ b/docs/architecture/router/explanation/request-lifecycle.md @@ -0,0 +1,70 @@ +# The life of a request + +This page follows one `await router.acompletion(model="gpt-4o", messages=...)` +from the facade, through the core, to the provider and back. The other endpoint +methods (`aembedding`, `aresponses`, `aanthropic_messages`, the files and +batches wrappers and the rest) take the same path with a different operation. + +## The shape + +Fallbacks wrap retries, and retries wrap a single attempt. Each attempt picks a +deployment and calls `litellm.` with that deployment's params, so a retry +can land on a different deployment. + +![Router.acompletion passes through fallbacks, retries and one attempt to +litellm.acompletion and the provider HTTP call](../img/request-lifecycle-1.svg) + +The router sets `max_retries` to 0 for the provider call, so neither the +provider SDK nor the HTTP client retries on its own. The router's loop is the +only retry loop, which is what lets each retry pick a different deployment. + +`litellm.acompletion` is the same function SDK users call. The router does not +talk to providers itself. It chooses the arguments, and the provider +transformation layer does the rest. + +The response carries the picked deployment's id in +`response._hidden_params["model_id"]`. The proxy reports it in the +`x-litellm-model-id` header, together with the model group, the retries and the +fallbacks the call used. + +## Step by step + +`RustCore.acompletion` turns the call into one +`RouterCoreHandle.start("acompletion", request)` and drives the steps it +returns. The loop itself is in `litellm-rust/crates/router-core/src/call.rs`, +and the pick it runs per attempt is in `select.rs`. + +![Numbered sequence of one acompletion call: the Router, the RustCore driver, +the core's steps, the cache reads and writes, the attempt through PythonHost to +the provider, and the response with routing +headers](../img/request-lifecycle-2.svg) + +Some details worth knowing. + +The core turns an attempt's error into a typed `AttemptError` with a failure +kind (rate limit, context window, content policy, auth, timeout, ...), the +status code and any `Retry-After`. The retry, fallback and cooldown rules read +those facts. + +The core records each outcome itself, through cache steps, right after the +attempt, so nothing depends on a logging callback firing. Routing events +(`Selected`, `AttemptFailed`, `CooldownSet`, `Retry`, `Fallback`, +`RetriesExhausted` and others) ride along on every step. The driver hands them +to the host's observers, and uses them to stamp the request metadata and to +raise the exception a caller expects. + +A streamed attempt is deferred. The core counts the request when the stream +opens. When the stream ends, the host's stream wrapper reports the success with +the final usage, or the failure if the stream broke. A stream that breaks before +the caller got any content asks the core what may still take over +(`call/stream.rs`). + +Stateful strategies, such as latency and usage-based routing, learn from +litellm's logging callbacks and keep their counters in Python. They run in +`litellm/router_rust/steps/selection.py` when the core asks for a pick. + +## The sync path + +`router.completion()` and the other sync methods run the same loop. +`RustCore._call_sync` drives the same steps on the calling thread, and a sleep +step blocks that thread. The proxy uses the async methods for every request. diff --git a/docs/architecture/router/explanation/selection.md b/docs/architecture/router/explanation/selection.md new file mode 100644 index 00000000000..e87950db533 --- /dev/null +++ b/docs/architecture/router/explanation/selection.md @@ -0,0 +1,109 @@ +# How a deployment is picked + +A pick turns a requested name into one deployment. It runs inside every attempt, +so a retry picks again and can land somewhere else. `router.select()` runs the +same pick without calling the provider. + +## The filter chain + +Resolution produces the candidates, and a chain of filters removes the ones that +must not serve this request right now. A filter can only remove candidates. The +strategy at the end of the chain sees what is left. + +![The filter chain from pre-routing and name resolution through every filter to +the strategy, or to the no deployments available error](../img/selection-1.svg) + +## The filters + +**Route capability filter.** A call that only some providers serve skips the +deployments whose provider cannot serve it, as long as another candidate +remains. For example, a `/v1/decisions` call to a group with an OpenAI +deployment and an Anthropic deployment skips the Anthropic one, since litellm +has no Decisions config for that provider. + +**Health check filter.** With `enable_health_check_routing`, the results of the +background health checks count too. A deployment that the last check marked +unhealthy is dropped, as long as that result is younger than +`health_check_staleness_threshold`. `background_health_check_model_groups` +limits this to some model groups. If the filter would leave nothing, it keeps +every candidate instead, so a broken health checker cannot take a whole group +down. + +**Cooldown filter.** Drops every deployment with a live +`deployment:{id}:cooldown` key in the routing cache. Cooldowns are how failures +on one request protect the next ones, see [Retries, fallbacks and +cooldowns](reliability.md). When health checks drive routing with an +`allowed_fails_policy` and the cooldown filter would leave nothing, the router +ignores cooldowns for that request. + +**Blocked filter.** Drops deployments with `model_info.blocked`, which an admin +sets to pause a deployment without deleting it. If every deployment of a group +is blocked, the error says so. + +**Callback filters.** Callbacks can narrow the candidates too, and they run in +the chain like any other filter. The budget limiter drops deployments whose +provider is over its `provider_budget_config` budget. Deployment and session +affinity keep a conversation on the deployment that served it before by removing +all others from consideration as long as the previous deployment is healthy. + +**Pre-call checks.** With `enable_pre_call_checks`, the router drops deployments +whose context window the prompt exceeds, deployments outside the request's +`allowed_model_region`, and deployments that cannot take a requested parameter. +The prompt's token count and each deployment's limits come from the model cost +map. When nothing is left, the error says which check removed the last +deployment. A context window error, for example, is one the fallback walk can +route to `context_window_fallbacks`. + +**Tag filter.** With `enable_tag_filtering`, a request's tags keep only the +deployments whose `litellm_params.tags` include them. Deployments tagged +`default` serve untagged requests. `!tag` excludes deployments that carry that +tag, and `&tag` requires it. + +**Routing plugin candidates.** When routing plugins ran during pre-routing, only +the deployments of the models they left remain. + +**Text-only filter.** A request that carries an image skips deployments that +declare no vision support, as long as another candidate remains. + +**Order filter.** Keeps only the deployments with the lowest +`litellm_params.order` among those left, or with the order an order fallback +targets. See below. + +**Excluded and retry-skipped ids.** A retry skips the deployments that already +failed for this request, but only while something else remains. Weighted +failover excludes ids outright, and an excluded id is never picked. + +## Priority with `order` + +`order` is how in-group priority works. Deployments with `order: 1` take all +traffic while any of them passes the filters. Deployments with `order: 2` see +traffic only when the fallback walk targets the group at that order, after every +`order: 1` deployment failed for the request (see [Retries, fallbacks and +cooldowns](reliability.md)). Mixing deployments with and without an order is +easy to get wrong. While any candidate sets an order, this filter drops a +deployment without one, and the order fallbacks never target it either. So give +every deployment of a group an order, or give none of them one. A priority +routing group serves each member's priority as its deployments' `order`, so the +same filter walks the priorities (see [Strategies, routing groups and +sub-routers](strategies.md)). + +![Priority with order: normal picks go to the two order 1 deployments, and the +order 2 deployment only gets a request after order 1 +failed](../img/selection-2.svg) + +## Shortcuts + +A name that resolves to one deployment by its id is served even while that +deployment cools down. The callback filters and the strategy are skipped, since +there is nothing to choose. A custom strategy skips the whole chain, see +[Strategies, routing groups and sub-routers](strategies.md). + +## When nothing is left + +The pick fails with "No deployments available for selected model". The error +lists the deployments in cooldown and the seconds until the soonest cooldown +ends, and it says whether pre-call checks were on, since they are a common +cause. The fallback walk treats this error like a rate limit, so a group with +every deployment cooling down falls back to its fallback groups. [Find out why a +request went where it did](../how-to/debug-a-routing-decision.md) shows how to +find which filter removed a deployment. diff --git a/docs/architecture/router/explanation/shared-state.md b/docs/architecture/router/explanation/shared-state.md new file mode 100644 index 00000000000..ebabb52aa6a --- /dev/null +++ b/docs/architecture/router/explanation/shared-state.md @@ -0,0 +1,65 @@ +# Shared state across proxy workers + +A production proxy runs many workers, each with its own Router built from the +same config. They agree on routing through Redis. This page explains what is +shared, what is not, and how fresh each piece is guaranteed to be. + +![Three proxy workers, each with a Router and its own in-memory DualCache tier, +all syncing with one shared Redis that holds cooldowns, fail counters, usage and +strategy state](../img/shared-state-1.svg) + +## The routing cache + +The router keeps its routing state in a `DualCache` +(`litellm/caching/dual_cache.py`) built from the `redis_*` arguments. The cache +has two tiers, an in-memory tier and a Redis tier. Writes go to both. Reads +check memory first, go to Redis only on a miss, and then copy the Redis value +into memory. Without Redis settings, the router runs with the in-memory tier +alone. That is fine for one process, but it means every process routes on its +own view. + +The cache is returned from `router.health().store`. + +Reading memory first keeps Redis off the hot path, at the cost of freshness. A +value another process wrote is seen only once the local copy has expired, or +when there never was a local copy. The router handles this differently for each +kind of state. + +| State | Key | Tiers | How fresh other processes see it | +| --- | --- | --- | --- | +| Cooldown | `deployment:{id}:cooldown` | memory + Redis | the local copy is re-checked against Redis at least every `DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS` (1s) | +| Allowed fails counter | `deployment:{id}:allowed_fails` | memory + Redis, incremented in Redis | fleet-wide at increment time, falling back to the local count if Redis fails | +| Success / fail counts for the error-rate rule | `{id}:successes`, `{id}:fails` | memory only, 60s TTL | never, each process judges on its own traffic | +| Usage per minute | `global_router:{id}:{model}:tpm:{HH-MM}`, `...:rpm:...` | Redis increment after each response | at once in Redis. A worker's copy catches up on its next write or the next minute | +| Strategy state (usage-based v2, latency, least-busy) | strategy keys | memory, pushed to Redis every `DEFAULT_REDIS_SYNC_INTERVAL` (1s) or on write | about a second behind | + +See [Routing state and cache keys](../reference/state-and-cache-keys.md) for the +full list with TTLs. + +## What this means in practice + +Cooldowns are shared. When one worker cools a deployment down, the others stop +picking it within about a second. + +Rate limits through `rpm` and `tpm` are soft. Processes do not reserve capacity +before a call. They read counters that are incremented after responses, so a +burst can overshoot a deployment's limit by roughly what the fleet sends in one +sync interval. The proxy's own rate limiter for keys and teams is a separate +system that uses atomic Lua scripts, and the router does not use it. + +The default error-rate cooldown is decided per process. A deployment that fails +half its traffic everywhere gets cooled down by each process independently, and +the resulting cooldown key is shared. With `allowed_fails` or an +`allowed_fails_policy` set, the counter is in Redis and the threshold is +fleet-wide. + +![Sequence in which one worker sets a deployment cooldown in Redis after a 429, +a second worker reads it and filters the deployment too, and the key expires on +both tiers](../img/shared-state-2.svg) + +## Redis outages + +The router keeps routing when Redis fails. The cache logs the failure and serves +from memory, and the `allowed_fails` increment falls back to the process's own +count. Routing quality drops to what each process knows alone until Redis +returns. diff --git a/docs/architecture/router/explanation/strategies.md b/docs/architecture/router/explanation/strategies.md new file mode 100644 index 00000000000..e645c653b2b --- /dev/null +++ b/docs/architecture/router/explanation/strategies.md @@ -0,0 +1,126 @@ +# Strategies, routing groups and sub-routers + +Once the filters have left a list of deployments that may serve a request, a +strategy picks one. Routing groups let different model groups use different +strategies, and sub-routers decide which model group a request should go to +before any of that happens. This page covers all three, plus custom strategies +and routing plugins. + +![A request is routed by a sub-router to a model group, resolved and filtered, +then the chosen strategy picks one deployment](../img/strategies-1.svg) + +## Strategies + +| Strategy | Picks | Reads | +| --- | --- | --- | +| `simple-shuffle` (default) | a weighted random deployment | per-request weights, then `weight`, then `rpm`, then `tpm`, else uniform | +| `least-busy` | the deployment with the fewest requests in flight | in-flight counters | +| `usage-based-routing-v2` | the deployment with the lowest tpm this minute that stays under its tpm and rpm | per-minute usage counters | +| `usage-based-routing` | the v1 version of the above, kept for old configs | per-minute usage | +| `latency-based-routing` | the deployment with the lowest recent latency (time to first token for streams), within a buffer | latency samples | +| `cost-based-routing` | the cheapest deployment still under its limits | model cost map, usage | +| `lar1` | a deployment by the caller's confidence level, with thresholds from `routing_strategy_args` | `metadata.lar1` on the request | + +`simple-shuffle` needs no state. The other strategies keep their own counters, +fed by litellm's logging callbacks. Every strategy runs after the filters, so it +only ever sees deployments the filters already approved. + +### Weights in `simple-shuffle` + +![How simple-shuffle picks its weights: request weights, else weight, else rpm, +else tpm, else uniform random](../img/strategies-2.svg) + +The first source with a positive value on any candidate is used for all +candidates, and a candidate that does not set it gets weight 0. Per-request +weights come from key or team level `router_settings`, and the proxy puts them +on the request as `{model_group: {deployment_id: weight}}`. One team can then +send most of a group's traffic to one deployment while everyone else gets the +global split, without a separate router per team. + +## Which strategy applies + +![Which strategy applies: the request's routing_strategy, else the routing +group's, else the router's, else simple-shuffle](../img/strategies-3.svg) + +A per-request `routing_strategy`, which the proxy forwards from key or team +`router_settings`, wins. Next comes the strategy of the routing group the +resolved name belongs to, then the router's own `routing_strategy`. Stateful +strategies keep separate state per scope, so the same strategy used by two +routing groups does not mix their counters. + +## Routing groups + +A routing group is a named set of model groups with its own strategy. + +```yaml +routing_groups: + - group_name: fast + models: [gpt-4o-mini, claude-haiku] + routing_strategy: latency-based-routing + - group_name: tiers + models: [gpt-4o-ptu, gpt-4o-paygo] + routing_strategy: priority + model_priorities: {gpt-4o-ptu: 1, gpt-4o-paygo: 2} +``` + +The routing group's name is callable. A request for `fast` resolves to the +deployments of both members, and latency-based routing picks among them. A +direct request for `gpt-4o-mini` also uses latency-based routing, since that +model group belongs to `fast`. Every model group that is not a member of a +configured routing group belongs to the implicit default routing group, which +uses the router's `routing_strategy`. + +A priority group is the exception. It does not pick by a strategy of its own. +Instead it serves each member's priority as its deployments' `order`, so the +order filter sends all traffic to priority 1 while any of it is healthy, and +order fallbacks walk down the priorities when it fails. See [How a deployment is +picked](selection.md) for the order filter and [Retries, fallbacks and +cooldowns](reliability.md) for order fallbacks. + +## Sub-routers + +The auto router, complexity router, quality router and adaptive router are +sub-routers, not strategies. They choose which model group should serve a +request, before the name is resolved. Each is configured as a deployment whose +`litellm_params.model` starts with `auto_router/`. +`auto_router/complexity_router...`, `auto_router/quality_router...` and +`auto_router/adaptive_router...` select those three routers, and any other name +under `auto_router/` is the auto router. A request for that deployment's +`model_name` runs the sub-router. + +| Sub-router | Chooses by | +| --- | --- | +| auto router | how closely the prompt matches the example prompts configured for each route, by embedding similarity | +| complexity router | how hard the prompt looks, sending simple prompts to a cheaper group | +| quality router | the prompt's complexity mapped to a quality tier, then the model whose routing preferences match that tier | +| adaptive router | the kind of request (code, writing, reasoning and so on) and a bandit that learns from past responses which model serves each kind best, weighed against cost | + +![A request for smart goes to the auto router, which classifies the prompt and +returns gpt-4o-mini, and the router picks within that +group](../img/strategies-4.svg) + +The router asks the sub-router once per pick, before resolution, and resolves +whatever model group it returns. The attempt still reports the requested name, +`smart`, as its model group, so headers and spend logs show what the caller +asked for. Marker deployments are dropped from every candidate list, so they are +never called directly. Whether a caller may use a sub-router that belongs to a +team depends on the proxy's team checks, which the router asks the proxy for. + +Session binding happens at the same point: a request that belongs to a coding +session can be bound to the model group that served the session so far. + +## Routing plugins + +`Router(plugins=[...])` takes routing plugins: objects that look at a request +and narrow which model groups or deployments may serve it, or that attach +signals for later stages to read. The plugins run during pre-routing, and the +router then keeps only the deployments of the models they left. + +## Custom strategies + +`router.register_plugin(strategy)` with a `CustomRoutingStrategyBase` does not +slot into the strategy step. It replaces the whole pick. The router serves the +deployment the custom strategy returns as is, without resolution, filters or a +strategy of its own. The custom strategy may even return a deployment the +registry does not hold. `lar1` works this way. See [Add a custom routing +strategy](../how-to/add-a-routing-strategy.md). diff --git a/docs/architecture/router/explanation/teams-and-access.md b/docs/architecture/router/explanation/teams-and-access.md new file mode 100644 index 00000000000..87d8b0d731e --- /dev/null +++ b/docs/architecture/router/explanation/teams-and-access.md @@ -0,0 +1,89 @@ +# Teams and access + +The proxy serves many teams from one router, and not every deployment is for +everyone. Four rules decide which of a model group's deployments a caller may +use. They all run during resolution and right after it, before any health or +load filter, so a deployment the caller may not use never shows up in the +caller's candidates, errors or fallbacks. + +![Resolved deployments pass access windows, access groups, team deployments and +PTU shares in order before the health and load +filters](../img/teams-and-access-1.svg) + +The caller facts come in on the request: `team_id`, whether the caller is a +proxy admin, and the models the key and team may use. The proxy fills them in +from the key that made the request. + +## Team deployments + +A deployment with `model_info.team_id` belongs to one team. Only requests +carrying that team id can use it, and everyone else's picks skip it. A team +deployment can also carry `model_info.team_public_model_name`, the name the +team's members call it by. A request for that name from a member of the team +resolves to the team's deployment, even though no model group has that name. + +```yaml +- model_name: team-a-gpt-4o-ft # the internal group name + litellm_params: {model: azure/ft-gpt-4o-team-a, ...} + model_info: + team_id: team-a + team_public_model_name: gpt-4o # what team-a's members ask for +``` + +When the requested model group has both team deployments and shared ones, the +caller's own team deployments come first. A proxy admin without a team may reach +team deployments by name, as long as they all belong to one team. If they belong +to several teams, the pick fails, since there is no way to tell which team was +meant. + +## Access groups + +`model_info.access_groups` lists the access groups a deployment belongs to. A +key or team can be given access to an access group instead of to model names. +When a caller reaches the requested model only through access groups, the router +limits the caller to the deployments in those groups. Two keys that both "have +gpt-4o" can therefore be sent to different deployments of it. + +## Access windows + +`model_info.access_windows` reserves a deployment for some teams during a daily +wall-clock window, for example a provisioned deployment that one team owns +during its business hours. + +```yaml +model_info: + access_windows: + - start: "09:00" + end: "17:00" + timezone: America/New_York + team_ids: [team-a] +``` + +While a window is active, callers outside its `team_ids` do not see the +deployment. Outside the window, everyone does. When all of a group's deployments +are reserved for someone else right now, the pick fails with an error that names +the window instead of a generic "no deployments". + +## PTU shares + +`model_info.ptu_shares` splits one provisioned deployment across teams, as a map +from team id to a whole number of PTUs. A deployment with shares serves only the +teams that hold one. A team that holds a share on any deployment of the group is +limited to those deployments, and a team without a share sees only the +deployments that are not split. + +```yaml +model_info: + ptu_shares: + team-a: 100 + team-b: 50 +``` + +## Fallbacks respect the same rules + +A fallback to another model group resolves the target name for the same caller, +so the same rules apply to the target. Before a cross-group fallback, the router +also checks whether this caller may use the target group at all and whether the +caller's budget allows it (`fallback_access_check` and `fallback_budget_check`). +A target the caller may not use is skipped, see [Retries, fallbacks and +cooldowns](reliability.md). diff --git a/docs/architecture/router/how-to/add-a-routing-strategy.md b/docs/architecture/router/how-to/add-a-routing-strategy.md new file mode 100644 index 00000000000..6a77d683730 --- /dev/null +++ b/docs/architecture/router/how-to/add-a-routing-strategy.md @@ -0,0 +1,89 @@ +# Add a custom routing strategy + +Use this guide when none of the built-in strategies picks deployments the way +you need, and you want to plug in your own without changing the router. You +write a `CustomRoutingStrategyBase` and register it with +`router.register_plugin()`. + +## Know what you replace + +`register_plugin` replaces the router's whole pick, not only the strategy at its +end. The router calls your method and serves the deployment it returns as is. +The pre-routing hook, the filters and the built-in strategy no longer run unless +you call them. + +![The built-in pre-routing, filters and strategy, next to a registered plugin's +async_get_available_deployment that replaces them and can call router.select to +keep the filters](../img/add-a-routing-strategy-1.svg) + +To keep cooldowns, team checks and the other filters, ask the router for the +filtered candidates with `select(kind="healthy")` and choose only among them, as +the example below does. + +## Write the strategy + +```python +from litellm import Router +from litellm.types.router import CustomRoutingStrategyBase + + +class LowestIdStrategy(CustomRoutingStrategyBase): + def __init__(self, router: Router) -> None: + self.router = router + + async def async_get_available_deployment( + self, model, messages=None, input=None, specific_deployment=False, request_kwargs=None + ): + healthy = await self.router.select( + model, + request_kwargs, + kind="healthy", + messages=messages, + input=input, + specific_deployment=specific_deployment, + ) + if isinstance(healthy, dict): # the name resolved to one specific deployment + return healthy + return min(healthy, key=lambda d: d["model_info"]["id"]) + + def get_available_deployment( + self, model, messages=None, input=None, specific_deployment=False, request_kwargs=None + ): + raise NotImplementedError("this strategy serves the async path only") +``` + +The method returns one deployment dict, normally one from the model list. The +router also serves a deployment that the registry does not hold. +`select(kind="healthy")` raises the router's usual no-deployments error when +everything is filtered out, so you do not need to raise your own. + +## Register it + +```python +router = Router(model_list=[...]) +router.register_plugin(LowestIdStrategy(router)) +``` + +From here on, `acompletion`, `aembedding` and the other async endpoint methods +all route through it. `get_available_deployment` serves `router.completion()` +and the other sync methods, so implement it too if you use them. + +## Check it + +`router.select(model)` now returns your pick without calling the provider, which +makes the strategy easy to unit test with mock deployments. [The +tutorial](../tutorial/route-your-first-requests.md) shows how to build a Router +with `mock_response` deployments. + +## When a plugin is the better fit + +If you only need to narrow the candidates, or to send some requests to a +different model group, you do not need to replace the pick. A deployment-level +filter fits as a `CustomLogger` that implements `async_filter_deployments` and +is added to `litellm.callbacks`. The router calls it in its callback filter +stage. Choosing a model group per request is what the sub-routers in +`litellm/router_strategy/auto_router/` and its siblings do in the pre-routing +hook. Both approaches keep the rest of the pipeline intact, including the +built-in strategy. See [How a deployment is picked](../explanation/selection.md) +and [Strategies, routing groups and sub-routers](../explanation/strategies.md) +for where each stage runs. diff --git a/docs/architecture/router/how-to/change-the-router-at-runtime.md b/docs/architecture/router/how-to/change-the-router-at-runtime.md new file mode 100644 index 00000000000..d97d321eb44 --- /dev/null +++ b/docs/architecture/router/how-to/change-the-router-at-runtime.md @@ -0,0 +1,89 @@ +# Change deployments or settings at runtime + +Use this guide to add, replace or remove deployments, or to change router +settings, on a Router that is already serving traffic. All changes go through +`router.apply()` with a `ChangeSet`. Do not assign Router attributes or mutate +the lists that `settings()` and `registry()` return. + +![Sequence in which apply(ChangeSet) runs assign, settings, upsert, delete and +redis_cache, bumps the registry version per deployment change, and returns +ApplyResult](../img/change-the-router-at-runtime-1.svg) + +## Add or replace a deployment + +`upsert` matches on `model_info.id`. A deployment with a new id is added, and +one with an existing id replaces the old one. + +```python +from litellm.router_api import ChangeSet +from litellm.types.router import Deployment + +result = router.apply(ChangeSet(upsert=[ + Deployment( + model_name="gpt-4o", + litellm_params={"model": "azure/gpt-4o-westus", "api_base": "...", "api_key": "..."}, + model_info={"id": "gpt-4o-westus"}, + ) +])) +previous = result.upserted[0] # the deployment it replaced, or None +``` + +## Remove a deployment + +```python +result = router.apply(ChangeSet(delete=["gpt-4o-westus"])) +removed = result.deleted[0] # the removed deployment, or None if the id was unknown +``` + +## Change settings + +Use `settings` for the validated runtime settings that `update_settings` +accepted, and `assign` to replace a field from `SETTING_FIELDS` as it is. + +```python +router.apply(ChangeSet(settings={"num_retries": 3, "routing_strategy": "latency-based-routing"})) +router.apply(ChangeSet(assign={"cooldown_time": 30, "fallbacks": [{"gpt-4o": ["claude-sonnet"]}]})) +``` + +`assign` takes names from `SETTING_FIELDS`, the plain settings in +`PLAIN_SETTING_ATTRIBUTES` (`num_retries`, `max_fallbacks`, `retry_after`, +`timeout`) and `model_group_alias`, and rejects anything else with a +`ValueError`. The values are copied, so changing your own dict afterwards does +not reach the router. `settings` is quieter: it skips keys outside +`RUNTIME_UPDATABLE_ROUTER_SETTINGS` with only a debug log, so check the name if +a change seems to do nothing. `settings` also casts the numeric values, rebuilds +routing groups when the strategy changes, and switches the strategy's loggers. +`assign` does none of that. + +## Changes that need I/O + +Re-reading discovered model info is async, so it needs `aapply`. + +```python +await router.aapply(ChangeSet(refresh_model_info=True)) +``` + +## Several changes at once + +One `ChangeSet` can carry all of these fields. They are applied in the order +`assign`, `settings`, `upsert`, `delete`, `redis_cache`, and +`ApplyResult.version` is the registry version after the last one. The fields are +not swapped in as one, so a request that runs meanwhile can see part of the +change set, and an error part way through leaves the earlier fields applied. An +unknown `assign` name is refused before anything changes. + +## Old code you may find + +| Old | New | +| --- | --- | +| `router.upsert_deployment(d)` | `router.apply(ChangeSet(upsert=[d]))` | +| `router.delete_deployment(id=i)` | `router.apply(ChangeSet(delete=[i]))` | +| `router.update_settings(**s)` | `router.apply(ChangeSet(settings=s))` | +| `router.cooldown_time = 30` | `router.apply(ChangeSet(assign={"cooldown_time": 30}))` | +| `router.update_redis_cache(cache=c)` | `router.apply(ChangeSet(redis_cache=c))` | +| `await router.arefresh_model_info()` | `await router.aapply(ChangeSet(refresh_model_info=True))` | + +The old forms still work through `DeprecatedRouterMembers` and warn. A write to +an old setting name becomes an `apply(ChangeSet(assign=...))` and takes effect +like one. Router settings are meant to be fixed after construction except +through `apply`, so new code should call `apply` directly. diff --git a/docs/architecture/router/how-to/debug-a-routing-decision.md b/docs/architecture/router/how-to/debug-a-routing-decision.md new file mode 100644 index 00000000000..b2a3112343b --- /dev/null +++ b/docs/architecture/router/how-to/debug-a-routing-decision.md @@ -0,0 +1,96 @@ +# Find out why a request went where it did + +Use this guide when a request landed on an unexpected deployment, fell back when +it should not have, or failed with "No deployments available for selected +model". Work from what the response tells you toward what the router saw. + +![Debugging unexpected routing: read the headers, check how the name resolved or +list the candidates the filters left, then check cooldowns, blocks, team and +order, or the strategy inputs](../img/debug-a-routing-decision-1.svg) + +## 1. Read the response headers + +Through the proxy, every response says what happened. + +| Header | Tells you | +| --- | --- | +| `x-litellm-model-id` | the deployment that answered (`model_info.id`) | +| `x-litellm-model-group` | the model group that answered, which differs from the requested one after a fallback | +| `x-litellm-model-api-base` | the deployment's endpoint | +| `x-litellm-attempted-retries` | how many retries ran | +| `x-litellm-attempted-fallbacks` | how many fallback hops ran | + +From the SDK, `response._hidden_params["model_id"]` holds the deployment id. + +## 2. Check how the name resolved + +```python +registry = router.registry() +registry.resolve("gpt-4", kind="alias") # alias target, or None +registry.resolve("gpt-4", kind="group") # the group the name routes to +registry.resolve("gpt-4", kind="recognized") # does the router know the name at all +registry.names("gpt-4", kind="route_candidates", team_id=team_id) # deployment ids it may reach +``` + +A name that is not a model group can still route through an alias, a routing +group, a team public name, a wildcard such as `openai/*`, or +`default_deployment`. [How a name finds its +deployments](../explanation/names.md) has the order. + +## 3. List the candidates the filters left + +`select(kind="healthy")` runs the whole filter chain for a request without +calling anything. Pass the same `request_kwargs` the real request had, since the +team filter, the tag filter and the per-request settings read them. + +```python +candidates = await router.select( + "gpt-4o", + {"metadata": {"user_api_key_team_id": team_id, "tags": ["prod"]}}, + kind="healthy", + messages=messages, +) +print([d["model_info"]["id"] for d in candidates]) + +picked = await router.select("gpt-4o", {"metadata": {...}}, messages=messages) +``` + +The router does not log its filter stages. Its filters run in the order that +[How a deployment is picked](../explanation/selection.md) shows, so compare the +`kind="healthy"` result with the group's deployments and walk through the checks +below. + +## 4. Check what removes a deployment + +```python +health = router.health() +ids = list(registry.names("gpt-4o", kind="ids")) +await health.cooldowns.async_get_active_cooldowns(model_ids=ids, parent_otel_span=None) +health.is_blocked(registry.deployment("gpt-4o-eastus")) +``` + +![Why a deployment is missing from the candidates: a cooldown entry, then +model_info.blocked, another team_id, a lower order in the group, and finally the +pre-call checks](../img/debug-a-routing-decision-2.svg) + +A cooldown entry records the failure that caused it. The rules for when a +failure counts are in [Retries, fallbacks and +cooldowns](../explanation/reliability.md). Deployments with a higher `order` get +traffic only once every lower-order deployment of the group has failed for the +request. + +The error "No deployments available for selected model" is a +`RouterRateLimitError`. Its message includes the cooldown list and the seconds +until the soonest cooldown ends, and it says whether pre-call checks were on, +since they are a common cause. + +## 5. Check the strategy's inputs + +If the right deployments made it through and the pick still looks wrong, read +what the strategy reads. `router.settings().routing_strategy` says which +strategy ran, unless a routing group or a per-request `routing_strategy` +overrode it. For `simple-shuffle`, check `weight`, `rpm` and `tpm` on the +deployments and any per-team `weights` in key or team `router_settings`. For +usage-based strategies, `await router.health().usage("gpt-4o")` shows this +minute's counters, and remember that workers see each other's counters about a +second late (see [Shared state](../explanation/shared-state.md)). diff --git a/docs/architecture/router/how-to/move-off-deprecated-members.md b/docs/architecture/router/how-to/move-off-deprecated-members.md new file mode 100644 index 00000000000..11a658da615 --- /dev/null +++ b/docs/architecture/router/how-to/move-off-deprecated-members.md @@ -0,0 +1,80 @@ +# Move a caller off a deprecated Router member + +Use this guide when you see +`DeprecationWarning: Router. is deprecated; use Router.`, or +when you want to remove an entry from the surface allowlist. The old members +still work, through `DeprecatedRouterMembers` in `litellm/router_deprecated.py`, +and will be removed once callers have moved. + +![Moving off a deprecated Router member: the warning names the replacement, you +rewrite the call, run the tests, then router_surface_check.py reports clean or +an unused entry to remove with +--write](../img/move-off-deprecated-members-1.svg) + +## 1. Find the replacement + +The warning text names it. Without a warning at hand, search +`litellm/router_deprecated.py` for the old name: every member there calls +`_note(name, replacement)`. [Entry points](../reference/entry-points.md) has the +same mapping in table form. + +To see which old names are still used at runtime, run the code path and inspect +the counter. + +```python +from litellm.router_deprecated import DEPRECATED_ROUTER_MEMBER_USES + +print(DEPRECATED_ROUTER_MEMBER_USES.most_common()) +``` + +## 2. Rewrite the call + +Most rewrites are mechanical. + +| Before | After | +| --- | --- | +| `llm_router.model_list` | `llm_router.registry().all()` | +| `llm_router.get_model_list(model_name=m)` | `llm_router.registry().deployments(m)` | +| `llm_router.get_deployment(model_id=i)` | `llm_router.registry().deployment(i)` | +| `llm_router.get_model_names()` | `llm_router.registry().names()` | +| `llm_router.model_names` | `llm_router.registry().names(kind="group_set")` | +| `llm_router.get_model_group_info(m)` | `llm_router.registry().group(m)` | +| `llm_router.routing_strategy` | `llm_router.settings().routing_strategy` | +| `llm_router.get_settings()` | `llm_router.settings().as_dict()` | +| `llm_router.cache` | `llm_router.health().store` | +| `llm_router.cooldown_cache` | `llm_router.health().cooldowns` | +| `await llm_router.async_get_available_deployment(model=m, request_kwargs=k)` | `await llm_router.select(m, k)` | +| `llm_router.upsert_deployment(deployment=d)` | `llm_router.apply(ChangeSet(upsert=[d]))` | +| `llm_router.get_routing_group(name)` | `llm_router.registry().routing_group(name)` | +| `llm_router.discard()` | `llm_router.retire()` | + +Return types are the same as before, so type checks keep passing. Where a query +is overloaded on `kind`, `source` or `field`, pass the literal directly so that +pyright chooses the right overload. + +## 3. Check the surface + +```bash +uv run python tests/code_coverage_tests/router_surface_check.py +``` + +The check fails in two cases. The first is a new member or `router_utils` import +used from outside the router, which you fix by using an entry point. The second +is a listed entry that nothing outside the router uses anymore, which means your +change removed its last use. In that case, shrink the allowlist in the same PR. + +```bash +uv run python tests/code_coverage_tests/router_surface_check.py --write +git diff tests/code_coverage_tests/router_surface_allowlist.json +``` + +The diff should only remove lines. If `--write` adds an entry, a new outside use +slipped in, and the right fix is in the caller, not the allowlist. + +## Inside the router family + +Modules under `litellm/router_utils/`, `litellm/router_strategy/` and +`litellm/router_rust/` are the router's own. They do not trigger the warning, +and the surface check lists their uses separately (`members_router_family`). +Routing plugins should take a `RoutingContext` and read through it, so that they +stop depending on Router members at all. diff --git a/docs/architecture/router/how-to/read-router-state.md b/docs/architecture/router/how-to/read-router-state.md new file mode 100644 index 00000000000..a7e8fb3ef40 --- /dev/null +++ b/docs/architecture/router/how-to/read-router-state.md @@ -0,0 +1,72 @@ +# Read router state from outside the router + +Use this guide when code outside the router needs to know something about it, +such as which deployments a model group has, what the fallbacks are, or whether +a deployment is cooled down. Read through the entry points, never through Router +attributes or `_core_*` members. The surface check in CI rejects new uses of +anything else. + +## Pick the entry point + +![Which view answers which question: router.registry(), router.settings(), +router.health() or router.select(), and otherwise a new query on an existing +view](../img/read-router-state-1.svg) + +## Get the router + +In the proxy, the Router is `litellm.proxy.proxy_server.llm_router`, which can +be None before the config loads. Import it inside the function that uses it, as +the existing proxy modules do. + +```python +from litellm.proxy.proxy_server import llm_router + +if llm_router is None: + raise HTTPException(status_code=500, detail="router not initialized") +``` + +## Common reads + +```python +registry = llm_router.registry() + +groups = registry.names(kind="group_set") # configured model group names +deployments = registry.deployments("gpt-4o", team_id=team_id) # what get_model_list returned +deployment = registry.deployment(model_id) # Deployment or None +group_name = registry.resolve("gpt-4", kind="alias") # alias target or None +info = registry.group("gpt-4o", field="info") # merged ModelGroupInfo +routing_group = registry.routing_group("smart-tier") # RoutingGroup or None +team_names = registry.names(kind="team_public", team_id=team_id) + +settings = llm_router.settings() +strategy = settings.routing_strategy +fallbacks = settings.fallbacks + +health = llm_router.health() +paused = health.is_blocked(deployment) +tpm, rpm = await health.usage("gpt-4o") +cooling = await health.cooldowns.async_get_active_cooldowns(model_ids=ids, parent_otel_span=None) +failures = health.failures_this_minute(deployment.model_info.id) +``` + +Each query's docstring in `litellm/router_api.py` names the old member it +replaces, and [Entry points](../reference/entry-points.md) lists them all in one +table. + +## Things to watch + +Treat what you get back as read-only. Many of the results are views over the +router's state, and changing one does not change the router. Use `apply()` +instead, see [Change deployments or settings at +runtime](change-the-router-at-runtime.md). + +`settings()` is a snapshot. Read it again after an `apply()` instead of holding +one across requests. To detect deployment changes, compare `registry().version` +with the value you saw last. + +Inside a routing plugin you get a `RoutingContext`, not the Router. It has the +same `registry()`, `settings()` and `health()`. + +Module-level helpers in `litellm/router_api.py`, such as `fallback_chain`, +`deployment_id` and `router_init_args`, answer questions that need no Router at +all. diff --git a/docs/architecture/router/img/add-a-routing-strategy-1.svg b/docs/architecture/router/img/add-a-routing-strategy-1.svg new file mode 100644 index 00000000000..bcacf9f021d --- /dev/null +++ b/docs/architecture/router/img/add-a-routing-strategy-1.svg @@ -0,0 +1,65 @@ + +Built-in selection versus a registered strategy + + + + + + + +BUILT-IN + +Pre-routing hook + +Filters +team, cooldown, blocked, ... + +Strategy + + + +AFTER register_plugin + +Your async_get_available_deployment + +router.select(kind='healthy') + +call it to keep the filters + diff --git a/docs/architecture/router/img/change-the-router-at-runtime-1.svg b/docs/architecture/router/img/change-the-router-at-runtime-1.svg new file mode 100644 index 00000000000..0450466e91e --- /dev/null +++ b/docs/architecture/router/img/change-the-router-at-runtime-1.svg @@ -0,0 +1,67 @@ + +Applying a ChangeSet to a running Router + + + + + + + + + + +Your code + +Router + +Registry + +apply(ChangeSet(...)) + +assign, then settings, then upsert, +then delete, then redis_cache + +version += 1 per deployment change + +ApplyResult(version, upserted, deleted) + +The cached settings() snapshot is dropped, +the next read builds a new one + diff --git a/docs/architecture/router/img/code-map-1.svg b/docs/architecture/router/img/code-map-1.svg new file mode 100644 index 00000000000..acfbe9e9f8b --- /dev/null +++ b/docs/architecture/router/img/code-map-1.svg @@ -0,0 +1,115 @@ + +Router code map +Python files of the router and the Rust crates they call, with which file uses which. + + + + + + + + +PYTHON + +RUST + + + +litellm/router_api.py +RouterRegistry, RouterSettings, +RouterHealth, ChangeSet, +RoutingContext + + +litellm/router.py +Router facade + + +litellm/router_config.py +RouterConfig, RouterDynamicConfig + + + +litellm/router_core.py +RouterCore, make_core + + +litellm/router_rust/core.py +RustCore step driver + + + +litellm/router_rust/ +steps, invoke, surface + + +litellm/router_attempt.py +one attempt's kwargs + + +litellm/router_python_host.py +PythonHost + + +litellm/router_strategy/ +strategies and sub-routers + + + +litellm-rust/crates/router-core +the routing core + + +litellm-rust/crates/python-bridge/src/router +RouterCoreHandle, RouterCall + + + + + + + + + + + + +select steps + diff --git a/docs/architecture/router/img/concepts-1.svg b/docs/architecture/router/img/concepts-1.svg new file mode 100644 index 00000000000..48d3726be6e --- /dev/null +++ b/docs/architecture/router/img/concepts-1.svg @@ -0,0 +1,105 @@ + +Router concepts and how they fit together + + + + + + + + + +WHAT IS CONFIGURED + +Deployments +one place that can serve a model + +Model groups +deployments sharing a model_name + +Names +aliases, wildcards, routing groups, +team names, deployment ids + + + + +Request +model='gpt-4o', +team, tags, messages + +Resolve the name + +Who may use what +teams, access groups, +windows, PTU shares + +Filters +health, cooldown, blocked, +pre-call checks, +tags, order + +Strategy +picks one deployment + + + + + + + + +Shared routing state + +Retries, fallbacks, +cooldowns + +Attempt + + + +failed + +writes + +read by + +try again + diff --git a/docs/architecture/router/img/configuration-1.svg b/docs/architecture/router/img/configuration-1.svg new file mode 100644 index 00000000000..e7285d7c60a --- /dev/null +++ b/docs/architecture/router/img/configuration-1.svg @@ -0,0 +1,95 @@ + +Router config arguments by area +Router config splits into five areas: deployments, selection, reliability, state and health, each with its constructor arguments. + + + + + + +Router config + + + + + + +Deployments + +model_list + +model_group_alias + +routing_groups + +search_tools, guardrail_list + +Selection + +routing_strategy + +routing_strategy_args + +plugins + +enable_pre_call_checks + +enable_tag_filtering + +Reliability + +num_retries, retry_policy + +fallbacks, max_fallbacks + +cooldown_time, allowed_fails + +State + +redis_url / host / port + +cache_responses + +Health + +enable_health_check_routing + +background_health_check_model_groups + diff --git a/docs/architecture/router/img/core-and-host-1.svg b/docs/architecture/router/img/core-and-host-1.svg new file mode 100644 index 00000000000..eb29bc5e5a9 --- /dev/null +++ b/docs/architecture/router/img/core-and-host-1.svg @@ -0,0 +1,100 @@ + +Router core and host interfaces +Router uses RouterCore, picked once by make_core. RustCore implements RouterCore and holds a RouterCoreHandle to the Rust RustRouter, which calls the Host trait. PythonHost implements Host through the step driver. + + + + + + + +PYTHON + +RUST + +Router +«Python facade» ++from_config(config, host) ++endpoint methods ++8 entry points + +RouterCore +«Protocol, +litellm/router_core.py» ++registry() settings() health() ++select(model, request_kwargs) ++apply(changes) ++observe() register_plugin() ++retire() + +RustCore +«litellm/router_rust/core.py» +drives RouterCall steps + +RustRouter +«router-core Router» ++call(op, request) ++select(op, request) healthy() ++registry() settings() health() ++report_success() report_failure() ++apply(ChangeSet) RegistryVersion ++retire() + +Host +«trait, host.rs» ++invoke(attempt) ++select(stage, ctx, candidates) ++pre_route(ask) ++allow_fallback(request, target) ++sleep() observe() cache() + +PythonHost + +picked once by +make_core + +implements + +RouterCoreHandle + + +implements, through +the step driver + diff --git a/docs/architecture/router/img/core-and-hosts-1.svg b/docs/architecture/router/img/core-and-hosts-1.svg new file mode 100644 index 00000000000..f50c90cd203 --- /dev/null +++ b/docs/architecture/router/img/core-and-hosts-1.svg @@ -0,0 +1,76 @@ + +The facade, the Rust core and the host +litellm.Router calls RustCore, which drives litellm-router-core over PyO3; the core asks the Python host for steps, and the host calls the provider through litellm.<op>. + + + + + + +litellm.Router +endpoint methods + 8 entry points + +RustCore +litellm/router_rust/core.py +the step driver + + +LITELLM-ROUTER-CORE (RUST) + +registry, build + +resolve, filters, simple-shuffle + +call loop: retries, fallbacks + +health: cooldowns, usage + +PyO3: RouterCoreHandle, RouterCall + +PythonHost and +litellm/router_rust/ +steps/, invoke/, surface/ + +steps + +provider + +litellm.<op> + diff --git a/docs/architecture/router/img/core-and-hosts-2.svg b/docs/architecture/router/img/core-and-hosts-2.svg new file mode 100644 index 00000000000..151f5a91dad --- /dev/null +++ b/docs/architecture/router/img/core-and-hosts-2.svg @@ -0,0 +1,111 @@ + +Driving the core: one call with a retry on 429 +The Router facade calls the RustCore driver, which loops on RouterCall.next and answers each step through PythonHost: read cooldowns, attempt deployment a, get a 429, set its cooldown, attempt deployment b, return the response. + + + + + + + + + +Router facade + + +RustCore driver + + +RouterCall (Rust) + + +PythonHost + +1 +acompletion(model, messages) + +2 +next(None) + +3 +step cache: read cooldowns + +4 +DualCache batch get + +5 +next(values) + +6 +step attempt on deployment a + +7 +router_attempt + litellm.acompletion + +8 +429 + +9 +next(failure) + +10 +step cache: set deployment:a:cooldown + +11 +next(ok) + +12 +step attempt on deployment b + +13 +router_attempt + litellm.acompletion + +14 +response + +15 +next(response) + +16 +step done + +17 +response with routing headers + diff --git a/docs/architecture/router/img/core-and-hosts-3.svg b/docs/architecture/router/img/core-and-hosts-3.svg new file mode 100644 index 00000000000..154bca3e069 --- /dev/null +++ b/docs/architecture/router/img/core-and-hosts-3.svg @@ -0,0 +1,69 @@ + +Static and dynamic router settings +Router(**arguments) is frozen into RouterConfig by build_router_config; process globals are snapshotted per version into RouterDynamicConfig; both, plus apply(ChangeSet), feed the core. + + + + + + + +Router(**arguments) + +litellm.num_retries, litellm.fallbacks, +litellm.model_cost, callbacks, ... + +apply(ChangeSet) + +RouterConfig +frozen for the router's life + +RouterDynamicConfig + +core + +build_router_config + +snapshot per version + + + + diff --git a/docs/architecture/router/img/debug-a-routing-decision-1.svg b/docs/architecture/router/img/debug-a-routing-decision-1.svg new file mode 100644 index 00000000000..6aab0f362be --- /dev/null +++ b/docs/architecture/router/img/debug-a-routing-decision-1.svg @@ -0,0 +1,86 @@ + +Debugging a routing decision + + + + + + +Unexpected routing + + +1. Read the +response headers + + +Right model group? + + +2. Check how the +name resolved + + +3. List the candidates +the filters left + + +Expected deployment +missing? + + +4. Check cooldowns, +blocks, team and order + + +5. Check the +strategy's inputs + + + + +no + +yes + + + +yes +no + diff --git a/docs/architecture/router/img/debug-a-routing-decision-2.svg b/docs/architecture/router/img/debug-a-routing-decision-2.svg new file mode 100644 index 00000000000..c3221254602 --- /dev/null +++ b/docs/architecture/router/img/debug-a-routing-decision-2.svg @@ -0,0 +1,77 @@ + +Why a deployment is missing from candidates + + + + + + +deployment missing +from candidates + +cooldown entry? + +its exception_received and status_code +say which failure cooled it + + +yes + +no + +model_info.blocked? + +model_info.team_id +set to another team? + +a lower +litellm_params.order +in the group? + +pre-call checks: +context window, +rpm / tpm, region, tags + +no + +no + +no + diff --git a/docs/architecture/router/img/deployments-and-model-groups-1.svg b/docs/architecture/router/img/deployments-and-model-groups-1.svg new file mode 100644 index 00000000000..2d57e394dce --- /dev/null +++ b/docs/architecture/router/img/deployments-and-model-groups-1.svg @@ -0,0 +1,73 @@ + +Deployment, litellm_params and model_info +A deployment has model_name, litellm_params and model_info. litellm_params and model_info are parts of the deployment, each with its own fields. + + + + + + +litellm_params +model: "provider/model" +api_base, api_key, api_version +rpm, tpm, weight, order +timeout, stream_timeout, +max_parallel_requests +tags, use_in_pass_through +mock_response + +Deployment +model_name +litellm_params +model_info + +model_info +id +team_id, team_public_model_name +access_groups, access_windows, +ptu_shares +blocked, cooldown_time +allowed_fails, +allowed_fails_policy + +has + +has + diff --git a/docs/architecture/router/img/deployments-and-model-groups-2.svg b/docs/architecture/router/img/deployments-and-model-groups-2.svg new file mode 100644 index 00000000000..a0ea5383bbf --- /dev/null +++ b/docs/architecture/router/img/deployments-and-model-groups-2.svg @@ -0,0 +1,72 @@ + +How the router builds each deployment + + + + + + +FOR EACH +model_list entry, in order + +Resolve secrets +os.environ/ +references to values + + +Validate +returns the +normalized entry + + +Find provider +e.g. azure + + +Add it +returns the entry as +the registry keeps it + + +Give it an id + + +Join the registry + diff --git a/docs/architecture/router/img/deployments-and-model-groups-3.svg b/docs/architecture/router/img/deployments-and-model-groups-3.svg new file mode 100644 index 00000000000..33ad7088c51 --- /dev/null +++ b/docs/architecture/router/img/deployments-and-model-groups-3.svg @@ -0,0 +1,71 @@ + +Model groups, deployments and fallback +A request for model gpt-4o goes to the gpt-4o model group of three deployments; that group falls back to the claude-sonnet group of two deployments. + + + + + + +Request +model=gpt-4o + + +MODEL GROUP +gpt-4o + +gpt-4o-eastus +azure, order 1 + +gpt-4o-westus +azure, order 1 + +gpt-4o-openai +openai, order 2 + +fallback + +MODEL GROUP +claude-sonnet + +anthropic + +bedrock us-east-1 + diff --git a/docs/architecture/router/img/entry-points-1.svg b/docs/architecture/router/img/entry-points-1.svg new file mode 100644 index 00000000000..df745110cac --- /dev/null +++ b/docs/architecture/router/img/entry-points-1.svg @@ -0,0 +1,108 @@ + +Router entry points +The Router facade and the views and change set its entry points return or take. + + + + + + +Router +acompletion() +and the other endpoint methods +registry() RouterRegistry +settings() RouterSettings +health() RouterHealth +retire() +select(model, request_kwargs, kind) deployment +apply(ChangeSet) ApplyResult +aapply(ChangeSet) ApplyResult +observe(logging_obj, request_scoped) +RoutingObservers +register_plugin(CustomRoutingStrategyBase) + +ChangeSet +assign +settings +upsert +delete +redis_cache +refresh_model_info + +apply + +RouterRegistry +version int +deployment(id) +deployments(model_name, source) +names(model_name, kind) +resolve(name, kind) +group(name, field) +access_groups() +credentials(id, form) +plugins(kind) +routing_group(name) +client(deployment) +aliases() patterns() +routing_groups() index() + +RouterSettings +frozen, static +routing_strategy +fallbacks ... +is_set(name) +as_dict() +allowed_fails_for(exception) +guardrail(name) + +RouterHealth +store routing cache +cooldowns +health_checks +latency +is_blocked(d) all_blocked(ds) +usage(model_group, kind) +fully_blocked_model_names() +failures_this_minute(id) +record_failure(id, exc, status) + + + + diff --git a/docs/architecture/router/img/move-off-deprecated-members-1.svg b/docs/architecture/router/img/move-off-deprecated-members-1.svg new file mode 100644 index 00000000000..d3d5664e835 --- /dev/null +++ b/docs/architecture/router/img/move-off-deprecated-members-1.svg @@ -0,0 +1,66 @@ + +Moving a caller off a deprecated member + + + + + + +DeprecationWarning +names the replacement + +Rewrite the call + +Run the caller's tests + +Run +router_surface_check.py + +Remove it with --write + +Done + + + + +listed entry now unused + +clean + diff --git a/docs/architecture/router/img/names-1.svg b/docs/architecture/router/img/names-1.svg new file mode 100644 index 00000000000..df620108e44 --- /dev/null +++ b/docs/architecture/router/img/names-1.svg @@ -0,0 +1,117 @@ + +How a requested name resolves to deployments + + + + + + + +requested +name + +specific_deployment? + +not a group name, +but a deployment id? + +swap in the alias +target, if any + +deployments whose litellm model +is exactly that name + +that one +deployment + +a routing +group? + +a configured +group? + +team public name, +then wildcard, then +default_deployment + +the group's deployments, +team ones first, kept to +the caller's access groups + +empty? + +those deployments + +deployments whose litellm model +is the name, then the first +default_fallbacks group + +error: no deployments +for this model + +the members' +deployments + + +no + +no + +yes + +yes + + +no + +no + +yes + + +no + +yes + + +yes + +still empty + diff --git a/docs/architecture/router/img/read-router-state-1.svg b/docs/architecture/router/img/read-router-state-1.svg new file mode 100644 index 00000000000..f1a3e22ab0e --- /dev/null +++ b/docs/architecture/router/img/read-router-state-1.svg @@ -0,0 +1,93 @@ + +Pick the entry point for reading router state + + + + + + +What do you need to know? + +Nothing fits: discuss adding a query +to an existing view before widening the surface + +What is configured? +deployments, names, +aliases, credentials + +yes + +router.registry() + +no + +How is the router +configured? +strategy, fallbacks, +cooldown time + +yes + +router.settings() + +no + +What state are +deployments in? +cooldowns, blocked, +usage, health checks + +yes + +router.health() + +no + +Which deployment would +a request get? + +yes + +await router.select( +model, request_kwargs) + + +no + diff --git a/docs/architecture/router/img/reliability-1.svg b/docs/architecture/router/img/reliability-1.svg new file mode 100644 index 00000000000..c940ba35071 --- /dev/null +++ b/docs/architecture/router/img/reliability-1.svg @@ -0,0 +1,115 @@ + +A retry inside a model group, then a fallback out of it +Inside each model group the strategy picks one deployment. The strategy picks D1 in gpt-4o, which returns 429 and cools down. The retry stays in gpt-4o, the strategy picks D3, and it also returns 429. The request falls back to opus, the strategy picks D5, and D5 returns 200 to the router. + + + + + + +Router + +MODEL GROUP: GPT-4O +retries stay in this group + +MODEL GROUP: OPUS + +strategy picks one deployment here + +strategy picks here + +D1 +429, cools down + +D2 + +D3 +429 + +D4 + +D5 +returns 200 + +D6 + + + + +retry + +fallback + + +1 + +2 + +3 + +4 + +5 + +6 + +7 + +1 +The strategy picks D1 from gpt-4o, and the router calls it + +2 +D1 returns 429, so it goes into cooldown + +3 +Retry in the same group: the strategy picks again and gets D3 + +4 +D3 returns 429, and the retries are spent + +5 +Fallback: the request moves to model group opus + +6 +The strategy picks D5 from opus, and the router calls it + +7 +D5 returns 200, which goes back to the caller + diff --git a/docs/architecture/router/img/reliability-2.svg b/docs/architecture/router/img/reliability-2.svg new file mode 100644 index 00000000000..0fde54d5554 --- /dev/null +++ b/docs/architecture/router/img/reliability-2.svg @@ -0,0 +1,89 @@ + +Retry loop after a failed attempt + + + + + + + +Attempt failed + +Guardrail intervention? + +Number of retries +retry_policy for this failure kind, unless the request +set num_retries=0, else the request's num_retries, +else the deployment's, else the router's num_retries + +Give up, so the fallback walk sees it + +Worth retrying? + +Sleep +0 if other healthy deployments +remain, else Retry-After +or exponential backoff + +Select again, skipping +deployments that already failed + +Success? + +Return, with x-litellm-attempted-retries + + +no + +yes + + +no + +yes + + + +no, retries left + +no, none left + +yes + diff --git a/docs/architecture/router/img/reliability-3.svg b/docs/architecture/router/img/reliability-3.svg new file mode 100644 index 00000000000..afefbbaaa19 --- /dev/null +++ b/docs/architecture/router/img/reliability-3.svg @@ -0,0 +1,123 @@ + +Where the router goes after retries run out + + + + + + + +Retries exhausted, +failure e + +disable_fallbacks on the +request, or a guardrail +intervention? + +Several order values in the group, +and e is not context window / content policy? + +raise e + +Re-pick across the +other deployments of +the group first + +enable_weighted_failover +with simple-shuffle? + +The next orders inside +the same group, then +the group's fallbacks + +What is e? + +context_window_fallbacks +set? + +fallbacks: the group's entry, +else the '*' entry +(default_fallbacks) + +content_policy_fallbacks +set? + +Its entry for this group +(no entry: raise e) + +Walk the targets in order + +Its entry for this group +(no entry: raise e) + + +no + +yes + +no + +yes + +yes + + +no + +context window + +content policy + +anything else + +no + +no + +yes + +yes + +no entry + + + + + diff --git a/docs/architecture/router/img/reliability-4.svg b/docs/architecture/router/img/reliability-4.svg new file mode 100644 index 00000000000..0ebf9e0ebd9 --- /dev/null +++ b/docs/architecture/router/img/reliability-4.svg @@ -0,0 +1,94 @@ + +Fallback from group A to group B +A 503 from group A survives its retries, the fallback checks allow B, and B answers with 200. + + + + + + + + + + + + +Caller + +Fallback walk + +Retry loop + +Fallback checks + + +1 +model=A + + +2 +group A, num_retries=2 + + +3 +503 after 3 attempts + + +4 +fallback_access_check, +fallback_budget_check for B + + +5 +allowed + + +6 +group B, num_retries=2 + + +7 +200 + + +8 +response from B, +x-litellm-attempted-fallbacks: 1 + diff --git a/docs/architecture/router/img/reliability-5.svg b/docs/architecture/router/img/reliability-5.svg new file mode 100644 index 00000000000..1e750c0114d --- /dev/null +++ b/docs/architecture/router/img/reliability-5.svg @@ -0,0 +1,83 @@ + +How a broken stream is handled + + + + + + + + + +Stream opened, +request counted + + +Stream breaks + + + +The router picks a retry or fallback, +the caller sees one stream + + +Continue on a fallback deployment +with the text so far as a prefill + + +The stream raises to the caller + + +before any content + +after text, continuation on + +after content, continuation off + + +Stream ends + + + +Success is recorded with the final usage + + diff --git a/docs/architecture/router/img/reliability-6.svg b/docs/architecture/router/img/reliability-6.svg new file mode 100644 index 00000000000..06c45a86dda --- /dev/null +++ b/docs/architecture/router/img/reliability-6.svg @@ -0,0 +1,117 @@ + +When a failure cools a deployment down +Decision flow of the failure handler: skip checks, whether the failure counts, the allowed fails counter, and the error-rate rule over the last 60 seconds. + + + + + + + +failure on deployment d + + +cooldowns disabled, or a failure +outside d (selection, caller credentials, ...)? + + +does this failure count? + + +allowed_fails or a policy set +(router or deployment)? + + +increment +deployment:d:allowed_fails +(TTL = cooldown time) + + +count > allowed fails? + + +this process's +last 60s for d + + +cool down + + +no cooldown + + + + +no + +429, 401, 402, 404, 408, any 5xx + +no + + + +yes + +connection error, +or 4xx other than 401, 402, 404, 408, 429 + + + +yes + + +no + +yes + + + +429 and group has 2+ deployments + +100% failed and 1000+ requests + +over 50% failed, 5+ requests, +group has 2+ deployments + +status should not be retried + +otherwise + diff --git a/docs/architecture/router/img/request-lifecycle-1.svg b/docs/architecture/router/img/request-lifecycle-1.svg new file mode 100644 index 00000000000..30786fb19bb --- /dev/null +++ b/docs/architecture/router/img/request-lifecycle-1.svg @@ -0,0 +1,69 @@ + +The shape of a router request +Router.acompletion goes through fallbacks, retries and a single attempt, which calls litellm.acompletion and then the provider. + + + + + + +Router.acompletion +facade + +Fallbacks +move to another model group when retries are spent + +Retries +loop in the same model group, +with backoff + +Attempt +select, build the deployment's +kwargs, call + +litellm.acompletion(**deployment params, **request) + +Provider HTTP call + + + + + + diff --git a/docs/architecture/router/img/request-lifecycle-2.svg b/docs/architecture/router/img/request-lifecycle-2.svg new file mode 100644 index 00000000000..2a552ed4a08 --- /dev/null +++ b/docs/architecture/router/img/request-lifecycle-2.svg @@ -0,0 +1,161 @@ + +Request lifecycle, step by step +Sequence of one acompletion call through the RustCore driver, router-core, the Python host, DualCache and the provider. + + + + + + + + + + + + + +Router +facade + +RustCore +driver + +router-core + +PythonHost + +router_rust + +DualCache + +Provider + +acompletion(model, +messages) + +1 + +start("acompletion", request) + +2 + +pre_route step +(sub-routers, custom strategy) + +3 + +run them + +4 + +routed group, or none + +5 + +resolve the name, team, PTU and access filters + +6 + +cache step: read cooldowns + +7 + +batch get + +8 + +drop cooled-down and blocked deployments + +9 + +select steps: callback filters, +plugins, tags + +10 + +run the Python filters + +11 + +pre-call checks, order filter, retry skips + +12 + +simple-shuffle picks +(or a select step for other strategies) + +13 + +attempt step on deployment a + +14 + +router_attempt builds kwargs, invoke("acompletion") + +15 + +litellm.acompletion(**kwargs) + +16 + +response or error + +17 + +output, or a typed failure + +18 + +next(outcome) + +19 + +cache steps: usage, or fail counts +and maybe a cooldown + +20 + +done + +21 + +response with routing headers + +22 + diff --git a/docs/architecture/router/img/route-your-first-requests-1.svg b/docs/architecture/router/img/route-your-first-requests-1.svg new file mode 100644 index 00000000000..04db061965b --- /dev/null +++ b/docs/architecture/router/img/route-your-first-requests-1.svg @@ -0,0 +1,70 @@ + +Tutorial steps: route your first requests +Five tutorial steps in order: build a Router, send requests, look inside, break a deployment, break the group. + + + + + + + +1 +Build a Router + + + +2 +Send requests + + + +3 +Look inside + + + +4 +Break a deployment + + + +5 +Break the group + diff --git a/docs/architecture/router/img/route-your-first-requests-2.svg b/docs/architecture/router/img/route-your-first-requests-2.svg new file mode 100644 index 00000000000..5f4266ef4c4 --- /dev/null +++ b/docs/architecture/router/img/route-your-first-requests-2.svg @@ -0,0 +1,86 @@ + +A 429 on deployment a: retry on b and cooldown for a + + + + + + + + + + + + +Router + +deployment a + +deployment b + +routing cache + + +attempt 1 +1 + + +429 RateLimitError +2 + + +record_failure sets deployment:a:cooldown (TTL 5s) +3 + + +retry 1 of 1, skip a +4 + + +attempt 2 +5 + + +200 +6 + + +Later requests read the cooldown and only see b + diff --git a/docs/architecture/router/img/route-your-first-requests-3.svg b/docs/architecture/router/img/route-your-first-requests-3.svg new file mode 100644 index 00000000000..99441ff71b4 --- /dev/null +++ b/docs/architecture/router/img/route-your-first-requests-3.svg @@ -0,0 +1,65 @@ + +Retries spent in chat, fallback to backup + + + + + + + + +acompletion(model='chat') + +Fallback walk + +Retry loop in 'chat' + +Retry loop in 'backup' + +'hi from backup' + + + +a: 429, b: 429 + +fallbacks: chat to backup + +c: 200 + diff --git a/docs/architecture/router/img/router-overview.svg b/docs/architecture/router/img/router-overview.svg new file mode 100644 index 00000000000..c894842a133 --- /dev/null +++ b/docs/architecture/router/img/router-overview.svg @@ -0,0 +1,62 @@ + +Router overview + + + + + + +Proxy route or SDK caller + +Routing cache +memory + Redis + +Provider API + +litellm.Router +picks a deployment: names, filters, strategy, retries, fallbacks, cooldowns + +acompletion(model='gpt-4o') + +cooldowns, usage, +fail counts + +litellm.acompletion(...) + diff --git a/docs/architecture/router/img/selection-1.svg b/docs/architecture/router/img/selection-1.svg new file mode 100644 index 00000000000..91e4bf7a128 --- /dev/null +++ b/docs/architecture/router/img/selection-1.svg @@ -0,0 +1,127 @@ + +The filter chain of a pick +A requested name passes pre-routing and resolution, then a chain of filters removes candidates; if anything is left a strategy picks one, otherwise the pick fails with no deployments available. + + + + + + + +requested name + request + +Pre-routing +sub-routers, routing plugins, +session binding +may route to another group + +Resolve the name +see How a name finds +its deployments + + + + +Team, PTU share and +access filters +see Teams and access + +Route capability filter +skips providers that cannot +serve this call + +Health check filter +only with +enable_health_check_routing + + + + +Cooldown filter +drops ids with a live +cooldown key + +Blocked filter +model_info.blocked +set by an admin + +Callback filters +budget limiter, affinity, ... + + + + +Pre-call checks +context window, region, params +only with enable_pre_call_checks + +Tag filter +only with tag filtering on + +Routing plugin candidates + + + + +Text-only filter +an image request skips +deployments without vision + +Order filter +the lowest order left + +Excluded and +retry-skipped ids + + + + +anything left? + +no deployments available, +with the soonest cooldown expiry + +Strategy picks one +see Strategies + + +no +yes + diff --git a/docs/architecture/router/img/selection-2.svg b/docs/architecture/router/img/selection-2.svg new file mode 100644 index 00000000000..edbf00941e0 --- /dev/null +++ b/docs/architecture/router/img/selection-2.svg @@ -0,0 +1,65 @@ + +Priority with order in a model group +A request picks normally between ptu-eastus and ptu-westus at order 1, and reaches paygo at order 2 only after order 1 failed. + + + + + + +MODEL GROUP +gpt-4o + +ptu-eastus +order 1 + +ptu-westus +order 1 + +paygo +order 2 + +Request + + +normal pick + +only after order 1 failed + diff --git a/docs/architecture/router/img/shared-state-1.svg b/docs/architecture/router/img/shared-state-1.svg new file mode 100644 index 00000000000..befdc0fffe8 --- /dev/null +++ b/docs/architecture/router/img/shared-state-1.svg @@ -0,0 +1,75 @@ + +Shared state across proxy workers + + + + + + +PROXY WORKER 1 + +Router + +DualCache +in-memory tier + + + +PROXY WORKER 2 + +Router + +DualCache +in-memory tier + + + +PROXY WORKER 3 + +Router + +DualCache +in-memory tier + + + +Redis +cooldowns, allowed_fails counters, tpm / rpm per minute, strategy state + diff --git a/docs/architecture/router/img/shared-state-2.svg b/docs/architecture/router/img/shared-state-2.svg new file mode 100644 index 00000000000..decca8a52e9 --- /dev/null +++ b/docs/architecture/router/img/shared-state-2.svg @@ -0,0 +1,92 @@ + +Cooldown shared between workers through Redis + + + + + + + + + + +Worker 1 + +Redis + +Worker 2 + + + +1 +deployment d returns 429 + + + +2 +SET deployment:d:cooldown (TTL 5s) + + + +3 +memory copy, d filtered here at once + + +still has no local +cooldown copy for d + + + +4 +cooldown read (at most every 1s) + + + +5 +d is cooling + + + +6 +d filtered here too + + +5s later the key expires on both tiers, d is eligible again + diff --git a/docs/architecture/router/img/state-and-cache-keys-1.svg b/docs/architecture/router/img/state-and-cache-keys-1.svg new file mode 100644 index 00000000000..875d811ab05 --- /dev/null +++ b/docs/architecture/router/img/state-and-cache-keys-1.svg @@ -0,0 +1,68 @@ + +Routing state: shared and process-local keys +Six cache keys shared through Redis when configured, and two kinds of state kept in this process only. + + + + + + +SHARED THROUGH REDIS WHEN CONFIGURED + +deployment:{id}:cooldown + +deployment:{id}:allowed_fails + +global_router:{id}:{model}:tpm:{minute} +...:rpm, :itpm, :otpm + +{id}:{model}:tpm:{minute} +{id}:{model}:rpm:{minute} + +{model_group}_map + +{model_group}_request_count:{id} + +THIS PROCESS ONLY + +{id}:successes / {id}:fails + +registry and settings snapshot + diff --git a/docs/architecture/router/img/strategies-1.svg b/docs/architecture/router/img/strategies-1.svg new file mode 100644 index 00000000000..ccb38c3c4a7 --- /dev/null +++ b/docs/architecture/router/img/strategies-1.svg @@ -0,0 +1,66 @@ + +From requested name to a picked deployment + + + + + + +request +model=smart + +Sub-router +which group? + +resolve and filter +the group's deployments + +which strategy? +request override, +routing group, router + +strategy +picks one + + + + +gpt-4o-mini + diff --git a/docs/architecture/router/img/strategies-2.svg b/docs/architecture/router/img/strategies-2.svg new file mode 100644 index 00000000000..d10bdb932fd --- /dev/null +++ b/docs/architecture/router/img/strategies-2.svg @@ -0,0 +1,65 @@ + +Weight sources in simple-shuffle +Request weights from key or team router_settings, else weight, else rpm, else tpm, else uniform random. + + + + + + + + + + +Request weights +key or team router_settings +weight +rpm +tpm +Uniform random + +else + +else + +else + +else + diff --git a/docs/architecture/router/img/strategies-3.svg b/docs/architecture/router/img/strategies-3.svg new file mode 100644 index 00000000000..1abe042f656 --- /dev/null +++ b/docs/architecture/router/img/strategies-3.svg @@ -0,0 +1,76 @@ + +Which routing strategy applies + + + + + + +Request +routing_strategy? + +Name in a +routing group? + +Router +routing_strategy? + +simple-shuffle + +Use it + +The group's +routing_strategy + +Use it + +no + +no + +no + +set + +yes + +set + diff --git a/docs/architecture/router/img/strategies-4.svg b/docs/architecture/router/img/strategies-4.svg new file mode 100644 index 00000000000..e566efc84bf --- /dev/null +++ b/docs/architecture/router/img/strategies-4.svg @@ -0,0 +1,64 @@ + +Auto router sequence +A request for model "smart" reaches the router, the router has the auto router "smart" classify the prompt, the model group gpt-4o-mini comes back, and the router resolves, filters and runs the strategy on it. + + + + + + + + + +Caller + +Router + +Auto router "smart" + +request for model="smart" + +classify the prompt + +model group gpt-4o-mini + +resolve, filters and strategy on gpt-4o-mini + diff --git a/docs/architecture/router/img/teams-and-access-1.svg b/docs/architecture/router/img/teams-and-access-1.svg new file mode 100644 index 00000000000..8444b5948b2 --- /dev/null +++ b/docs/architecture/router/img/teams-and-access-1.svg @@ -0,0 +1,69 @@ + +Access rules that narrow the candidate deployments + + + + + + +deployments the name +resolved to + +Access windows +withhold deployments reserved +for other teams right now + +Access groups +keep a caller to its groups + +Team deployments +a team's deployment serves +only that team + +PTU shares +split a deployment across teams + +health and load filters + + + + + + diff --git a/docs/architecture/router/reference/code-map.md b/docs/architecture/router/reference/code-map.md new file mode 100644 index 00000000000..19437d31bad --- /dev/null +++ b/docs/architecture/router/reference/code-map.md @@ -0,0 +1,79 @@ +# Code map + +Where the router's code lives. Paths are relative to the repo root. + +![The router code map: the Python facade and its modules lead to the RustCore +step driver, the router_rust step modules, the python-bridge crate and the +router-core crate](../img/code-map-1.svg) + +## Python + +| File | Holds | +| --- | --- | +| `litellm/router.py` | `Router`, the facade. It builds the config, chooses the core and delegates every member | +| `litellm/router_api.py` | the entry point views and types, see [Entry points](entry-points.md), plus `ROUTER_INIT_ARGS` and `apply_changes` | +| `litellm/router_core.py` | the `RouterCore` protocol and `make_core` | +| `litellm/router_config.py` | `RouterConfig`, `RouterDynamicConfig`, and the only router reads of `litellm.*` settings | +| `litellm/router_deprecated.py` | `DeprecatedRouterMembers`, the old member names with a `DeprecationWarning` | +| `litellm/router_python_host.py` | `PythonHost`, the host inside a Python process: endpoint functions, sleep, dynamic config, callbacks, caches | +| `litellm/router_host.py` | `RouterHost`, what only the application can answer, and `litellm/proxy/_router_host.py` with the proxy's version | +| `litellm/router_strategy/` | the stateful strategies, `lar1`, the sub-routers, the budget limiter and tag routing. The core reaches them through the select and pre-route steps | +| `litellm/router_utils/` | helpers the host uses, such as the cooldown cache, the health state cache, pre-call checks and the response headers | +| `litellm/litellm_core_utils/` | helpers the router and other modules share, such as wildcard matching, PTU shares and the retry and fallback headers | +| `litellm/types/router.py` | `Deployment`, `RoutingGroup`, `RetryPolicy` and the other config types | +| `litellm/proxy/route_llm_request.py` | maps a proxy request to a Router endpoint method | + +### `litellm/router_rust/` + +Everything the core needs on the Python side. + +| Path | Holds | +| --- | --- | +| `core.py` | `RustCore`, the step driver for async and sync calls | +| `steps/` | answers to the steps that need live Python objects: `selection.py` (strategies), `prerouting.py` (sub-routers, session binding, custom strategies), `precall.py` (pre-call facts), `plugins.py` (`Router(plugins=[...])`), `callbacks.py` (`CustomLogger` filters), `access.py` (caller access facts) | +| `invoke/` | one attempt: `endpoints.py` (the endpoint wrappers), `shaping.py` (request and response shaping), `streaming.py` (stream wrappers that report the outcome when a stream ends), `slots.py` (`max_parallel_requests`), `walk.py` (retry and fallback metadata), `prompt_management.py`, `websocket.py` | +| `surface/` | the Python `Router` API over the core's state: `registry.py`, `model_list.py`, `indexes.py`, `members.py`, `model_info.py`, `settings.py`, `health.py`, `health_routing.py`, `retries.py`, `fallbacks.py`, `alerting.py`, `replay.py` | + +`litellm/router_attempt.py` builds the keyword arguments of one attempt for the +deployment the core picked. It does not pick, retry or record anything. + +## Rust + +`litellm-rust/crates/router-core/src/`. + +| File | Holds | +| --- | --- | +| `lib.rs` | `Router`, `RouterBuilder`, and the crate overview | +| `config.rs` | `RouterConfig`, `RouterDynamicConfig`, `Op`, `RoutingStrategy`, retry and allowed fails policies | +| `registry.rs` | `Registry`, `Deployment`, `ChangeSet`, `deployment_id` | +| `build.rs` | building deployments from config with the host's `secrets`, `validate`, `provider` and `added` steps | +| `resolve.rs` | name resolution, a port of `_common_checks_available_deployment` | +| `select.rs` | the filter chain and `simple-shuffle` | +| `access.rs`, `ptu.rs`, `health_routing.rs`, `precall.rs` | the access group and window, PTU share, health check and pre-call filters | +| `prerouting.rs`, `routing_plugins.rs`, `routing_groups.rs` | pre-routing, routing plugin candidates and routing group members | +| `strategy.rs`, `weights.rs` | which strategy a pick uses, and per-request weights | +| `call.rs`, `call/stream.rs` | the call loop with retries, fallbacks and weighted failover, and stream recovery | +| `retry.rs`, `fallback.rs`, `mock.rs` | retry rules, the fallback lists, and `mock_testing_*` failures | +| `health.rs` | cooldowns, allowed fails, usage, and the cache keys | +| `cache.rs` | `RouterCache` and an in-memory implementation for tests | +| `settings.rs` | the frozen `Settings` view | +| `host.rs` | `Host`, `Attempt`, `RouterRequest`, `RouterEvent`, `RoutingContext` | +| `lookup.rs`, `host_index.rs` | the indexes and listings the Python surface reads | +| `model_info.rs`, `group_info.rs` | model info and model group info merges | +| `error.rs` | `ConfigError`, `Error`, `AttemptError`, `FailureKind` | + +`litellm-rust/crates/python-bridge/src/router/` exposes the core to Python as +`RouterCoreHandle` and `RouterCall`, turns Python's config and dynamic snapshot +into the core's types, and turns each hostcall into a step. + +## Tests and checks + +| Path | What it covers | +| --- | --- | +| `tests/unit/test_router/core/` | the core suite, run against the core with a scripted host: selection, retries and cooldowns, fallbacks, registry change sets, endpoints | +| `litellm-rust/crates/router-core` unit tests | the Rust modules, with seeded randomness and a test clock | +| `tests/unit/test_router/` | router unit tests through the facade, including `test_router_entry_points.py` | +| `tests/code_coverage_tests/router_surface_check.py` | the shrink-only surface allowlist, run in CI | +| `tests/code_coverage_tests/router_litellm_globals_check.py` | checks that only `litellm/router_config.py` reads `litellm.*` settings | +| `tests/code_coverage_tests/router_code_coverage.py` | checks that every router function has a test | +| litefuzz `translation-diff` suite | differential fuzzing of routing behavior, run on every change to the core | diff --git a/docs/architecture/router/reference/configuration.md b/docs/architecture/router/reference/configuration.md new file mode 100644 index 00000000000..1327d89860d --- /dev/null +++ b/docs/architecture/router/reference/configuration.md @@ -0,0 +1,115 @@ +# Configuration and defaults + +The constructor arguments of `litellm.Router`, grouped by what they control. In +proxy config they sit under `router_settings`, with `model_list` at the top +level. `litellm.router_api.router_init_args()` lists every accepted name. + +`Router(**arguments)` freezes the arguments into a `RouterConfig`, and they stay +fixed for the router's life. Change them with `apply()`, see [Change deployments +or settings at runtime](../how-to/change-the-router-at-runtime.md). Where a +default below says `litellm.`, the value comes from the process settings, +which the router reads as one `RouterDynamicConfig` snapshot per call. + +![The Router constructor arguments grouped into five areas: deployments, +selection, reliability, state and health](../img/configuration-1.svg) + +## Deployments + +| Argument | Default | Meaning | +| --- | --- | --- | +| `model_list` | None | the deployments, see [Concepts](../explanation/concepts.md) | +| `model_group_alias` | `{}` | name to group, or `{"model": group, "hidden": true}` | +| `routing_groups` | None | groups of model groups with their own strategy | +| `default_litellm_params` | `{}` | merged into every call. The router adds `max_retries: 0` to it | +| `assistants_config`, `search_tools`, `guardrail_list` | None | non-model resources the router serves | +| `ignore_invalid_deployments` | False | skip a bad deployment instead of failing construction | + +## Selection + +| Argument | Default | Meaning | +| --- | --- | --- | +| `routing_strategy` | `"simple-shuffle"` | see the strategy table in [Strategies, routing groups and sub-routers](../explanation/strategies.md) | +| `routing_strategy_args` | `{}` | strategy options, for example `ttl` and `lowest_latency_buffer` for latency routing | +| `enable_pre_call_checks` | False | run context window, region and rate limit checks per request | +| `optional_pre_call_checks` | None | extra checks, such as prompt caching or Responses API affinity | +| `enable_tag_filtering`, `tag_filtering_match_any`, `tag_routing_prefix` | False, True, `""` | route by request tags against deployment tags | +| `plugins` | None | routing plugins that narrow candidates or attach signals | +| `default_max_parallel_requests` | None | per-deployment concurrency cap when a deployment sets none | +| `provider_budget_config` | None | per-provider budgets enforced by the budget limiter filter | +| `deployment_affinity_ttl_seconds`, `model_group_affinity_config`, `session_affinity_repin_on_failover` | 3600, None, False | sticky routing for sessions and response chains | +| `default_priority`, `polling_interval` | None, None | the priority scheduler queue | + +## Reliability + +| Argument | Default | Meaning | +| --- | --- | --- | +| `num_retries` | `litellm.num_retries`, else 2 | retries in the same group | +| `retry_policy`, `model_group_retry_policy` | None, `{}` | retries per error type, router-wide or per group | +| `retry_after` | 0 | minimum seconds to wait before a backoff retry | +| `timeout`, `stream_timeout` | None | per-call timeouts. A deployment's own values win | +| `fallbacks` | `[]` | `[{"group": ["other", ...]}]` for any error | +| `context_window_fallbacks` | `[]` | the same, for `ContextWindowExceededError` | +| `content_policy_fallbacks` | `[]` | the same, for `ContentPolicyViolationError` | +| `default_fallbacks` | None | becomes the `"*"` entry of `fallbacks` | +| `max_fallbacks` | `litellm.max_fallbacks`, else 5 | hop limit for fallbacks | +| `fallback_on_unsupported_params` | True | fall back on an unsupported-parameter 400 | +| `fallback_access_check`, `fallback_budget_check` | None | callables that may veto a fallback target | +| `enable_weighted_failover` | False | pick again in the group before falling back, `simple-shuffle` only | +| `enable_mid_stream_fallback_continuation` | False | continue a broken stream on a fallback group | +| `cooldown_time` | 5 seconds | how long a deployment stays cooled down | +| `allowed_fails` | `litellm.allowed_fails` (3) | fails allowed per cooldown window, used only when set explicitly or changed from the default | +| `allowed_fails_policy` | None | allowed fails per error type | +| `disable_cooldowns` | None | turn cooldowns off | + +## State + +| Argument | Default | Meaning | +| --- | --- | --- | +| `redis_url`, `redis_host`, `redis_port`, `redis_password`, `redis_db` | None | the Redis behind the routing state, see [Shared state](../explanation/shared-state.md) | +| `cache_kwargs` | `{}` | extra arguments for the Redis client | +| `cache_responses`, `caching_groups` | False, None | response caching through `litellm.cache` | +| `client_ttl` | 3600 | seconds before a cached provider client is rebuilt | + +## Health + +| Argument | Default | Meaning | +| --- | --- | --- | +| `enable_health_check_routing` | False | drop deployments failing background health checks, and with an allowed fails policy ignore cooldowns when they would leave nothing | +| `health_check_staleness_threshold` | None | how old a health result may be and still count | +| `health_check_ignore_transient_errors` | False | do not mark a deployment unhealthy for transient errors | +| `background_health_check_model_groups` | None | limit health checks to these groups | +| `alerting_config` | None | Slack alerting on routing events | + +## Per-deployment overrides + +These go in a deployment's `litellm_params` (some also in `model_info`) and win +over the router-wide value. + +| Key | Effect | +| --- | --- | +| `rpm`, `tpm` | limits for usage-aware strategies and pre-call checks, and simple-shuffle weights | +| `weight` | simple-shuffle weight | +| `order` | in-group priority, lowest first | +| `cooldown_time` | this deployment's cooldown time, which wins over the `Retry-After` header and the router value | +| `num_retries` | retries for requests that land here | +| `max_parallel_requests` | concurrency cap | +| `timeout`, `stream_timeout` | per-call timeouts | +| `model_info.team_id` | restrict the deployment to one team | +| `model_info.blocked` | paused by an admin | + +## Constants + +From `litellm/constants.py`. Each can be overridden with the environment +variable of the same name. + +| Constant | Default | Used for | +| --- | --- | --- | +| `DEFAULT_COOLDOWN_TIME_SECONDS` | 5 | router `cooldown_time` default | +| `DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS` | 1.0 | how often the cooldown cache re-reads Redis | +| `DEFAULT_FAILURE_THRESHOLD_PERCENT` | 0.5 | error-rate cooldown threshold | +| `DEFAULT_FAILURE_THRESHOLD_MINIMUM_REQUESTS` | 5 | minimum requests in the window before the error rate counts | +| `SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD` | 1000 | requests needed before a lone deployment can be cooled down at 100% failures | +| `ROUTER_MAX_FALLBACKS` | 5 | `max_fallbacks` default | +| `DEFAULT_REDIS_SYNC_INTERVAL` | 1 | seconds between usage counter pushes to Redis | +| `DEFAULT_MAX_RETRIES` | 2 | the default retry count elsewhere in LiteLLM. The router uses the OpenAI SDK's value, which is also 2 | +| `INITIAL_RETRY_DELAY`, `MAX_RETRY_DELAY`, `JITTER` | 0.5, 8.0, 0.75 | the retry backoff in seconds | diff --git a/docs/architecture/router/reference/core-and-host.md b/docs/architecture/router/reference/core-and-host.md new file mode 100644 index 00000000000..658b6c39c5a --- /dev/null +++ b/docs/architecture/router/reference/core-and-host.md @@ -0,0 +1,192 @@ +# Core, config and host interfaces + +The interfaces between the Python facade, the Rust core and the host it runs +under. [The core and the host](../explanation/core-and-hosts.md) explains why +they are shaped this way. + +![The Router facade uses the RouterCore protocol; RustCore implements it and +holds a handle to the Rust RustRouter, which calls the Host trait that +PythonHost implements](../img/core-and-host-1.svg) + +## Python side + +| Name | File | What it is | +| --- | --- | --- | +| `Router` | `litellm/router.py` | the facade. `Router(**arguments)` and `Router.from_config(config, host=python_host)` | +| `RouterCore` | `litellm/router_core.py` | the protocol the facade delegates to | +| `make_core(config, host)` | `litellm/router_core.py` | builds the core once, when the Router is built | +| `RustCore` | `litellm/router_rust/core.py` | holds one `RouterCoreHandle` and answers its steps | +| `RustBridgeDeclined` | `litellm.rust_bridge._native` | raised when the handle cannot read a config | +| `RouterConfig` | `litellm/router_config.py` | the constructor arguments, frozen, with plain containers copied (`build_router_config`) | +| `RouterDynamicConfig` | `litellm/router_config.py` | a versioned snapshot of the process settings the router reads per call (`current_dynamic_config()`) | +| `PythonHost` | `litellm/router_python_host.py` | `invoke(op)` returns the `litellm` endpoint function, plus `sleep`, `dynamic()`, the callback lists, the global cache and `routing_cache(config)` | +| `router_attempt` | `litellm/router_attempt.py` | builds one attempt's keyword arguments for the deployment the core picked | +| `RouterHost` | `litellm/router_host.py` | what only the application can answer (vertex pass-through credentials, member auto-router authorization) | +| `DeprecatedRouterMembers` | `litellm/router_deprecated.py` | the old Router member names, served with a `DeprecationWarning` | + +`RouterDynamicConfig` fields are `num_retries`, `max_fallbacks`, +`router_max_fallbacks`, `allowed_fails`, `request_timeout`, `fallbacks`, +`default_fallbacks`, `context_window_fallbacks`, `content_policy_fallbacks`, +`drop_params`, `provider_list`, `expose_router_debug_in_errors`, +`redact_user_api_key_info`, `model_alias_map`, `model_cost`, +`known_custom_logger_compatible_callbacks`, `exception_types`, `callbacks`, +`success_callback`, `failure_callback`, `input_callback`, `callback_manager` and +`cache`, plus its `version`. + +## The step protocol + +`RouterCoreHandle(config, dynamic)` in `litellm.rust_bridge._native` wraps one +Rust `Router`. `start(op, request)`, `select(model, request)` and +`healthy(model, request)` return a `RouterCall`, and so do the report and model +info queries. The driver calls `RouterCall.next(answer)` until it gets `done` or +`failed`. Every step carries the routing events since the last step. + +| Step | The core needs | The driver answers with | +| --- | --- | --- | +| `attempt` | one attempt run on the named deployment | the response and its total tokens, or the typed failure | +| `sleep` | a backoff of `seconds` | nothing, after sleeping through the host | +| `allow_fallback` | whether a cross-group fallback `target` is allowed | a bool from the fallback access and budget checks | +| `pre_route` | the sub-routers, session binding or a custom strategy before a pick resolves `model` | a model group, a deployment the custom strategy picked, or `None` | +| `select` | one host stage run on the candidates, named by `stage` | see the stage table below | +| `cache` | a routing cache read or write (`get`, `set`, `batch_get`, `increment` and the rest) | the values read, or an acknowledgement | +| `secrets`, `validate`, `provider`, `added` | one deployment's build, while a config or change set is applied | the resolved secrets, the validated entry, the provider, the added entry | +| `model_info` | a cost map or capability lookup during a model info query | the lookup's documents | +| `done` | nothing more | the result goes back to the caller, with routing headers | +| `failed` | nothing more | the error goes back to the caller, by its `error_kind` | + +| `select` stage | The host returns | +| --- | --- | +| `routing_plugins` | the candidate models the routing plugins left, after running them when `run` is set | +| `callback_filters` | the deployment ids the `CustomLogger` filters left, or a failure | +| `plugins` | the deployments `Router(plugins=[...])` objects left | +| `pre_call` | the facts the pre-call checks decide by, such as token counts | +| `tags` | the deployments tag routing left | +| `text_only` | the candidates that declare no vision support | +| `strategy` | the candidates with the pick first, for a strategy the core does not implement | + +`apply(changes)` returns the new registry version, `registry_version()` reads +it, and `retire()` retires the router. + +## Rust side + +All of these live in `litellm-rust/crates/router-core/src/`. + +| Type | Module | What it is | +| --- | --- | --- | +| `Router`, `RouterBuilder` | `lib.rs` | `Router::new(config, dynamic, host)`, or the builder with `randomness` and `clock` for tests | +| `RouterConfig`, `RouterDynamicConfig`, `Op`, `RoutingStrategy`, `RetryPolicy`, `AllowedFailsPolicy` | `config.rs` | static and dynamic config, the operation name, the strategies (`is_native` is true for `simple-shuffle`) | +| `Registry`, `Deployment`, `ChangeSet`, `RegistryVersion`, `deployment_id` | `registry.rs` | the immutable, indexed deployments | +| `Settings` | `settings.rs` | the frozen settings view | +| `Health`, `Outcome`, `Clock`, `cooldown_key`, `usage_key` | `health.rs` | cooldowns, allowed fails, usage and the outcome counters | +| `RouterCache`, `InMemoryRouterCache` | `cache.rs` | the routing state store the host provides | +| `Randomness`, `SeededRandomness`, `simple_shuffle` | `select.rs` | the filter chain and `simple-shuffle` | +| `resolve_request`, `all_deployments` | `resolve.rs` | name resolution, a port of `_common_checks_available_deployment` | +| `routing_context`, `request_override` | `strategy.rs` | which strategy a pick uses | +| `RouterWeights`, `validate` | `weights.rs` | per-request weights | +| `RouterResponse` | `call.rs` | the call loop's result with the deployment, group, retries and fallbacks used | +| `StreamPolicy`, `StreamOrigin` | `call/stream.rs` | what may take over a stream that broke | +| `Fallbacks`, `FallbackTarget` | `fallback.rs` | the parsed fallback lists | +| `Host`, `Attempt`, `RouterRequest`, `Output`, `RouterEvent`, `RoutingContext`, `SelectStage` | `host.rs` | the host seam | +| `Error`, `ConfigError`, `Unsupported`, `AttemptError`, `FailureKind` | `error.rs` | errors at build time, at call time, and per attempt | + +The other modules hold one stage or query each: `access.rs` (access groups and +windows), `ptu.rs` (PTU shares), `health_routing.rs`, `precall.rs`, +`prerouting.rs`, `routing_groups.rs`, `routing_plugins.rs`, `retry.rs`, +`mock.rs` (`mock_testing_*` failures), `build.rs` (the deployment build), +`lookup.rs` and `host_index.rs` (the indexes the Python surface reads), +`model_info.rs` and `group_info.rs` (model info merges). + +`Op` is the operation's name as a string, such as `acompletion`, `aembedding`, +`aresponses` or `aanthropic_messages`. The core passes it to the host and never +interprets it. + +### `Host` + +| Method | Called for | +| --- | --- | +| `invoke(attempt)` | every attempt. There is no native attempt path | +| `select(stage, ctx, candidates)` | the `Plugins`, `Strategy`, `Tags` and `TextOnly` stages | +| `pre_route(ask)` | pre-routing, and the whole pick for a custom strategy | +| `routing_plugin_models`, `callback_filters`, `pre_call_facts` | the stages that need live Python objects | +| `allow_fallback(request, target)` | the access and budget checks before a cross-group fallback | +| `sleep(duration)`, `observe(event)`, `cache()` | backoff, routing events, the routing state store | +| `secrets`, `validate`, `provider`, `added` | the deployment build | +| `model_info(lookup)` | cost map and capability lookups | +| `supports(op, provider)`, `can_select()` | what the host can serve. Both default to true, and `PythonHost` serves everything | + +### `RouterRequest` + +| Field | Meaning | +| --- | --- | +| `model` | the requested name, a model group, alias, routing group, deployment id or wildcard match | +| `body` | the request params the core's own checks read | +| `team_id`, `proxy_admin`, `allowed_models`, `admin_model` | the caller facts the team and access filters read | +| `specific_deployment` | the caller named a deployment's `litellm_params.model` | +| `pinned_deployment_id` | a deployment the session or affinity binding pinned | +| `num_retries`, `model_group_retry_policy`, `retry_policy_raises` | per-request retry settings, which win over the router's | +| `disable_fallbacks`, `fallbacks`, `context_window_fallbacks`, `content_policy_fallbacks`, `max_fallbacks` | per-request fallback settings | +| `cooldown_time` | a per-request cooldown | +| `router_weights` | per-team weights, model group to deployment id to weight | +| `routing_strategy` | a per-request strategy override | +| `filter_tags`, `has_image` | inputs to the tag and text-only stages | +| `pass_through` | a pass-through route, which serves only pass-through deployments | +| `target_order`, `excluded_ids`, `retry_skipped_ids` | the order an order fallback targets, and ids the pick must skip | +| `pre_routes`, `unrouted`, `skip_weighted_failover` | whether and how pre-routing and weighted failover run | +| `mock_failure`, `health_check_probe` | test failures and health check probes | +| `payload` | what the host needs to run the call itself, such as the Python kwargs | + +### `Attempt` and `Output` + +`Attempt` carries `op`, `request`, `deployment`, `name` (the requested name or +the fallback target), `model_group` (the group served, after fallbacks), +`target_order`, `candidate_ids`, `retry` (0 for the first attempt in this +group), `fallback_depth`, `fallback_level` (0 for the requested name), +`timeout`, `params`, `scope`, `max_retries` and `model_group_size`. + +`Output` carries the `value` the host hands back unchanged, `total_tokens`, +`blocked` (`ContentFilter` or `Refusal`, which the fallback rules read) and +`deferred`, which is set for a stream or realtime session whose outcome the host +reports when it ends. + +### `RouterEvent` + +| Event | Fields | +| --- | --- | +| `Selected` | `model_group`, `deployment_id` | +| `Succeeded` | `deployment_id` | +| `AttemptFailed` | `deployment_id`, `kind`, `status` | +| `CooldownSet` | `deployment_id`, `cooldown` | +| `Retry` | `model_group`, `retry`, `delay` | +| `Fallback` | `from`, `to`, `order`, `depth`, `level`, `scope`, `parent`, `max_fallbacks` | +| `FallbackDone` | `from`, `to`, `level`, `scope`, `parent`, `succeeded` | +| `Blocked` | `level`, `model`, `blocked` | +| `RetryLogged` | `scope`, `level`, `model`, `attempted_retries`, `cause` | +| `LevelFailed` | `level`, `model`, `error`, `handling` | +| `MockFailed` | `mock`, `level`, `model`, `num_retries` | +| `RetryPolicyRaised` | `level`, `model`, `cause` | +| `OrderChainFailed` | `level`, `last_hop` | +| `RetriesExhausted` | `model_group`, `attempted`, `max`, `level`, `picked` | + +The driver uses the later events to raise the exception a caller expects at that +point, with the retry and fallback attributes set. + +### `FailureKind` + +The kinds are `RateLimit`, `ContextWindowExceeded`, `ContentPolicyViolation`, +`Authentication`, `PermissionDenied`, `NotFound`, `BadRequest`, `Timeout`, +`ApiConnection`, `InternalServer`, `ServiceUnavailable`, `BadGateway`, +`GuardrailIntervention` (never retried or fallen back from), `Declined` (litellm +rejected the request before calling, and it never cools a deployment down), +`CallerCredential` (the caller's own credential failed) and `Other`. +`AttemptError` adds `status`, `retry_after`, `num_retries`, the provider's error +text for the cooldown record, and `agentic_followup`, which marks an agentic +follow-up call that failed after the first call was served and is never retried. + +### Build-time errors + +`Router::new` returns `ConfigError` when the registry cannot be built, for +example from a malformed deployment or a duplicate id, or when the host cannot +serve the config. `Unsupported` names what is missing: a strategy or plugin on a +host that cannot select, or an operation and provider pair the host does not +support. Under `PythonHost` neither happens, since it serves every operation and +strategy. diff --git a/docs/architecture/router/reference/entry-points.md b/docs/architecture/router/reference/entry-points.md new file mode 100644 index 00000000000..bcccb225276 --- /dev/null +++ b/docs/architecture/router/reference/entry-points.md @@ -0,0 +1,160 @@ +# Entry points + +`litellm.Router` exposes its endpoint methods plus the entry points below, and +nothing else. The views and types live in `litellm/router_api.py`. The "Was" +columns name the Router member from v1.108 and below which each query replaced. +The old names still work through `DeprecatedRouterMembers`, with a +`DeprecationWarning`, see [Move a caller off a deprecated Router +member](../how-to/move-off-deprecated-members.md). + +![The Router facade and the RouterRegistry, RouterSettings and RouterHealth +views it returns, plus the ChangeSet that apply takes, with each type's +members](../img/entry-points-1.svg) + +## `registry()` returns `RouterRegistry` + +Read-only queries over the configured deployments. The router keeps the registry +as an immutable, versioned snapshot, and each query reads the current one. + +| Query | Returns | Was | +| --- | --- | --- | +| `version` | an int that goes up on every deployment change | | +| `deployment(id)` | `Deployment` or None | `get_deployment` | +| `has_deployment(id)` | bool | `has_model_id` | +| `deployment_dict(id)` | the deployment as its model_list dict | `get_model_info` | +| `deployment_model_info(id, model_name)` | model info merged with the pricing map | `get_deployment_model_info` | +| `router_model_info(...)`, `discovered_model_info(id)`, `zero_cost()` | routing model info, provider-discovered limits, the zero-cost lookup | `get_router_model_info`, `get_discovered_model_info`, `_zero_cost_cache` | +| `deployments(name, source="group")` | the group, its aliases, routing groups, then wildcards | `get_model_list` | +| `deployments(name, source="all")` | every deployment of the group | `get_all_deployments` | +| `deployments(source="config")` | deployments from config, not the DB | `config_deployments` | +| `deployments(name, source="routable", request=, call_type=)` | deployments that pass the routing checks for a request | `get_routable_deployments` | +| `deployments(name, source="first_in_group")` | the first deployment of a group | `get_deployment_by_model_group_name` | +| `deployments(name, source=...)` | also `named`, `alias`, `routing_groups`, `routed_group`, `wildcard`, `for_request` | see the docstring | +| `all()` | the live model list | `model_list` | +| `names(kind="groups")` | every routable model name | `get_model_names` | +| `names(kind="group_set")` | the set of configured group names | `model_names` | +| `names(name, kind="ids")` | deployment ids, of one group or all | `get_model_ids` | +| `names(kind="team_public", team_id=)` | the public model names of a team's deployments | `team_public_model_names_of` | +| `names(kind=...)` | also `deployment_names`, `litellm_models`, `litellm_base_models`, `wildcard`, `route_candidates` | see the docstring | +| `resolve(name, kind="group")` | the model group a name routes to | `routable_model_group` | +| `resolve(name, kind="alias")` | the alias target, or None | `get_model_from_alias` | +| `resolve(name, kind="team_model", team_id=)` | a team's public name to its group | `map_team_model` | +| `resolve(name, kind="recognized")` | whether the router knows the name | `is_recognized_model` | +| `resolve(id, kind="group_of_id")` | the group of a deployment id | `get_model_group` | +| `resolve(name, kind=...)` | also `served`, `from_id`, `upstream`, `wildcard_access_group`, `team_model_alternatives`, `routing_group_alternatives`, `search_tool_support` | see the docstring | +| `group(name, field="info")` | merged `ModelGroupInfo` | `get_model_group_info` | +| `group(name, field=...)` | also `cached_info`, `display_name`, `mode`, `service_tiers`, `max_input_tokens`, `token_limits`, `listing_info` | see the docstring | +| `access_groups(...)` | access group name to model names | `get_model_access_groups` | +| `credentials(id, form="params")` | a deployment's credentials | `get_deployment_credentials` | +| `plugins(kind)` | `auto`, `complexity`, `quality`, `adaptive` sub-routers, `routing` plugins, `budget_limiter`, `prompt_caching_pins` | `auto_routers` and friends | +| `routing_group(name)` | the routing group callable as that name, or None | `get_routing_group` | +| `client(deployment, client_type=None)` | the SDK client the router cached for a deployment, or None | `_get_client` | +| `aliases()`, `patterns()`, `routing_groups()`, `index(name)` | the alias map, the wildcard patterns, routing groups, the name index | the old attributes | + +Module-level helpers in the same file need no Router. + +| Function | Returns | +| --- | --- | +| `deployment_id(model_group, litellm_params)` | the stable id of a deployment that has none, the same hash the router uses (was `Router.generate_model_id`) | +| `deployment_usable_by_team(deployment, team_id)` | whether a team may use a deployment | +| `router_init_args()` | the arguments `Router()` accepts (was `Router.get_valid_args`) | +| `fallback_chain(fallbacks, model_group)`, `request_fallback_chain(fallbacks, request_kwargs, model_group)` | the fallbacks configured for a group, or for one request, and the index of the `*` entry | +| `fallbacks_disabled(request_kwargs)` | whether the request turned fallbacks off | +| `references_provider_scoped_resource(request_kwargs)` | whether the request names a file, batch or fine-tuning job only its own provider knows | +| `deployment_metadata(request_kwargs)` | the metadata bucket the router stamped the picked deployment into | +| `mark_advisor_orchestration_failure(exception)` | tags an advisor orchestration failure, so the router does not cool down the deployment it picked | + +## `settings()` returns `RouterSettings` + +A frozen dataclass with one field per readable setting, listed in +`SETTING_FIELDS`. It is built from the router's static `RouterConfig` and the +current dynamic snapshot, and it changes only through `apply()`. A setting the +router never set reads as None. Treat the field values as read-only. + +`routing_strategy`, `fallbacks`, `context_window_fallbacks`, +`content_policy_fallbacks`, `default_deployment`, `default_litellm_params`, +`default_max_parallel_requests`, `cooldown_time`, `allowed_fails`, +`allowed_fails_policy`, `disable_cooldowns`, `enable_pre_call_checks`, +`enable_health_check_routing`, `health_check_ignore_transient_errors`, +`background_health_check_model_groups`, `enable_tag_filtering`, +`tag_filtering_match_any`, `tag_routing_prefix`, `router_general_settings`, +`provider_budget_config`, `provider_default_deployment_ids`, +`fallback_access_check`, `fallback_budget_check`, `kubernetes_pod_discovery`, +`search_tools`, `guardrail_list`, `cache_responses`. + +| Method | Returns | Was | +| --- | --- | --- | +| `is_set(name)` | whether the router has this setting at all | | +| `as_dict()` | the settings as plain values | `get_settings` | +| `allowed_fails_for(exception)` | allowed fails for an error under `allowed_fails_policy` | `get_allowed_fails_from_policy` | +| `guardrail(name)` | a configured guardrail | `get_available_guardrail` | + +## `health()` returns `RouterHealth` + +| Member | Returns | Was | +| --- | --- | --- | +| `store` | the routing cache, see [Routing state and cache keys](state-and-cache-keys.md) | `cache` | +| `cooldowns` | active cooldowns, `await cooldowns.async_get_active_cooldowns(model_ids=, parent_otel_span=)` | `cooldown_cache` | +| `health_checks` | background health check results | `health_state_cache` | +| `latency` | latency samples by deployment | `deployment_latency_map` | +| `is_blocked(deployment)`, `all_blocked(deployments)` | whether an admin paused them | `is_deployment_blocked`, `are_all_deployments_blocked` | +| `fully_blocked_model_names()` | groups with every deployment paused | `get_fully_blocked_model_names` | +| `await fully_unhealthy_model_names()` | groups with every deployment failing health checks | `async_get_fully_unhealthy_model_names` | +| `await usage(group, kind="current")` | tpm and rpm this minute | `get_model_group_usage` | +| `await usage(group, kind="remaining")` | what is left of the group's limits | `get_remaining_model_group_usage` | +| `team_ptu_ceiling(team_id, group)` | a team's PTU share ceiling | `team_ptu_ceiling` | +| `split_priority(model, kwargs)` | the request priority and the kwargs without it | `split_request_priority` | +| `failures_this_minute(id)` | failures counted against a deployment this minute | `get_deployment_failures_for_current_minute` | +| `record_failure(id, exception, status)` | counts a failure the router did not see itself, cools the deployment down once it is over its allowed fails, and says whether it did | `increment_deployment_failures_for_current_minute` then `set_cooldown_deployments` | +| `priority_header_fields(model, kwargs, header)` | the request fields a priority header sets | `priority_header_update` | + +## `select(model, request_kwargs=None, *, kind="pick", ...)` + +Runs the pick without calling the provider. `kind="pick"` returns one deployment +dict, exactly what a real request would get (was +`async_get_available_deployment`). `kind="healthy"` returns the filtered +candidates before the strategy (was `async_get_healthy_deployments`). +`kind="pass_through"` picks among deployments with `use_in_pass_through=True`. +Optional keyword arguments are `messages`, `input`, `specific_deployment`, +`parent_otel_span` and `health_check_probe`. + +## `apply(changes)` and `aapply(changes)` + +The only way to change a Router after it is built. The router rejects an +`assign` name it does not know before changing anything, then applies the fields +in the order below. A change set is not one swap, so a request that runs while +the change set is applied can see the earlier fields changed and not the later +ones, and an error part way through leaves the earlier changes in place. The +fields are. + +| Field | Effect | Was | +| --- | --- | --- | +| `assign` | replace settings as they are. Names from `SETTING_FIELDS`, `PLAIN_SETTING_ATTRIBUTES` (`num_retries`, `max_fallbacks`, `retry_after`, `timeout`) or `ASSIGNABLE_ALIASES` (`model_group_alias`) | `router. = value` | +| `settings` | validated runtime settings | `update_settings(**settings)` | +| `upsert` | add or replace deployments, by `model_info.id` | `upsert_deployment` | +| `delete` | remove deployments by id | `delete_deployment` | +| `redis_cache` | point routing state at a new Redis cache | `update_redis_cache` | +| `refresh_model_info`, `refresh_client` | re-read discovered model info, optionally with your HTTP client, `aapply` only | `arefresh_model_info` | + +Both return `ApplyResult(version, upserted, deleted)`, with the registry version +after the change and, per upsert or delete, the deployment that was there +before. + +## Lifecycle + +`observe(logging_obj=None, *, request_scoped=False)` returns the router's +`RoutingObservers`, which receive the router's routing events. With +`request_scoped=True` the router stops registering its callbacks globally, and +passing a request's `logging_obj` binds them to that request. +`register_plugin(strategy)` puts a `CustomRoutingStrategyBase` in place of the +whole pick, see [Add a custom routing +strategy](../how-to/add-a-routing-strategy.md). `retire()` releases the router's +global callback registrations and routing state, for a Router that is being +replaced (was `discard`). Calls on a retired router fail. + +## `RoutingContext` + +What a routing plugin receives instead of the Router. It has `registry()`, +`settings()` and `health()`, plus `select`, `observers`, `belongs_to`, +`explicit_allowed_fails`, `pre_call_checks`, `count_pre_call_check_tokens`, +`update_kwargs_before_fallbacks`, `pre_routing_hook` and `log_retry`. diff --git a/docs/architecture/router/reference/state-and-cache-keys.md b/docs/architecture/router/reference/state-and-cache-keys.md new file mode 100644 index 00000000000..25314fc5188 --- /dev/null +++ b/docs/architecture/router/reference/state-and-cache-keys.md @@ -0,0 +1,37 @@ +# Routing state and cache keys + +Everything the router remembers between requests, where it lives, and how long +it lasts. This state lives in the router's routing cache, a `DualCache`. Every +worker uses the same keys, encodings and TTLs, so a fleet shares one Redis. In +the keys below, `{id}` is the deployment's `model_info.id` and `{minute}` is the +UTC time as `HH-MM`. + +![The routing cache keys shared through Redis when it is configured, beside the +state kept only in this process](../img/state-and-cache-keys-1.svg) + +| Key | Written by | Read by | Value | TTL | Tiers | +| --- | --- | --- | --- | --- | --- | +| `deployment:{id}:cooldown` | `Health::add_cooldown` | the cooldown filter, `health().cooldowns` | `{exception_received, status_code, timestamp, cooldown_time}` | the cooldown time | memory + Redis, memory re-checked against Redis every 1s | +| `deployment:{id}:allowed_fails` | `Health::record_failure` | the same | fail count | the cooldown time | memory + Redis | +| `deployment:{id}:allowed_fails:{suffix}` | the same, with a deployment-level policy | the same | fail count per error type | the cooldown time | memory + Redis | +| `{id}:successes`, `{id}:fails` | `Health::record_success`, `Health::record_failure` | the default error-rate cooldown rule, `health().failures_this_minute()` | count | 60s | memory only | +| `global_router:{id}:{model}:tpm:{minute}`, `:rpm:` | `Health::record_success` after each response | pre-call rate limit checks, `health().usage()` | count | 60s | Redis increment, memory | +| `global_router:{id}:{model}:itpm:{minute}`, `:otpm:` | the input and output token rate limit check | the same check | input and output token counts | 60s | memory + Redis | +| `{id}:{model}:tpm:{minute}`, `{id}:{model}:rpm:{minute}` | the `usage-based-routing-v2` strategy | the same | count | 60s | memory, pushed to Redis every 1s | +| `{model_group}_map` | the `latency-based-routing` strategy | the same | latency and ttft samples per deployment | `routing_strategy_args.ttl`, 1 hour | memory + Redis | +| `{model_group}_request_count:{id}` | the `least-busy` strategy | the same | requests in flight | 1 hour | memory + Redis | + +Strategies and plugins keep their own keys (the strategy counters above, the +budget limiter, prompt cache pins, deployment and session affinity). + +## Process-local state + +Some state is per router and is never shared. + +- the registry, with its indexes and `registry().version`, and the settings + view, each held as one snapshot that `apply()` swaps +- provider clients, cached per deployment for `client_ttl` seconds +- the background health check results in `health().health_checks` + +This state is rebuilt from config in every process, which is why deployment ids +must be stable across processes. diff --git a/docs/architecture/router/tutorial/route-your-first-requests.md b/docs/architecture/router/tutorial/route-your-first-requests.md new file mode 100644 index 00000000000..f7c35b22ea2 --- /dev/null +++ b/docs/architecture/router/tutorial/route-your-first-requests.md @@ -0,0 +1,187 @@ +# Route your first requests + +In this tutorial you build a Router with three mock deployments and watch it +balance load, retry, cool a deployment down and fall back to another model +group. Every deployment uses `mock_response`, so nothing leaves your machine and +you need no API keys. + +You need a checkout of this repo with its environment installed (`uv sync`). Run +each step from the repo root with `uv run python`, or paste the steps one after +another into a single file. + +![The five tutorial steps in order: build a Router, send requests, look inside, +break a deployment, break the group](../img/route-your-first-requests-1.svg) + +## 1. Build a Router + +Create `tutorial.py` with a model list. Two deployments, `a` and `b`, serve the +model group `chat`. A third, `c`, serves the model group `backup`. The router +retries once inside a group, then falls back from `chat` to `backup`. + +```python +import asyncio +import random +from collections import Counter + +from litellm import Router +from litellm.router_api import ChangeSet +from litellm.types.router import Deployment + +random.seed(0) + + +def mock(id: str, group: str, response: str) -> dict: + return { + "model_name": group, + "litellm_params": {"model": "openai/gpt-4o-mini", "api_key": "sk-fake", "mock_response": response}, + "model_info": {"id": id}, + } + + +router = Router( + model_list=[ + mock("a", "chat", "hi from a"), + mock("b", "chat", "hi from b"), + mock("c", "backup", "hi from backup"), + ], + fallbacks=[{"chat": ["backup"]}], + num_retries=1, +) + + +async def ask() -> tuple[str, str]: + response = await router.acompletion(model="chat", messages=[{"role": "user", "content": "hello"}]) + return response._hidden_params["model_id"], response.choices[0].message.content +``` + +`random.seed(0)` makes the router's random picks repeat, so your output matches +this page. `model_name` is the name callers send. `litellm_params` is what the +router passes to `litellm.acompletion` once it picks the deployment. +`model_info.id` names the deployment. When you leave it out, the router hashes +the group and params into an id. + +`Router(...)` freezes these arguments into a `RouterConfig`, and settings stay +fixed after that. + +## 2. Send requests + +Add a `main` coroutine and send twenty requests. + +```python +async def main() -> None: + print("picked:", Counter([(await ask())[0] for _ in range(20)])) + + +asyncio.run(main()) +``` + +Run it. You see both deployments picked, in a split close to even. + +``` +picked: Counter({'b': 12, 'a': 8}) +``` + +No strategy was configured, so the router used `simple-shuffle`, the default. +Neither deployment sets `weight`, `rpm` or `tpm`, so each pick is a uniform +random choice among the healthy deployments of the group. Each attempt still +went through `litellm.acompletion`, so `mock_response` worked as it does without +a router. + +## 3. Look inside + +The router answers questions about itself through its entry points. Add these +lines to `main`. + +```python + registry = router.registry() + print("groups:", sorted(registry.names(kind="group_set"))) + print("chat deployments:", [d["model_info"]["id"] for d in registry.deployments("chat")]) + print("strategy:", router.settings().routing_strategy) + picked = await router.select("chat") + print("select picked:", picked["model_info"]["id"]) +``` + +``` +groups: ['backup', 'chat'] +chat deployments: ['a', 'b'] +strategy: simple-shuffle +select picked: b +``` + +`registry()` holds the configured deployments, `settings()` is a frozen view of +the configuration, and `select()` runs the same pick a real request runs, +without calling the provider. [Entry points](../reference/entry-points.md) lists +everything they offer. + +## 4. Break a deployment + +Replace deployment `a` with one that always answers 429. Router settings are +static after construction, so changes go through `apply()` with a `ChangeSet`. +The router replaces the deployment and bumps the registry version. The exact +number depends on how many deployments were added before. What matters is that +it goes up. + +```python + broken_a = Deployment(**mock("a", "chat", "litellm.RateLimitError")) + print("version after apply:", router.apply(ChangeSet(upsert=[broken_a])).version) + print("picked:", Counter([(await ask())[0] for _ in range(10)])) + print("cooldowns:", await router.health().cooldowns.async_get_active_cooldowns( + model_ids=["a", "b", "c"], parent_otel_span=None + )) +``` + +``` +version after apply: 7 +picked: Counter({'b': 10}) +cooldowns: [('a', {'exception_received': 'litellm.RateLimitError: ...', 'status_code': '429', 'timestamp': ..., 'cooldown_time': 5})] +``` + +Every request succeeded. The first one that landed on `a` failed with a 429, and +two things happened. + +![Sequence in which the first attempt on deployment a gets a 429, the router +sets a cooldown on a, retries on b and gets a 200, and later requests only see +b](../img/route-your-first-requests-2.svg) + +First, the router retried in the same group and skipped `a`, so the caller got +`b`'s answer. Second, it recorded the failure and put `a` into cooldown, because +a 429 in a group with more than one deployment cools that deployment down at +once. For the next five seconds, the default cooldown time, the cooldown filter +removes `a` before the strategy ever sees it. [Retries, fallbacks and +cooldowns](../explanation/reliability.md) explains the rules. + +## 5. Break the group + +Break `b` too and send one more request. + +```python + broken_b = Deployment(**mock("b", "chat", "litellm.RateLimitError")) + router.apply(ChangeSet(upsert=[broken_b])) + print("answer:", await ask()) +``` + +``` +answer: ('c', 'hi from backup') +``` + +With both `chat` deployments failing, the retry has nowhere left to go in the +group. Once retries are spent, the router's fallback walk looks up `chat` in +`fallbacks`, finds `backup`, and runs the whole retry loop again against that +group. + +![A request for chat retries a and b, both return 429, then falls back to +backup, where c returns 200 with 'hi from +backup'](../img/route-your-first-requests-3.svg) + +## What you saw + +A Router owns a list of deployments grouped by `model_name` into model groups. +Each request is resolved to a model group, the group's deployments are filtered, +and a strategy picks one. Failures feed back into shared state (the cooldown you +saw) and into the retry and fallback loop around the attempt. Code outside the +router reads and changes all of this through a handful of entry points. + +Next, read [Router concepts](../explanation/concepts.md) for the ideas behind +each step, [The life of a request](../explanation/request-lifecycle.md) for the +full path, or [Configuration and defaults](../reference/configuration.md) for +the knobs you skipped here.