scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-14 09:03:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 214

Merge 624a80b59b into e84859ae4e

Browse source
This commit is contained in:
Nate Armstrong 2026-11-07 00:05:45 +00:00 • committed by GitHub
commit f370c41e5b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
67 changed files with 5922 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
ARCHITECTURE.md
View file

@ -124,7 +124,7 @@ graph TD
- `proxy/proxy_server.py` - Main API endpoints
- `proxy/auth/` - Authentication (API keys, JWT, OAuth2)
- `proxy/hooks/` - Proxy-level callbacks
- `router.py` - Load balancing, fallbacks
- `router.py` - Load balancing, fallbacks (see [docs/architecture/router](docs/architecture/router/README.md))
- `router_strategy/` - Routing algorithms (`lowest_latency.py`, `simple_shuffle.py`, etc.)
**LLM-specific proxy endpoints:**

25
docs/architecture/README.md Normal file
View file

@ -0,0 +1,25 @@
# Architecture
This folder explains how the parts of LiteLLM work inside, for the people who
change them. It covers one component so far, the router. More will follow as the
Rust migration moves each module behind a public interface and into Rust.
Each component's docs follow the [Diataxis](https://diataxis.fr/) layout, so you
can choose the kind of page that fits what you need right now.
![A two-by-two grid of the four kinds of pages: tutorials, how-to guides,
explanation and reference, placed on a studying-to-working axis and a
theory-to-practice axis](img/diataxis-quadrants.svg)
| Kind | Read it when | Router |
| --- | --- | --- |
| Tutorial | you are new to the component and want to see it work | [Route your first requests](router/tutorial/route-your-first-requests.md) |
| How-to guide | you have a specific task to get done | [router/how-to](router/how-to/) |
| Reference | you need an exact name, default, key or signature | [router/reference](router/reference/) |
| Explanation | you want to know why it works the way it does | [router/explanation](router/explanation/) |
Decisions that are hard to reverse are recorded separately as ADRs in
[`docs/adrs/`](../adrs/AGENTS.md). These pages describe the code as it is on the
branch they ship with, and they link to the source so you can check them. When
the code and a page disagree, the code is right, and the page needs a fix in the
same PR that changed the behavior.

17
docs/architecture/img/README.md Normal file
View file

@ -0,0 +1,17 @@
# Diagrams
The architecture docs use hand-laid-out SVGs instead of Mermaid, so each diagram
fits a docs column without wasted space.
- Every SVG embeds [diagram-style.css](diagram-style.css) verbatim in one
`<style>` element. The stylesheet is based on the litellm-docs theme (Geist
and Geist Mono, the accent `#0117be` and its greys) and switches to the dark
palette with `prefers-color-scheme`
- Use only its classes, with no inline styles or hard-coded colors. The box
classes mean: `box core` the Rust router core, `box host` the host and Python
side, `box store` caches and Redis, `box warn` errors, `box ok` results, and
`box decision` a question
- Keep the `viewBox` at most 880 wide and as short as the content allows. Start
with the `canvas` rect, and give the root element `role="img"` and a `<title>`
- Edit a diagram by changing its SVG directly. A page's diagrams live in an
`img/` folder next to it

38
docs/architecture/img/diagram-style.css Normal file
View file

@ -0,0 +1,38 @@
/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }

61
docs/architecture/img/diataxis-quadrants.svg Normal file
View file

@ -0,0 +1,61 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 220" role="img" aria-labelledby="t">
<title id="t">Four kinds of pages</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="639" height="219" rx="8"/>
<text class="title" x="100" y="30">Four kinds of pages</text>
<rect class="box" x="100" y="46" width="254" height="60" rx="6"/>
<text class="title center" x="227" y="80">Tutorials</text>
<rect class="box" x="366" y="46" width="254" height="60" rx="6"/>
<text class="title center" x="493" y="80">How-to guides</text>
<rect class="box" x="100" y="118" width="254" height="60" rx="6"/>
<text class="title center" x="227" y="152">Explanation</text>
<rect class="box" x="366" y="118" width="254" height="60" rx="6"/>
<text class="title center" x="493" y="152">Reference</text>
<text class="muted center" x="52" y="60">Practice</text>
<path class="edge" d="M52,160 V70" marker-start="url(#arrow)" marker-end="url(#arrow)"/>
<text class="muted center" x="52" y="174">Theory</text>
<text class="muted" x="100" y="202">Studying</text>
<path class="edge" d="M162,198 H556" marker-start="url(#arrow)" marker-end="url(#arrow)"/>
<text class="muted end" x="620" y="202">Working</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4 KiB

70
docs/architecture/router/README.md Normal file
View file

@ -0,0 +1,70 @@
# Router
`litellm.Router` decides which deployment serves a request, makes the call, and
handles what happens when the call fails. It retries in the same model group,
falls back to other model groups, and puts failing deployments into cooldown so
that later requests avoid them. The proxy builds one Router from its config and
sends every model request through it. SDK users can build their own.
![A caller calls litellm.Router, which picks a deployment, calls the provider,
and keeps cooldowns, usage and fail counts in the routing
cache](img/router-overview.svg)
## Concepts
Start with [Router concepts](explanation/concepts.md), which names the ideas and
shows how they fit together. Each idea then has its own page.
- [Deployments and model groups](explanation/deployments-and-model-groups.md),
what is configured and how a deployment gets its identity
- [How a name finds its deployments](explanation/names.md), aliases, wildcards,
routing groups, team names, deployment ids and defaults
- [Teams and access](explanation/teams-and-access.md), team deployments, access
groups, access windows and PTU shares
- [How a deployment is picked](explanation/selection.md), the filter chain and
priority with `order`
- [Strategies, routing groups and sub-routers](explanation/strategies.md), how
one deployment is chosen and how a request is sent to the right group
- [Retries, fallbacks and cooldowns](explanation/reliability.md), including
allowed fails, order fallbacks, weighted failover and streams
- [Shared state across proxy workers](explanation/shared-state.md)
Two pages cover how the router is built.
- [The core and the host](explanation/core-and-hosts.md), what the Rust core
owns and what it asks its host for
- [The life of a request](explanation/request-lifecycle.md), from `acompletion`
through the core to the provider
## Tutorial
New to the router? Work through [Route your first
requests](tutorial/route-your-first-requests.md). It builds a Router with mock
deployments, so it needs no API keys. It shows load balancing, retries,
cooldowns and fallbacks in about fifteen minutes.
## How-to guides
- [Read router state from outside the router](how-to/read-router-state.md)
- [Change deployments or settings at
runtime](how-to/change-the-router-at-runtime.md)
- [Move a caller off a deprecated Router
member](how-to/move-off-deprecated-members.md)
- [Add a custom routing strategy](how-to/add-a-routing-strategy.md)
- [Find out why a request went where it did](how-to/debug-a-routing-decision.md)
## Reference
- [Entry points](reference/entry-points.md), every method on the public surface
and what it replaced
- [Core, config and host interfaces](reference/core-and-host.md), the step
protocol, the Rust `Router` and the `Host` trait
- [Configuration and defaults](reference/configuration.md), constructor
arguments, strategies and constants
- [Routing state and cache keys](reference/state-and-cache-keys.md), what the
router stores and for how long
- [Code map](reference/code-map.md), which file holds what
## Status
These pages describe the branch `nate_router_step6_rust_core_v2` (#49513).

74
docs/architecture/router/explanation/concepts.md Normal file
View file

@ -0,0 +1,74 @@
# Router concepts
The router is responsible for answering one question for every request: which
deployment should serve it. It then has to deal with what happens when that
deployment fails, retrying the request, and falling back to other deployments.
There are a few main ideas that are referenced through the rest of these docs.
This page gives a brief overview of them.
![The router concepts in order: a name resolves to deployments, access rules,
filters and a strategy pick one for an attempt, and a failure goes to retries,
fallbacks and cooldowns, which write the shared state the filters
read](../img/concepts-1.svg)
## What is configured
A **deployment** is one place that can serve a model, such as an Azure OpenAI
resource in one region or a vLLM cluster behind a URL. It is one entry of
`model_list`, with the provider call's arguments in `litellm_params` and
metadata in `model_info`. Its `model_info.id` is the key for all of its routing
state. See [Deployments and model groups](deployments-and-model-groups.md).
A **model group** is every deployment that shares a `model_name`. Callers ask
for a model group, never for a deployment. Load balancing, retries and cooldowns
happen inside one group, and fallbacks move a request between groups.
A **name** is what a request asks for. It is usually a model group, but it can
also be an alias, a wildcard such as `openai/*`, a routing group, a team's
public name for a model, or a deployment id. **Resolution** turns a name into
the list of deployments that may serve the request. See [How a name finds its
deployments](names.md).
## Who may use what
A deployment can belong to one **team**, be reserved for some teams during an
**access window**, be split across teams by **PTU shares**, or sit behind an
**access group** that only some keys reach. These rules run right after
resolution and remove the deployments the caller may not use. See [Teams and
access](teams-and-access.md).
## How one deployment is picked
A **filter** removes deployments that must not serve this request right now:
ones that are failing health checks, cooling down, blocked by an admin, too
small for the prompt, missing a tag, or at a higher **order** (a lower priority)
than another healthy one. See [How a deployment is picked](selection.md).
A **strategy** picks one deployment from the remaining eligible ones.
`simple-shuffle`, the default, picks a weighted random one. Others pick by load,
latency, usage or cost. A **routing group** gives a set of model groups its own
strategy, and **sub-routers** such as the auto router choose which model group a
request goes to before any of this runs. See [Strategies, routing groups and
sub-routers](strategies.md).
## What happens on failure
A **retry** runs the pick again in the same model group, so it can land on a
different deployment. A **fallback** moves the request to another model group
once retries are spent. A **cooldown** takes one failing deployment out of every
pick for a few seconds, and **allowed fails** decide how many failures that
takes. See [Retries, fallbacks and cooldowns](reliability.md).
## State
Cooldowns, failure counters and usage live in a **routing cache** with an
in-memory tier and an optional Redis tier, so a fleet of proxy workers shares
one view of which deployments are healthy. See [Shared state across proxy
workers](shared-state.md).
## How it is built
You do not need this section to use the router. For how the router is built
inside and how it runs a request, see [The core and the host](core-and-hosts.md)
and [The life of a request](request-lifecycle.md).

136
docs/architecture/router/explanation/core-and-hosts.md Normal file
View file

@ -0,0 +1,136 @@
# The core and the host
The router has three layers. `litellm.Router` is a thin Python facade that
callers use. Behind it, the Rust crate `litellm-router-core` (the core) makes
every routing decision and owns the state those decisions need. The core reaches
outside itself only through a host, which runs the provider calls and anything
else that needs live objects from the process the core runs in. This page
explains that split and why the line is drawn where it is.
![litellm.Router and the RustCore driver over the Rust crate, which hands steps
to PythonHost, which calls the provider](../img/core-and-hosts-1.svg)
## What the core owns
The core owns the routing decisions and their state.
- the registry of deployments, its indexes and the deployment build
- name resolution, the filter chain and `simple-shuffle`
- teams, access groups, access windows and PTU shares
- routing groups, pre-call check decisions, health-check routing and the order
filter
- the call loop: retries, retry policies, fallbacks, order fallbacks, weighted
failover and stream recovery
- cooldowns, allowed fails counters and usage, written through the routing cache
- model info and model group info, merged from what the host reports
Everything the core decides is a function of its config, the request, the
routing cache, and the answers the host gave. It never reads process globals,
never calls a provider and never awaits Python. That is what makes the core
testable with a scripted host and portable to a host that is not Python.
## What the host answers
The core asks for what it needs as a step, and the host answers it. Every
attempt is a step: the core picks the deployment, the host runs the call, and
the core records the outcome. There is no second attempt path inside the core.
The `Host` trait (`litellm-rust/crates/router-core/src/host.rs`) groups the
hostcalls.
| Hostcall | What the core needs |
| --- | --- |
| `invoke` | run one attempt on one deployment |
| `select` | a strategy implemented in Python, or the routing plugin, tag or text-only stage |
| `pre_route` | the sub-routers and session binding before a pick resolves its name, or a whole pick by a custom strategy |
| `pre_call_facts`, `callback_filters`, `routing_plugin_models` | the facts and filters that need live objects, such as token counts and `CustomLogger` filters |
| `allow_fallback` | the access and budget checks before a cross-group fallback |
| `secrets`, `validate`, `provider`, `added` | the deployment build, one deployment at a time |
| `model_info` | cost map and capability lookups during a model info query |
| `sleep`, `observe`, `cache` | backoff, routing events, and the shared store all routing state lives in |
The rule for where a piece of logic goes is simple. If a decision needs only
config, the request, routing state and plain facts, it belongs in the core. If
it needs a live object, such as a stateful strategy, a logging callback or the
provider SDK, the host runs that part and hands the core a plain answer. The
core still makes the decision. Pre-call checks show the pattern: the host counts
the prompt's tokens and looks up context windows, and the core decides which
deployments to drop.
## The host today
Today the core runs inside the Python process, under `PythonHost`
(`litellm/router_python_host.py`) and the modules under `litellm/router_rust/`.
- `invoke/` runs one attempt. `litellm/router_attempt.py` builds the keyword
arguments for the deployment the core picked, and `PythonHost.invoke(op)`
calls `litellm.<op>` with them. The endpoint wrappers, stream wrappers,
`max_parallel_requests` slots and retry and fallback metadata live here.
Nothing here picks, retries, falls back or records cooldowns
- `steps/` answers the hostcalls that need live Python objects: the stateful
strategies, the sub-routers, pre-call facts, `Router(plugins=[...])` objects,
`CustomLogger` filters and caller access facts
- `surface/` keeps the Python `Router` API over the core's state. `registry()`,
`model_list`, the indexes and `names()` read the core, and every write to them
becomes a change set
A host that answers these steps natively, inside a Rust gateway, would need none
of `surface/`. Through `Host::supports` and `Host::can_select`, it would refuse
a config that needs Python objects when the router is built, rather than failing
partway through a request.
## Driving the core
`RustCore` holds one `RouterCoreHandle`. A call starts with
`RouterCoreHandle.start(op, request)` and then loops: each
`RouterCall.next(answer)` runs the core until it needs something only the host
can give, and returns that need as a step. `select()` and the model info queries
are driven the same way.
![The step loop for one call: the driver reads cooldowns, tries deployment a and
gets a 429, records a cooldown, tries deployment b, and returns its response
with routing headers](../img/core-and-hosts-2.svg)
Every step also carries the routing events since the last one (`Selected`,
`AttemptFailed`, `CooldownSet`, `Retry`, `Fallback` and others). The driver
hands them to the host's observers, which feed alerting and metrics, and uses
them to stamp the request's metadata and to raise the exception a caller
expects.
## Outcomes are recorded by the core
The core records each attempt's outcome itself, through cache steps, right after
the attempt, so nothing depends on a logging callback firing. A streamed attempt
is deferred: the core counts the request when the stream opens, and when the
stream ends the host's stream wrapper reports either the success, with the final
usage, or the failure.
## The public surface
Code outside the router uses the endpoint methods and eight entry points, and
nothing else: `registry()`, `settings()`, `health()`, `select()`, `apply()`,
`observe()`, `register_plugin()` and `retire()`. Routing plugins get a
`RoutingContext` with the same views instead of the Router.
`tests/code_coverage_tests/router_surface_check.py` keeps an allowlist of the
members that outside code touches, and the list may only shrink. See [Entry
points](../reference/entry-points.md).
## Settings are static
`Router(**arguments)` freezes its arguments into a `RouterConfig`, and the core
is built from it. After that, `apply(ChangeSet(...))` is the only way to change
deployments or settings. The process settings the router reads while serving,
such as `litellm.num_retries`, the fallback lists, the model cost map and the
callback lists, arrive as a versioned `RouterDynamicConfig` snapshot, so the
core never reads a global.
![How the router's settings reach the core: constructor arguments frozen into
RouterConfig, process settings snapshotted into RouterDynamicConfig, and
apply(ChangeSet) for later changes](../img/core-and-hosts-3.svg)
## The application host
Separately, the router needs a few answers that only the proxy has, such as
vertex pass-through credentials and whether a caller may use a team's auto
router. It asks a `RouterHost` (`litellm/router_host.py`). That is the proxy's
`ProxyRouterHost` when the proxy can be imported, and otherwise a default that
refuses team auto routers.

102
docs/architecture/router/explanation/deployments-and-model-groups.md Normal file
View file

@ -0,0 +1,102 @@
# Deployments and model groups
Everything the router routes to is a deployment, and every request asks for a
name, which is usually a model group. This page explains those, lists the
deployment fields the router itself reads, and shows how a deployment gets its
identity.
## Deployment
A deployment is one place that can serve a model, such as an Azure OpenAI
resource in `eastus`, a Bedrock model in one AWS account, or a vLLM cluster
behind a URL. In config, it is one entry of `model_list`.
```yaml
model_list:
- model_name: gpt-4o # the model group callers ask for
litellm_params: # the provider call's arguments
model: azure/gpt-4o-eastus
api_base: https://eastus.example.openai.azure.com
api_key: os.environ/AZURE_EASTUS_KEY
rpm: 600
order: 1
model_info: # metadata about the deployment
id: gpt-4o-eastus
```
![A deployment and its two parts: litellm_params, the provider call's arguments,
and model_info, its metadata](../img/deployments-and-model-groups-1.svg)
`litellm_params` holds mostly the arguments for the provider call, which the
router passes on unchanged when the deployment is picked. The router reads a few
of them itself.
| Field | What the router does with it |
| --- | --- |
| `weight`, `rpm`, `tpm` | `simple-shuffle` weights, in that order of preference. `rpm` and `tpm` are also limits for usage-aware strategies and pre-call checks |
| `order` | in-group priority, lowest first. See [How a deployment is picked](selection.md) |
| `tags` | which tagged requests it serves, when tag filtering is on |
| `use_in_pass_through` | whether [pass-through routes](../../../../litellm/proxy/pass_through_endpoints/architecture.md) may use it |
| `timeout`, `stream_timeout` | the attempt's timeout, which wins over the router's |
| `max_parallel_requests` | how many attempts may run on it at once |
`model_info` is metadata. The router reads who may use the deployment
(`team_id`, `access_groups`, `access_windows`, `ptu_shares`, see [Teams and
access](teams-and-access.md)), whether an admin paused it (`blocked`), and how
it cools down (`cooldown_time`, `allowed_fails`, `allowed_fails_policy`, see
[Retries, fallbacks and cooldowns](reliability.md)). Everything else is kept as
written and handed back to callers that read the registry.
## Identity
`model_info.id` is the deployment's identity. Cooldowns, failure counters, usage
and health results are all keyed by it, and a change set replaces or deletes a
deployment by it. When the config leaves it out, the router derives one with
`deployment_id()`: a SHA-256 over the model group name and every
`litellm_params` key and value, in order. The same config gets the same id on
every worker, which is what lets workers share routing state. Two entries with
the same id both stay in the group and share that state, and a lookup by id
returns the later one.
## How a deployment is built
The router builds deployments from config in order. For each entry it resolves
`os.environ/` secret references, validates the entry, resolves the provider that
serves it, and records it. An entry with an `organization` list becomes one
deployment per organization. A deployment joins the registry as soon as it is
accepted, so a later entry sees the earlier ones, and an error points at the
entry that caused it. An entry that fails to build is skipped, and the rest
still build.
![For each model_list entry the router resolves secrets, validates the entry,
finds the provider, records it, then gives it an id and adds it to the
registry](../img/deployments-and-model-groups-2.svg)
## Model group
A model group is every deployment that shares a `model_name`. Callers ask for a
group, never for a deployment, and the router picks a deployment inside it. Load
balancing, retries and cooldowns all happen inside one group, and fallbacks move
a request from one group to another.
![A request for gpt-4o reaches the gpt-4o model group of three deployments at
two orders, which falls back to the claude-sonnet model
group](../img/deployments-and-model-groups-3.svg)
Group size changes behavior in a few places. In a group of one: a 429 does not
cool the only deployment down, an auth error is not retried, and the default
error-rate rule leaves the lone deployment alone unless it saw 1,000 requests
that all failed. Those rules exist because cooling down the only deployment
would fail every request for the group. [Retries, fallbacks and
cooldowns](reliability.md) has the details.
## The registry
All deployments live in the registry, an immutable, indexed snapshot held by the
router. It is indexed by id, model group, alias, wildcard pattern and team, and
it carries a version that goes up on every change. A change replaces the
snapshot, so a request that already read the registry keeps a consistent view
while the change lands. Code outside the router reads it through
`router.registry()` and changes it through `router.apply()`, see [Read router
state](../how-to/read-router-state.md) and [Change deployments or settings at
runtime](../how-to/change-the-router-at-runtime.md).

76
docs/architecture/router/explanation/names.md Normal file
View file

@ -0,0 +1,76 @@
# How a name finds its deployments
A request names a model, and the name does not have to be a model group. It can
be an alias, a wildcard match, a routing group, a team's own name for a model,
or a deployment id. Resolution turns the name into the list of deployments that
may serve this caller, which is used for later steps. Resolution runs at the
start of every pick, so a retry or a fallback resolves the name again.
## The kinds of name
| Name | Example | Request names | Resolves to |
| --- | --- | --- | --- |
| Model group | `gpt-4o` | `gpt-4o` | the deployments with that `model_name` |
| Alias | `model_group_alias={"gpt-4": "gpt-4o"}` | `gpt-4` | the target group. `{"model": "gpt-4o", "hidden": true}` also hides the alias from model listings |
| Routing group | `routing_groups=[{"group_name": "fast", "models": [...]}]` | `fast` | the deployments of every member group, see [Strategies, routing groups and sub-routers](strategies.md) |
| Wildcard | a deployment with `model_name: "openai/*"` | `openai/gpt-4o-mini` | that deployment, with the requested model filled into its params |
| Team public name | `model_info.team_public_model_name: "my-gpt"` | `my-gpt` | the team's deployment, for members of that team |
| Deployment id | `gpt-4o-eastus` | `gpt-4o-eastus` | that one deployment, even while it cools down |
| Specific deployment | a deployment's `litellm_params.model` | `azure/gpt-4o`, with `specific_deployment=True` | the deployments whose litellm model is exactly that name |
| Anything else | | `unknown-model` | the `default_deployment` if one is set, else the first `default_fallbacks` group, else an error |
## The order
The checks run in a fixed order, and the first match wins.
![How a name resolves, in order: specific_deployment, deployment id, alias,
routing group, configured group, then the litellm model name, the first
default_fallbacks group, or an error](../img/names-1.svg)
Two rules apply on every path. Deployments that are sub-router markers, such as
`auto_router/...`, are never returned, since they are not meant to be called.
Deployments that an access window reserves for other teams right now are
withheld, see [Teams and access](teams-and-access.md).
## Aliases
An alias maps one name to a model group. Aliases come from the router's
`model_group_alias` and from the process's model alias map, which the router
reads through its dynamic config. Resolution swaps the alias for its target
before it looks anything else up, so an alias can also point at a routing group.
The attempt still reports the requested name to the caller.
## Wildcards
A deployment whose `model_name` contains `*`, such as `openai/*` or
`anthropic/claude-*`, serves any name it matches that no model group claims. The
registry keeps the patterns ordered from most to least specific, so
`anthropic/claude-3-*` wins over `anthropic/*`. The matched part is substituted
into the deployment's `litellm_params.model`: a deployment with
`model_name: openai/*` and `model: openai/*` serves a request for
`openai/gpt-4o-mini` by calling `openai/gpt-4o-mini`. Wildcards are checked only
for a name that is not a configured model group, so a configured group always
wins over a pattern that also matches it.
## Deployment ids
A name that is not a model group but is a deployment id resolves to that one
deployment. The router serves it even while it cools down, and skips the
strategy, since there is nothing to choose. This is how a caller pins a request
to one deployment.
## Default deployment and default fallbacks
`default_deployment` is a deployment that serves any name nothing else matched,
with the requested model filled in. Without one, a name that resolves to nothing
tries `default_fallbacks`, the catch-all fallback list, and resolves to its
first model group. If that list is empty too, the pick fails with an error that
says no deployments are configured for the model.
## The result
Resolution returns three things: the deployments, which kind of name matched,
and the model group the attempt reports. For an alias or a routing group, that
model group is the requested name, so headers and logs show what the caller
asked for while the deployments come from the target. [How a deployment is
picked](selection.md) takes it from there.

200
docs/architecture/router/explanation/reliability.md Normal file
View file

@ -0,0 +1,200 @@
# Retries, fallbacks and cooldowns
The router has three tools for failures, and each works at a different scope. A
retry tries the same model group again, and the new pick can land on a different
deployment. A fallback moves the request to a different model group. A cooldown
takes one deployment out of every pick, for every request, for a few seconds.
![Inside each model group the strategy picks one deployment. The strategy picks
D1 in gpt-4o, which returns 429 and cools down, the retry picks D3 in the same
group, which also returns 429, then the request falls back to opus, where the
strategy picks D5, which returns 200 to the router](../img/reliability-1.svg)
Every decision reads three facts about the failure: its kind (rate limit,
context window, content policy, auth, not found, timeout, ...), its status code
and its `Retry-After`. The router takes these from the provider's exception and
decides from them alone.
The router is the only retry loop. It sets `max_retries` to 0 on the provider
call, so neither the provider SDK nor the HTTP client retries on its own, and
each retry can pick a different deployment.
## Retries
The retry loop runs the first attempt. When that attempt fails, the router works
out how many retries this request gets, decides whether this failure is worth
retrying, waits, and loops.
![The retry loop: a guardrail intervention gives up at once, otherwise the
router works out the retries, checks the failure, sleeps, selects again skipping
failed deployments, and returns, loops or hands the failure to the fallback
walk](../img/reliability-2.svg)
A failure is not worth retrying in these cases.
- a context window error when `context_window_fallbacks` is set, or a content
policy error when `content_policy_fallbacks` is set, because a fallback will
do better than another deployment of the same model
- a 4xx status other than 401, 403, 408, 409 and 429
- a not found error
- a rate limit when no healthy deployment is left and fallbacks exist
- an auth or permission error in a group with a single deployment
- any failure when no healthy deployment is left
The wait is zero while the group has more than one deployment and some are still
healthy, since the next attempt can go somewhere else. Otherwise the router
honors the provider's `Retry-After`, up to one minute. Without a `Retry-After`,
it backs off exponentially from 0.5 seconds up to 8 seconds, never less than the
router's `retry_after`, plus up to 0.75 seconds of jitter.
### Retry policies
`retry_policy` sets the number of retries per failure kind. For example, more
retries for rate limits than for timeouts. `model_group_retry_policy` does the
same per model group.
```python
Router(
model_list=...,
retry_policy={"RateLimitErrorRetries": 3, "TimeoutErrorRetries": 1, "ContentPolicyViolationErrorRetries": 0},
model_group_retry_policy={"gpt-4o": {"RateLimitErrorRetries": 5}},
)
```
A policy's count for the failure's kind wins over `num_retries`, unless the
request itself turned retries off with `num_retries=0`.
## Fallbacks
When the retry loop gives up, the router chooses where to go next.
![Where the router goes once retries are exhausted: raise on disable_fallbacks
or a guardrail, else the next orders in the group, weighted failover, then the
context window, content policy or generic fallbacks by failure
kind](../img/reliability-3.svg)
Each fallback list is a list of single-key dicts, each mapping a model group to
an ordered list of model groups to try, for example
`fallbacks=[{"gpt-4o": ["claude-sonnet", "gpt-4o-mini"]}]`. A target can also be
a dict such as `{"model": "gpt-4o-mini", "temperature": 0}`, whose other keys
override the request's params for that hop. `default_fallbacks=["gpt-4o-mini"]`
becomes the catch-all entry `{"*": ["gpt-4o-mini"]}`. The router looks up the
group being served first and then the group the request asked for, so a fallback
target's own fallbacks apply on the next hop. The lists come from the router's
arguments or, when the router sets none, from the process settings in the
dynamic snapshot. A request can also carry its own `fallbacks`, which win for
that request.
The fallback walk takes each target in order. Before a target in another model
group, it runs `fallback_access_check` (may this caller use that group) and
`fallback_budget_check`, which use the proxy's key and team data. It skips
targets this request already tried, then runs the whole retry loop again against
the target, which can fall back further. A depth counter stops the walk at
`max_fallbacks` (default 5, or the `ROUTER_MAX_FALLBACKS` environment variable).
Each hop is reported as a `Fallback` routing event with its source, target,
order, depth and level.
![A fallback in sequence: model A fails with a 503 after three attempts, the
fallback checks allow group B, and B answers with a
200](../img/reliability-4.svg)
With `fallbacks=[{"A": ["B"]}]`, `context_window_fallbacks=[{"A": ["A-long"]}]`
and `num_retries=2`, a 503 from A is retried twice in A, then the request goes
to B with two retries of its own. A context window error from A is not retried
at all, since a context window fallback exists. The request goes straight to
A-long.
### Order fallbacks
When a model group's deployments use more than one `order`, a failure first
walks down the orders inside the same group before it leaves the group. With
`order: 1` PTU deployments and an `order: 2` pay-as-you-go deployment, a 429 on
the PTU tier sends the request to pay-as-you-go in the same group, and only then
to the group's configured fallbacks. Each order level gets its own retry loop. A
context window or content policy error skips the order walk, since a lower
priority deployment of the same model would fail in the same way.
### Weighted failover
With `enable_weighted_failover` and `simple-shuffle`, a failed deployment is
excluded and the router picks again among the group's remaining deployments that
are not cooling down, before any configured fallback runs. This spreads a
failing deployment's traffic over its healthy siblings, by their weights,
instead of sending it to another group. Context window and content policy errors
skip weighted failover for the same reason as above.
## Streams
A streamed attempt succeeds as soon as the provider opens the stream, but it can
still fail while the caller reads it. The router wraps every stream it returns.
When the stream breaks before the caller has received any content, the router
takes over with a retry in the same group or the next fallback, as if the
attempt had failed up front. Once content has reached the caller, a retry would
repeat output, so the stream fails, unless
`enable_mid_stream_fallback_continuation` is set. With that setting, a chat
completions stream that broke after plain assistant text continues on a fallback
deployment that supports assistant prefill: the router sends the text so far as
a prefill, so the new deployment carries on instead of starting over, and the
caller sees one response. A stream that has already emitted tool calls, thinking
blocks, audio or images, or constrained output such as JSON mode, is never
continued. Without an eligible deployment, the original error is raised.
![What happens when a stream breaks: before any content the router retries or
falls back, after text with continuation on it continues on a fallback
deployment with a prefill, otherwise it raises to the
caller](../img/reliability-5.svg)
## Cooldowns
A cooldown is a key `deployment:{id}:cooldown` in the routing cache, with a TTL
equal to the cooldown time. While the key exists, the cooldown filter drops the
deployment from every pick, on every process that shares the Redis. When the TTL
expires, the deployment is simply eligible again. There is no probe request and
no early reset on success.
The failure handler decides whether a failure cools the deployment down. It runs
two checks: first on the failure itself, then on the deployment's recent
history.
![How a failure on a deployment leads to a cooldown: the skip and counting
checks, then either the allowed_fails counter or the last 60 seconds error-rate
rules](../img/reliability-6.svg)
The cooldown time comes from the first of these that is set: the deployment's
own `cooldown_time`, then the provider's `Retry-After`, then the router's
`cooldown_time`, which defaults to 5 seconds. Cooldown alerts and metrics fire
from the `CooldownSet` routing event when a deployment is cooled down.
### Allowed fails
`allowed_fails` lets a deployment fail a few times before it cools down. Each
counted failure increments `deployment:{id}:allowed_fails`, which expires with
the cooldown time, and the deployment cools down once the count goes over the
limit. `allowed_fails_policy` sets the limit per failure kind, for example
tolerating a few timeouts but cooling down on the first auth error.
```python
Router(
model_list=...,
allowed_fails_policy={"TimeoutErrorAllowedFails": 5, "AuthenticationErrorAllowedFails": 0, "RateLimitErrorAllowedFails": 2},
)
```
A deployment can set its own `allowed_fails` or `allowed_fails_policy` in
`model_info`, and that setting wins over the router's. The router-wide
`allowed_fails` takes part only when it was set explicitly or changed from its
default, so a router that never mentions it uses the error-rate rule below.
Two details are easy to miss. First, the success and failure counts for the
error-rate rule are kept in memory only, per process, with a 60 second TTL, so
on a fleet each worker judges a deployment by the traffic it saw itself. The
`allowed_fails` counter and the cooldown key live in Redis when it is
configured, so they apply to the whole fleet. Second, by default a lone
deployment is protected from the error-rate rule, since cooling down the only
deployment of a group would fail every request for that group.
When every candidate is cooled down, the pick raises `RouterRateLimitError` with
the time until the soonest cooldown ends, and the fallback walk handles it like
any rate limit. `enable_health_check_routing` together with an allowed fails
policy changes this: when every candidate is cooled down, the router picks one
of them anyway instead of raising.

70
docs/architecture/router/explanation/request-lifecycle.md Normal file
View file

@ -0,0 +1,70 @@
# The life of a request
This page follows one `await router.acompletion(model="gpt-4o", messages=...)`
from the facade, through the core, to the provider and back. The other endpoint
methods (`aembedding`, `aresponses`, `aanthropic_messages`, the files and
batches wrappers and the rest) take the same path with a different operation.
## The shape
Fallbacks wrap retries, and retries wrap a single attempt. Each attempt picks a
deployment and calls `litellm.<op>` with that deployment's params, so a retry
can land on a different deployment.
![Router.acompletion passes through fallbacks, retries and one attempt to
litellm.acompletion and the provider HTTP call](../img/request-lifecycle-1.svg)
The router sets `max_retries` to 0 for the provider call, so neither the
provider SDK nor the HTTP client retries on its own. The router's loop is the
only retry loop, which is what lets each retry pick a different deployment.
`litellm.acompletion` is the same function SDK users call. The router does not
talk to providers itself. It chooses the arguments, and the provider
transformation layer does the rest.
The response carries the picked deployment's id in
`response._hidden_params["model_id"]`. The proxy reports it in the
`x-litellm-model-id` header, together with the model group, the retries and the
fallbacks the call used.
## Step by step
`RustCore.acompletion` turns the call into one
`RouterCoreHandle.start("acompletion", request)` and drives the steps it
returns. The loop itself is in `litellm-rust/crates/router-core/src/call.rs`,
and the pick it runs per attempt is in `select.rs`.
![Numbered sequence of one acompletion call: the Router, the RustCore driver,
the core's steps, the cache reads and writes, the attempt through PythonHost to
the provider, and the response with routing
headers](../img/request-lifecycle-2.svg)
Some details worth knowing.
The core turns an attempt's error into a typed `AttemptError` with a failure
kind (rate limit, context window, content policy, auth, timeout, ...), the
status code and any `Retry-After`. The retry, fallback and cooldown rules read
those facts.
The core records each outcome itself, through cache steps, right after the
attempt, so nothing depends on a logging callback firing. Routing events
(`Selected`, `AttemptFailed`, `CooldownSet`, `Retry`, `Fallback`,
`RetriesExhausted` and others) ride along on every step. The driver hands them
to the host's observers, and uses them to stamp the request metadata and to
raise the exception a caller expects.
A streamed attempt is deferred. The core counts the request when the stream
opens. When the stream ends, the host's stream wrapper reports the success with
the final usage, or the failure if the stream broke. A stream that breaks before
the caller got any content asks the core what may still take over
(`call/stream.rs`).
Stateful strategies, such as latency and usage-based routing, learn from
litellm's logging callbacks and keep their counters in Python. They run in
`litellm/router_rust/steps/selection.py` when the core asks for a pick.
## The sync path
`router.completion()` and the other sync methods run the same loop.
`RustCore._call_sync` drives the same steps on the calling thread, and a sleep
step blocks that thread. The proxy uses the async methods for every request.

109
docs/architecture/router/explanation/selection.md Normal file
View file

@ -0,0 +1,109 @@
# How a deployment is picked
A pick turns a requested name into one deployment. It runs inside every attempt,
so a retry picks again and can land somewhere else. `router.select()` runs the
same pick without calling the provider.
## The filter chain
Resolution produces the candidates, and a chain of filters removes the ones that
must not serve this request right now. A filter can only remove candidates. The
strategy at the end of the chain sees what is left.
![The filter chain from pre-routing and name resolution through every filter to
the strategy, or to the no deployments available error](../img/selection-1.svg)
## The filters
**Route capability filter.** A call that only some providers serve skips the
deployments whose provider cannot serve it, as long as another candidate
remains. For example, a `/v1/decisions` call to a group with an OpenAI
deployment and an Anthropic deployment skips the Anthropic one, since litellm
has no Decisions config for that provider.
**Health check filter.** With `enable_health_check_routing`, the results of the
background health checks count too. A deployment that the last check marked
unhealthy is dropped, as long as that result is younger than
`health_check_staleness_threshold`. `background_health_check_model_groups`
limits this to some model groups. If the filter would leave nothing, it keeps
every candidate instead, so a broken health checker cannot take a whole group
down.
**Cooldown filter.** Drops every deployment with a live
`deployment:{id}:cooldown` key in the routing cache. Cooldowns are how failures
on one request protect the next ones, see [Retries, fallbacks and
cooldowns](reliability.md). When health checks drive routing with an
`allowed_fails_policy` and the cooldown filter would leave nothing, the router
ignores cooldowns for that request.
**Blocked filter.** Drops deployments with `model_info.blocked`, which an admin
sets to pause a deployment without deleting it. If every deployment of a group
is blocked, the error says so.
**Callback filters.** Callbacks can narrow the candidates too, and they run in
the chain like any other filter. The budget limiter drops deployments whose
provider is over its `provider_budget_config` budget. Deployment and session
affinity keep a conversation on the deployment that served it before by removing
all others from consideration as long as the previous deployment is healthy.
**Pre-call checks.** With `enable_pre_call_checks`, the router drops deployments
whose context window the prompt exceeds, deployments outside the request's
`allowed_model_region`, and deployments that cannot take a requested parameter.
The prompt's token count and each deployment's limits come from the model cost
map. When nothing is left, the error says which check removed the last
deployment. A context window error, for example, is one the fallback walk can
route to `context_window_fallbacks`.
**Tag filter.** With `enable_tag_filtering`, a request's tags keep only the
deployments whose `litellm_params.tags` include them. Deployments tagged
`default` serve untagged requests. `!tag` excludes deployments that carry that
tag, and `&tag` requires it.
**Routing plugin candidates.** When routing plugins ran during pre-routing, only
the deployments of the models they left remain.
**Text-only filter.** A request that carries an image skips deployments that
declare no vision support, as long as another candidate remains.
**Order filter.** Keeps only the deployments with the lowest
`litellm_params.order` among those left, or with the order an order fallback
targets. See below.
**Excluded and retry-skipped ids.** A retry skips the deployments that already
failed for this request, but only while something else remains. Weighted
failover excludes ids outright, and an excluded id is never picked.
## Priority with `order`
`order` is how in-group priority works. Deployments with `order: 1` take all
traffic while any of them passes the filters. Deployments with `order: 2` see
traffic only when the fallback walk targets the group at that order, after every
`order: 1` deployment failed for the request (see [Retries, fallbacks and
cooldowns](reliability.md)). Mixing deployments with and without an order is
easy to get wrong. While any candidate sets an order, this filter drops a
deployment without one, and the order fallbacks never target it either. So give
every deployment of a group an order, or give none of them one. A priority
routing group serves each member's priority as its deployments' `order`, so the
same filter walks the priorities (see [Strategies, routing groups and
sub-routers](strategies.md)).
![Priority with order: normal picks go to the two order 1 deployments, and the
order 2 deployment only gets a request after order 1
failed](../img/selection-2.svg)
## Shortcuts
A name that resolves to one deployment by its id is served even while that
deployment cools down. The callback filters and the strategy are skipped, since
there is nothing to choose. A custom strategy skips the whole chain, see
[Strategies, routing groups and sub-routers](strategies.md).
## When nothing is left
The pick fails with "No deployments available for selected model". The error
lists the deployments in cooldown and the seconds until the soonest cooldown
ends, and it says whether pre-call checks were on, since they are a common
cause. The fallback walk treats this error like a rate limit, so a group with
every deployment cooling down falls back to its fallback groups. [Find out why a
request went where it did](../how-to/debug-a-routing-decision.md) shows how to
find which filter removed a deployment.

65
docs/architecture/router/explanation/shared-state.md Normal file
View file

@ -0,0 +1,65 @@
# Shared state across proxy workers
A production proxy runs many workers, each with its own Router built from the
same config. They agree on routing through Redis. This page explains what is
shared, what is not, and how fresh each piece is guaranteed to be.
![Three proxy workers, each with a Router and its own in-memory DualCache tier,
all syncing with one shared Redis that holds cooldowns, fail counters, usage and
strategy state](../img/shared-state-1.svg)
## The routing cache
The router keeps its routing state in a `DualCache`
(`litellm/caching/dual_cache.py`) built from the `redis_*` arguments. The cache
has two tiers, an in-memory tier and a Redis tier. Writes go to both. Reads
check memory first, go to Redis only on a miss, and then copy the Redis value
into memory. Without Redis settings, the router runs with the in-memory tier
alone. That is fine for one process, but it means every process routes on its
own view.
The cache is returned from `router.health().store`.
Reading memory first keeps Redis off the hot path, at the cost of freshness. A
value another process wrote is seen only once the local copy has expired, or
when there never was a local copy. The router handles this differently for each
kind of state.
| State | Key | Tiers | How fresh other processes see it |
| --- | --- | --- | --- |
| Cooldown | `deployment:{id}:cooldown` | memory + Redis | the local copy is re-checked against Redis at least every `DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS` (1s) |
| Allowed fails counter | `deployment:{id}:allowed_fails` | memory + Redis, incremented in Redis | fleet-wide at increment time, falling back to the local count if Redis fails |
| Success / fail counts for the error-rate rule | `{id}:successes`, `{id}:fails` | memory only, 60s TTL | never, each process judges on its own traffic |
| Usage per minute | `global_router:{id}:{model}:tpm:{HH-MM}`, `...:rpm:...` | Redis increment after each response | at once in Redis. A worker's copy catches up on its next write or the next minute |
| Strategy state (usage-based v2, latency, least-busy) | strategy keys | memory, pushed to Redis every `DEFAULT_REDIS_SYNC_INTERVAL` (1s) or on write | about a second behind |
See [Routing state and cache keys](../reference/state-and-cache-keys.md) for the
full list with TTLs.
## What this means in practice
Cooldowns are shared. When one worker cools a deployment down, the others stop
picking it within about a second.
Rate limits through `rpm` and `tpm` are soft. Processes do not reserve capacity
before a call. They read counters that are incremented after responses, so a
burst can overshoot a deployment's limit by roughly what the fleet sends in one
sync interval. The proxy's own rate limiter for keys and teams is a separate
system that uses atomic Lua scripts, and the router does not use it.
The default error-rate cooldown is decided per process. A deployment that fails
half its traffic everywhere gets cooled down by each process independently, and
the resulting cooldown key is shared. With `allowed_fails` or an
`allowed_fails_policy` set, the counter is in Redis and the threshold is
fleet-wide.
![Sequence in which one worker sets a deployment cooldown in Redis after a 429,
a second worker reads it and filters the deployment too, and the key expires on
both tiers](../img/shared-state-2.svg)
## Redis outages
The router keeps routing when Redis fails. The cache logs the failure and serves
from memory, and the `allowed_fails` increment falls back to the process's own
count. Routing quality drops to what each process knows alone until Redis
returns.

126
docs/architecture/router/explanation/strategies.md Normal file
View file

@ -0,0 +1,126 @@
# Strategies, routing groups and sub-routers
Once the filters have left a list of deployments that may serve a request, a
strategy picks one. Routing groups let different model groups use different
strategies, and sub-routers decide which model group a request should go to
before any of that happens. This page covers all three, plus custom strategies
and routing plugins.
![A request is routed by a sub-router to a model group, resolved and filtered,
then the chosen strategy picks one deployment](../img/strategies-1.svg)
## Strategies
| Strategy | Picks | Reads |
| --- | --- | --- |
| `simple-shuffle` (default) | a weighted random deployment | per-request weights, then `weight`, then `rpm`, then `tpm`, else uniform |
| `least-busy` | the deployment with the fewest requests in flight | in-flight counters |
| `usage-based-routing-v2` | the deployment with the lowest tpm this minute that stays under its tpm and rpm | per-minute usage counters |
| `usage-based-routing` | the v1 version of the above, kept for old configs | per-minute usage |
| `latency-based-routing` | the deployment with the lowest recent latency (time to first token for streams), within a buffer | latency samples |
| `cost-based-routing` | the cheapest deployment still under its limits | model cost map, usage |
| `lar1` | a deployment by the caller's confidence level, with thresholds from `routing_strategy_args` | `metadata.lar1` on the request |
`simple-shuffle` needs no state. The other strategies keep their own counters,
fed by litellm's logging callbacks. Every strategy runs after the filters, so it
only ever sees deployments the filters already approved.
### Weights in `simple-shuffle`
![How simple-shuffle picks its weights: request weights, else weight, else rpm,
else tpm, else uniform random](../img/strategies-2.svg)
The first source with a positive value on any candidate is used for all
candidates, and a candidate that does not set it gets weight 0. Per-request
weights come from key or team level `router_settings`, and the proxy puts them
on the request as `{model_group: {deployment_id: weight}}`. One team can then
send most of a group's traffic to one deployment while everyone else gets the
global split, without a separate router per team.
## Which strategy applies
![Which strategy applies: the request's routing_strategy, else the routing
group's, else the router's, else simple-shuffle](../img/strategies-3.svg)
A per-request `routing_strategy`, which the proxy forwards from key or team
`router_settings`, wins. Next comes the strategy of the routing group the
resolved name belongs to, then the router's own `routing_strategy`. Stateful
strategies keep separate state per scope, so the same strategy used by two
routing groups does not mix their counters.
## Routing groups
A routing group is a named set of model groups with its own strategy.
```yaml
routing_groups:
- group_name: fast
models: [gpt-4o-mini, claude-haiku]
routing_strategy: latency-based-routing
- group_name: tiers
models: [gpt-4o-ptu, gpt-4o-paygo]
routing_strategy: priority
model_priorities: {gpt-4o-ptu: 1, gpt-4o-paygo: 2}
```
The routing group's name is callable. A request for `fast` resolves to the
deployments of both members, and latency-based routing picks among them. A
direct request for `gpt-4o-mini` also uses latency-based routing, since that
model group belongs to `fast`. Every model group that is not a member of a
configured routing group belongs to the implicit default routing group, which
uses the router's `routing_strategy`.
A priority group is the exception. It does not pick by a strategy of its own.
Instead it serves each member's priority as its deployments' `order`, so the
order filter sends all traffic to priority 1 while any of it is healthy, and
order fallbacks walk down the priorities when it fails. See [How a deployment is
picked](selection.md) for the order filter and [Retries, fallbacks and
cooldowns](reliability.md) for order fallbacks.
## Sub-routers
The auto router, complexity router, quality router and adaptive router are
sub-routers, not strategies. They choose which model group should serve a
request, before the name is resolved. Each is configured as a deployment whose
`litellm_params.model` starts with `auto_router/`.
`auto_router/complexity_router...`, `auto_router/quality_router...` and
`auto_router/adaptive_router...` select those three routers, and any other name
under `auto_router/` is the auto router. A request for that deployment's
`model_name` runs the sub-router.
| Sub-router | Chooses by |
| --- | --- |
| auto router | how closely the prompt matches the example prompts configured for each route, by embedding similarity |
| complexity router | how hard the prompt looks, sending simple prompts to a cheaper group |
| quality router | the prompt's complexity mapped to a quality tier, then the model whose routing preferences match that tier |
| adaptive router | the kind of request (code, writing, reasoning and so on) and a bandit that learns from past responses which model serves each kind best, weighed against cost |
![A request for smart goes to the auto router, which classifies the prompt and
returns gpt-4o-mini, and the router picks within that
group](../img/strategies-4.svg)
The router asks the sub-router once per pick, before resolution, and resolves
whatever model group it returns. The attempt still reports the requested name,
`smart`, as its model group, so headers and spend logs show what the caller
asked for. Marker deployments are dropped from every candidate list, so they are
never called directly. Whether a caller may use a sub-router that belongs to a
team depends on the proxy's team checks, which the router asks the proxy for.
Session binding happens at the same point: a request that belongs to a coding
session can be bound to the model group that served the session so far.
## Routing plugins
`Router(plugins=[...])` takes routing plugins: objects that look at a request
and narrow which model groups or deployments may serve it, or that attach
signals for later stages to read. The plugins run during pre-routing, and the
router then keeps only the deployments of the models they left.
## Custom strategies
`router.register_plugin(strategy)` with a `CustomRoutingStrategyBase` does not
slot into the strategy step. It replaces the whole pick. The router serves the
deployment the custom strategy returns as is, without resolution, filters or a
strategy of its own. The custom strategy may even return a deployment the
registry does not hold. `lar1` works this way. See [Add a custom routing
strategy](../how-to/add-a-routing-strategy.md).

89
docs/architecture/router/explanation/teams-and-access.md Normal file
View file

@ -0,0 +1,89 @@
# Teams and access
The proxy serves many teams from one router, and not every deployment is for
everyone. Four rules decide which of a model group's deployments a caller may
use. They all run during resolution and right after it, before any health or
load filter, so a deployment the caller may not use never shows up in the
caller's candidates, errors or fallbacks.
![Resolved deployments pass access windows, access groups, team deployments and
PTU shares in order before the health and load
filters](../img/teams-and-access-1.svg)
The caller facts come in on the request: `team_id`, whether the caller is a
proxy admin, and the models the key and team may use. The proxy fills them in
from the key that made the request.
## Team deployments
A deployment with `model_info.team_id` belongs to one team. Only requests
carrying that team id can use it, and everyone else's picks skip it. A team
deployment can also carry `model_info.team_public_model_name`, the name the
team's members call it by. A request for that name from a member of the team
resolves to the team's deployment, even though no model group has that name.
```yaml
- model_name: team-a-gpt-4o-ft # the internal group name
litellm_params: {model: azure/ft-gpt-4o-team-a, ...}
model_info:
team_id: team-a
team_public_model_name: gpt-4o # what team-a's members ask for
```
When the requested model group has both team deployments and shared ones, the
caller's own team deployments come first. A proxy admin without a team may reach
team deployments by name, as long as they all belong to one team. If they belong
to several teams, the pick fails, since there is no way to tell which team was
meant.
## Access groups
`model_info.access_groups` lists the access groups a deployment belongs to. A
key or team can be given access to an access group instead of to model names.
When a caller reaches the requested model only through access groups, the router
limits the caller to the deployments in those groups. Two keys that both "have
gpt-4o" can therefore be sent to different deployments of it.
## Access windows
`model_info.access_windows` reserves a deployment for some teams during a daily
wall-clock window, for example a provisioned deployment that one team owns
during its business hours.
```yaml
model_info:
access_windows:
- start: "09:00"
end: "17:00"
timezone: America/New_York
team_ids: [team-a]
```
While a window is active, callers outside its `team_ids` do not see the
deployment. Outside the window, everyone does. When all of a group's deployments
are reserved for someone else right now, the pick fails with an error that names
the window instead of a generic "no deployments".
## PTU shares
`model_info.ptu_shares` splits one provisioned deployment across teams, as a map
from team id to a whole number of PTUs. A deployment with shares serves only the
teams that hold one. A team that holds a share on any deployment of the group is
limited to those deployments, and a team without a share sees only the
deployments that are not split.
```yaml
model_info:
ptu_shares:
team-a: 100
team-b: 50
```
## Fallbacks respect the same rules
A fallback to another model group resolves the target name for the same caller,
so the same rules apply to the target. Before a cross-group fallback, the router
also checks whether this caller may use the target group at all and whether the
caller's budget allows it (`fallback_access_check` and `fallback_budget_check`).
A target the caller may not use is skipped, see [Retries, fallbacks and
cooldowns](reliability.md).

89
docs/architecture/router/how-to/add-a-routing-strategy.md Normal file
View file

@ -0,0 +1,89 @@
# Add a custom routing strategy
Use this guide when none of the built-in strategies picks deployments the way
you need, and you want to plug in your own without changing the router. You
write a `CustomRoutingStrategyBase` and register it with
`router.register_plugin()`.
## Know what you replace
`register_plugin` replaces the router's whole pick, not only the strategy at its
end. The router calls your method and serves the deployment it returns as is.
The pre-routing hook, the filters and the built-in strategy no longer run unless
you call them.
![The built-in pre-routing, filters and strategy, next to a registered plugin's
async_get_available_deployment that replaces them and can call router.select to
keep the filters](../img/add-a-routing-strategy-1.svg)
To keep cooldowns, team checks and the other filters, ask the router for the
filtered candidates with `select(kind="healthy")` and choose only among them, as
the example below does.
## Write the strategy
```python
from litellm import Router
from litellm.types.router import CustomRoutingStrategyBase
class LowestIdStrategy(CustomRoutingStrategyBase):
def __init__(self, router: Router) -> None:
self.router = router
async def async_get_available_deployment(
self, model, messages=None, input=None, specific_deployment=False, request_kwargs=None
):
healthy = await self.router.select(
model,
request_kwargs,
kind="healthy",
messages=messages,
input=input,
specific_deployment=specific_deployment,
)
if isinstance(healthy, dict): # the name resolved to one specific deployment
return healthy
return min(healthy, key=lambda d: d["model_info"]["id"])
def get_available_deployment(
self, model, messages=None, input=None, specific_deployment=False, request_kwargs=None
):
raise NotImplementedError("this strategy serves the async path only")
```
The method returns one deployment dict, normally one from the model list. The
router also serves a deployment that the registry does not hold.
`select(kind="healthy")` raises the router's usual no-deployments error when
everything is filtered out, so you do not need to raise your own.
## Register it
```python
router = Router(model_list=[...])
router.register_plugin(LowestIdStrategy(router))
```
From here on, `acompletion`, `aembedding` and the other async endpoint methods
all route through it. `get_available_deployment` serves `router.completion()`
and the other sync methods, so implement it too if you use them.
## Check it
`router.select(model)` now returns your pick without calling the provider, which
makes the strategy easy to unit test with mock deployments. [The
tutorial](../tutorial/route-your-first-requests.md) shows how to build a Router
with `mock_response` deployments.
## When a plugin is the better fit
If you only need to narrow the candidates, or to send some requests to a
different model group, you do not need to replace the pick. A deployment-level
filter fits as a `CustomLogger` that implements `async_filter_deployments` and
is added to `litellm.callbacks`. The router calls it in its callback filter
stage. Choosing a model group per request is what the sub-routers in
`litellm/router_strategy/auto_router/` and its siblings do in the pre-routing
hook. Both approaches keep the rest of the pipeline intact, including the
built-in strategy. See [How a deployment is picked](../explanation/selection.md)
and [Strategies, routing groups and sub-routers](../explanation/strategies.md)
for where each stage runs.

89
docs/architecture/router/how-to/change-the-router-at-runtime.md Normal file
View file

@ -0,0 +1,89 @@
# Change deployments or settings at runtime
Use this guide to add, replace or remove deployments, or to change router
settings, on a Router that is already serving traffic. All changes go through
`router.apply()` with a `ChangeSet`. Do not assign Router attributes or mutate
the lists that `settings()` and `registry()` return.
![Sequence in which apply(ChangeSet) runs assign, settings, upsert, delete and
redis_cache, bumps the registry version per deployment change, and returns
ApplyResult](../img/change-the-router-at-runtime-1.svg)
## Add or replace a deployment
`upsert` matches on `model_info.id`. A deployment with a new id is added, and
one with an existing id replaces the old one.
```python
from litellm.router_api import ChangeSet
from litellm.types.router import Deployment
result = router.apply(ChangeSet(upsert=[
Deployment(
model_name="gpt-4o",
litellm_params={"model": "azure/gpt-4o-westus", "api_base": "...", "api_key": "..."},
model_info={"id": "gpt-4o-westus"},
)
]))
previous = result.upserted[0] # the deployment it replaced, or None
```
## Remove a deployment
```python
result = router.apply(ChangeSet(delete=["gpt-4o-westus"]))
removed = result.deleted[0] # the removed deployment, or None if the id was unknown
```
## Change settings
Use `settings` for the validated runtime settings that `update_settings`
accepted, and `assign` to replace a field from `SETTING_FIELDS` as it is.
```python
router.apply(ChangeSet(settings={"num_retries": 3, "routing_strategy": "latency-based-routing"}))
router.apply(ChangeSet(assign={"cooldown_time": 30, "fallbacks": [{"gpt-4o": ["claude-sonnet"]}]}))
```
`assign` takes names from `SETTING_FIELDS`, the plain settings in
`PLAIN_SETTING_ATTRIBUTES` (`num_retries`, `max_fallbacks`, `retry_after`,
`timeout`) and `model_group_alias`, and rejects anything else with a
`ValueError`. The values are copied, so changing your own dict afterwards does
not reach the router. `settings` is quieter: it skips keys outside
`RUNTIME_UPDATABLE_ROUTER_SETTINGS` with only a debug log, so check the name if
a change seems to do nothing. `settings` also casts the numeric values, rebuilds
routing groups when the strategy changes, and switches the strategy's loggers.
`assign` does none of that.
## Changes that need I/O
Re-reading discovered model info is async, so it needs `aapply`.
```python
await router.aapply(ChangeSet(refresh_model_info=True))
```
## Several changes at once
One `ChangeSet` can carry all of these fields. They are applied in the order
`assign`, `settings`, `upsert`, `delete`, `redis_cache`, and
`ApplyResult.version` is the registry version after the last one. The fields are
not swapped in as one, so a request that runs meanwhile can see part of the
change set, and an error part way through leaves the earlier fields applied. An
unknown `assign` name is refused before anything changes.
## Old code you may find
| Old | New |
| --- | --- |
| `router.upsert_deployment(d)` | `router.apply(ChangeSet(upsert=[d]))` |
| `router.delete_deployment(id=i)` | `router.apply(ChangeSet(delete=[i]))` |
| `router.update_settings(**s)` | `router.apply(ChangeSet(settings=s))` |
| `router.cooldown_time = 30` | `router.apply(ChangeSet(assign={"cooldown_time": 30}))` |
| `router.update_redis_cache(cache=c)` | `router.apply(ChangeSet(redis_cache=c))` |
| `await router.arefresh_model_info()` | `await router.aapply(ChangeSet(refresh_model_info=True))` |
The old forms still work through `DeprecatedRouterMembers` and warn. A write to
an old setting name becomes an `apply(ChangeSet(assign=...))` and takes effect
like one. Router settings are meant to be fixed after construction except
through `apply`, so new code should call `apply` directly.

96
docs/architecture/router/how-to/debug-a-routing-decision.md Normal file
View file

@ -0,0 +1,96 @@
# Find out why a request went where it did
Use this guide when a request landed on an unexpected deployment, fell back when
it should not have, or failed with "No deployments available for selected
model". Work from what the response tells you toward what the router saw.
![Debugging unexpected routing: read the headers, check how the name resolved or
list the candidates the filters left, then check cooldowns, blocks, team and
order, or the strategy inputs](../img/debug-a-routing-decision-1.svg)
## 1. Read the response headers
Through the proxy, every response says what happened.
| Header | Tells you |
| --- | --- |
| `x-litellm-model-id` | the deployment that answered (`model_info.id`) |
| `x-litellm-model-group` | the model group that answered, which differs from the requested one after a fallback |
| `x-litellm-model-api-base` | the deployment's endpoint |
| `x-litellm-attempted-retries` | how many retries ran |
| `x-litellm-attempted-fallbacks` | how many fallback hops ran |
From the SDK, `response._hidden_params["model_id"]` holds the deployment id.
## 2. Check how the name resolved
```python
registry = router.registry()
registry.resolve("gpt-4", kind="alias") # alias target, or None
registry.resolve("gpt-4", kind="group") # the group the name routes to
registry.resolve("gpt-4", kind="recognized") # does the router know the name at all
registry.names("gpt-4", kind="route_candidates", team_id=team_id) # deployment ids it may reach
```
A name that is not a model group can still route through an alias, a routing
group, a team public name, a wildcard such as `openai/*`, or
`default_deployment`. [How a name finds its
deployments](../explanation/names.md) has the order.
## 3. List the candidates the filters left
`select(kind="healthy")` runs the whole filter chain for a request without
calling anything. Pass the same `request_kwargs` the real request had, since the
team filter, the tag filter and the per-request settings read them.
```python
candidates = await router.select(
"gpt-4o",
{"metadata": {"user_api_key_team_id": team_id, "tags": ["prod"]}},
kind="healthy",
messages=messages,
)
print([d["model_info"]["id"] for d in candidates])
picked = await router.select("gpt-4o", {"metadata": {...}}, messages=messages)
```
The router does not log its filter stages. Its filters run in the order that
[How a deployment is picked](../explanation/selection.md) shows, so compare the
`kind="healthy"` result with the group's deployments and walk through the checks
below.
## 4. Check what removes a deployment
```python
health = router.health()
ids = list(registry.names("gpt-4o", kind="ids"))
await health.cooldowns.async_get_active_cooldowns(model_ids=ids, parent_otel_span=None)
health.is_blocked(registry.deployment("gpt-4o-eastus"))
```
![Why a deployment is missing from the candidates: a cooldown entry, then
model_info.blocked, another team_id, a lower order in the group, and finally the
pre-call checks](../img/debug-a-routing-decision-2.svg)
A cooldown entry records the failure that caused it. The rules for when a
failure counts are in [Retries, fallbacks and
cooldowns](../explanation/reliability.md). Deployments with a higher `order` get
traffic only once every lower-order deployment of the group has failed for the
request.
The error "No deployments available for selected model" is a
`RouterRateLimitError`. Its message includes the cooldown list and the seconds
until the soonest cooldown ends, and it says whether pre-call checks were on,
since they are a common cause.
## 5. Check the strategy's inputs
If the right deployments made it through and the pick still looks wrong, read
what the strategy reads. `router.settings().routing_strategy` says which
strategy ran, unless a routing group or a per-request `routing_strategy`
overrode it. For `simple-shuffle`, check `weight`, `rpm` and `tpm` on the
deployments and any per-team `weights` in key or team `router_settings`. For
usage-based strategies, `await router.health().usage("gpt-4o")` shows this
minute's counters, and remember that workers see each other's counters about a
second late (see [Shared state](../explanation/shared-state.md)).

80
docs/architecture/router/how-to/move-off-deprecated-members.md Normal file
View file

@ -0,0 +1,80 @@
# Move a caller off a deprecated Router member
Use this guide when you see
`DeprecationWarning: Router.<name> is deprecated; use Router.<replacement>`, or
when you want to remove an entry from the surface allowlist. The old members
still work, through `DeprecatedRouterMembers` in `litellm/router_deprecated.py`,
and will be removed once callers have moved.
![Moving off a deprecated Router member: the warning names the replacement, you
rewrite the call, run the tests, then router_surface_check.py reports clean or
an unused entry to remove with
--write](../img/move-off-deprecated-members-1.svg)
## 1. Find the replacement
The warning text names it. Without a warning at hand, search
`litellm/router_deprecated.py` for the old name: every member there calls
`_note(name, replacement)`. [Entry points](../reference/entry-points.md) has the
same mapping in table form.
To see which old names are still used at runtime, run the code path and inspect
the counter.
```python
from litellm.router_deprecated import DEPRECATED_ROUTER_MEMBER_USES
print(DEPRECATED_ROUTER_MEMBER_USES.most_common())
```
## 2. Rewrite the call
Most rewrites are mechanical.
| Before | After |
| --- | --- |
| `llm_router.model_list` | `llm_router.registry().all()` |
| `llm_router.get_model_list(model_name=m)` | `llm_router.registry().deployments(m)` |
| `llm_router.get_deployment(model_id=i)` | `llm_router.registry().deployment(i)` |
| `llm_router.get_model_names()` | `llm_router.registry().names()` |
| `llm_router.model_names` | `llm_router.registry().names(kind="group_set")` |
| `llm_router.get_model_group_info(m)` | `llm_router.registry().group(m)` |
| `llm_router.routing_strategy` | `llm_router.settings().routing_strategy` |
| `llm_router.get_settings()` | `llm_router.settings().as_dict()` |
| `llm_router.cache` | `llm_router.health().store` |
| `llm_router.cooldown_cache` | `llm_router.health().cooldowns` |
| `await llm_router.async_get_available_deployment(model=m, request_kwargs=k)` | `await llm_router.select(m, k)` |
| `llm_router.upsert_deployment(deployment=d)` | `llm_router.apply(ChangeSet(upsert=[d]))` |
| `llm_router.get_routing_group(name)` | `llm_router.registry().routing_group(name)` |
| `llm_router.discard()` | `llm_router.retire()` |
Return types are the same as before, so type checks keep passing. Where a query
is overloaded on `kind`, `source` or `field`, pass the literal directly so that
pyright chooses the right overload.
## 3. Check the surface
```bash
uv run python tests/code_coverage_tests/router_surface_check.py
```
The check fails in two cases. The first is a new member or `router_utils` import
used from outside the router, which you fix by using an entry point. The second
is a listed entry that nothing outside the router uses anymore, which means your
change removed its last use. In that case, shrink the allowlist in the same PR.
```bash
uv run python tests/code_coverage_tests/router_surface_check.py --write
git diff tests/code_coverage_tests/router_surface_allowlist.json
```
The diff should only remove lines. If `--write` adds an entry, a new outside use
slipped in, and the right fix is in the caller, not the allowlist.
## Inside the router family
Modules under `litellm/router_utils/`, `litellm/router_strategy/` and
`litellm/router_rust/` are the router's own. They do not trigger the warning,
and the surface check lists their uses separately (`members_router_family`).
Routing plugins should take a `RoutingContext` and read through it, so that they
stop depending on Router members at all.

72
docs/architecture/router/how-to/read-router-state.md Normal file
View file

@ -0,0 +1,72 @@
# Read router state from outside the router
Use this guide when code outside the router needs to know something about it,
such as which deployments a model group has, what the fallbacks are, or whether
a deployment is cooled down. Read through the entry points, never through Router
attributes or `_core_*` members. The surface check in CI rejects new uses of
anything else.
## Pick the entry point
![Which view answers which question: router.registry(), router.settings(),
router.health() or router.select(), and otherwise a new query on an existing
view](../img/read-router-state-1.svg)
## Get the router
In the proxy, the Router is `litellm.proxy.proxy_server.llm_router`, which can
be None before the config loads. Import it inside the function that uses it, as
the existing proxy modules do.
```python
from litellm.proxy.proxy_server import llm_router
if llm_router is None:
raise HTTPException(status_code=500, detail="router not initialized")
```
## Common reads
```python
registry = llm_router.registry()
groups = registry.names(kind="group_set") # configured model group names
deployments = registry.deployments("gpt-4o", team_id=team_id) # what get_model_list returned
deployment = registry.deployment(model_id) # Deployment or None
group_name = registry.resolve("gpt-4", kind="alias") # alias target or None
info = registry.group("gpt-4o", field="info") # merged ModelGroupInfo
routing_group = registry.routing_group("smart-tier") # RoutingGroup or None
team_names = registry.names(kind="team_public", team_id=team_id)
settings = llm_router.settings()
strategy = settings.routing_strategy
fallbacks = settings.fallbacks
health = llm_router.health()
paused = health.is_blocked(deployment)
tpm, rpm = await health.usage("gpt-4o")
cooling = await health.cooldowns.async_get_active_cooldowns(model_ids=ids, parent_otel_span=None)
failures = health.failures_this_minute(deployment.model_info.id)
```
Each query's docstring in `litellm/router_api.py` names the old member it
replaces, and [Entry points](../reference/entry-points.md) lists them all in one
table.
## Things to watch
Treat what you get back as read-only. Many of the results are views over the
router's state, and changing one does not change the router. Use `apply()`
instead, see [Change deployments or settings at
runtime](change-the-router-at-runtime.md).
`settings()` is a snapshot. Read it again after an `apply()` instead of holding
one across requests. To detect deployment changes, compare `registry().version`
with the value you saw last.
Inside a routing plugin you get a `RoutingContext`, not the Router. It has the
same `registry()`, `settings()` and `health()`.
Module-level helpers in `litellm/router_api.py`, such as `fallback_chain`,
`deployment_id` and `router_init_args`, answer questions that need no Router at
all.

65
docs/architecture/router/img/add-a-routing-strategy-1.svg Normal file
View file

@ -0,0 +1,65 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 228" role="img" aria-labelledby="t">
<title id="t">Built-in selection versus a registered strategy</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="759" height="227" rx="8"/>
<rect class="group" x="16" y="14" width="728" height="94" rx="8"/>
<text class="group-label" x="30" y="32">BUILT-IN</text>
<rect class="box host" x="40" y="44" width="170" height="52" rx="6"/>
<text class="title center" x="125" y="74">Pre-routing hook</text>
<rect class="box core" x="290" y="44" width="200" height="52" rx="6"/>
<text class="title center" x="390" y="65">Filters</text>
<text class="small muted center" x="390" y="83">team, cooldown, blocked, ...</text>
<rect class="box core" x="570" y="44" width="150" height="52" rx="6"/>
<text class="title center" x="645" y="74">Strategy</text>
<path class="edge" d="M210,70 H288" marker-end="url(#arrow)"/>
<path class="edge" d="M490,70 H568" marker-end="url(#arrow)"/>
<rect class="group" x="16" y="120" width="728" height="94" rx="8"/>
<text class="group-label" x="30" y="138">AFTER <tspan class="mono">register_plugin</tspan></text>
<rect class="box host" x="40" y="150" width="290" height="52" rx="6"/>
<text class="center" x="185" y="180">Your <tspan class="mono">async_get_available_deployment</tspan></text>
<rect class="box" x="480" y="150" width="240" height="52" rx="6"/>
<text class="mono center" x="600" y="181">router.select(kind='healthy')</text>
<path class="edge dashed" d="M330,176 H478" marker-end="url(#arrow)"/>
<text class="edge-label center" x="404" y="168">call it to keep the filters</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.6 KiB

67
docs/architecture/router/img/change-the-router-at-runtime-1.svg Normal file
View file

@ -0,0 +1,67 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 790 300" role="img" aria-labelledby="t">
<title id="t">Applying a ChangeSet to a running Router</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="789" height="299" rx="8"/>
<line class="lifeline" x1="90" y1="56" x2="90" y2="284"/>
<line class="lifeline" x1="400" y1="56" x2="400" y2="284"/>
<line class="lifeline" x1="700" y1="56" x2="700" y2="284"/>
<rect class="box" x="30" y="16" width="120" height="40" rx="6"/>
<text class="title center" x="90" y="40.5">Your code</text>
<rect class="box core" x="340" y="16" width="120" height="40" rx="6"/>
<text class="title center" x="400" y="40.5">Router</text>
<rect class="box store" x="640" y="16" width="120" height="40" rx="6"/>
<text class="title center" x="700" y="40.5">Registry</text>
<path class="edge" d="M90,92 H398" marker-end="url(#arrow)"/>
<text class="edge-label mono small center" x="245" y="84">apply(ChangeSet(...))</text>
<path class="edge" d="M400,114 H436 V146 H402" marker-end="url(#arrow)"/>
<text class="edge-label" x="448" y="126"><tspan class="mono small">assign</tspan>, then <tspan class="mono small">settings</tspan>, then <tspan class="mono small">upsert</tspan>,</text>
<text class="edge-label" x="448" y="142">then <tspan class="mono small">delete</tspan>, then <tspan class="mono small">redis_cache</tspan></text>
<path class="edge" d="M400,182 H698" marker-end="url(#arrow)"/>
<text class="edge-label center" x="550" y="174"><tspan class="mono small">version += 1</tspan> per deployment change</text>
<path class="edge dashed" d="M400,220 H92" marker-end="url(#arrow)"/>
<text class="edge-label mono small center" x="245" y="212">ApplyResult(version, upserted, deleted)</text>
<rect class="note" x="250" y="236" width="300" height="46" rx="6"/>
<text class="small center" x="400" y="255">The cached <tspan class="mono small">settings()</tspan> snapshot is dropped,</text>
<text class="small center" x="400" y="271">the next read builds a new one</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 5 KiB

115
docs/architecture/router/img/code-map-1.svg Normal file
View file

@ -0,0 +1,115 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 430" role="img" aria-labelledby="t">
<title id="t">Router code map</title>
<desc>Python files of the router and the Rust crates they call, with which file uses which.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="839" height="429" rx="8"/>
<rect class="group" x="16" y="16" width="808" height="302" rx="8"/>
<text class="group-label" x="32" y="34">PYTHON</text>
<rect class="group core" x="16" y="330" width="808" height="84" rx="8"/>
<text class="group-label" x="32" y="376">RUST</text>
<!-- row 1 -->
<rect class="box host" x="36" y="48" width="244" height="84" rx="6"/>
<text class="mono title" x="48" y="67">litellm/router_api.py</text>
<text class="mono small" x="48" y="85">RouterRegistry, RouterSettings,</text>
<text class="mono small" x="48" y="101">RouterHealth, ChangeSet,</text>
<text class="mono small" x="48" y="117">RoutingContext</text>
<rect class="box host" x="345" y="64" width="142" height="52" rx="6"/>
<text class="mono title center" x="416" y="85">litellm/router.py</text>
<text class="small center" x="416" y="102"><tspan class="mono small">Router</tspan> facade</text>
<rect class="box host" x="556" y="64" width="248" height="52" rx="6"/>
<text class="mono title center" x="680" y="85">litellm/router_config.py</text>
<text class="mono small center" x="680" y="102">RouterConfig, RouterDynamicConfig</text>
<!-- row 2 -->
<rect class="box host" x="326" y="158" width="180" height="52" rx="6"/>
<text class="mono title center" x="416" y="179">litellm/router_core.py</text>
<text class="mono small center" x="416" y="196">RouterCore, make_core</text>
<rect class="box host" x="592" y="158" width="212" height="52" rx="6"/>
<text class="mono title center" x="698" y="179">litellm/router_rust/core.py</text>
<text class="small center" x="698" y="196"><tspan class="mono small">RustCore</tspan> step driver</text>
<!-- step drivers and their targets -->
<rect class="box host" x="366" y="257" width="170" height="52" rx="6"/>
<text class="mono title center" x="451" y="278">litellm/router_rust/</text>
<text class="mono small center" x="451" y="295">steps, invoke, surface</text>
<rect class="box host" x="36" y="148" width="244" height="46" rx="6"/>
<text class="mono title" x="48" y="167">litellm/router_attempt.py</text>
<text class="small" x="48" y="183">one attempt's kwargs</text>
<rect class="box host" x="36" y="204" width="244" height="46" rx="6"/>
<text class="mono title" x="48" y="223">litellm/router_python_host.py</text>
<text class="mono small" x="48" y="239">PythonHost</text>
<rect class="box host" x="36" y="260" width="244" height="46" rx="6"/>
<text class="mono title" x="48" y="279">litellm/router_strategy/</text>
<text class="small" x="48" y="295">strategies and sub-routers</text>
<!-- rust -->
<rect class="box core" x="160" y="346" width="244" height="52" rx="6"/>
<text class="mono title center" x="282" y="367">litellm-rust/crates/router-core</text>
<text class="small center" x="282" y="384">the routing core</text>
<rect class="box core" x="468" y="346" width="336" height="52" rx="6"/>
<text class="mono title center" x="636" y="367">litellm-rust/crates/python-bridge/src/router</text>
<text class="mono small center" x="636" y="384">RouterCoreHandle, RouterCall</text>
<!-- edges -->
<path class="edge" d="M345,90 H282" marker-end="url(#arrow)"/>
<path class="edge" d="M487,90 H554" marker-end="url(#arrow)"/>
<path class="edge" d="M416,116 V156" marker-end="url(#arrow)"/>
<path class="edge" d="M506,184 H590" marker-end="url(#arrow)"/>
<path class="edge" d="M640,210 V283 H538" marker-end="url(#arrow)"/>
<path class="edge" d="M760,210 V344" marker-end="url(#arrow)"/>
<path class="edge core" d="M468,372 H406" marker-end="url(#arrow-core)"/>
<path class="edge" d="M396,257 V227 H282" marker-end="url(#arrow)"/>
<path class="edge" d="M306,227 V171 H282" marker-end="url(#arrow)"/>
<path class="edge dashed" d="M366,283 H282" marker-end="url(#arrow)"/>
<text class="edge-label center" x="323" y="277">select steps</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 7 KiB

105
docs/architecture/router/img/concepts-1.svg Normal file
View file

@ -0,0 +1,105 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 876 400" role="img" aria-labelledby="t">
<title id="t">Router concepts and how they fit together</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="875" height="399" rx="8"/>
<rect class="group" x="20" y="14" width="744" height="106" rx="8"/>
<text class="group-label" x="32" y="32">WHAT IS CONFIGURED</text>
<rect class="box" x="32" y="42" width="210" height="62" rx="6"/>
<text class="title center" x="137" y="66">Deployments</text>
<text class="small muted center" x="137" y="84">one place that can serve a model</text>
<rect class="box" x="272" y="42" width="230" height="62" rx="6"/>
<text class="title center" x="387" y="66">Model groups</text>
<text class="small muted center" x="387" y="84">deployments sharing a <tspan class="mono">model_name</tspan></text>
<rect class="box" x="532" y="42" width="220" height="62" rx="6"/>
<text class="title center" x="642" y="62">Names</text>
<text class="small muted center" x="642" y="79">aliases, wildcards, routing groups,</text>
<text class="small muted center" x="642" y="95">team names, deployment ids</text>
<path class="edge" d="M242,73 H270" marker-end="url(#arrow)"/>
<path class="edge" d="M502,73 H530" marker-end="url(#arrow)"/>
<rect class="box" x="20" y="170" width="140" height="80" rx="6"/>
<text class="title center" x="90" y="203">Request</text>
<text class="mono small muted center" x="90" y="221">model='gpt-4o',</text>
<text class="small muted center" x="90" y="237">team, tags, messages</text>
<rect class="box core" x="186" y="170" width="136" height="80" rx="6"/>
<text class="title center" x="254" y="215">Resolve the name</text>
<rect class="box core" x="348" y="170" width="150" height="80" rx="6"/>
<text class="title center" x="423" y="203">Who may use what</text>
<text class="small muted center" x="423" y="221">teams, access groups,</text>
<text class="small muted center" x="423" y="237">windows, PTU shares</text>
<rect class="box core" x="524" y="170" width="165" height="80" rx="6"/>
<text class="title center" x="606.5" y="191">Filters</text>
<text class="small muted center" x="606.5" y="208">health, cooldown, blocked,</text>
<text class="small muted center" x="606.5" y="224">pre-call checks,</text>
<text class="small muted center" x="606.5" y="240">tags, order</text>
<rect class="box core" x="715" y="170" width="140" height="80" rx="6"/>
<text class="title center" x="785" y="206">Strategy</text>
<text class="small muted center" x="785" y="224">picks one deployment</text>
<path class="edge" d="M160,210 H184" marker-end="url(#arrow)"/>
<path class="edge core" d="M322,210 H346" marker-end="url(#arrow-core)"/>
<path class="edge core" d="M498,210 H522" marker-end="url(#arrow-core)"/>
<path class="edge core" d="M689,210 H713" marker-end="url(#arrow-core)"/>
<path class="edge dashed" d="M600,104 V140 H254 V168" marker-end="url(#arrow)"/>
<rect class="box store" x="300" y="310" width="150" height="56" rx="6"/>
<text class="title center" x="375" y="343">Shared routing state</text>
<rect class="box core" x="524" y="310" width="165" height="56" rx="6"/>
<text class="title center" x="606.5" y="333">Retries, fallbacks,</text>
<text class="title center" x="606.5" y="350">cooldowns</text>
<rect class="box host" x="735" y="310" width="100" height="56" rx="6"/>
<text class="title center" x="785" y="343">Attempt</text>
<path class="edge" d="M785,250 V308" marker-end="url(#arrow)"/>
<path class="edge warn" d="M735,338 H691" marker-end="url(#arrow-warn)"/>
<text class="edge-label center" x="713" y="330">failed</text>
<path class="edge dashed" d="M524,338 H452" marker-end="url(#arrow)"/>
<text class="edge-label center" x="488" y="330">writes</text>
<path class="edge dashed" d="M430,310 V280 H570 V252" marker-end="url(#arrow)"/>
<text class="edge-label center" x="500" y="274">read by</text>
<path class="edge core" d="M570,366 V384 H254 V252" marker-end="url(#arrow-core)"/>
<text class="edge-label end" x="248" y="320">try again</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 7.2 KiB

95
docs/architecture/router/img/configuration-1.svg Normal file
View file

@ -0,0 +1,95 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 809 388" role="img" aria-labelledby="t">
<title id="t">Router config arguments by area</title>
<desc>Router config splits into five areas: deployments, selection, reliability, state and health, each with its constructor arguments.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="808" height="387" rx="8"/>
<rect class="box core" x="20" y="170.0" width="104" height="48" rx="24"/>
<text class="title center" x="72.0" y="198.5">Router config</text>
<path class="edge" d="M124,194.0 C154,194.0 152,54.0 182,54.0" marker-end="url(#arrow)"/>
<path class="edge" d="M124,194.0 C154,194.0 152,140.0 182,140.0" marker-end="url(#arrow)"/>
<path class="edge" d="M124,194.0 C154,194.0 152,226.0 182,226.0" marker-end="url(#arrow)"/>
<path class="edge" d="M124,194.0 C154,194.0 152,296.0 182,296.0" marker-end="url(#arrow)"/>
<path class="edge" d="M124,194.0 C154,194.0 152,350.0 182,350.0" marker-end="url(#arrow)"/>
<rect class="box" x="184" y="16" width="609" height="76" rx="6"/>
<text class="title" x="198" y="58.5">Deployments</text>
<rect class="box" x="292" y="25" width="90" height="26" rx="4"/>
<text class="mono small center" x="337.0" y="42">model_list</text>
<rect class="box" x="390" y="25" width="139" height="26" rx="4"/>
<text class="mono small center" x="459.5" y="42">model_group_alias</text>
<rect class="box" x="537" y="25" width="118" height="26" rx="4"/>
<text class="mono small center" x="596.0" y="42">routing_groups</text>
<rect class="box" x="292" y="57" width="216" height="26" rx="4"/>
<text class="mono small center" x="400.0" y="74">search_tools, guardrail_list</text>
<rect class="box" x="184" y="102" width="609" height="76" rx="6"/>
<text class="title" x="198" y="144.5">Selection</text>
<rect class="box" x="292" y="111" width="132" height="26" rx="4"/>
<text class="mono small center" x="358.0" y="128">routing_strategy</text>
<rect class="box" x="432" y="111" width="167" height="26" rx="4"/>
<text class="mono small center" x="515.5" y="128">routing_strategy_args</text>
<rect class="box" x="607" y="111" width="69" height="26" rx="4"/>
<text class="mono small center" x="641.5" y="128">plugins</text>
<rect class="box" x="292" y="143" width="174" height="26" rx="4"/>
<text class="mono small center" x="379.0" y="160">enable_pre_call_checks</text>
<rect class="box" x="474" y="143" width="160" height="26" rx="4"/>
<text class="mono small center" x="554.0" y="160">enable_tag_filtering</text>
<rect class="box" x="184" y="188" width="609" height="76" rx="6"/>
<text class="title" x="198" y="230.5">Reliability</text>
<rect class="box" x="292" y="197" width="195" height="26" rx="4"/>
<text class="mono small center" x="389.5" y="214">num_retries, retry_policy</text>
<rect class="box" x="495" y="197" width="188" height="26" rx="4"/>
<text class="mono small center" x="589.0" y="214">fallbacks, max_fallbacks</text>
<rect class="box" x="292" y="229" width="216" height="26" rx="4"/>
<text class="mono small center" x="400.0" y="246">cooldown_time, allowed_fails</text>
<rect class="box store" x="184" y="274" width="609" height="44" rx="6"/>
<text class="title" x="198" y="300.5">State</text>
<rect class="box" x="292" y="283" width="181" height="26" rx="4"/>
<text class="mono small center" x="382.5" y="300">redis_url / host / port</text>
<rect class="box" x="481" y="283" width="125" height="26" rx="4"/>
<text class="mono small center" x="543.5" y="300">cache_responses</text>
<rect class="box" x="184" y="328" width="609" height="44" rx="6"/>
<text class="title" x="198" y="354.5">Health</text>
<rect class="box" x="292" y="337" width="209" height="26" rx="4"/>
<text class="mono small center" x="396.5" y="354">enable_health_check_routing</text>
<rect class="box" x="509" y="337" width="272" height="26" rx="4"/>
<text class="mono small center" x="645.0" y="354">background_health_check_model_groups</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 6.7 KiB

100
docs/architecture/router/img/core-and-host-1.svg Normal file
View file

@ -0,0 +1,100 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 880 424" role="img" aria-labelledby="t">
<title id="t">Router core and host interfaces</title>
<desc>Router uses RouterCore, picked once by make_core. RustCore implements RouterCore and holds a RouterCoreHandle to the Rust RustRouter, which calls the Host trait. PythonHost implements Host through the step driver.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="879" height="423" rx="8"/>
<rect class="group" x="10" y="10" width="860" height="192" rx="8"/>
<text class="group-label" x="22" y="28">PYTHON</text>
<rect class="group core" x="295" y="220" width="575" height="192" rx="8"/>
<text class="group-label" x="307" y="238">RUST</text>
<rect class="box host" x="20" y="40" width="202" height="100" rx="6"/>
<text class="mono title center" x="121.0" y="62">Router</text>
<text class="small muted center" x="121.0" y="78">«Python facade»</text>
<text class="mono small" x="32" y="99">+from_config(config, host)</text>
<text class="mono small" x="32" y="115">+endpoint methods</text>
<text class="mono small" x="32" y="131">+8 entry points</text>
<rect class="box host" x="315" y="40" width="240" height="152" rx="6"/>
<text class="mono title center" x="435.0" y="62">RouterCore</text>
<text class="small muted center" x="435.0" y="78">«Protocol,</text>
<text class="small muted center" x="435.0" y="93"><tspan class="mono">litellm/router_core.py</tspan>»</text>
<text class="mono small" x="327" y="114">+registry() settings() health()</text>
<text class="mono small" x="327" y="130">+select(model, request_kwargs)</text>
<text class="mono small" x="327" y="146">+apply(changes)</text>
<text class="mono small" x="327" y="162">+observe() register_plugin()</text>
<text class="mono small" x="327" y="178">+retire()</text>
<rect class="box host" x="630" y="40" width="228" height="74" rx="6"/>
<text class="mono title center" x="744.0" y="62">RustCore</text>
<text class="small muted center" x="744.0" y="78">«<tspan class="mono">litellm/router_rust/core.py</tspan>»</text>
<text class="small" x="642" y="99">drives <tspan class="mono">RouterCall</tspan> steps</text>
<rect class="box core" x="600" y="248" width="258" height="152" rx="6"/>
<text class="mono title center" x="729.0" y="270">RustRouter</text>
<text class="small muted center" x="729.0" y="286">«<tspan class="mono">router-core</tspan> Router»</text>
<text class="mono small" x="612" y="307">+call(op, request)</text>
<text class="mono small" x="612" y="323">+select(op, request) healthy()</text>
<text class="mono small" x="612" y="339">+registry() settings() health()</text>
<text class="mono small" x="612" y="355">+report_success() report_failure()</text>
<text class="mono small" x="612" y="371">+apply(ChangeSet) RegistryVersion</text>
<text class="mono small" x="612" y="387">+retire()</text>
<rect class="box core" x="310" y="248" width="245" height="136" rx="6"/>
<text class="mono title center" x="432.5" y="270">Host</text>
<text class="small muted center" x="432.5" y="286">«trait, <tspan class="mono">host.rs</tspan>»</text>
<text class="mono small" x="322" y="307">+invoke(attempt)</text>
<text class="mono small" x="322" y="323">+select(stage, ctx, candidates)</text>
<text class="mono small" x="322" y="339">+pre_route(ask)</text>
<text class="mono small" x="322" y="355">+allow_fallback(request, target)</text>
<text class="mono small" x="322" y="371">+sleep() observe() cache()</text>
<rect class="box host" x="20" y="290" width="120" height="40" rx="6"/>
<text class="mono title center" x="80" y="314">PythonHost</text>
<path class="edge" d="M222,90 H313" marker-end="url(#arrow)"/>
<text class="edge-label center" x="268" y="67">picked once by</text>
<text class="edge-label center" x="268" y="82"><tspan class="mono">make_core</tspan></text>
<path class="edge dashed" d="M630,77 H557" marker-end="url(#arrow)"/>
<text class="edge-label center" x="592" y="69">implements</text>
<path class="edge" d="M744,114 V246" marker-end="url(#arrow)"/>
<text class="edge-label mono end" x="734" y="184">RouterCoreHandle</text>
<path class="edge core" d="M600,310 H557" marker-end="url(#arrow-core)"/>
<path class="edge dashed" d="M140,310 H308" marker-end="url(#arrow)"/>
<text class="edge-label center" x="217" y="286">implements, through</text>
<text class="edge-label center" x="217" y="301">the step driver</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 7.3 KiB

76
docs/architecture/router/img/core-and-hosts-1.svg Normal file
View file

@ -0,0 +1,76 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 276" role="img" aria-labelledby="t">
<title id="t">The facade, the Rust core and the host</title>
<desc>litellm.Router calls RustCore, which drives litellm-router-core over PyO3; the core asks the Python host for steps, and the host calls the provider through litellm.&lt;op&gt;.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="799" height="275" rx="8"/>
<rect class="box" x="20" y="20" width="240" height="72" rx="6"/>
<text class="mono title center" x="140" y="52">litellm.Router</text>
<text class="small muted center" x="140" y="71">endpoint methods + 8 entry points</text>
<rect class="box host" x="310" y="20" width="220" height="72" rx="6"/>
<text class="title center" x="420" y="43">RustCore</text>
<text class="mono small center" x="420" y="62">litellm/router_rust/core.py</text>
<text class="small muted center" x="420" y="80">the step driver</text>
<path class="edge" d="M260,56 H308" marker-end="url(#arrow)"/>
<rect class="group core" x="20" y="124" width="510" height="132" rx="8"/>
<text class="group-label" x="36" y="143">LITELLM-ROUTER-CORE (RUST)</text>
<rect class="box core" x="36" y="154" width="233" height="40" rx="6"/>
<text class="center" x="152.5" y="178">registry, build</text>
<rect class="box core" x="281" y="154" width="233" height="40" rx="6"/>
<text class="center" x="397.5" y="178">resolve, filters, simple-shuffle</text>
<rect class="box core" x="36" y="204" width="233" height="40" rx="6"/>
<text class="center" x="152.5" y="228">call loop: retries, fallbacks</text>
<rect class="box core" x="281" y="204" width="233" height="40" rx="6"/>
<text class="center" x="397.5" y="228">health: cooldowns, usage</text>
<path class="edge" d="M420,92 V122" marker-end="url(#arrow)"/>
<text class="edge-label mono end" x="410" y="112">PyO3: RouterCoreHandle, RouterCall</text>
<rect class="box host" x="580" y="154" width="200" height="72" rx="6"/>
<text class="title center" x="680" y="177">PythonHost and</text>
<text class="mono small center" x="680" y="196">litellm/router_rust/</text>
<text class="mono small center" x="680" y="214">steps/, invoke/, surface/</text>
<path class="edge" d="M530,190 H578" marker-end="url(#arrow)"/>
<text class="edge-label center" x="554" y="182">steps</text>
<rect class="box" x="620" y="20" width="120" height="72" rx="6"/>
<text class="title center" x="680" y="60">provider</text>
<path class="edge" d="M680,154 V94" marker-end="url(#arrow)"/>
<text class="edge-label mono" x="690" y="128">litellm.&lt;op&gt;</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 5.4 KiB

111
docs/architecture/router/img/core-and-hosts-2.svg Normal file
View file

@ -0,0 +1,111 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 522" role="img" aria-labelledby="t">
<title id="t">Driving the core: one call with a retry on 429</title>
<desc>The Router facade calls the RustCore driver, which loops on RouterCall.next and answers each step through PythonHost: read cooldowns, attempt deployment a, get a 429, set its cooldown, attempt deployment b, return the response.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="799" height="521" rx="8"/>
<line class="lifeline" x1="80" y1="52" x2="80" y2="512"/>
<rect class="box" x="10" y="16" width="140" height="36" rx="6"/>
<text class="title center" x="80" y="38.5">Router facade</text>
<line class="lifeline" x1="240" y1="52" x2="240" y2="512"/>
<rect class="box host" x="170" y="16" width="140" height="36" rx="6"/>
<text class="title center" x="240" y="38.5">RustCore driver</text>
<line class="lifeline" x1="520" y1="52" x2="520" y2="512"/>
<rect class="box core" x="445" y="16" width="150" height="36" rx="6"/>
<text class="title center" x="520" y="38.5">RouterCall (Rust)</text>
<line class="lifeline" x1="710" y1="52" x2="710" y2="512"/>
<rect class="box host" x="640" y="16" width="140" height="36" rx="6"/>
<text class="title center" x="710" y="38.5">PythonHost</text>
<path class="edge" d="M80,88 H239" marker-end="url(#arrow)"/>
<circle class="step" cx="92" cy="77" r="8"/><text class="step-text" x="92" y="80.5">1</text>
<text class="edge-label" x="104" y="81"><tspan class="mono small">acompletion(model, messages)</tspan></text>
<path class="edge core" d="M240,114 H519" marker-end="url(#arrow-core)"/>
<circle class="step" cx="252" cy="103" r="8"/><text class="step-text" x="252" y="106.5">2</text>
<text class="edge-label" x="264" y="107"><tspan class="mono small">next(None)</tspan></text>
<path class="edge core dashed" d="M520,140 H241" marker-end="url(#arrow-core)"/>
<circle class="step" cx="508" cy="129" r="8"/><text class="step-text" x="508" y="132.5">3</text>
<text class="edge-label end" x="496" y="133">step cache: read cooldowns</text>
<path class="edge" d="M240,166 H709" marker-end="url(#arrow)"/>
<circle class="step" cx="252" cy="155" r="8"/><text class="step-text" x="252" y="158.5">4</text>
<text class="edge-label" x="264" y="159"><tspan class="mono small">DualCache</tspan> batch get</text>
<path class="edge core" d="M240,192 H519" marker-end="url(#arrow-core)"/>
<circle class="step" cx="252" cy="181" r="8"/><text class="step-text" x="252" y="184.5">5</text>
<text class="edge-label" x="264" y="185"><tspan class="mono small">next(values)</tspan></text>
<path class="edge core dashed" d="M520,218 H241" marker-end="url(#arrow-core)"/>
<circle class="step" cx="508" cy="207" r="8"/><text class="step-text" x="508" y="210.5">6</text>
<text class="edge-label end" x="496" y="211">step attempt on deployment a</text>
<path class="edge" d="M240,244 H709" marker-end="url(#arrow)"/>
<circle class="step" cx="252" cy="233" r="8"/><text class="step-text" x="252" y="236.5">7</text>
<text class="edge-label" x="264" y="237"><tspan class="mono small">router_attempt</tspan> + <tspan class="mono small">litellm.acompletion</tspan></text>
<path class="edge warn dashed" d="M710,270 H241" marker-end="url(#arrow-warn)"/>
<circle class="step" cx="698" cy="259" r="8"/><text class="step-text" x="698" y="262.5">8</text>
<text class="edge-label end" x="686" y="263">429</text>
<path class="edge core" d="M240,296 H519" marker-end="url(#arrow-core)"/>
<circle class="step" cx="252" cy="285" r="8"/><text class="step-text" x="252" y="288.5">9</text>
<text class="edge-label" x="264" y="289"><tspan class="mono small">next(failure)</tspan></text>
<path class="edge core dashed" d="M520,322 H241" marker-end="url(#arrow-core)"/>
<circle class="step" cx="508" cy="311" r="8"/><text class="step-text" x="508" y="314.5">10</text>
<text class="edge-label end" x="496" y="315">step cache: set <tspan class="mono small">deployment:a:cooldown</tspan></text>
<path class="edge core" d="M240,348 H519" marker-end="url(#arrow-core)"/>
<circle class="step" cx="252" cy="337" r="8"/><text class="step-text" x="252" y="340.5">11</text>
<text class="edge-label" x="264" y="341"><tspan class="mono small">next(ok)</tspan></text>
<path class="edge core dashed" d="M520,374 H241" marker-end="url(#arrow-core)"/>
<circle class="step" cx="508" cy="363" r="8"/><text class="step-text" x="508" y="366.5">12</text>
<text class="edge-label end" x="496" y="367">step attempt on deployment b</text>
<path class="edge" d="M240,400 H709" marker-end="url(#arrow)"/>
<circle class="step" cx="252" cy="389" r="8"/><text class="step-text" x="252" y="392.5">13</text>
<text class="edge-label" x="264" y="393"><tspan class="mono small">router_attempt</tspan> + <tspan class="mono small">litellm.acompletion</tspan></text>
<path class="edge dashed" d="M710,426 H241" marker-end="url(#arrow)"/>
<circle class="step" cx="698" cy="415" r="8"/><text class="step-text" x="698" y="418.5">14</text>
<text class="edge-label end" x="686" y="419">response</text>
<path class="edge core" d="M240,452 H519" marker-end="url(#arrow-core)"/>
<circle class="step" cx="252" cy="441" r="8"/><text class="step-text" x="252" y="444.5">15</text>
<text class="edge-label" x="264" y="445"><tspan class="mono small">next(response)</tspan></text>
<path class="edge core dashed" d="M520,478 H241" marker-end="url(#arrow-core)"/>
<circle class="step" cx="508" cy="467" r="8"/><text class="step-text" x="508" y="470.5">16</text>
<text class="edge-label end" x="496" y="471">step done</text>
<path class="edge dashed" d="M240,504 H81" marker-end="url(#arrow)"/>
<circle class="step" cx="228" cy="493" r="8"/><text class="step-text" x="228" y="496.5">17</text>
<text class="edge-label end" x="216" y="497">response with routing headers</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 8.9 KiB

69
docs/architecture/router/img/core-and-hosts-3.svg Normal file
View file

@ -0,0 +1,69 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 820 218" role="img" aria-labelledby="t">
<title id="t">Static and dynamic router settings</title>
<desc>Router(**arguments) is frozen into RouterConfig by build_router_config; process globals are snapshotted per version into RouterDynamicConfig; both, plus apply(ChangeSet), feed the core.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="819" height="217" rx="8"/>
<rect class="box" x="20" y="20" width="300" height="50" rx="6"/>
<text class="mono center" x="170" y="49">Router(**arguments)</text>
<rect class="box host" x="20" y="86" width="300" height="56" rx="6"/>
<text class="mono small center" x="170" y="109">litellm.num_retries, litellm.fallbacks,</text>
<text class="mono small center" x="170" y="126">litellm.model_cost, callbacks, ...</text>
<rect class="box" x="20" y="158" width="300" height="40" rx="6"/>
<text class="mono center" x="170" y="182">apply(ChangeSet)</text>
<rect class="box" x="482" y="20" width="188" height="50" rx="6"/>
<text class="mono title center" x="576" y="40">RouterConfig</text>
<text class="small muted center" x="576" y="58">frozen for the router's life</text>
<rect class="box" x="482" y="86" width="188" height="56" rx="6"/>
<text class="mono title center" x="576" y="118">RouterDynamicConfig</text>
<rect class="box core" x="720" y="20" width="80" height="178" rx="6"/>
<text class="title center" x="760" y="113">core</text>
<path class="edge" d="M320,45 H480" marker-end="url(#arrow)"/>
<text class="edge-label mono small center" x="401" y="38">build_router_config</text>
<path class="edge" d="M320,114 H480" marker-end="url(#arrow)"/>
<text class="edge-label center" x="401" y="107">snapshot per version</text>
<path class="edge" d="M670,45 H718" marker-end="url(#arrow)"/>
<path class="edge" d="M670,114 H718" marker-end="url(#arrow)"/>
<path class="edge" d="M320,178 H718" marker-end="url(#arrow)"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.9 KiB

86
docs/architecture/router/img/debug-a-routing-decision-1.svg Normal file
View file

@ -0,0 +1,86 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 844 238" role="img" aria-labelledby="t">
<title id="t">Debugging a routing decision</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="843" height="237" rx="8"/>
<rect class="box" x="20" y="26" width="186" height="48" rx="6"/>
<text class="center" x="113" y="54.5">Unexpected routing</text>
<rect class="box" x="256" y="26" width="180" height="48" rx="6"/>
<text class="center" x="346" y="46">1. Read the</text>
<text class="center" x="346" y="63">response headers</text>
<polygon class="box decision" points="484,26 616,26 630,50 616,74 484,74 470,50"/>
<text class="title center" x="550" y="54.5">Right model group?</text>
<rect class="box" x="664" y="26" width="160" height="48" rx="6"/>
<text class="center" x="744" y="46">2. Check how the</text>
<text class="center" x="744" y="63">name resolved</text>
<rect class="box" x="470" y="139" width="160" height="48" rx="6"/>
<text class="center" x="550" y="159">3. List the candidates</text>
<text class="center" x="550" y="176">the filters left</text>
<polygon class="box decision" points="270,139 422,139 436,163 422,187 270,187 256,163"/>
<text class="title center" x="346" y="159">Expected deployment</text>
<text class="title center" x="346" y="176">missing?</text>
<rect class="box" x="20" y="108" width="186" height="48" rx="6"/>
<text class="center" x="113" y="128">4. Check cooldowns,</text>
<text class="center" x="113" y="145">blocks, team and order</text>
<rect class="box" x="20" y="170" width="186" height="48" rx="6"/>
<text class="center" x="113" y="190">5. Check the</text>
<text class="center" x="113" y="207">strategy's inputs</text>
<path class="edge" d="M206,50 H254" marker-end="url(#arrow)"/>
<path class="edge" d="M436,50 H468" marker-end="url(#arrow)"/>
<path class="edge" d="M630,50 H662" marker-end="url(#arrow)"/>
<text class="edge-label center" x="647" y="43">no</text>
<path class="edge" d="M550,74 V137" marker-end="url(#arrow)"/>
<text class="edge-label " x="557" y="110">yes</text>
<path class="edge" d="M470,163 H438" marker-end="url(#arrow)"/>
<path class="edge" d="M256,163 H231 V132 H208" marker-end="url(#arrow)"/>
<path class="edge" d="M231,163 V194 H208" marker-end="url(#arrow)"/>
<text class="edge-label " x="237" y="150">yes</text>
<text class="edge-label " x="237" y="184">no</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 5.2 KiB

77
docs/architecture/router/img/debug-a-routing-decision-2.svg Normal file
View file

@ -0,0 +1,77 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 820 192" role="img" aria-labelledby="t">
<title id="t">Why a deployment is missing from candidates</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="819" height="191" rx="8"/>
<rect class="box" x="16" y="16" width="160" height="50" rx="6"/>
<text class="center" x="96" y="37">deployment missing</text>
<text class="center" x="96" y="54">from candidates</text>
<rect class="box decision" x="212" y="16" width="150" height="50" rx="14"/>
<text class="title center" x="287" y="46">cooldown entry?</text>
<rect class="box store" x="420" y="16" width="300" height="50" rx="6"/>
<text class="center" x="570" y="37">its <tspan class="mono">exception_received</tspan> and <tspan class="mono">status_code</tspan></text>
<text class="center" x="570" y="54">say which failure cooled it</text>
<path class="edge" d="M176,41 H210" marker-end="url(#arrow)"/>
<path class="edge" d="M362,41 H418" marker-end="url(#arrow)"/>
<text class="edge-label center" x="391" y="34">yes</text>
<path class="edge" d="M287,66 V88 H96 V108" marker-end="url(#arrow)"/>
<text class="edge-label" x="294" y="82">no</text>
<rect class="box decision" x="16" y="110" width="160" height="66" rx="14"/>
<text class="mono center" x="96" y="147">model_info.blocked?</text>
<rect class="box decision" x="212" y="110" width="170" height="66" rx="14"/>
<text class="mono center" x="297" y="139">model_info.team_id</text>
<text class="center" x="297" y="156">set to another team?</text>
<rect class="box decision" x="418" y="110" width="170" height="66" rx="14"/>
<text class="center" x="503" y="130">a lower</text>
<text class="mono center" x="503" y="147">litellm_params.order</text>
<text class="center" x="503" y="164">in the group?</text>
<rect class="box" x="624" y="110" width="180" height="66" rx="6"/>
<text class="title center" x="714" y="130">pre-call checks:</text>
<text class="center" x="714" y="147">context window,</text>
<text class="center" x="714" y="164">rpm / tpm, region, tags</text>
<path class="edge" d="M176,143 H210" marker-end="url(#arrow)"/>
<text class="edge-label center" x="194" y="136">no</text>
<path class="edge" d="M382,143 H416" marker-end="url(#arrow)"/>
<text class="edge-label center" x="400" y="136">no</text>
<path class="edge" d="M588,143 H622" marker-end="url(#arrow)"/>
<text class="edge-label center" x="606" y="136">no</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 5.2 KiB

73
docs/architecture/router/img/deployments-and-model-groups-1.svg Normal file
View file

@ -0,0 +1,73 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 820 208" role="img" aria-labelledby="t">
<title id="t">Deployment, litellm_params and model_info</title>
<desc>A deployment has model_name, litellm_params and model_info. litellm_params and model_info are parts of the deployment, each with its own fields.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="819" height="207" rx="8"/>
<rect class="box" x="20" y="20" width="260" height="168" rx="6"/>
<text class="mono title" x="36" y="44">litellm_params</text>
<text class="mono" x="36" y="68">model: &quot;provider/model&quot;</text>
<text class="mono" x="36" y="86">api_base, api_key, api_version</text>
<text class="mono" x="36" y="104">rpm, tpm, weight, order</text>
<text class="mono" x="36" y="122">timeout, stream_timeout,</text>
<text class="mono" x="36" y="140">max_parallel_requests</text>
<text class="mono" x="36" y="158">tags, use_in_pass_through</text>
<text class="mono" x="36" y="176">mock_response</text>
<rect class="box" x="335" y="57" width="150" height="94" rx="6"/>
<text class="title" x="351" y="81">Deployment</text>
<text class="mono" x="351" y="105">model_name</text>
<text class="mono" x="351" y="123">litellm_params</text>
<text class="mono" x="351" y="141">model_info</text>
<rect class="box" x="540" y="20" width="260" height="168" rx="6"/>
<text class="mono title" x="556" y="44">model_info</text>
<text class="mono" x="556" y="68">id</text>
<text class="mono" x="556" y="86">team_id, team_public_model_name</text>
<text class="mono" x="556" y="104">access_groups, access_windows,</text>
<text class="mono" x="556" y="122">ptu_shares</text>
<text class="mono" x="556" y="140">blocked, cooldown_time</text>
<text class="mono" x="556" y="158">allowed_fails,</text>
<text class="mono" x="556" y="176">allowed_fails_policy</text>
<path class="edge" d="M335,119 H282" marker-end="url(#arrow)"/>
<text class="edge-label center" x="308" y="112">has</text>
<path class="edge" d="M485,137 H538" marker-end="url(#arrow)"/>
<text class="edge-label center" x="512" y="130">has</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.9 KiB

72
docs/architecture/router/img/deployments-and-model-groups-2.svg Normal file
View file

@ -0,0 +1,72 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 872 146" role="img" aria-labelledby="t">
<title id="t">How the router builds each deployment</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="871" height="145" rx="8"/>
<rect class="group" x="14" y="14" width="844" height="118" rx="6"/>
<text class="group-label" x="28" y="34">FOR EACH</text>
<text class="small muted" x="92" y="34"><tspan class="mono">model_list</tspan> entry, in order</text>
<rect class="box" x="30" y="50" width="136" height="70" rx="6"/>
<text class="title center" x="98" y="73.5">Resolve secrets</text>
<text class="small muted center" x="98" y="91.5"><tspan class="mono">os.environ/</tspan></text>
<text class="small muted center" x="98" y="106.5">references to values</text>
<path class="edge" d="M166,85 H180" marker-end="url(#arrow)"/>
<rect class="box" x="182" y="50" width="122" height="70" rx="6"/>
<text class="title center" x="243" y="73.5">Validate</text>
<text class="small muted center" x="243" y="91.5">returns the</text>
<text class="small muted center" x="243" y="106.5">normalized entry</text>
<path class="edge" d="M304,85 H318" marker-end="url(#arrow)"/>
<rect class="box" x="320" y="50" width="112" height="70" rx="6"/>
<text class="title center" x="376" y="81.5">Find provider</text>
<text class="small muted center" x="376" y="99.5">e.g. azure</text>
<path class="edge" d="M432,85 H446" marker-end="url(#arrow)"/>
<rect class="box" x="448" y="50" width="130" height="70" rx="6"/>
<text class="title center" x="513" y="73.5">Add it</text>
<text class="small muted center" x="513" y="91.5">returns the entry as</text>
<text class="small muted center" x="513" y="106.5">the registry keeps it</text>
<path class="edge" d="M578,85 H592" marker-end="url(#arrow)"/>
<rect class="box" x="594" y="50" width="104" height="70" rx="6"/>
<text class="title center" x="646" y="89.5">Give it an id</text>
<path class="edge" d="M698,85 H712" marker-end="url(#arrow)"/>
<rect class="box store" x="714" y="50" width="128" height="70" rx="6"/>
<text class="title center" x="778" y="89.5">Join the registry</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.9 KiB

71
docs/architecture/router/img/deployments-and-model-groups-3.svg Normal file
View file

@ -0,0 +1,71 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 730 232" role="img" aria-labelledby="t">
<title id="t">Model groups, deployments and fallback</title>
<desc>A request for model gpt-4o goes to the gpt-4o model group of three deployments; that group falls back to the claude-sonnet group of two deployments.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="729" height="231" rx="8"/>
<rect class="box" x="20" y="100" width="150" height="46" rx="6"/>
<text class="title center" x="95" y="120">Request</text>
<text class="mono small center" x="95" y="136">model=gpt-4o</text>
<path class="edge" d="M170,123 H208" marker-end="url(#arrow)"/>
<rect class="group" x="210" y="16" width="210" height="200" rx="8"/>
<text class="group-label" x="225" y="36">MODEL GROUP</text>
<text class="mono small accent" x="318" y="36">gpt-4o</text>
<rect class="box" x="225" y="48" width="180" height="46" rx="6"/>
<text class="mono center" x="315" y="68">gpt-4o-eastus</text>
<text class="small muted center" x="315" y="84">azure, order 1</text>
<rect class="box" x="225" y="102" width="180" height="46" rx="6"/>
<text class="mono center" x="315" y="122">gpt-4o-westus</text>
<text class="small muted center" x="315" y="138">azure, order 1</text>
<rect class="box" x="225" y="156" width="180" height="46" rx="6"/>
<text class="mono center" x="315" y="176">gpt-4o-openai</text>
<text class="small muted center" x="315" y="192">openai, order 2</text>
<path class="edge dashed" d="M420,123 H498" marker-end="url(#arrow)"/>
<text class="edge-label center" x="459" y="114">fallback</text>
<rect class="group" x="500" y="16" width="210" height="200" rx="8"/>
<text class="group-label" x="515" y="36">MODEL GROUP</text>
<text class="mono small accent" x="608" y="36">claude-sonnet</text>
<rect class="box" x="515" y="75" width="180" height="46" rx="6"/>
<text class="center" x="605" y="103">anthropic</text>
<rect class="box" x="515" y="129" width="180" height="46" rx="6"/>
<text class="center" x="605" y="157">bedrock us-east-1</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.9 KiB

108
docs/architecture/router/img/entry-points-1.svg Normal file
View file

@ -0,0 +1,108 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 820 496" role="img" aria-labelledby="t">
<title id="t">Router entry points</title>
<desc>The Router facade and the views and change set its entry points return or take.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="819" height="495" rx="8"/>
<rect class="box host" x="20" y="20" width="580" height="144" rx="6"/>
<text class="title mono center" x="310" y="42">Router</text>
<text class="mono small" x="36" y="66">acompletion()</text>
<text class="small muted" x="36" y="83">and the other endpoint methods</text>
<text class="mono small" x="36" y="100">registry() <tspan class="muted">RouterRegistry</tspan></text>
<text class="mono small" x="36" y="117">settings() <tspan class="muted">RouterSettings</tspan></text>
<text class="mono small" x="36" y="134">health() <tspan class="muted">RouterHealth</tspan></text>
<text class="mono small" x="36" y="151">retire()</text>
<text class="mono small" x="258" y="66">select(model, request_kwargs, kind) <tspan class="muted">deployment</tspan></text>
<text class="mono small" x="258" y="83">apply(ChangeSet) <tspan class="muted">ApplyResult</tspan></text>
<text class="mono small" x="258" y="100">aapply(ChangeSet) <tspan class="muted">ApplyResult</tspan></text>
<text class="mono small" x="258" y="117">observe(logging_obj, request_scoped)</text>
<text class="mono small muted" x="282" y="134">RoutingObservers</text>
<text class="mono small" x="258" y="151">register_plugin(CustomRoutingStrategyBase)</text>
<rect class="box" x="640" y="20" width="160" height="144" rx="6"/>
<text class="title mono center" x="720" y="42">ChangeSet</text>
<text class="mono small" x="656" y="66">assign</text>
<text class="mono small" x="656" y="83">settings</text>
<text class="mono small" x="656" y="100">upsert</text>
<text class="mono small" x="656" y="117">delete</text>
<text class="mono small" x="656" y="134">redis_cache</text>
<text class="mono small" x="656" y="151">refresh_model_info</text>
<path class="edge dashed" d="M600,92 H638" marker-end="url(#arrow)"/>
<text class="edge-label center" x="620" y="84">apply</text>
<rect class="box store" x="20" y="214" width="250" height="262" rx="6"/>
<text class="title mono center" x="145" y="236">RouterRegistry</text>
<text class="mono small" x="36" y="260">version <tspan class="muted">int</tspan></text>
<text class="mono small" x="36" y="277">deployment(id)</text>
<text class="mono small" x="36" y="294">deployments(model_name, source)</text>
<text class="mono small" x="36" y="311">names(model_name, kind)</text>
<text class="mono small" x="36" y="328">resolve(name, kind)</text>
<text class="mono small" x="36" y="345">group(name, field)</text>
<text class="mono small" x="36" y="362">access_groups()</text>
<text class="mono small" x="36" y="379">credentials(id, form)</text>
<text class="mono small" x="36" y="396">plugins(kind)</text>
<text class="mono small" x="36" y="413">routing_group(name)</text>
<text class="mono small" x="36" y="430">client(deployment)</text>
<text class="mono small" x="36" y="447">aliases() patterns()</text>
<text class="mono small" x="36" y="464">routing_groups() index()</text>
<rect class="box store" x="295" y="214" width="230" height="157" rx="6"/>
<text class="title mono center" x="410" y="236">RouterSettings</text>
<text class="small muted center" x="410" y="254">frozen, static</text>
<text class="mono small" x="311" y="274">routing_strategy</text>
<text class="mono small" x="311" y="291">fallbacks ...</text>
<text class="mono small" x="311" y="308">is_set(name)</text>
<text class="mono small" x="311" y="325">as_dict()</text>
<text class="mono small" x="311" y="342">allowed_fails_for(exception)</text>
<text class="mono small" x="311" y="359">guardrail(name)</text>
<rect class="box store" x="550" y="214" width="250" height="194" rx="6"/>
<text class="title mono center" x="675" y="236">RouterHealth</text>
<text class="mono small" x="566" y="260">store <tspan class="muted">routing cache</tspan></text>
<text class="mono small" x="566" y="277">cooldowns</text>
<text class="mono small" x="566" y="294">health_checks</text>
<text class="mono small" x="566" y="311">latency</text>
<text class="mono small" x="566" y="328">is_blocked(d) all_blocked(ds)</text>
<text class="mono small" x="566" y="345">usage(model_group, kind)</text>
<text class="mono small" x="566" y="362">fully_blocked_model_names()</text>
<text class="mono small" x="566" y="379">failures_this_minute(id)</text>
<text class="mono small" x="566" y="396">record_failure(id, exc, status)</text>
<path class="edge" d="M145,164 V212" marker-end="url(#arrow)"/>
<path class="edge" d="M410,164 V212" marker-end="url(#arrow)"/>
<path class="edge" d="M580,164 V212" marker-end="url(#arrow)"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 7.5 KiB

66
docs/architecture/router/img/move-off-deprecated-members-1.svg Normal file
View file

@ -0,0 +1,66 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 815 194" role="img" aria-labelledby="t">
<title id="t">Moving a caller off a deprecated member</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="814" height="193" rx="8"/>
<rect class="box" x="20" y="20" width="170" height="56" rx="6"/>
<text class="mono center" x="105" y="44">DeprecationWarning</text>
<text class="center" x="105" y="61">names the replacement</text>
<rect class="box" x="230" y="20" width="130" height="56" rx="6"/>
<text class="center" x="295" y="52">Rewrite the call</text>
<rect class="box" x="400" y="20" width="165" height="56" rx="6"/>
<text class="center" x="482.5" y="52">Run the caller's tests</text>
<rect class="box decision" x="605" y="20" width="190" height="56" rx="6"/>
<text class="center" x="700" y="44">Run</text>
<text class="mono center" x="700" y="61">router_surface_check.py</text>
<rect class="box" x="300" y="130" width="180" height="44" rx="6"/>
<text class="center" x="390" y="156">Remove it with <tspan class="mono">--write</tspan></text>
<rect class="box ok" x="680" y="130" width="100" height="44" rx="6"/>
<text class="title center" x="730" y="156">Done</text>
<path class="edge" d="M190,48 H228" marker-end="url(#arrow)"/>
<path class="edge" d="M360,48 H398" marker-end="url(#arrow)"/>
<path class="edge" d="M565,48 H603" marker-end="url(#arrow)"/>
<path class="edge" d="M650,76 V152 H482" marker-end="url(#arrow)"/>
<text class="edge-label center" x="566" y="144">listed entry now unused</text>
<path class="edge" d="M730,76 V128" marker-end="url(#arrow)"/>
<text class="edge-label" x="738" y="106">clean</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.5 KiB

117
docs/architecture/router/img/names-1.svg Normal file
View file

@ -0,0 +1,117 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 376" role="img" aria-labelledby="t">
<title id="t">How a requested name resolves to deployments</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="839" height="375" rx="8"/>
<rect class="box" x="20" y="20" width="90" height="46" rx="6"/>
<text class="center" x="65" y="39.5">requested</text>
<text class="center" x="65" y="55.5">name</text>
<rect class="box decision" x="150" y="25" width="165" height="36" rx="18"/>
<text class="center" x="232.5" y="47.5"><tspan class="mono">specific_deployment?</tspan></text>
<rect class="box decision" x="360" y="20" width="160" height="46" rx="18"/>
<text class="center" x="440" y="39.5">not a group name,</text>
<text class="center" x="440" y="55.5">but a deployment id?</text>
<rect class="box" x="705" y="20" width="130" height="46" rx="6"/>
<text class="center" x="770" y="39.5">swap in the alias</text>
<text class="center" x="770" y="55.5">target, if any</text>
<rect class="box ok" x="117.5" y="106" width="230" height="46" rx="6"/>
<text class="center" x="232.5" y="125.5">deployments whose litellm model</text>
<text class="center" x="232.5" y="141.5">is exactly that name</text>
<rect class="box ok" x="390.0" y="106" width="100" height="46" rx="6"/>
<text class="center" x="440" y="125.5">that one</text>
<text class="center" x="440" y="141.5">deployment</text>
<rect class="box decision" x="715" y="200" width="110" height="46" rx="18"/>
<text class="center" x="770" y="219.5">a routing</text>
<text class="center" x="770" y="235.5">group?</text>
<rect class="box decision" x="565" y="200" width="105" height="46" rx="18"/>
<text class="center" x="617.5" y="219.5">a configured</text>
<text class="center" x="617.5" y="235.5">group?</text>
<rect class="box" x="539.5" y="98" width="156" height="62" rx="6"/>
<text class="center" x="617.5" y="117.5">team public name,</text>
<text class="center" x="617.5" y="133.5">then wildcard, then</text>
<text class="center" x="617.5" y="149.5"><tspan class="mono">default_deployment</tspan></text>
<rect class="box" x="325" y="192" width="195" height="62" rx="6"/>
<text class="center" x="422.5" y="211.5">the group's deployments,</text>
<text class="center" x="422.5" y="227.5">team ones first, kept to</text>
<text class="center" x="422.5" y="243.5">the caller's access groups</text>
<rect class="box decision" x="200" y="205" width="80" height="36" rx="18"/>
<text class="center" x="240" y="227.5">empty?</text>
<rect class="box ok" x="20" y="200" width="135" height="46" rx="6"/>
<text class="center" x="87.5" y="227.5">those deployments</text>
<rect class="box" x="122.0" y="294" width="236" height="62" rx="6"/>
<text class="center" x="240" y="313.5">deployments whose litellm model</text>
<text class="center" x="240" y="329.5">is the name, then the first</text>
<text class="center" x="240" y="345.5"><tspan class="mono">default_fallbacks</tspan> group</text>
<rect class="box warn" x="446.0" y="302" width="170" height="46" rx="6"/>
<text class="center" x="531" y="321.5">error: no deployments</text>
<text class="center" x="531" y="337.5">for this model</text>
<rect class="box" x="715.0" y="302" width="110" height="46" rx="6"/>
<text class="center" x="770" y="321.5">the members'</text>
<text class="center" x="770" y="337.5">deployments</text>
<path class="edge" d="M110,43.0 H148" marker-end="url(#arrow)"/>
<path class="edge" d="M315,43.0 H358" marker-end="url(#arrow)"/>
<text class="edge-label center" x="337.5" y="37">no</text>
<path class="edge" d="M520,43.0 H703" marker-end="url(#arrow)"/>
<text class="edge-label center" x="612.5" y="37">no</text>
<path class="edge" d="M232.5,61 V104" marker-end="url(#arrow)"/>
<text class="edge-label" x="238.5" y="87.5">yes</text>
<path class="edge" d="M440.0,66 V104" marker-end="url(#arrow)"/>
<text class="edge-label" x="446" y="90">yes</text>
<path class="edge" d="M770.0,66 V198" marker-end="url(#arrow)"/>
<path class="edge" d="M715,223.0 H672" marker-end="url(#arrow)"/>
<text class="edge-label center" x="692.5" y="217">no</text>
<path class="edge" d="M617.5,200 V162" marker-end="url(#arrow)"/>
<text class="edge-label" x="623.5" y="184">no</text>
<path class="edge" d="M565,223.0 H522" marker-end="url(#arrow)"/>
<text class="edge-label center" x="542.5" y="217">yes</text>
<path class="edge" d="M325,223.0 H282" marker-end="url(#arrow)"/>
<path class="edge" d="M200,223.0 H157" marker-end="url(#arrow)"/>
<text class="edge-label center" x="177.5" y="217">no</text>
<path class="edge" d="M770.0,246 V300" marker-end="url(#arrow)"/>
<text class="edge-label" x="776" y="278">yes</text>
<path class="edge" d="M715.0,325.0 H693.0 V274 H302.5 V223.0 H282" marker-end="url(#arrow)"/>
<path class="edge" d="M240.0,241 V292" marker-end="url(#arrow)"/>
<text class="edge-label" x="246" y="286">yes</text>
<path class="edge warn" d="M358.0,325.0 H444.0" marker-end="url(#arrow-warn)"/>
<text class="edge-label center" x="402" y="319">still empty</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 7.9 KiB

93
docs/architecture/router/img/read-router-state-1.svg Normal file
View file

@ -0,0 +1,93 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 300" role="img" aria-labelledby="t">
<title id="t">Pick the entry point for reading router state</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="859" height="299" rx="8"/>
<rect class="box" x="14" y="16" width="196" height="52" rx="6"/>
<text class="title center" x="112" y="46">What do you need to know?</text>
<rect class="box" x="444" y="16" width="396" height="52" rx="6"/>
<text class="center" x="642" y="37">Nothing fits: discuss adding a query</text>
<text class="center" x="642" y="55">to an existing view before widening the surface</text>
<rect class="box decision" x="20" y="100" width="184" height="88" rx="6"/>
<text class="title center" x="112" y="132.5">What is configured?</text>
<text class="small muted center" x="112" y="152.5">deployments, names,</text>
<text class="small muted center" x="112" y="168.5">aliases, credentials</text>
<path class="edge" d="M112,188 V226" marker-end="url(#arrow)"/>
<text class="edge-label" x="118" y="211">yes</text>
<rect class="box ok" x="20" y="228" width="184" height="52" rx="6"/>
<text class="mono center" x="112" y="258">router.registry()</text>
<path class="edge" d="M204,144 H230" marker-end="url(#arrow)"/>
<text class="edge-label center" x="218" y="138">no</text>
<rect class="box decision" x="232" y="100" width="184" height="88" rx="6"/>
<text class="title center" x="324" y="124">How is the router</text>
<text class="title center" x="324" y="141">configured?</text>
<text class="small muted center" x="324" y="161">strategy, fallbacks,</text>
<text class="small muted center" x="324" y="177">cooldown time</text>
<path class="edge" d="M324,188 V226" marker-end="url(#arrow)"/>
<text class="edge-label" x="330" y="211">yes</text>
<rect class="box ok" x="232" y="228" width="184" height="52" rx="6"/>
<text class="mono center" x="324" y="258">router.settings()</text>
<path class="edge" d="M416,144 H442" marker-end="url(#arrow)"/>
<text class="edge-label center" x="430" y="138">no</text>
<rect class="box decision" x="444" y="100" width="184" height="88" rx="6"/>
<text class="title center" x="536" y="124">What state are</text>
<text class="title center" x="536" y="141">deployments in?</text>
<text class="small muted center" x="536" y="161">cooldowns, blocked,</text>
<text class="small muted center" x="536" y="177">usage, health checks</text>
<path class="edge" d="M536,188 V226" marker-end="url(#arrow)"/>
<text class="edge-label" x="542" y="211">yes</text>
<rect class="box ok" x="444" y="228" width="184" height="52" rx="6"/>
<text class="mono center" x="536" y="258">router.health()</text>
<path class="edge" d="M628,144 H654" marker-end="url(#arrow)"/>
<text class="edge-label center" x="642" y="138">no</text>
<rect class="box decision" x="656" y="100" width="184" height="88" rx="6"/>
<text class="title center" x="748" y="140">Which deployment would</text>
<text class="title center" x="748" y="157">a request get?</text>
<path class="edge" d="M748,188 V226" marker-end="url(#arrow)"/>
<text class="edge-label" x="754" y="211">yes</text>
<rect class="box ok" x="656" y="228" width="184" height="52" rx="6"/>
<text class="mono center" x="748" y="250">await router.select(</text>
<text class="mono center" x="748" y="266">model, request_kwargs)</text>
<path class="edge" d="M112,68 V98" marker-end="url(#arrow)"/>
<path class="edge" d="M748,100 V70" marker-end="url(#arrow)"/>
<text class="edge-label" x="754" y="89">no</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 6.3 KiB

115
docs/architecture/router/img/reliability-1.svg Normal file
View file

@ -0,0 +1,115 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 880 424" role="img" aria-labelledby="t">
<title id="t">A retry inside a model group, then a fallback out of it</title>
<desc>Inside each model group the strategy picks one deployment. The strategy picks D1 in gpt-4o, which returns 429 and cools down. The retry stays in gpt-4o, the strategy picks D3, and it also returns 429. The request falls back to opus, the strategy picks D5, and D5 returns 200 to the router.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="879" height="423" rx="8"/>
<circle class="box core" cx="80" cy="150" r="56"/>
<text class="title center" x="80" y="155">Router</text>
<rect class="group core" x="190" y="20" width="360" height="260" rx="8"/>
<text class="group-label" x="204" y="42">MODEL GROUP: GPT-4O</text>
<text class="edge-label end" x="536" y="42">retries stay in this group</text>
<rect class="group core" x="640" y="20" width="220" height="260" rx="8"/>
<text class="group-label" x="654" y="42">MODEL GROUP: OPUS</text>
<rect class="box decision" x="250" y="136" width="250" height="32" rx="16"/>
<text class="small center" x="375" y="156">strategy picks one deployment here</text>
<rect class="box decision" x="675" y="136" width="150" height="32" rx="16"/>
<text class="small center" x="750" y="156">strategy picks here</text>
<rect class="box warn" x="220" y="60" width="140" height="64" rx="6"/>
<text class="title center" x="290" y="87">D1</text>
<text class="small center" x="290" y="106">429, cools down</text>
<rect class="box" x="390" y="60" width="140" height="64" rx="6"/>
<text class="title center" x="460" y="97">D2</text>
<rect class="box warn" x="220" y="180" width="140" height="64" rx="6"/>
<text class="title center" x="290" y="207">D3</text>
<text class="small center" x="290" y="226">429</text>
<rect class="box" x="390" y="180" width="140" height="64" rx="6"/>
<text class="title center" x="460" y="217">D4</text>
<rect class="box ok" x="675" y="60" width="150" height="64" rx="6"/>
<text class="title center" x="750" y="87">D5</text>
<text class="small center" x="750" y="106">returns 200</text>
<rect class="box" x="675" y="180" width="150" height="64" rx="6"/>
<text class="title center" x="750" y="217">D6</text>
<path class="edge" d="M120,111 L218,76" marker-end="url(#arrow)"/>
<path class="edge" d="M220,108 L138,140" marker-end="url(#arrow)"/>
<path class="edge" d="M132,171 L218,196" marker-end="url(#arrow)"/>
<path class="edge" d="M220,228 L118,192" marker-end="url(#arrow)"/>
<text class="edge-label" x="186" y="177">retry</text>
<path class="edge" d="M550,150 H655 V92 H673" marker-end="url(#arrow)"/>
<text class="edge-label center" x="594" y="134">fallback</text>
<path class="edge" d="M825,92 H846 V296 H80 V208" marker-end="url(#arrow)"/>
<circle class="step" cx="169" cy="93" r="8"/>
<text class="step-text" x="169" y="96.5">1</text>
<circle class="step" cx="179" cy="124" r="8"/>
<text class="step-text" x="179" y="127.5">2</text>
<circle class="step" cx="175" cy="183" r="8"/>
<text class="step-text" x="175" y="186.5">3</text>
<circle class="step" cx="169" cy="210" r="8"/>
<text class="step-text" x="169" y="213.5">4</text>
<circle class="step" cx="594" cy="150" r="8"/>
<text class="step-text" x="594" y="153.5">5</text>
<circle class="step" cx="655" cy="118" r="8"/>
<text class="step-text" x="655" y="121.5">6</text>
<circle class="step" cx="460" cy="296" r="8"/>
<text class="step-text" x="460" y="299.5">7</text>
<circle class="step" cx="36" cy="330" r="8"/>
<text class="step-text" x="36" y="333.5">1</text>
<text class="small" x="52" y="334">The strategy picks D1 from gpt-4o, and the router calls it</text>
<circle class="step" cx="36" cy="356" r="8"/>
<text class="step-text" x="36" y="359.5">2</text>
<text class="small" x="52" y="360">D1 returns 429, so it goes into cooldown</text>
<circle class="step" cx="36" cy="382" r="8"/>
<text class="step-text" x="36" y="385.5">3</text>
<text class="small" x="52" y="386">Retry in the same group: the strategy picks again and gets D3</text>
<circle class="step" cx="36" cy="408" r="8"/>
<text class="step-text" x="36" y="411.5">4</text>
<text class="small" x="52" y="412">D3 returns 429, and the retries are spent</text>
<circle class="step" cx="460" cy="330" r="8"/>
<text class="step-text" x="460" y="333.5">5</text>
<text class="small" x="476" y="334">Fallback: the request moves to model group opus</text>
<circle class="step" cx="460" cy="356" r="8"/>
<text class="step-text" x="460" y="359.5">6</text>
<text class="small" x="476" y="360">The strategy picks D5 from opus, and the router calls it</text>
<circle class="step" cx="460" cy="382" r="8"/>
<text class="step-text" x="460" y="385.5">7</text>
<text class="small" x="476" y="386">D5 returns 200, which goes back to the caller</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 7.6 KiB

89
docs/architecture/router/img/reliability-2.svg Normal file
View file

@ -0,0 +1,89 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 372" role="img" aria-labelledby="t">
<title id="t">Retry loop after a failed attempt</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="859" height="371" rx="8"/>
<rect class="box" x="20" y="42" width="108" height="44" rx="6"/>
<text class="center" x="74" y="68">Attempt failed</text>
<polygon class="box decision" points="155,64 260,30 365,64 260,98"/>
<text class="center" x="260" y="68">Guardrail intervention?</text>
<rect class="box" x="400" y="20" width="440" height="88" rx="6"/>
<text class="title" x="416" y="42">Number of retries</text>
<text class="small" x="416" y="61"><tspan class="mono">retry_policy</tspan> for this failure kind, unless the request</text>
<text class="small" x="416" y="78">set <tspan class="mono">num_retries=0</tspan>, else the request's <tspan class="mono">num_retries</tspan>,</text>
<text class="small" x="416" y="95">else the deployment's, else the router's <tspan class="mono">num_retries</tspan></text>
<rect class="box warn" x="20" y="158" width="300" height="44" rx="6"/>
<text class="center" x="150" y="184">Give up, so the fallback walk sees it</text>
<polygon class="box decision" points="430,180 510,150 590,180 510,210"/>
<text class="center" x="510" y="184">Worth retrying?</text>
<rect class="box" x="620" y="137" width="220" height="86" rx="6"/>
<text class="title" x="636" y="159">Sleep</text>
<text class="small" x="636" y="178">0 if other healthy deployments</text>
<text class="small" x="636" y="195">remain, else <tspan class="mono">Retry-After</tspan></text>
<text class="small" x="636" y="212">or exponential backoff</text>
<rect class="box" x="620" y="255" width="220" height="50" rx="6"/>
<text class="center" x="730" y="276">Select again, skipping</text>
<text class="small center" x="730" y="293">deployments that already failed</text>
<polygon class="box decision" points="450,280 510,252 570,280 510,308"/>
<text class="center" x="510" y="284">Success?</text>
<rect class="box ok" x="20" y="310" width="310" height="44" rx="6"/>
<text class="center" x="175" y="336">Return, with <tspan class="mono">x-litellm-attempted-retries</tspan></text>
<path class="edge" d="M128,64 H153" marker-end="url(#arrow)"/>
<path class="edge" d="M365,64 H398" marker-end="url(#arrow)"/>
<text class="edge-label center" x="382" y="56">no</text>
<path class="edge warn" d="M260,98 V156" marker-end="url(#arrow-warn)"/>
<text class="edge-label" x="268" y="132">yes</text>
<path class="edge" d="M510,108 V148" marker-end="url(#arrow)"/>
<path class="edge warn" d="M430,180 H322" marker-end="url(#arrow-warn)"/>
<text class="edge-label center" x="376" y="172">no</text>
<path class="edge" d="M590,180 H618" marker-end="url(#arrow)"/>
<text class="edge-label center" x="604" y="172">yes</text>
<path class="edge" d="M730,223 V253" marker-end="url(#arrow)"/>
<path class="edge" d="M620,280 H572" marker-end="url(#arrow)"/>
<path class="edge" d="M510,252 V212" marker-end="url(#arrow)"/>
<text class="edge-label" x="518" y="236">no, retries left</text>
<path class="edge warn" d="M450,280 H290 V204" marker-end="url(#arrow-warn)"/>
<text class="edge-label center" x="370" y="272">no, none left</text>
<path class="edge" d="M510,308 V332 H332" marker-end="url(#arrow)"/>
<text class="edge-label center" x="420" y="325">yes</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 6.3 KiB

123
docs/architecture/router/img/reliability-3.svg Normal file
View file

@ -0,0 +1,123 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 520" role="img" aria-labelledby="t">
<title id="t">Where the router goes after retries run out</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="859" height="519" rx="8"/>
<rect class="box" x="20" y="20" width="130" height="72" rx="6"/>
<text class="center" x="85" y="52">Retries exhausted,</text>
<text class="center" x="85" y="69">failure e</text>
<rect class="box decision" x="184" y="20" width="226" height="72" rx="6"/>
<text class="center" x="297" y="43.5"><tspan class="mono">disable_fallbacks</tspan> on the</text>
<text class="center" x="297" y="60.5">request, or a guardrail</text>
<text class="center" x="297" y="77.5">intervention?</text>
<rect class="box decision" x="446" y="20" width="394" height="72" rx="6"/>
<text class="center" x="643" y="52">Several <tspan class="mono">order</tspan> values in the group,</text>
<text class="center" x="643" y="69">and e is not context window / content policy?</text>
<rect class="box warn" x="20" y="140" width="130" height="36" rx="6"/>
<text class="center" x="85" y="162.5"><tspan class="mono">raise e</tspan></text>
<rect class="box" x="236" y="126" width="180" height="64" rx="6"/>
<text class="center" x="326" y="145.5">Re-pick across the</text>
<text class="center" x="326" y="162.5">other deployments of</text>
<text class="center" x="326" y="179.5">the group first</text>
<rect class="box decision" x="446" y="130" width="200" height="56" rx="6"/>
<text class="center" x="546" y="154"><tspan class="mono">enable_weighted_failover</tspan></text>
<text class="center" x="546" y="171">with <tspan class="mono">simple-shuffle</tspan>?</text>
<rect class="box" x="676" y="126" width="164" height="64" rx="6"/>
<text class="center" x="758" y="145.5">The next orders inside</text>
<text class="center" x="758" y="162.5">the same group, then</text>
<text class="center" x="758" y="179.5">the group's fallbacks</text>
<rect class="box decision" x="296" y="242" width="180" height="40" rx="6"/>
<text class="center" x="386" y="266.5">What is e?</text>
<rect class="box decision" x="20" y="328" width="196" height="52" rx="6"/>
<text class="center" x="118" y="350"><tspan class="mono">context_window_fallbacks</tspan></text>
<text class="center" x="118" y="367">set?</text>
<rect class="box" x="256" y="322" width="244" height="64" rx="6"/>
<text class="center" x="378" y="341.5"><tspan class="mono">fallbacks</tspan>: the group's entry,</text>
<text class="center" x="378" y="358.5">else the <tspan class="mono">'*'</tspan> entry</text>
<text class="center" x="378" y="375.5">(<tspan class="mono">default_fallbacks</tspan>)</text>
<rect class="box decision" x="540" y="328" width="210" height="52" rx="6"/>
<text class="center" x="645" y="350"><tspan class="mono">content_policy_fallbacks</tspan></text>
<text class="center" x="645" y="367">set?</text>
<rect class="box" x="20" y="426" width="196" height="52" rx="6"/>
<text class="center" x="118" y="448">Its entry for this group</text>
<text class="center small muted" x="118" y="465">(no entry: <tspan class="mono">raise e</tspan>)</text>
<rect class="box ok" x="256" y="432" width="244" height="40" rx="6"/>
<text class="center" x="378" y="456.5">Walk the targets in order</text>
<rect class="box" x="540" y="426" width="210" height="52" rx="6"/>
<text class="center" x="645" y="448">Its entry for this group</text>
<text class="center small muted" x="645" y="465">(no entry: <tspan class="mono">raise e</tspan>)</text>
<path class="edge" d="M150,56 H182" marker-end="url(#arrow)"/>
<path class="edge" d="M410,56 H444" marker-end="url(#arrow)"/>
<text class="edge-label center" x="428" y="48">no</text>
<path class="edge warn" d="M200,92 V158 H152" marker-end="url(#arrow-warn)"/>
<text class="edge-label" x="208" y="128">yes</text>
<path class="edge" d="M546,92 V128" marker-end="url(#arrow)"/>
<text class="edge-label" x="554" y="115">no</text>
<path class="edge" d="M758,92 V124" marker-end="url(#arrow)"/>
<text class="edge-label" x="766" y="113">yes</text>
<path class="edge" d="M446,158 H418" marker-end="url(#arrow)"/>
<text class="edge-label center" x="431" y="150">yes</text>
<path class="edge" d="M326,190 V240" marker-end="url(#arrow)"/>
<path class="edge" d="M462,186 V240" marker-end="url(#arrow)"/>
<text class="edge-label" x="470" y="218">no</text>
<path class="edge" d="M296,262 H118 V326" marker-end="url(#arrow)"/>
<text class="edge-label center" x="190" y="254">context window</text>
<path class="edge" d="M476,262 H645 V326" marker-end="url(#arrow)"/>
<text class="edge-label center" x="560" y="254">content policy</text>
<path class="edge" d="M386,282 V320" marker-end="url(#arrow)"/>
<text class="edge-label" x="394" y="306">anything else</text>
<path class="edge" d="M216,354 H254" marker-end="url(#arrow)"/>
<text class="edge-label center" x="236" y="346">no</text>
<path class="edge" d="M540,354 H502" marker-end="url(#arrow)"/>
<text class="edge-label center" x="520" y="346">no</text>
<path class="edge" d="M118,380 V424" marker-end="url(#arrow)"/>
<text class="edge-label" x="126" y="407">yes</text>
<path class="edge" d="M645,380 V424" marker-end="url(#arrow)"/>
<text class="edge-label" x="653" y="407">yes</text>
<path class="edge warn" d="M270,322 V212 H85 V178" marker-end="url(#arrow-warn)"/>
<text class="edge-label center" x="180" y="207">no entry</text>
<path class="edge" d="M378,386 V430" marker-end="url(#arrow)"/>
<path class="edge" d="M216,452 H254" marker-end="url(#arrow)"/>
<path class="edge" d="M540,452 H502" marker-end="url(#arrow)"/>
<path class="edge" d="M800,190 V500 H378 V474" marker-end="url(#arrow)"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 8.6 KiB

94
docs/architecture/router/img/reliability-4.svg Normal file
View file

@ -0,0 +1,94 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 790 452" role="img" aria-labelledby="t">
<title id="t">Fallback from group A to group B</title>
<desc>A 503 from group A survives its retries, the fallback checks allow B, and B answers with 200.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="789" height="451" rx="8"/>
<line class="lifeline" x1="80" y1="58" x2="80" y2="434"/>
<line class="lifeline" x1="350" y1="58" x2="350" y2="434"/>
<line class="lifeline" x1="545" y1="58" x2="545" y2="434"/>
<line class="lifeline" x1="708" y1="58" x2="708" y2="434"/>
<rect class="box" x="10" y="18" width="140" height="40" rx="6"/>
<text class="title center" x="80" y="43">Caller</text>
<rect class="box core" x="280" y="18" width="140" height="40" rx="6"/>
<text class="title center" x="350" y="43">Fallback walk</text>
<rect class="box core" x="475" y="18" width="140" height="40" rx="6"/>
<text class="title center" x="545" y="43">Retry loop</text>
<rect class="box host" x="638" y="18" width="140" height="40" rx="6"/>
<text class="title center" x="708" y="43">Fallback checks</text>
<path class="edge core" d="M106,96 H348" marker-end="url(#arrow-core)"/>
<circle class="step" cx="94" cy="96" r="8"/>
<text class="step-text" x="94" y="99.7">1</text>
<text class="edge-label center" x="225.0" y="88"><tspan class="mono">model=A</tspan></text>
<path class="edge core" d="M376,138 H543" marker-end="url(#arrow-core)"/>
<circle class="step" cx="364" cy="138" r="8"/>
<text class="step-text" x="364" y="141.7">2</text>
<text class="edge-label center" x="457.5" y="130"><tspan>group A, </tspan><tspan class="mono">num_retries=2</tspan></text>
<path class="edge warn dashed" d="M519,180 H352" marker-end="url(#arrow-warn)"/>
<circle class="step" cx="531" cy="180" r="8"/>
<text class="step-text" x="531" y="183.7">3</text>
<text class="edge-label center" x="437.5" y="172"><tspan>503 after 3 attempts</tspan></text>
<path class="edge" d="M376,238 H706" marker-end="url(#arrow)"/>
<circle class="step" cx="364" cy="238" r="8"/>
<text class="step-text" x="364" y="241.7">4</text>
<text class="edge-label center" x="539.0" y="215"><tspan class="mono">fallback_access_check,</tspan></text>
<text class="edge-label center" x="539.0" y="230"><tspan class="mono">fallback_budget_check</tspan><tspan> for B</tspan></text>
<path class="edge dashed" d="M682,280 H352" marker-end="url(#arrow)"/>
<circle class="step" cx="694" cy="280" r="8"/>
<text class="step-text" x="694" y="283.7">5</text>
<text class="edge-label center" x="519.0" y="272"><tspan>allowed</tspan></text>
<path class="edge core" d="M376,322 H543" marker-end="url(#arrow-core)"/>
<circle class="step" cx="364" cy="322" r="8"/>
<text class="step-text" x="364" y="325.7">6</text>
<text class="edge-label center" x="457.5" y="314"><tspan>group B, </tspan><tspan class="mono">num_retries=2</tspan></text>
<path class="edge dashed" d="M519,364 H352" marker-end="url(#arrow)"/>
<circle class="step" cx="531" cy="364" r="8"/>
<text class="step-text" x="531" y="367.7">7</text>
<text class="edge-label center" x="437.5" y="356"><tspan>200</tspan></text>
<path class="edge dashed" d="M324,420 H82" marker-end="url(#arrow)"/>
<circle class="step" cx="336" cy="420" r="8"/>
<text class="step-text" x="336" y="423.7">8</text>
<text class="edge-label center" x="205.0" y="397"><tspan>response from B,</tspan></text>
<text class="edge-label center" x="205.0" y="412"><tspan class="mono">x-litellm-attempted-fallbacks: 1</tspan></text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 6.7 KiB

83
docs/architecture/router/img/reliability-5.svg Normal file
View file

@ -0,0 +1,83 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 820 248" role="img" aria-labelledby="t">
<title id="t">How a broken stream is handled</title>
<style>
/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="819" height="247" rx="8"/>
<rect class="box" x="20" y="81" width="150" height="48" rx="6"/>
<text class="center" x="95" y="101">Stream opened,</text>
<text class="center" x="95" y="117">request counted</text>
<rect class="box decision" x="200" y="85" width="120" height="40" rx="20"/>
<text class="title center" x="260" y="110">Stream breaks</text>
<path class="edge" d="M170,105 H198" marker-end="url(#arrow)"/>
<rect class="box core" x="540" y="16" width="260" height="50" rx="6"/>
<text x="552" y="37">The router picks a retry or fallback,</text>
<text x="552" y="54">the caller sees one stream</text>
<rect class="box core" x="540" y="80" width="260" height="50" rx="6"/>
<text x="552" y="101">Continue on a fallback deployment</text>
<text x="552" y="118">with the text so far as a prefill</text>
<rect class="box warn" x="540" y="144" width="260" height="36" rx="6"/>
<text x="552" y="166">The stream raises to the caller</text>
<path class="edge core" d="M320,105 H350 V41 H538" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="444" y="35">before any content</text>
<path class="edge core" d="M350,105 H538" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="444" y="99">after text, continuation on</text>
<path class="edge warn" d="M350,105 V162 H538" marker-end="url(#arrow-warn)"/>
<text class="edge-label center" x="444" y="156">after content, continuation off</text>
<rect class="box" x="20" y="196" width="150" height="36" rx="6"/>
<text class="center" x="95" y="218">Stream ends</text>
<path class="edge" d="M95,129 V194" marker-end="url(#arrow)"/>
<rect class="box ok" x="200" y="196" width="320" height="36" rx="6"/>
<text x="212" y="218">Success is recorded with the final usage</text>
<path class="edge" d="M170,214 H198" marker-end="url(#arrow)"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 5.2 KiB

117
docs/architecture/router/img/reliability-6.svg Normal file
View file

@ -0,0 +1,117 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 820 610" role="img" aria-labelledby="t">
<title id="t">When a failure cools a deployment down</title>
<desc>Decision flow of the failure handler: skip checks, whether the failure counts, the allowed fails counter, and the error-rate rule over the last 60 seconds.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="819" height="609" rx="8"/>
<rect class="box warn" x="85" y="20" width="200" height="36" rx="6"/>
<text class="center" x="185" y="42.5">failure on deployment d</text>
<rect class="box decision" x="20" y="76" width="330" height="50" rx="6"/>
<text class="center" x="185" y="96">cooldowns disabled, or a failure</text>
<text class="center" x="185" y="114">outside d (selection, caller credentials, ...)?</text>
<rect class="box decision" x="75" y="152" width="220" height="36" rx="6"/>
<text class="center" x="185" y="174.5">does this failure count?</text>
<rect class="box decision" x="25" y="234" width="320" height="50" rx="6"/>
<text class="center" x="185" y="254"><tspan class="mono">allowed_fails</tspan> or a policy set</text>
<text class="center" x="185" y="272">(router or deployment)?</text>
<rect class="box store" x="460" y="226" width="220" height="66" rx="6"/>
<text class="center" x="570" y="246">increment</text>
<text class="mono center" x="570" y="263">deployment:d:allowed_fails</text>
<text class="small muted center" x="570" y="280">(TTL = cooldown time)</text>
<rect class="box decision" x="475" y="316" width="190" height="36" rx="6"/>
<text class="center" x="570" y="338.5">count &gt; allowed fails?</text>
<rect class="box decision" x="105" y="380" width="160" height="210" rx="6"/>
<text class="center" x="185" y="481">this process's</text>
<text class="center" x="185" y="498">last 60s for d</text>
<rect class="box warn" x="500" y="380" width="140" height="170" rx="6"/>
<text class="title center" x="570" y="469.5">cool down</text>
<rect class="box ok" x="700" y="76" width="100" height="514" rx="6"/>
<text class="title center" x="750" y="337.5">no cooldown</text>
<!-- spine -->
<path class="edge" d="M185,56 V74" marker-end="url(#arrow)"/>
<path class="edge" d="M185,126 V150" marker-end="url(#arrow)"/>
<text class="edge-label" x="193" y="142">no</text>
<path class="edge" d="M185,188 V232" marker-end="url(#arrow)"/>
<text class="edge-label" x="193" y="214">429, 401, 402, 404, 408, any 5xx</text>
<path class="edge" d="M185,284 V378" marker-end="url(#arrow)"/>
<text class="edge-label" x="193" y="334">no</text>
<!-- to no cooldown -->
<path class="edge" d="M350,101 H698" marker-end="url(#arrow)"/>
<text class="edge-label" x="370" y="94">yes</text>
<path class="edge" d="M295,170 H698" marker-end="url(#arrow)"/>
<text class="edge-label" x="310" y="149">connection error,</text>
<text class="edge-label" x="310" y="163">or 4xx other than 401, 402, 404, 408, 429</text>
<!-- allowed fails branch -->
<path class="edge" d="M345,259 H458" marker-end="url(#arrow)"/>
<text class="edge-label" x="360" y="252">yes</text>
<path class="edge" d="M570,292 V314" marker-end="url(#arrow)"/>
<path class="edge" d="M665,334 H698" marker-end="url(#arrow)"/>
<text class="edge-label center" x="682" y="327">no</text>
<path class="edge" d="M570,352 V378" marker-end="url(#arrow)"/>
<text class="edge-label" x="578" y="370">yes</text>
<!-- error-rate rule -->
<path class="edge" d="M265,405 H498" marker-end="url(#arrow)"/>
<text class="edge-label" x="278" y="398">429 and group has 2+ deployments</text>
<path class="edge" d="M265,445 H498" marker-end="url(#arrow)"/>
<text class="edge-label" x="278" y="438">100% failed and 1000+ requests</text>
<path class="edge" d="M265,495 H498" marker-end="url(#arrow)"/>
<text class="edge-label" x="278" y="474">over 50% failed, 5+ requests,</text>
<text class="edge-label" x="278" y="488">group has 2+ deployments</text>
<path class="edge" d="M265,535 H498" marker-end="url(#arrow)"/>
<text class="edge-label" x="278" y="528">status should not be retried</text>
<path class="edge" d="M265,575 H698" marker-end="url(#arrow)"/>
<text class="edge-label" x="278" y="568">otherwise</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 6.9 KiB

69
docs/architecture/router/img/request-lifecycle-1.svg Normal file
View file

@ -0,0 +1,69 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 870 224" role="img" aria-labelledby="t">
<title id="t">The shape of a router request</title>
<desc>Router.acompletion goes through fallbacks, retries and a single attempt, which calls litellm.acompletion and then the provider.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="869" height="223" rx="8"/>
<rect class="box host" x="20" y="20" width="190" height="72" rx="6"/>
<text class="mono title center" x="115" y="51">Router.acompletion</text>
<text class="small muted center" x="115" y="69">facade</text>
<rect class="box core" x="250" y="20" width="354" height="72" rx="6"/>
<text class="title center" x="427" y="51">Fallbacks</text>
<text class="small muted center" x="427" y="69">move to another model group when retries are spent</text>
<rect class="box core" x="644" y="20" width="206" height="72" rx="6"/>
<text class="title center" x="747" y="43">Retries</text>
<text class="small muted center" x="747" y="61">loop in the same model group,</text>
<text class="small muted center" x="747" y="76">with backoff</text>
<rect class="box core" x="644" y="132" width="206" height="72" rx="6"/>
<text class="title center" x="747" y="155">Attempt</text>
<text class="small muted center" x="747" y="173">select, build the deployment's</text>
<text class="small muted center" x="747" y="188">kwargs, call</text>
<rect class="box host" x="212" y="132" width="400" height="72" rx="6"/>
<text class="mono center" x="412" y="172">litellm.acompletion(**deployment params, **request)</text>
<rect class="box" x="20" y="132" width="160" height="72" rx="6"/>
<text class="title center" x="100" y="172">Provider HTTP call</text>
<path class="edge" d="M210,56 H248" marker-end="url(#arrow)"/>
<path class="edge" d="M604,56 H642" marker-end="url(#arrow)"/>
<path class="edge" d="M747,92 V130" marker-end="url(#arrow)"/>
<path class="edge" d="M644,168 H614" marker-end="url(#arrow)"/>
<path class="edge" d="M212,168 H182" marker-end="url(#arrow)"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.8 KiB

161
docs/architecture/router/img/request-lifecycle-2.svg Normal file
View file

@ -0,0 +1,161 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 880 800" role="img" aria-labelledby="t">
<title id="t">Request lifecycle, step by step</title>
<desc>Sequence of one acompletion call through the RustCore driver, router-core, the Python host, DualCache and the provider.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="879" height="799" rx="8"/>
<line class="lifeline" x1="48" y1="54" x2="48" y2="788"/>
<line class="lifeline" x1="236" y1="54" x2="236" y2="788"/>
<line class="lifeline" x1="462" y1="54" x2="462" y2="788"/>
<line class="lifeline" x1="604" y1="54" x2="604" y2="788"/>
<line class="lifeline" x1="714" y1="54" x2="714" y2="788"/>
<line class="lifeline" x1="832" y1="54" x2="832" y2="788"/>
<rect class="box" x="6" y="12" width="84" height="42" rx="6"/>
<text class="title center" x="48" y="29">Router</text>
<text class="title center" x="48" y="44">facade</text>
<rect class="box host" x="188" y="12" width="96" height="42" rx="6"/>
<text class="title center" x="236" y="29"><tspan class="mono">RustCore</tspan></text>
<text class="title center" x="236" y="44">driver</text>
<rect class="box core" x="407" y="12" width="110" height="42" rx="6"/>
<text class="title mono center" x="462" y="37.5">router-core</text>
<rect class="box host" x="546" y="12" width="116" height="42" rx="6"/>
<text class="title mono center" x="604" y="29">PythonHost +</text>
<text class="title mono center" x="604" y="44">router_rust</text>
<rect class="box store" x="670" y="12" width="88" height="42" rx="6"/>
<text class="title mono center" x="714" y="37.5">DualCache</text>
<rect class="box" x="792" y="12" width="80" height="42" rx="6"/>
<text class="title center" x="832" y="37.5">Provider</text>
<path class="edge" d="M57,97 H235" marker-end="url(#arrow)"/>
<text class="edge-label center" x="146" y="76"><tspan class="mono">acompletion(model,</tspan></text>
<text class="edge-label center" x="146" y="90"><tspan class="mono">messages)</tspan></text>
<circle class="step" cx="48" cy="97" r="8"/>
<text class="step-text" x="48" y="100.8">1</text>
<path class="edge" d="M245,128 H461" marker-end="url(#arrow)"/>
<text class="edge-label center" x="353" y="121"><tspan class="mono">start("acompletion", request)</tspan></text>
<circle class="step" cx="236" cy="128" r="8"/>
<text class="step-text" x="236" y="131.8">2</text>
<path class="edge core dashed" d="M453,172 H237" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="345" y="151"><tspan class="mono">pre_route</tspan> step</text>
<text class="edge-label center" x="345" y="165">(sub-routers, custom strategy)</text>
<circle class="step" cx="462" cy="172" r="8"/>
<text class="step-text" x="462" y="175.8">3</text>
<path class="edge" d="M245,203 H603" marker-end="url(#arrow)"/>
<text class="edge-label center" x="424" y="196">run them</text>
<circle class="step" cx="236" cy="203" r="8"/>
<text class="step-text" x="236" y="206.8">4</text>
<path class="edge dashed" d="M595,234 H237" marker-end="url(#arrow)"/>
<text class="edge-label center" x="416" y="227">routed group, or none</text>
<circle class="step" cx="604" cy="234" r="8"/>
<text class="step-text" x="604" y="237.8">5</text>
<path class="edge core" d="M471,257 H486 V271 H464" marker-end="url(#arrow-core)"/>
<text class="edge-label" x="494" y="268">resolve the name, team, PTU and access filters</text>
<circle class="step" cx="462" cy="257" r="8"/>
<text class="step-text" x="462" y="260.8">6</text>
<path class="edge core dashed" d="M453,291 H237" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="345" y="284">cache step: read cooldowns</text>
<circle class="step" cx="462" cy="291" r="8"/>
<text class="step-text" x="462" y="294.8">7</text>
<path class="edge" d="M245,322 H713" marker-end="url(#arrow)"/>
<text class="edge-label center" x="479" y="315">batch get</text>
<circle class="step" cx="236" cy="322" r="8"/>
<text class="step-text" x="236" y="325.8">8</text>
<path class="edge core" d="M471,345 H486 V359 H464" marker-end="url(#arrow-core)"/>
<text class="edge-label" x="494" y="356">drop cooled-down and blocked deployments</text>
<circle class="step" cx="462" cy="345" r="8"/>
<text class="step-text" x="462" y="348.8">9</text>
<path class="edge core dashed" d="M453,392 H237" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="345" y="371">select steps: callback filters,</text>
<text class="edge-label center" x="345" y="385">plugins, tags</text>
<circle class="step" cx="462" cy="392" r="8"/>
<text class="step-text" x="462" y="395.8">10</text>
<path class="edge" d="M245,423 H603" marker-end="url(#arrow)"/>
<text class="edge-label center" x="424" y="416">run the Python filters</text>
<circle class="step" cx="236" cy="423" r="8"/>
<text class="step-text" x="236" y="426.8">11</text>
<path class="edge core" d="M471,446 H486 V460 H464" marker-end="url(#arrow-core)"/>
<text class="edge-label" x="494" y="457">pre-call checks, order filter, retry skips</text>
<circle class="step" cx="462" cy="446" r="8"/>
<text class="step-text" x="462" y="449.8">12</text>
<path class="edge core" d="M471,472 H486 V486 H464" marker-end="url(#arrow-core)"/>
<text class="edge-label" x="494" y="476">simple-shuffle picks</text>
<text class="edge-label" x="494" y="490">(or a select step for other strategies)</text>
<circle class="step" cx="462" cy="472" r="8"/>
<text class="step-text" x="462" y="475.8">13</text>
<path class="edge core dashed" d="M453,512 H237" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="345" y="505">attempt step on deployment a</text>
<circle class="step" cx="462" cy="512" r="8"/>
<text class="step-text" x="462" y="515.8">14</text>
<path class="edge" d="M245,543 H603" marker-end="url(#arrow)"/>
<text class="edge-label center" x="424" y="536"><tspan class="mono">router_attempt</tspan> builds kwargs, <tspan class="mono">invoke("acompletion")</tspan></text>
<circle class="step" cx="236" cy="543" r="8"/>
<text class="step-text" x="236" y="546.8">15</text>
<path class="edge" d="M613,574 H831" marker-end="url(#arrow)"/>
<text class="edge-label center" x="722" y="567"><tspan class="mono">litellm.acompletion(**kwargs)</tspan></text>
<circle class="step" cx="604" cy="574" r="8"/>
<text class="step-text" x="604" y="577.8">16</text>
<path class="edge dashed" d="M823,605 H605" marker-end="url(#arrow)"/>
<text class="edge-label center" x="714" y="598">response or error</text>
<circle class="step" cx="832" cy="605" r="8"/>
<text class="step-text" x="832" y="608.8">17</text>
<path class="edge dashed" d="M595,636 H237" marker-end="url(#arrow)"/>
<text class="edge-label center" x="416" y="629">output, or a typed failure</text>
<circle class="step" cx="604" cy="636" r="8"/>
<text class="step-text" x="604" y="639.8">18</text>
<path class="edge" d="M245,667 H461" marker-end="url(#arrow)"/>
<text class="edge-label center" x="353" y="660"><tspan class="mono">next(outcome)</tspan></text>
<circle class="step" cx="236" cy="667" r="8"/>
<text class="step-text" x="236" y="670.8">19</text>
<path class="edge core dashed" d="M453,711 H237" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="345" y="690">cache steps: usage, or fail counts</text>
<text class="edge-label center" x="345" y="704">and maybe a cooldown</text>
<circle class="step" cx="462" cy="711" r="8"/>
<text class="step-text" x="462" y="714.8">20</text>
<path class="edge core dashed" d="M453,742 H237" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="345" y="735">done</text>
<circle class="step" cx="462" cy="742" r="8"/>
<text class="step-text" x="462" y="745.8">21</text>
<path class="edge dashed" d="M227,773 H49" marker-end="url(#arrow)"/>
<text class="edge-label center" x="138" y="766">response with routing headers</text>
<circle class="step" cx="236" cy="773" r="8"/>
<text class="step-text" x="236" y="776.8">22</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 11 KiB

70
docs/architecture/router/img/route-your-first-requests-1.svg Normal file
View file

@ -0,0 +1,70 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 832 76" role="img" aria-labelledby="t">
<title id="t">Tutorial steps: route your first requests</title>
<desc>Five tutorial steps in order: build a Router, send requests, look inside, break a deployment, break the group.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="831" height="75" rx="8"/>
<rect class="box" x="14" y="16" width="132" height="44" rx="6"/>
<circle class="step" cx="31" cy="38" r="9"/>
<text class="step-text" x="31" y="41.8">1</text>
<text class="title" x="46" y="42.5">Build a Router</text>
<path class="edge" d="M146,38 H170" marker-end="url(#arrow)"/>
<rect class="box" x="172" y="16" width="136" height="44" rx="6"/>
<circle class="step" cx="189" cy="38" r="9"/>
<text class="step-text" x="189" y="41.8">2</text>
<text class="title" x="204" y="42.5">Send requests</text>
<path class="edge" d="M308,38 H332" marker-end="url(#arrow)"/>
<rect class="box" x="334" y="16" width="117" height="44" rx="6"/>
<circle class="step" cx="351" cy="38" r="9"/>
<text class="step-text" x="351" y="41.8">3</text>
<text class="title" x="366" y="42.5">Look inside</text>
<path class="edge" d="M451,38 H475" marker-end="url(#arrow)"/>
<rect class="box" x="477" y="16" width="170" height="44" rx="6"/>
<circle class="step" cx="494" cy="38" r="9"/>
<text class="step-text" x="494" y="41.8">4</text>
<text class="title" x="509" y="42.5">Break a deployment</text>
<path class="edge" d="M647,38 H671" marker-end="url(#arrow)"/>
<rect class="box" x="673" y="16" width="145" height="44" rx="6"/>
<circle class="step" cx="690" cy="38" r="9"/>
<text class="step-text" x="690" y="41.8">5</text>
<text class="title" x="705" y="42.5">Break the group</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.6 KiB

86
docs/architecture/router/img/route-your-first-requests-2.svg Normal file
View file

@ -0,0 +1,86 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 750 360" role="img" aria-labelledby="t">
<title id="t">A 429 on deployment a: retry on b and cooldown for a</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="749" height="359" rx="8"/>
<line class="lifeline" x1="100" y1="58" x2="100" y2="312"/>
<line class="lifeline" x1="290" y1="58" x2="290" y2="312"/>
<line class="lifeline" x1="470" y1="58" x2="470" y2="312"/>
<line class="lifeline" x1="650" y1="58" x2="650" y2="312"/>
<rect class="box core" x="30" y="18" width="140" height="40" rx="6"/>
<text class="title center" x="100" y="43">Router</text>
<rect class="box" x="220" y="18" width="140" height="40" rx="6"/>
<text class="title center" x="290" y="43">deployment a</text>
<rect class="box" x="400" y="18" width="140" height="40" rx="6"/>
<text class="title center" x="470" y="43">deployment b</text>
<rect class="box store" x="580" y="18" width="140" height="40" rx="6"/>
<text class="title center" x="650" y="43">routing cache</text>
<path class="edge core" d="M100,92 H289" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="195" y="85">attempt 1</text>
<circle class="step" cx="80" cy="92" r="9"/><text class="step-text" x="80" y="96">1</text>
<path class="edge warn dashed" d="M290,128 H101" marker-end="url(#arrow-warn)"/>
<text class="edge-label center" x="195" y="121">429 <tspan class="mono">RateLimitError</tspan></text>
<circle class="step" cx="80" cy="128" r="9"/><text class="step-text" x="80" y="132">2</text>
<path class="edge core" d="M100,168 H649" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="375" y="161"><tspan class="mono">record_failure</tspan> sets <tspan class="mono">deployment:a:cooldown</tspan> (TTL 5s)</text>
<circle class="step" cx="80" cy="168" r="9"/><text class="step-text" x="80" y="172">3</text>
<path class="edge core" d="M100,196 H136 V222 H101" marker-end="url(#arrow-core)"/>
<text class="edge-label" x="146" y="213">retry 1 of 1, skip a</text>
<circle class="step" cx="80" cy="209" r="9"/><text class="step-text" x="80" y="213">4</text>
<path class="edge core" d="M100,256 H469" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="380" y="249">attempt 2</text>
<circle class="step" cx="80" cy="256" r="9"/><text class="step-text" x="80" y="260">5</text>
<path class="edge dashed" d="M470,292 H101" marker-end="url(#arrow)"/>
<text class="edge-label center" x="380" y="285">200</text>
<circle class="step" cx="80" cy="292" r="9"/><text class="step-text" x="80" y="296">6</text>
<rect class="note" x="30" y="312" width="690" height="30" rx="5"/>
<text class="center" x="375" y="331">Later requests read the cooldown and only see b</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 5.9 KiB

65
docs/architecture/router/img/route-your-first-requests-3.svg Normal file
View file

@ -0,0 +1,65 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 828 188" role="img" aria-labelledby="t">
<title id="t">Retries spent in chat, fallback to backup</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="827" height="187" rx="8"/>
<rect class="box" x="20" y="72" width="204" height="44" rx="6"/>
<text class="mono center" x="122" y="98">acompletion(model='chat')</text>
<rect class="box core" x="264" y="72" width="126" height="44" rx="6"/>
<text class="title center" x="327" y="98">Fallback walk</text>
<rect class="box core" x="450" y="20" width="166" height="44" rx="6"/>
<text class="title center" x="533" y="46">Retry loop in 'chat'</text>
<rect class="box core" x="450" y="124" width="166" height="44" rx="6"/>
<text class="title center" x="533" y="150">Retry loop in 'backup'</text>
<rect class="box ok" x="672" y="124" width="136" height="44" rx="6"/>
<text class="mono center" x="740" y="150">'hi from backup'</text>
<path class="edge" d="M224,94 H262" marker-end="url(#arrow)"/>
<path class="edge core" d="M320,72 V42 H448" marker-end="url(#arrow-core)"/>
<path class="edge warn" d="M533,64 V94 H392" marker-end="url(#arrow-warn)"/>
<text class="edge-label" x="541" y="86">a: 429, b: 429</text>
<path class="edge core" d="M320,116 V146 H448" marker-end="url(#arrow-core)"/>
<text class="edge-label end" x="312" y="150">fallbacks: chat to backup</text>
<path class="edge" d="M616,146 H670" marker-end="url(#arrow)"/>
<text class="edge-label center" x="643" y="139">c: 200</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.7 KiB

62
docs/architecture/router/img/router-overview.svg Normal file
View file

@ -0,0 +1,62 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 780 238" role="img" aria-labelledby="t">
<title id="t">Router overview</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="779" height="237" rx="8"/>
<rect class="box" x="20" y="20" width="190" height="56" rx="6"/>
<text class="title center" x="115" y="52">Proxy route or SDK caller</text>
<rect class="box store" x="260" y="20" width="210" height="56" rx="6"/>
<text class="title center" x="365" y="44">Routing cache</text>
<text class="small muted center" x="365" y="61">memory + Redis</text>
<rect class="box" x="530" y="20" width="230" height="56" rx="6"/>
<text class="title center" x="645" y="52">Provider API</text>
<rect class="box" x="20" y="158" width="740" height="60" rx="6"/>
<text class="mono title center" x="390" y="182">litellm.Router</text>
<text class="small muted center" x="390" y="201">picks a deployment: names, filters, strategy, retries, fallbacks, cooldowns</text>
<path class="edge" d="M115,76 V156" marker-end="url(#arrow)"/>
<text class="edge-label mono" x="125" y="121">acompletion(model='gpt-4o')</text>
<path class="edge" d="M365,78 V156" marker-start="url(#arrow)" marker-end="url(#arrow)"/>
<text class="edge-label" x="375" y="112">cooldowns, usage,</text>
<text class="edge-label" x="375" y="127">fail counts</text>
<path class="edge" d="M700,158 V78" marker-end="url(#arrow)"/>
<text class="edge-label mono end" x="690" y="121">litellm.acompletion(...)</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.3 KiB

127
docs/architecture/router/img/selection-1.svg Normal file
View file

@ -0,0 +1,127 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 820 630" role="img" aria-labelledby="t">
<title id="t">The filter chain of a pick</title>
<desc>A requested name passes pre-routing and resolution, then a chain of filters removes candidates; if anything is left a strategy picks one, otherwise the pick fails with no deployments available.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="819" height="629" rx="8"/>
<rect class="box " x="20" y="20" width="240" height="82" rx="6"/>
<text class="title" x="32" y="66.0">requested name + request</text>
<rect class="box" x="290" y="20" width="240" height="82" rx="6"/>
<text class="title" x="302" y="42.5">Pre-routing</text>
<text class="small" x="302" y="59.5">sub-routers, routing plugins,</text>
<text class="small" x="302" y="74.5">session binding</text>
<text class="small" x="302" y="89.5">may route to another group</text>
<rect class="box" x="560" y="20" width="240" height="82" rx="6"/>
<text class="title" x="572" y="50.0">Resolve the name</text>
<text class="small" x="572" y="67.0">see How a name finds</text>
<text class="small" x="572" y="82.0">its deployments</text>
<path class="edge" d="M260,61.0 H288" marker-end="url(#arrow)"/>
<path class="edge" d="M530,61.0 H558" marker-end="url(#arrow)"/>
<path class="edge" d="M680.0,102 V128" marker-end="url(#arrow)"/>
<rect class="box" x="560" y="130" width="240" height="68" rx="6"/>
<text class="title" x="572" y="152.5">Team, PTU share and</text>
<text class="title" x="572" y="168.5">access filters</text>
<text class="small" x="572" y="185.5">see Teams and access</text>
<rect class="box" x="290" y="130" width="240" height="68" rx="6"/>
<text class="title" x="302" y="153.0">Route capability filter</text>
<text class="small" x="302" y="170.0">skips providers that cannot</text>
<text class="small" x="302" y="185.0">serve this call</text>
<rect class="box" x="20" y="130" width="240" height="68" rx="6"/>
<text class="title" x="32" y="153.0">Health check filter</text>
<text class="small" x="32" y="170.0">only with</text>
<text class="mono small" x="32" y="185.0">enable_health_check_routing</text>
<path class="edge" d="M560,164.0 H532" marker-end="url(#arrow)"/>
<path class="edge" d="M290,164.0 H262" marker-end="url(#arrow)"/>
<path class="edge" d="M140.0,198 V224" marker-end="url(#arrow)"/>
<rect class="box" x="20" y="226" width="240" height="68" rx="6"/>
<text class="title" x="32" y="249.0">Cooldown filter</text>
<text class="small" x="32" y="266.0">drops ids with a live</text>
<text class="small" x="32" y="281.0">cooldown key</text>
<rect class="box" x="290" y="226" width="240" height="68" rx="6"/>
<text class="title" x="302" y="249.0">Blocked filter</text>
<text class="mono small" x="302" y="266.0">model_info.blocked</text>
<text class="small" x="302" y="281.0">set by an admin</text>
<rect class="box" x="560" y="226" width="240" height="68" rx="6"/>
<text class="title" x="572" y="256.5">Callback filters</text>
<text class="small" x="572" y="273.5">budget limiter, affinity, ...</text>
<path class="edge" d="M260,260.0 H288" marker-end="url(#arrow)"/>
<path class="edge" d="M530,260.0 H558" marker-end="url(#arrow)"/>
<path class="edge" d="M680.0,294 V320" marker-end="url(#arrow)"/>
<rect class="box" x="560" y="322" width="240" height="68" rx="6"/>
<text class="title" x="572" y="345.0">Pre-call checks</text>
<text class="small" x="572" y="362.0">context window, region, params</text>
<text class="small raw" x="572" y="377.0">only with <tspan class="mono small">enable_pre_call_checks</tspan></text>
<rect class="box" x="290" y="322" width="240" height="68" rx="6"/>
<text class="title" x="302" y="352.5">Tag filter</text>
<text class="small" x="302" y="369.5">only with tag filtering on</text>
<rect class="box" x="20" y="322" width="240" height="68" rx="6"/>
<text class="title" x="32" y="361.0">Routing plugin candidates</text>
<path class="edge" d="M560,356.0 H532" marker-end="url(#arrow)"/>
<path class="edge" d="M290,356.0 H262" marker-end="url(#arrow)"/>
<path class="edge" d="M140.0,390 V416" marker-end="url(#arrow)"/>
<rect class="box" x="20" y="418" width="240" height="68" rx="6"/>
<text class="title" x="32" y="441.0">Text-only filter</text>
<text class="small" x="32" y="458.0">an image request skips</text>
<text class="small" x="32" y="473.0">deployments without vision</text>
<rect class="box" x="290" y="418" width="240" height="68" rx="6"/>
<text class="title" x="302" y="448.5">Order filter</text>
<text class="small" x="302" y="465.5">the lowest order left</text>
<rect class="box" x="560" y="418" width="240" height="68" rx="6"/>
<text class="title" x="572" y="449.0">Excluded and</text>
<text class="title" x="572" y="465.0">retry-skipped ids</text>
<path class="edge" d="M260,452.0 H288" marker-end="url(#arrow)"/>
<path class="edge" d="M530,452.0 H558" marker-end="url(#arrow)"/>
<path class="edge" d="M680.0,486 V510" marker-end="url(#arrow)"/>
<path class="box decision" d="M610,550 L680.0,512.0 L750,550 L680.0,588.0 Z"/>
<text class="title center" x="680.0" y="554.5">anything left?</text>
<rect class="box warn" x="330" y="500" width="220" height="48" rx="6"/>
<text x="342" y="520">no deployments available,</text>
<text class="small" x="342" y="537">with the soonest cooldown expiry</text>
<rect class="box ok" x="330" y="562" width="220" height="48" rx="6"/>
<text class="title" x="342" y="582">Strategy picks one</text>
<text class="small" x="342" y="599">see Strategies</text>
<path class="edge warn" d="M625,541.9 V524 H552" marker-end="url(#arrow-warn)"/>
<path class="edge" d="M625,558.1 V586 H552" marker-end="url(#arrow)"/>
<text class="edge-label" x="588" y="518">no</text>
<text class="edge-label" x="588" y="602">yes</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 8.7 KiB

65
docs/architecture/router/img/selection-2.svg Normal file
View file

@ -0,0 +1,65 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 600 232" role="img" aria-labelledby="t">
<title id="t">Priority with order in a model group</title>
<desc>A request picks normally between ptu-eastus and ptu-westus at order 1, and reaches paygo at order 2 only after order 1 failed.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="599" height="231" rx="8"/>
<rect class="group" x="360" y="24" width="220" height="190" rx="8"/>
<text class="group-label" x="376" y="46">MODEL GROUP</text>
<text class="mono small muted" x="468" y="46">gpt-4o</text>
<rect class="box" x="380" y="60" width="180" height="40" rx="6"/>
<text class="mono" x="394" y="84">ptu-eastus</text>
<text class="small muted end" x="546" y="84">order 1</text>
<rect class="box" x="380" y="110" width="180" height="40" rx="6"/>
<text class="mono" x="394" y="134">ptu-westus</text>
<text class="small muted end" x="546" y="134">order 1</text>
<rect class="box" x="380" y="160" width="180" height="40" rx="6"/>
<text class="mono" x="394" y="184">paygo</text>
<text class="small muted end" x="546" y="184">order 2</text>
<rect class="box" x="20" y="86" width="110" height="40" rx="6"/>
<text class="title center" x="75" y="110">Request</text>
<path class="edge" d="M130,106 H250 V80 H378" marker-end="url(#arrow)"/>
<path class="edge" d="M250,106 V130 H378" marker-end="url(#arrow)"/>
<text class="edge-label center" x="190" y="98">normal pick</text>
<path class="edge dashed" d="M75,126 V180 H378" marker-end="url(#arrow)"/>
<text class="edge-label" x="90" y="172">only after order 1 failed</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.4 KiB

75
docs/architecture/router/img/shared-state-1.svg Normal file
View file

@ -0,0 +1,75 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 768 214" role="img" aria-labelledby="t">
<title id="t">Shared state across proxy workers</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="767" height="213" rx="8"/>
<rect class="group" x="20" y="14" width="232" height="92" rx="8"/>
<text class="group-label" x="32" y="32">PROXY WORKER 1</text>
<rect class="box core" x="32" y="44" width="76" height="48" rx="6"/>
<text class="title center" x="70" y="72">Router</text>
<rect class="box store" x="136" y="44" width="104" height="48" rx="6"/>
<text class="mono title center" x="188" y="64">DualCache</text>
<text class="small muted center" x="188" y="81">in-memory tier</text>
<path class="edge" d="M108,68 H135" marker-end="url(#arrow)"/>
<path class="edge" d="M188,93 V145" marker-start="url(#arrow)" marker-end="url(#arrow)"/>
<rect class="group" x="268" y="14" width="232" height="92" rx="8"/>
<text class="group-label" x="280" y="32">PROXY WORKER 2</text>
<rect class="box core" x="280" y="44" width="76" height="48" rx="6"/>
<text class="title center" x="318" y="72">Router</text>
<rect class="box store" x="384" y="44" width="104" height="48" rx="6"/>
<text class="mono title center" x="436" y="64">DualCache</text>
<text class="small muted center" x="436" y="81">in-memory tier</text>
<path class="edge" d="M356,68 H383" marker-end="url(#arrow)"/>
<path class="edge" d="M436,93 V145" marker-start="url(#arrow)" marker-end="url(#arrow)"/>
<rect class="group" x="516" y="14" width="232" height="92" rx="8"/>
<text class="group-label" x="528" y="32">PROXY WORKER 3</text>
<rect class="box core" x="528" y="44" width="76" height="48" rx="6"/>
<text class="title center" x="566" y="72">Router</text>
<rect class="box store" x="632" y="44" width="104" height="48" rx="6"/>
<text class="mono title center" x="684" y="64">DualCache</text>
<text class="small muted center" x="684" y="81">in-memory tier</text>
<path class="edge" d="M604,68 H631" marker-end="url(#arrow)"/>
<path class="edge" d="M684,93 V145" marker-start="url(#arrow)" marker-end="url(#arrow)"/>
<rect class="box store" x="32" y="146" width="704" height="50" rx="6"/>
<text class="title center" x="384" y="166">Redis</text>
<text class="small center" x="384" y="184">cooldowns, <tspan class="mono">allowed_fails</tspan> counters, tpm / rpm per minute, strategy state</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 5.2 KiB

92
docs/architecture/router/img/shared-state-2.svg Normal file
View file

@ -0,0 +1,92 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 376" role="img" aria-labelledby="t">
<title id="t">Cooldown shared between workers through Redis</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
<marker id="arrow-warn" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead warn" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="799" height="375" rx="8"/>
<line class="lifeline" x1="110" y1="60" x2="110" y2="356"/>
<line class="lifeline" x1="400" y1="60" x2="400" y2="356"/>
<line class="lifeline" x1="690" y1="60" x2="690" y2="356"/>
<rect class="box" x="55" y="20" width="110" height="40" rx="6"/>
<text class="title center" x="110" y="44.5">Worker 1</text>
<rect class="box store" x="345" y="20" width="110" height="40" rx="6"/>
<text class="title center" x="400" y="44.5">Redis</text>
<rect class="box" x="635" y="20" width="110" height="40" rx="6"/>
<text class="title center" x="690" y="44.5">Worker 2</text>
<path class="edge warn" d="M110,84 H146 V102 H112" marker-end="url(#arrow-warn)"/>
<circle class="step" cx="92" cy="93" r="8"/>
<text class="step-text" x="92" y="96.5">1</text>
<text class="edge-label" x="156" y="97">deployment d returns 429</text>
<path class="edge" d="M110,142 H398" marker-end="url(#arrow)"/>
<circle class="step" cx="92" cy="142" r="8"/>
<text class="step-text" x="92" y="145.5">2</text>
<text class="edge-label center" x="255" y="133"><tspan class="mono">SET deployment:d:cooldown</tspan> (TTL 5s)</text>
<path class="edge" d="M110,166 H146 V184 H112" marker-end="url(#arrow)"/>
<circle class="step" cx="92" cy="175" r="8"/>
<text class="step-text" x="92" y="178.5">3</text>
<text class="edge-label" x="156" y="179">memory copy, d filtered here at once</text>
<rect class="note" x="610" y="158" width="160" height="42" rx="4"/>
<text class="small center" x="690" y="175">still has no local</text>
<text class="small center" x="690" y="191">cooldown copy for d</text>
<path class="edge" d="M690,236 H402" marker-end="url(#arrow)"/>
<circle class="step" cx="708" cy="236" r="8"/>
<text class="step-text" x="708" y="239.5">4</text>
<text class="edge-label center" x="545" y="227">cooldown read (at most every 1s)</text>
<path class="edge dashed" d="M400,274 H688" marker-end="url(#arrow)"/>
<circle class="step" cx="382" cy="274" r="8"/>
<text class="step-text" x="382" y="277.5">5</text>
<text class="edge-label center" x="545" y="265">d is cooling</text>
<path class="edge" d="M690,298 H654 V316 H688" marker-end="url(#arrow)"/>
<circle class="step" cx="708" cy="307" r="8"/>
<text class="step-text" x="708" y="310.5">6</text>
<text class="edge-label end" x="644" y="311">d filtered here too</text>
<rect class="note" x="50" y="334" width="700" height="26" rx="4"/>
<text class="small center" x="400" y="351">5s later the key expires on both tiers, d is eligible again</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 5.7 KiB

68
docs/architecture/router/img/state-and-cache-keys-1.svg Normal file
View file

@ -0,0 +1,68 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 850 208" role="img" aria-labelledby="t">
<title id="t">Routing state: shared and process-local keys</title>
<desc>Six cache keys shared through Redis when configured, and two kinds of state kept in this process only.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="849" height="207" rx="8"/>
<rect class="group" x="12" y="12" width="580" height="182" rx="8"/>
<text class="group-label" x="22" y="30">SHARED THROUGH REDIS WHEN CONFIGURED</text>
<rect class="box store" x="22" y="40" width="300" height="36" rx="6"/>
<text class="mono center" x="172" y="62">deployment:{id}:cooldown</text>
<rect class="box store" x="332" y="40" width="250" height="36" rx="6"/>
<text class="mono center" x="457" y="62">deployment:{id}:allowed_fails</text>
<rect class="box store" x="22" y="86" width="300" height="52" rx="6"/>
<text class="mono center" x="172" y="108">global_router:{id}:{model}:tpm:{minute}</text>
<text class="mono center" x="172" y="124">...:rpm, :itpm, :otpm</text>
<rect class="box store" x="332" y="86" width="250" height="52" rx="6"/>
<text class="mono center" x="457" y="108">{id}:{model}:tpm:{minute}</text>
<text class="mono center" x="457" y="124">{id}:{model}:rpm:{minute}</text>
<rect class="box store" x="22" y="148" width="300" height="36" rx="6"/>
<text class="mono center" x="172" y="170">{model_group}_map</text>
<rect class="box store" x="332" y="148" width="250" height="36" rx="6"/>
<text class="mono center" x="457" y="170">{model_group}_request_count:{id}</text>
<rect class="group" x="604" y="12" width="234" height="182" rx="8"/>
<text class="group-label" x="614" y="30">THIS PROCESS ONLY</text>
<rect class="box store" x="614" y="40" width="214" height="36" rx="6"/>
<text class="mono center" x="721" y="62">{id}:successes / {id}:fails</text>
<rect class="box store" x="614" y="86" width="214" height="36" rx="6"/>
<text class=" center" x="721" y="108">registry and settings snapshot</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.8 KiB

66
docs/architecture/router/img/strategies-1.svg Normal file
View file

@ -0,0 +1,66 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 814 112" role="img" aria-labelledby="t">
<title id="t">From requested name to a picked deployment</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="813" height="111" rx="8"/>
<rect class="box" x="16" y="24" width="100" height="64" rx="6"/>
<text class="title center" x="66.0" y="52.0">request</text>
<text class="mono small center" x="66.0" y="69.0">model=smart</text>
<rect class="box host" x="146" y="24" width="104" height="64" rx="6"/>
<text class="title center" x="198.0" y="52.0">Sub-router</text>
<text class="small muted center" x="198.0" y="69.0">which group?</text>
<rect class="box core" x="350" y="24" width="150" height="64" rx="6"/>
<text class="title center" x="425.0" y="52.0">resolve and filter</text>
<text class="small muted center" x="425.0" y="69.0">the group's deployments</text>
<rect class="box decision" x="530" y="24" width="146" height="64" rx="6"/>
<text class="title center" x="603.0" y="43.5">which strategy?</text>
<text class="small muted center" x="603.0" y="60.5">request override,</text>
<text class="small muted center" x="603.0" y="77.5">routing group, router</text>
<rect class="box ok" x="706" y="24" width="92" height="64" rx="6"/>
<text class="title center" x="752.0" y="52.0">strategy</text>
<text class="title center" x="752.0" y="69.0">picks one</text>
<path class="edge" d="M116,56 H144" marker-end="url(#arrow)"/>
<path class="edge" d="M250,56 H348" marker-end="url(#arrow)"/>
<path class="edge" d="M500,56 H528" marker-end="url(#arrow)"/>
<path class="edge" d="M676,56 H704" marker-end="url(#arrow)"/>
<text class="edge-label mono center" x="300" y="48">gpt-4o-mini</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.5 KiB

65
docs/architecture/router/img/strategies-2.svg Normal file
View file

@ -0,0 +1,65 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 820 96" role="img" aria-labelledby="t">
<title id="t">Weight sources in simple-shuffle</title>
<desc>Request weights from key or team router_settings, else weight, else rpm, else tpm, else uniform random.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow-core" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead core" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="819" height="95" rx="8"/>
<rect class="box core" x="20" y="20" width="200" height="56" rx="6"/>
<rect class="box core" x="270" y="20" width="90" height="56" rx="6"/>
<rect class="box core" x="410" y="20" width="80" height="56" rx="6"/>
<rect class="box core" x="540" y="20" width="80" height="56" rx="6"/>
<rect class="box core" x="670" y="20" width="130" height="56" rx="6"/>
<text class="title center" x="120.0" y="43">Request weights</text>
<text class="small muted center" x="120.0" y="61">key or team <tspan class="mono">router_settings</tspan></text>
<text class="title mono center" x="315.0" y="52.0">weight</text>
<text class="title mono center" x="450.0" y="52.0">rpm</text>
<text class="title mono center" x="580.0" y="52.0">tpm</text>
<text class="title center" x="735.0" y="52.5">Uniform random</text>
<path class="edge core" d="M220,48.0 H268" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="245.0" y="40.0">else</text>
<path class="edge core" d="M360,48.0 H408" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="385.0" y="40.0">else</text>
<path class="edge core" d="M490,48.0 H538" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="515.0" y="40.0">else</text>
<path class="edge core" d="M620,48.0 H668" marker-end="url(#arrow-core)"/>
<text class="edge-label center" x="645.0" y="40.0">else</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.5 KiB

76
docs/architecture/router/img/strategies-3.svg Normal file
View file

@ -0,0 +1,76 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 180" role="img" aria-labelledby="t">
<title id="t">Which routing strategy applies</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="839" height="179" rx="8"/>
<rect class="box decision" x="20" y="20" width="170" height="56" rx="6"/>
<text class="title center" x="105" y="43">Request</text>
<text class="mono center" x="105" y="61">routing_strategy?</text>
<rect class="box decision" x="230" y="20" width="170" height="56" rx="6"/>
<text class="title center" x="315" y="43">Name in a</text>
<text class="title center" x="315" y="61">routing group?</text>
<rect class="box decision" x="440" y="20" width="170" height="56" rx="6"/>
<text class="title center" x="525" y="43">Router</text>
<text class="mono center" x="525" y="61">routing_strategy?</text>
<rect class="box ok" x="650" y="20" width="170" height="56" rx="6"/>
<text class="mono center" x="735" y="52">simple-shuffle</text>
<rect class="box ok" x="20" y="112" width="170" height="48" rx="6"/>
<text class="center" x="105" y="140">Use it</text>
<rect class="box ok" x="230" y="112" width="170" height="48" rx="6"/>
<text class="center" x="315" y="131">The group's</text>
<text class="mono center" x="315" y="148">routing_strategy</text>
<rect class="box ok" x="440" y="112" width="170" height="48" rx="6"/>
<text class="center" x="525" y="140">Use it</text>
<path class="edge" d="M190,48 H228" marker-end="url(#arrow)"/>
<text class="edge-label center" x="209" y="41">no</text>
<path class="edge" d="M400,48 H438" marker-end="url(#arrow)"/>
<text class="edge-label center" x="419" y="41">no</text>
<path class="edge" d="M610,48 H648" marker-end="url(#arrow)"/>
<text class="edge-label center" x="629" y="41">no</text>
<path class="edge" d="M105,76 V110" marker-end="url(#arrow)"/>
<text class="edge-label" x="113" y="97">set</text>
<path class="edge" d="M315,76 V110" marker-end="url(#arrow)"/>
<text class="edge-label" x="323" y="97">yes</text>
<path class="edge" d="M525,76 V110" marker-end="url(#arrow)"/>
<text class="edge-label" x="533" y="97">set</text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.9 KiB

64
docs/architecture/router/img/strategies-4.svg Normal file
View file

@ -0,0 +1,64 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 660 226" role="img" aria-labelledby="t">
<title id="t">Auto router sequence</title>
<desc>A request for model "smart" reaches the router, the router has the auto router "smart" classify the prompt, the model group gpt-4o-mini comes back, and the router resolves, filters and runs the strategy on it.</desc>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="659" height="225" rx="8"/>
<line class="lifeline" x1="100" y1="56" x2="100" y2="210"/>
<line class="lifeline" x1="330" y1="56" x2="330" y2="210"/>
<line class="lifeline" x1="560" y1="56" x2="560" y2="210"/>
<rect class="box" x="30" y="16" width="140" height="40" rx="6"/>
<text class="title center" x="100" y="41">Caller</text>
<rect class="box" x="260" y="16" width="140" height="40" rx="6"/>
<text class="title center" x="330" y="41">Router</text>
<rect class="box" x="475" y="16" width="170" height="40" rx="6"/>
<text class="title center" x="560" y="41">Auto router <tspan class="mono">"smart"</tspan></text>
<path class="edge" d="M101,92 H328" marker-end="url(#arrow)"/>
<text class="edge-label" x="114" y="84">request for <tspan class="mono">model="smart"</tspan></text>
<path class="edge" d="M331,124 H558" marker-end="url(#arrow)"/>
<text class="edge-label" x="344" y="116">classify the prompt</text>
<path class="edge dashed" d="M559,156 H332" marker-end="url(#arrow)"/>
<text class="edge-label" x="344" y="148">model group <tspan class="mono">gpt-4o-mini</tspan></text>
<path class="edge" d="M331,180 H370 V198 H333" marker-end="url(#arrow)"/>
<text class="edge-label" x="380" y="193">resolve, filters and strategy on <tspan class="mono">gpt-4o-mini</tspan></text>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.5 KiB

69
docs/architecture/router/img/teams-and-access-1.svg Normal file
View file

@ -0,0 +1,69 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 770 216" role="img" aria-labelledby="t">
<title id="t">Access rules that narrow the candidate deployments</title>
<style>/* litellm architecture diagrams v1, from the litellm-docs Aurora theme (src/css/custom.css). Embed verbatim in every SVG. */
svg { --ink:#272b37; --text:#3a3e4b; --muted:#646875; --line:#a3a8ba; --rule:#e3e5ee; --card:#ffffff; --canvas:#f7f8fd; --accent:#0117be; --accent-bg:#f2f4fd; --accent-line:#b9c1f0; --host:#1f6fa8; --host-bg:#eaf5fd; --store:#6b3fb8; --store-bg:#efe8fb; --warn:#b42318; --warn-bg:#fdecea; --ok:#1a7f4b; --ok-bg:#e7f6ee; }
@media (prefers-color-scheme: dark) {
svg { --ink:#ffffff; --text:#d6d4ea; --muted:#a6a4c6; --line:#5c6294; --rule:#1a1f45; --card:#0f1335; --canvas:#080b26; --accent:#9fabff; --accent-bg:#161b4a; --accent-line:#3d4790; --host:#7cc4f5; --host-bg:#0f2a44; --store:#c3a6f0; --store-bg:#23173f; --warn:#f19a9a; --warn-bg:#3a1622; --ok:#7ee2a8; --ok-bg:#0f2e25; }
}
text { font-family: Geist, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 13px; fill: var(--text); }
.mono { font-family: "Geist Mono", ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.title { font-weight: 600; fill: var(--ink); }
.small { font-size: 11.5px; }
.muted { fill: var(--muted); }
.accent { fill: var(--accent); }
.center { text-anchor: middle; }
.end { text-anchor: end; }
.canvas { fill: var(--canvas); stroke: var(--rule); stroke-width: 1; }
.box { fill: var(--card); stroke: var(--line); stroke-width: 1.2; }
.box.core { fill: var(--accent-bg); stroke: var(--accent); }
.box.host { fill: var(--host-bg); stroke: var(--host); }
.box.store { fill: var(--store-bg); stroke: var(--store); }
.box.warn { fill: var(--warn-bg); stroke: var(--warn); }
.box.ok { fill: var(--ok-bg); stroke: var(--ok); }
.box.decision { fill: var(--card); stroke: var(--ink); }
.group { fill: none; stroke: var(--line); stroke-width: 1; stroke-dasharray: 5 4; }
.group.core { stroke: var(--accent-line); }
.group-label { font-size: 11px; font-weight: 600; fill: var(--muted); letter-spacing: 0.06em; }
.edge { fill: none; stroke: var(--line); stroke-width: 1.4; }
.edge.strong { stroke: var(--ink); }
.edge.dashed { stroke-dasharray: 5 4; }
.edge.core { stroke: var(--accent); }
.edge.warn { stroke: var(--warn); }
.edge-label { font-size: 11.5px; fill: var(--muted); paint-order: stroke; stroke: var(--canvas); stroke-width: 4px; stroke-linejoin: round; }
.arrowhead { fill: var(--line); }
.arrowhead.strong { fill: var(--ink); }
.arrowhead.core { fill: var(--accent); }
.arrowhead.warn { fill: var(--warn); }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 3 4; }
.note { fill: var(--accent-bg); stroke: var(--accent-line); stroke-width: 1; }
.step { fill: var(--accent); }
.step-text { font-size: 10.5px; font-weight: 600; fill: var(--card); text-anchor: middle; }</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrowhead" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<rect class="canvas" x="0.5" y="0.5" width="769" height="215" rx="8"/>
<rect class="box" x="20" y="20" width="210" height="68" rx="6"/>
<text class="center" x="125" y="50">deployments the name</text>
<text class="center" x="125" y="68">resolved to</text>
<rect class="box core" x="280" y="20" width="210" height="68" rx="6"/>
<text class="title center" x="385" y="42">Access windows</text>
<text class="small muted center" x="385" y="59">withhold deployments reserved</text>
<text class="small muted center" x="385" y="74">for other teams right now</text>
<rect class="box core" x="540" y="20" width="210" height="68" rx="6"/>
<text class="title center" x="645" y="49">Access groups</text>
<text class="small muted center" x="645" y="66">keep a caller to its groups</text>
<rect class="box core" x="540" y="128" width="210" height="68" rx="6"/>
<text class="title center" x="645" y="150">Team deployments</text>
<text class="small muted center" x="645" y="167">a team's deployment serves</text>
<text class="small muted center" x="645" y="182">only that team</text>
<rect class="box core" x="280" y="128" width="210" height="68" rx="6"/>
<text class="title center" x="385" y="157">PTU shares</text>
<text class="small muted center" x="385" y="174">split a deployment across teams</text>
<rect class="box" x="20" y="128" width="210" height="68" rx="6"/>
<text class="center" x="125" y="167">health and load filters</text>
<path class="edge" d="M230,54 H278" marker-end="url(#arrow)"/>
<path class="edge" d="M490,54 H538" marker-end="url(#arrow)"/>
<path class="edge" d="M645,88 V126" marker-end="url(#arrow)"/>
<path class="edge" d="M540,162 H492" marker-end="url(#arrow)"/>
<path class="edge" d="M280,162 H232" marker-end="url(#arrow)"/>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 4.8 KiB

79
docs/architecture/router/reference/code-map.md Normal file
View file

@ -0,0 +1,79 @@
# Code map
Where the router's code lives. Paths are relative to the repo root.
![The router code map: the Python facade and its modules lead to the RustCore
step driver, the router_rust step modules, the python-bridge crate and the
router-core crate](../img/code-map-1.svg)
## Python
| File | Holds |
| --- | --- |
| `litellm/router.py` | `Router`, the facade. It builds the config, chooses the core and delegates every member |
| `litellm/router_api.py` | the entry point views and types, see [Entry points](entry-points.md), plus `ROUTER_INIT_ARGS` and `apply_changes` |
| `litellm/router_core.py` | the `RouterCore` protocol and `make_core` |
| `litellm/router_config.py` | `RouterConfig`, `RouterDynamicConfig`, and the only router reads of `litellm.*` settings |
| `litellm/router_deprecated.py` | `DeprecatedRouterMembers`, the old member names with a `DeprecationWarning` |
| `litellm/router_python_host.py` | `PythonHost`, the host inside a Python process: endpoint functions, sleep, dynamic config, callbacks, caches |
| `litellm/router_host.py` | `RouterHost`, what only the application can answer, and `litellm/proxy/_router_host.py` with the proxy's version |
| `litellm/router_strategy/` | the stateful strategies, `lar1`, the sub-routers, the budget limiter and tag routing. The core reaches them through the select and pre-route steps |
| `litellm/router_utils/` | helpers the host uses, such as the cooldown cache, the health state cache, pre-call checks and the response headers |
| `litellm/litellm_core_utils/` | helpers the router and other modules share, such as wildcard matching, PTU shares and the retry and fallback headers |
| `litellm/types/router.py` | `Deployment`, `RoutingGroup`, `RetryPolicy` and the other config types |
| `litellm/proxy/route_llm_request.py` | maps a proxy request to a Router endpoint method |
### `litellm/router_rust/`
Everything the core needs on the Python side.
| Path | Holds |
| --- | --- |
| `core.py` | `RustCore`, the step driver for async and sync calls |
| `steps/` | answers to the steps that need live Python objects: `selection.py` (strategies), `prerouting.py` (sub-routers, session binding, custom strategies), `precall.py` (pre-call facts), `plugins.py` (`Router(plugins=[...])`), `callbacks.py` (`CustomLogger` filters), `access.py` (caller access facts) |
| `invoke/` | one attempt: `endpoints.py` (the endpoint wrappers), `shaping.py` (request and response shaping), `streaming.py` (stream wrappers that report the outcome when a stream ends), `slots.py` (`max_parallel_requests`), `walk.py` (retry and fallback metadata), `prompt_management.py`, `websocket.py` |
| `surface/` | the Python `Router` API over the core's state: `registry.py`, `model_list.py`, `indexes.py`, `members.py`, `model_info.py`, `settings.py`, `health.py`, `health_routing.py`, `retries.py`, `fallbacks.py`, `alerting.py`, `replay.py` |
`litellm/router_attempt.py` builds the keyword arguments of one attempt for the
deployment the core picked. It does not pick, retry or record anything.
## Rust
`litellm-rust/crates/router-core/src/`.
| File | Holds |
| --- | --- |
| `lib.rs` | `Router`, `RouterBuilder`, and the crate overview |
| `config.rs` | `RouterConfig`, `RouterDynamicConfig`, `Op`, `RoutingStrategy`, retry and allowed fails policies |
| `registry.rs` | `Registry`, `Deployment`, `ChangeSet`, `deployment_id` |
| `build.rs` | building deployments from config with the host's `secrets`, `validate`, `provider` and `added` steps |
| `resolve.rs` | name resolution, a port of `_common_checks_available_deployment` |
| `select.rs` | the filter chain and `simple-shuffle` |
| `access.rs`, `ptu.rs`, `health_routing.rs`, `precall.rs` | the access group and window, PTU share, health check and pre-call filters |
| `prerouting.rs`, `routing_plugins.rs`, `routing_groups.rs` | pre-routing, routing plugin candidates and routing group members |
| `strategy.rs`, `weights.rs` | which strategy a pick uses, and per-request weights |
| `call.rs`, `call/stream.rs` | the call loop with retries, fallbacks and weighted failover, and stream recovery |
| `retry.rs`, `fallback.rs`, `mock.rs` | retry rules, the fallback lists, and `mock_testing_*` failures |
| `health.rs` | cooldowns, allowed fails, usage, and the cache keys |
| `cache.rs` | `RouterCache` and an in-memory implementation for tests |
| `settings.rs` | the frozen `Settings` view |
| `host.rs` | `Host`, `Attempt`, `RouterRequest`, `RouterEvent`, `RoutingContext` |
| `lookup.rs`, `host_index.rs` | the indexes and listings the Python surface reads |
| `model_info.rs`, `group_info.rs` | model info and model group info merges |
| `error.rs` | `ConfigError`, `Error`, `AttemptError`, `FailureKind` |
`litellm-rust/crates/python-bridge/src/router/` exposes the core to Python as
`RouterCoreHandle` and `RouterCall`, turns Python's config and dynamic snapshot
into the core's types, and turns each hostcall into a step.
## Tests and checks
| Path | What it covers |
| --- | --- |
| `tests/unit/test_router/core/` | the core suite, run against the core with a scripted host: selection, retries and cooldowns, fallbacks, registry change sets, endpoints |
| `litellm-rust/crates/router-core` unit tests | the Rust modules, with seeded randomness and a test clock |
| `tests/unit/test_router/` | router unit tests through the facade, including `test_router_entry_points.py` |
| `tests/code_coverage_tests/router_surface_check.py` | the shrink-only surface allowlist, run in CI |
| `tests/code_coverage_tests/router_litellm_globals_check.py` | checks that only `litellm/router_config.py` reads `litellm.*` settings |
| `tests/code_coverage_tests/router_code_coverage.py` | checks that every router function has a test |
| litefuzz `translation-diff` suite | differential fuzzing of routing behavior, run on every change to the core |

115
docs/architecture/router/reference/configuration.md Normal file
View file

@ -0,0 +1,115 @@
# Configuration and defaults
The constructor arguments of `litellm.Router`, grouped by what they control. In
proxy config they sit under `router_settings`, with `model_list` at the top
level. `litellm.router_api.router_init_args()` lists every accepted name.
`Router(**arguments)` freezes the arguments into a `RouterConfig`, and they stay
fixed for the router's life. Change them with `apply()`, see [Change deployments
or settings at runtime](../how-to/change-the-router-at-runtime.md). Where a
default below says `litellm.<name>`, the value comes from the process settings,
which the router reads as one `RouterDynamicConfig` snapshot per call.
![The Router constructor arguments grouped into five areas: deployments,
selection, reliability, state and health](../img/configuration-1.svg)
## Deployments
| Argument | Default | Meaning |
| --- | --- | --- |
| `model_list` | None | the deployments, see [Concepts](../explanation/concepts.md) |
| `model_group_alias` | `{}` | name to group, or `{"model": group, "hidden": true}` |
| `routing_groups` | None | groups of model groups with their own strategy |
| `default_litellm_params` | `{}` | merged into every call. The router adds `max_retries: 0` to it |
| `assistants_config`, `search_tools`, `guardrail_list` | None | non-model resources the router serves |
| `ignore_invalid_deployments` | False | skip a bad deployment instead of failing construction |
## Selection
| Argument | Default | Meaning |
| --- | --- | --- |
| `routing_strategy` | `"simple-shuffle"` | see the strategy table in [Strategies, routing groups and sub-routers](../explanation/strategies.md) |
| `routing_strategy_args` | `{}` | strategy options, for example `ttl` and `lowest_latency_buffer` for latency routing |
| `enable_pre_call_checks` | False | run context window, region and rate limit checks per request |
| `optional_pre_call_checks` | None | extra checks, such as prompt caching or Responses API affinity |
| `enable_tag_filtering`, `tag_filtering_match_any`, `tag_routing_prefix` | False, True, `""` | route by request tags against deployment tags |
| `plugins` | None | routing plugins that narrow candidates or attach signals |
| `default_max_parallel_requests` | None | per-deployment concurrency cap when a deployment sets none |
| `provider_budget_config` | None | per-provider budgets enforced by the budget limiter filter |
| `deployment_affinity_ttl_seconds`, `model_group_affinity_config`, `session_affinity_repin_on_failover` | 3600, None, False | sticky routing for sessions and response chains |
| `default_priority`, `polling_interval` | None, None | the priority scheduler queue |
## Reliability
| Argument | Default | Meaning |
| --- | --- | --- |
| `num_retries` | `litellm.num_retries`, else 2 | retries in the same group |
| `retry_policy`, `model_group_retry_policy` | None, `{}` | retries per error type, router-wide or per group |
| `retry_after` | 0 | minimum seconds to wait before a backoff retry |
| `timeout`, `stream_timeout` | None | per-call timeouts. A deployment's own values win |
| `fallbacks` | `[]` | `[{"group": ["other", ...]}]` for any error |
| `context_window_fallbacks` | `[]` | the same, for `ContextWindowExceededError` |
| `content_policy_fallbacks` | `[]` | the same, for `ContentPolicyViolationError` |
| `default_fallbacks` | None | becomes the `"*"` entry of `fallbacks` |
| `max_fallbacks` | `litellm.max_fallbacks`, else 5 | hop limit for fallbacks |
| `fallback_on_unsupported_params` | True | fall back on an unsupported-parameter 400 |
| `fallback_access_check`, `fallback_budget_check` | None | callables that may veto a fallback target |
| `enable_weighted_failover` | False | pick again in the group before falling back, `simple-shuffle` only |
| `enable_mid_stream_fallback_continuation` | False | continue a broken stream on a fallback group |
| `cooldown_time` | 5 seconds | how long a deployment stays cooled down |
| `allowed_fails` | `litellm.allowed_fails` (3) | fails allowed per cooldown window, used only when set explicitly or changed from the default |
| `allowed_fails_policy` | None | allowed fails per error type |
| `disable_cooldowns` | None | turn cooldowns off |
## State
| Argument | Default | Meaning |
| --- | --- | --- |
| `redis_url`, `redis_host`, `redis_port`, `redis_password`, `redis_db` | None | the Redis behind the routing state, see [Shared state](../explanation/shared-state.md) |
| `cache_kwargs` | `{}` | extra arguments for the Redis client |
| `cache_responses`, `caching_groups` | False, None | response caching through `litellm.cache` |
| `client_ttl` | 3600 | seconds before a cached provider client is rebuilt |
## Health
| Argument | Default | Meaning |
| --- | --- | --- |
| `enable_health_check_routing` | False | drop deployments failing background health checks, and with an allowed fails policy ignore cooldowns when they would leave nothing |
| `health_check_staleness_threshold` | None | how old a health result may be and still count |
| `health_check_ignore_transient_errors` | False | do not mark a deployment unhealthy for transient errors |
| `background_health_check_model_groups` | None | limit health checks to these groups |
| `alerting_config` | None | Slack alerting on routing events |
## Per-deployment overrides
These go in a deployment's `litellm_params` (some also in `model_info`) and win
over the router-wide value.
| Key | Effect |
| --- | --- |
| `rpm`, `tpm` | limits for usage-aware strategies and pre-call checks, and simple-shuffle weights |
| `weight` | simple-shuffle weight |
| `order` | in-group priority, lowest first |
| `cooldown_time` | this deployment's cooldown time, which wins over the `Retry-After` header and the router value |
| `num_retries` | retries for requests that land here |
| `max_parallel_requests` | concurrency cap |
| `timeout`, `stream_timeout` | per-call timeouts |
| `model_info.team_id` | restrict the deployment to one team |
| `model_info.blocked` | paused by an admin |
## Constants
From `litellm/constants.py`. Each can be overridden with the environment
variable of the same name.
| Constant | Default | Used for |
| --- | --- | --- |
| `DEFAULT_COOLDOWN_TIME_SECONDS` | 5 | router `cooldown_time` default |
| `DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS` | 1.0 | how often the cooldown cache re-reads Redis |
| `DEFAULT_FAILURE_THRESHOLD_PERCENT` | 0.5 | error-rate cooldown threshold |
| `DEFAULT_FAILURE_THRESHOLD_MINIMUM_REQUESTS` | 5 | minimum requests in the window before the error rate counts |
| `SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD` | 1000 | requests needed before a lone deployment can be cooled down at 100% failures |
| `ROUTER_MAX_FALLBACKS` | 5 | `max_fallbacks` default |
| `DEFAULT_REDIS_SYNC_INTERVAL` | 1 | seconds between usage counter pushes to Redis |
| `DEFAULT_MAX_RETRIES` | 2 | the default retry count elsewhere in LiteLLM. The router uses the OpenAI SDK's value, which is also 2 |
| `INITIAL_RETRY_DELAY`, `MAX_RETRY_DELAY`, `JITTER` | 0.5, 8.0, 0.75 | the retry backoff in seconds |

192
docs/architecture/router/reference/core-and-host.md Normal file
View file

@ -0,0 +1,192 @@
# Core, config and host interfaces
The interfaces between the Python facade, the Rust core and the host it runs
under. [The core and the host](../explanation/core-and-hosts.md) explains why
they are shaped this way.
![The Router facade uses the RouterCore protocol; RustCore implements it and
holds a handle to the Rust RustRouter, which calls the Host trait that
PythonHost implements](../img/core-and-host-1.svg)
## Python side
| Name | File | What it is |
| --- | --- | --- |
| `Router` | `litellm/router.py` | the facade. `Router(**arguments)` and `Router.from_config(config, host=python_host)` |
| `RouterCore` | `litellm/router_core.py` | the protocol the facade delegates to |
| `make_core(config, host)` | `litellm/router_core.py` | builds the core once, when the Router is built |
| `RustCore` | `litellm/router_rust/core.py` | holds one `RouterCoreHandle` and answers its steps |
| `RustBridgeDeclined` | `litellm.rust_bridge._native` | raised when the handle cannot read a config |
| `RouterConfig` | `litellm/router_config.py` | the constructor arguments, frozen, with plain containers copied (`build_router_config`) |
| `RouterDynamicConfig` | `litellm/router_config.py` | a versioned snapshot of the process settings the router reads per call (`current_dynamic_config()`) |
| `PythonHost` | `litellm/router_python_host.py` | `invoke(op)` returns the `litellm` endpoint function, plus `sleep`, `dynamic()`, the callback lists, the global cache and `routing_cache(config)` |
| `router_attempt` | `litellm/router_attempt.py` | builds one attempt's keyword arguments for the deployment the core picked |
| `RouterHost` | `litellm/router_host.py` | what only the application can answer (vertex pass-through credentials, member auto-router authorization) |
| `DeprecatedRouterMembers` | `litellm/router_deprecated.py` | the old Router member names, served with a `DeprecationWarning` |
`RouterDynamicConfig` fields are `num_retries`, `max_fallbacks`,
`router_max_fallbacks`, `allowed_fails`, `request_timeout`, `fallbacks`,
`default_fallbacks`, `context_window_fallbacks`, `content_policy_fallbacks`,
`drop_params`, `provider_list`, `expose_router_debug_in_errors`,
`redact_user_api_key_info`, `model_alias_map`, `model_cost`,
`known_custom_logger_compatible_callbacks`, `exception_types`, `callbacks`,
`success_callback`, `failure_callback`, `input_callback`, `callback_manager` and
`cache`, plus its `version`.
## The step protocol
`RouterCoreHandle(config, dynamic)` in `litellm.rust_bridge._native` wraps one
Rust `Router`. `start(op, request)`, `select(model, request)` and
`healthy(model, request)` return a `RouterCall`, and so do the report and model
info queries. The driver calls `RouterCall.next(answer)` until it gets `done` or
`failed`. Every step carries the routing events since the last step.
| Step | The core needs | The driver answers with |
| --- | --- | --- |
| `attempt` | one attempt run on the named deployment | the response and its total tokens, or the typed failure |
| `sleep` | a backoff of `seconds` | nothing, after sleeping through the host |
| `allow_fallback` | whether a cross-group fallback `target` is allowed | a bool from the fallback access and budget checks |
| `pre_route` | the sub-routers, session binding or a custom strategy before a pick resolves `model` | a model group, a deployment the custom strategy picked, or `None` |
| `select` | one host stage run on the candidates, named by `stage` | see the stage table below |
| `cache` | a routing cache read or write (`get`, `set`, `batch_get`, `increment` and the rest) | the values read, or an acknowledgement |
| `secrets`, `validate`, `provider`, `added` | one deployment's build, while a config or change set is applied | the resolved secrets, the validated entry, the provider, the added entry |
| `model_info` | a cost map or capability lookup during a model info query | the lookup's documents |
| `done` | nothing more | the result goes back to the caller, with routing headers |
| `failed` | nothing more | the error goes back to the caller, by its `error_kind` |
| `select` stage | The host returns |
| --- | --- |
| `routing_plugins` | the candidate models the routing plugins left, after running them when `run` is set |
| `callback_filters` | the deployment ids the `CustomLogger` filters left, or a failure |
| `plugins` | the deployments `Router(plugins=[...])` objects left |
| `pre_call` | the facts the pre-call checks decide by, such as token counts |
| `tags` | the deployments tag routing left |
| `text_only` | the candidates that declare no vision support |
| `strategy` | the candidates with the pick first, for a strategy the core does not implement |
`apply(changes)` returns the new registry version, `registry_version()` reads
it, and `retire()` retires the router.
## Rust side
All of these live in `litellm-rust/crates/router-core/src/`.
| Type | Module | What it is |
| --- | --- | --- |
| `Router`, `RouterBuilder` | `lib.rs` | `Router::new(config, dynamic, host)`, or the builder with `randomness` and `clock` for tests |
| `RouterConfig`, `RouterDynamicConfig`, `Op`, `RoutingStrategy`, `RetryPolicy`, `AllowedFailsPolicy` | `config.rs` | static and dynamic config, the operation name, the strategies (`is_native` is true for `simple-shuffle`) |
| `Registry`, `Deployment`, `ChangeSet`, `RegistryVersion`, `deployment_id` | `registry.rs` | the immutable, indexed deployments |
| `Settings` | `settings.rs` | the frozen settings view |
| `Health`, `Outcome`, `Clock`, `cooldown_key`, `usage_key` | `health.rs` | cooldowns, allowed fails, usage and the outcome counters |
| `RouterCache`, `InMemoryRouterCache` | `cache.rs` | the routing state store the host provides |
| `Randomness`, `SeededRandomness`, `simple_shuffle` | `select.rs` | the filter chain and `simple-shuffle` |
| `resolve_request`, `all_deployments` | `resolve.rs` | name resolution, a port of `_common_checks_available_deployment` |
| `routing_context`, `request_override` | `strategy.rs` | which strategy a pick uses |
| `RouterWeights`, `validate` | `weights.rs` | per-request weights |
| `RouterResponse` | `call.rs` | the call loop's result with the deployment, group, retries and fallbacks used |
| `StreamPolicy`, `StreamOrigin` | `call/stream.rs` | what may take over a stream that broke |
| `Fallbacks`, `FallbackTarget` | `fallback.rs` | the parsed fallback lists |
| `Host`, `Attempt`, `RouterRequest`, `Output`, `RouterEvent`, `RoutingContext`, `SelectStage` | `host.rs` | the host seam |
| `Error`, `ConfigError`, `Unsupported`, `AttemptError`, `FailureKind` | `error.rs` | errors at build time, at call time, and per attempt |
The other modules hold one stage or query each: `access.rs` (access groups and
windows), `ptu.rs` (PTU shares), `health_routing.rs`, `precall.rs`,
`prerouting.rs`, `routing_groups.rs`, `routing_plugins.rs`, `retry.rs`,
`mock.rs` (`mock_testing_*` failures), `build.rs` (the deployment build),
`lookup.rs` and `host_index.rs` (the indexes the Python surface reads),
`model_info.rs` and `group_info.rs` (model info merges).
`Op` is the operation's name as a string, such as `acompletion`, `aembedding`,
`aresponses` or `aanthropic_messages`. The core passes it to the host and never
interprets it.
### `Host`
| Method | Called for |
| --- | --- |
| `invoke(attempt)` | every attempt. There is no native attempt path |
| `select(stage, ctx, candidates)` | the `Plugins`, `Strategy`, `Tags` and `TextOnly` stages |
| `pre_route(ask)` | pre-routing, and the whole pick for a custom strategy |
| `routing_plugin_models`, `callback_filters`, `pre_call_facts` | the stages that need live Python objects |
| `allow_fallback(request, target)` | the access and budget checks before a cross-group fallback |
| `sleep(duration)`, `observe(event)`, `cache()` | backoff, routing events, the routing state store |
| `secrets`, `validate`, `provider`, `added` | the deployment build |
| `model_info(lookup)` | cost map and capability lookups |
| `supports(op, provider)`, `can_select()` | what the host can serve. Both default to true, and `PythonHost` serves everything |
### `RouterRequest`
| Field | Meaning |
| --- | --- |
| `model` | the requested name, a model group, alias, routing group, deployment id or wildcard match |
| `body` | the request params the core's own checks read |
| `team_id`, `proxy_admin`, `allowed_models`, `admin_model` | the caller facts the team and access filters read |
| `specific_deployment` | the caller named a deployment's `litellm_params.model` |
| `pinned_deployment_id` | a deployment the session or affinity binding pinned |
| `num_retries`, `model_group_retry_policy`, `retry_policy_raises` | per-request retry settings, which win over the router's |
| `disable_fallbacks`, `fallbacks`, `context_window_fallbacks`, `content_policy_fallbacks`, `max_fallbacks` | per-request fallback settings |
| `cooldown_time` | a per-request cooldown |
| `router_weights` | per-team weights, model group to deployment id to weight |
| `routing_strategy` | a per-request strategy override |
| `filter_tags`, `has_image` | inputs to the tag and text-only stages |
| `pass_through` | a pass-through route, which serves only pass-through deployments |
| `target_order`, `excluded_ids`, `retry_skipped_ids` | the order an order fallback targets, and ids the pick must skip |
| `pre_routes`, `unrouted`, `skip_weighted_failover` | whether and how pre-routing and weighted failover run |
| `mock_failure`, `health_check_probe` | test failures and health check probes |
| `payload` | what the host needs to run the call itself, such as the Python kwargs |
### `Attempt` and `Output`
`Attempt` carries `op`, `request`, `deployment`, `name` (the requested name or
the fallback target), `model_group` (the group served, after fallbacks),
`target_order`, `candidate_ids`, `retry` (0 for the first attempt in this
group), `fallback_depth`, `fallback_level` (0 for the requested name),
`timeout`, `params`, `scope`, `max_retries` and `model_group_size`.
`Output` carries the `value` the host hands back unchanged, `total_tokens`,
`blocked` (`ContentFilter` or `Refusal`, which the fallback rules read) and
`deferred`, which is set for a stream or realtime session whose outcome the host
reports when it ends.
### `RouterEvent`
| Event | Fields |
| --- | --- |
| `Selected` | `model_group`, `deployment_id` |
| `Succeeded` | `deployment_id` |
| `AttemptFailed` | `deployment_id`, `kind`, `status` |
| `CooldownSet` | `deployment_id`, `cooldown` |
| `Retry` | `model_group`, `retry`, `delay` |
| `Fallback` | `from`, `to`, `order`, `depth`, `level`, `scope`, `parent`, `max_fallbacks` |
| `FallbackDone` | `from`, `to`, `level`, `scope`, `parent`, `succeeded` |
| `Blocked` | `level`, `model`, `blocked` |
| `RetryLogged` | `scope`, `level`, `model`, `attempted_retries`, `cause` |
| `LevelFailed` | `level`, `model`, `error`, `handling` |
| `MockFailed` | `mock`, `level`, `model`, `num_retries` |
| `RetryPolicyRaised` | `level`, `model`, `cause` |
| `OrderChainFailed` | `level`, `last_hop` |
| `RetriesExhausted` | `model_group`, `attempted`, `max`, `level`, `picked` |
The driver uses the later events to raise the exception a caller expects at that
point, with the retry and fallback attributes set.
### `FailureKind`
The kinds are `RateLimit`, `ContextWindowExceeded`, `ContentPolicyViolation`,
`Authentication`, `PermissionDenied`, `NotFound`, `BadRequest`, `Timeout`,
`ApiConnection`, `InternalServer`, `ServiceUnavailable`, `BadGateway`,
`GuardrailIntervention` (never retried or fallen back from), `Declined` (litellm
rejected the request before calling, and it never cools a deployment down),
`CallerCredential` (the caller's own credential failed) and `Other`.
`AttemptError` adds `status`, `retry_after`, `num_retries`, the provider's error
text for the cooldown record, and `agentic_followup`, which marks an agentic
follow-up call that failed after the first call was served and is never retried.
### Build-time errors
`Router::new` returns `ConfigError` when the registry cannot be built, for
example from a malformed deployment or a duplicate id, or when the host cannot
serve the config. `Unsupported` names what is missing: a strategy or plugin on a
host that cannot select, or an operation and provider pair the host does not
support. Under `PythonHost` neither happens, since it serves every operation and
strategy.

160
docs/architecture/router/reference/entry-points.md Normal file
View file

@ -0,0 +1,160 @@
# Entry points
`litellm.Router` exposes its endpoint methods plus the entry points below, and
nothing else. The views and types live in `litellm/router_api.py`. The "Was"
columns name the Router member from v1.108 and below which each query replaced.
The old names still work through `DeprecatedRouterMembers`, with a
`DeprecationWarning`, see [Move a caller off a deprecated Router
member](../how-to/move-off-deprecated-members.md).
![The Router facade and the RouterRegistry, RouterSettings and RouterHealth
views it returns, plus the ChangeSet that apply takes, with each type's
members](../img/entry-points-1.svg)
## `registry()` returns `RouterRegistry`
Read-only queries over the configured deployments. The router keeps the registry
as an immutable, versioned snapshot, and each query reads the current one.
| Query | Returns | Was |
| --- | --- | --- |
| `version` | an int that goes up on every deployment change | |
| `deployment(id)` | `Deployment` or None | `get_deployment` |
| `has_deployment(id)` | bool | `has_model_id` |
| `deployment_dict(id)` | the deployment as its model_list dict | `get_model_info` |
| `deployment_model_info(id, model_name)` | model info merged with the pricing map | `get_deployment_model_info` |
| `router_model_info(...)`, `discovered_model_info(id)`, `zero_cost()` | routing model info, provider-discovered limits, the zero-cost lookup | `get_router_model_info`, `get_discovered_model_info`, `_zero_cost_cache` |
| `deployments(name, source="group")` | the group, its aliases, routing groups, then wildcards | `get_model_list` |
| `deployments(name, source="all")` | every deployment of the group | `get_all_deployments` |
| `deployments(source="config")` | deployments from config, not the DB | `config_deployments` |
| `deployments(name, source="routable", request=, call_type=)` | deployments that pass the routing checks for a request | `get_routable_deployments` |
| `deployments(name, source="first_in_group")` | the first deployment of a group | `get_deployment_by_model_group_name` |
| `deployments(name, source=...)` | also `named`, `alias`, `routing_groups`, `routed_group`, `wildcard`, `for_request` | see the docstring |
| `all()` | the live model list | `model_list` |
| `names(kind="groups")` | every routable model name | `get_model_names` |
| `names(kind="group_set")` | the set of configured group names | `model_names` |
| `names(name, kind="ids")` | deployment ids, of one group or all | `get_model_ids` |
| `names(kind="team_public", team_id=)` | the public model names of a team's deployments | `team_public_model_names_of` |
| `names(kind=...)` | also `deployment_names`, `litellm_models`, `litellm_base_models`, `wildcard`, `route_candidates` | see the docstring |
| `resolve(name, kind="group")` | the model group a name routes to | `routable_model_group` |
| `resolve(name, kind="alias")` | the alias target, or None | `get_model_from_alias` |
| `resolve(name, kind="team_model", team_id=)` | a team's public name to its group | `map_team_model` |
| `resolve(name, kind="recognized")` | whether the router knows the name | `is_recognized_model` |
| `resolve(id, kind="group_of_id")` | the group of a deployment id | `get_model_group` |
| `resolve(name, kind=...)` | also `served`, `from_id`, `upstream`, `wildcard_access_group`, `team_model_alternatives`, `routing_group_alternatives`, `search_tool_support` | see the docstring |
| `group(name, field="info")` | merged `ModelGroupInfo` | `get_model_group_info` |
| `group(name, field=...)` | also `cached_info`, `display_name`, `mode`, `service_tiers`, `max_input_tokens`, `token_limits`, `listing_info` | see the docstring |
| `access_groups(...)` | access group name to model names | `get_model_access_groups` |
| `credentials(id, form="params")` | a deployment's credentials | `get_deployment_credentials` |
| `plugins(kind)` | `auto`, `complexity`, `quality`, `adaptive` sub-routers, `routing` plugins, `budget_limiter`, `prompt_caching_pins` | `auto_routers` and friends |
| `routing_group(name)` | the routing group callable as that name, or None | `get_routing_group` |
| `client(deployment, client_type=None)` | the SDK client the router cached for a deployment, or None | `_get_client` |
| `aliases()`, `patterns()`, `routing_groups()`, `index(name)` | the alias map, the wildcard patterns, routing groups, the name index | the old attributes |
Module-level helpers in the same file need no Router.
| Function | Returns |
| --- | --- |
| `deployment_id(model_group, litellm_params)` | the stable id of a deployment that has none, the same hash the router uses (was `Router.generate_model_id`) |
| `deployment_usable_by_team(deployment, team_id)` | whether a team may use a deployment |
| `router_init_args()` | the arguments `Router()` accepts (was `Router.get_valid_args`) |
| `fallback_chain(fallbacks, model_group)`, `request_fallback_chain(fallbacks, request_kwargs, model_group)` | the fallbacks configured for a group, or for one request, and the index of the `*` entry |
| `fallbacks_disabled(request_kwargs)` | whether the request turned fallbacks off |
| `references_provider_scoped_resource(request_kwargs)` | whether the request names a file, batch or fine-tuning job only its own provider knows |
| `deployment_metadata(request_kwargs)` | the metadata bucket the router stamped the picked deployment into |
| `mark_advisor_orchestration_failure(exception)` | tags an advisor orchestration failure, so the router does not cool down the deployment it picked |
## `settings()` returns `RouterSettings`
A frozen dataclass with one field per readable setting, listed in
`SETTING_FIELDS`. It is built from the router's static `RouterConfig` and the
current dynamic snapshot, and it changes only through `apply()`. A setting the
router never set reads as None. Treat the field values as read-only.
`routing_strategy`, `fallbacks`, `context_window_fallbacks`,
`content_policy_fallbacks`, `default_deployment`, `default_litellm_params`,
`default_max_parallel_requests`, `cooldown_time`, `allowed_fails`,
`allowed_fails_policy`, `disable_cooldowns`, `enable_pre_call_checks`,
`enable_health_check_routing`, `health_check_ignore_transient_errors`,
`background_health_check_model_groups`, `enable_tag_filtering`,
`tag_filtering_match_any`, `tag_routing_prefix`, `router_general_settings`,
`provider_budget_config`, `provider_default_deployment_ids`,
`fallback_access_check`, `fallback_budget_check`, `kubernetes_pod_discovery`,
`search_tools`, `guardrail_list`, `cache_responses`.
| Method | Returns | Was |
| --- | --- | --- |
| `is_set(name)` | whether the router has this setting at all | |
| `as_dict()` | the settings as plain values | `get_settings` |
| `allowed_fails_for(exception)` | allowed fails for an error under `allowed_fails_policy` | `get_allowed_fails_from_policy` |
| `guardrail(name)` | a configured guardrail | `get_available_guardrail` |
## `health()` returns `RouterHealth`
| Member | Returns | Was |
| --- | --- | --- |
| `store` | the routing cache, see [Routing state and cache keys](state-and-cache-keys.md) | `cache` |
| `cooldowns` | active cooldowns, `await cooldowns.async_get_active_cooldowns(model_ids=, parent_otel_span=)` | `cooldown_cache` |
| `health_checks` | background health check results | `health_state_cache` |
| `latency` | latency samples by deployment | `deployment_latency_map` |
| `is_blocked(deployment)`, `all_blocked(deployments)` | whether an admin paused them | `is_deployment_blocked`, `are_all_deployments_blocked` |
| `fully_blocked_model_names()` | groups with every deployment paused | `get_fully_blocked_model_names` |
| `await fully_unhealthy_model_names()` | groups with every deployment failing health checks | `async_get_fully_unhealthy_model_names` |
| `await usage(group, kind="current")` | tpm and rpm this minute | `get_model_group_usage` |
| `await usage(group, kind="remaining")` | what is left of the group's limits | `get_remaining_model_group_usage` |
| `team_ptu_ceiling(team_id, group)` | a team's PTU share ceiling | `team_ptu_ceiling` |
| `split_priority(model, kwargs)` | the request priority and the kwargs without it | `split_request_priority` |
| `failures_this_minute(id)` | failures counted against a deployment this minute | `get_deployment_failures_for_current_minute` |
| `record_failure(id, exception, status)` | counts a failure the router did not see itself, cools the deployment down once it is over its allowed fails, and says whether it did | `increment_deployment_failures_for_current_minute` then `set_cooldown_deployments` |
| `priority_header_fields(model, kwargs, header)` | the request fields a priority header sets | `priority_header_update` |
## `select(model, request_kwargs=None, *, kind="pick", ...)`
Runs the pick without calling the provider. `kind="pick"` returns one deployment
dict, exactly what a real request would get (was
`async_get_available_deployment`). `kind="healthy"` returns the filtered
candidates before the strategy (was `async_get_healthy_deployments`).
`kind="pass_through"` picks among deployments with `use_in_pass_through=True`.
Optional keyword arguments are `messages`, `input`, `specific_deployment`,
`parent_otel_span` and `health_check_probe`.
## `apply(changes)` and `aapply(changes)`
The only way to change a Router after it is built. The router rejects an
`assign` name it does not know before changing anything, then applies the fields
in the order below. A change set is not one swap, so a request that runs while
the change set is applied can see the earlier fields changed and not the later
ones, and an error part way through leaves the earlier changes in place. The
fields are.
| Field | Effect | Was |
| --- | --- | --- |
| `assign` | replace settings as they are. Names from `SETTING_FIELDS`, `PLAIN_SETTING_ATTRIBUTES` (`num_retries`, `max_fallbacks`, `retry_after`, `timeout`) or `ASSIGNABLE_ALIASES` (`model_group_alias`) | `router.<field> = value` |
| `settings` | validated runtime settings | `update_settings(**settings)` |
| `upsert` | add or replace deployments, by `model_info.id` | `upsert_deployment` |
| `delete` | remove deployments by id | `delete_deployment` |
| `redis_cache` | point routing state at a new Redis cache | `update_redis_cache` |
| `refresh_model_info`, `refresh_client` | re-read discovered model info, optionally with your HTTP client, `aapply` only | `arefresh_model_info` |
Both return `ApplyResult(version, upserted, deleted)`, with the registry version
after the change and, per upsert or delete, the deployment that was there
before.
## Lifecycle
`observe(logging_obj=None, *, request_scoped=False)` returns the router's
`RoutingObservers`, which receive the router's routing events. With
`request_scoped=True` the router stops registering its callbacks globally, and
passing a request's `logging_obj` binds them to that request.
`register_plugin(strategy)` puts a `CustomRoutingStrategyBase` in place of the
whole pick, see [Add a custom routing
strategy](../how-to/add-a-routing-strategy.md). `retire()` releases the router's
global callback registrations and routing state, for a Router that is being
replaced (was `discard`). Calls on a retired router fail.
## `RoutingContext`
What a routing plugin receives instead of the Router. It has `registry()`,
`settings()` and `health()`, plus `select`, `observers`, `belongs_to`,
`explicit_allowed_fails`, `pre_call_checks`, `count_pre_call_check_tokens`,
`update_kwargs_before_fallbacks`, `pre_routing_hook` and `log_retry`.

37
docs/architecture/router/reference/state-and-cache-keys.md Normal file
View file

@ -0,0 +1,37 @@
# Routing state and cache keys
Everything the router remembers between requests, where it lives, and how long
it lasts. This state lives in the router's routing cache, a `DualCache`. Every
worker uses the same keys, encodings and TTLs, so a fleet shares one Redis. In
the keys below, `{id}` is the deployment's `model_info.id` and `{minute}` is the
UTC time as `HH-MM`.
![The routing cache keys shared through Redis when it is configured, beside the
state kept only in this process](../img/state-and-cache-keys-1.svg)
| Key | Written by | Read by | Value | TTL | Tiers |
| --- | --- | --- | --- | --- | --- |
| `deployment:{id}:cooldown` | `Health::add_cooldown` | the cooldown filter, `health().cooldowns` | `{exception_received, status_code, timestamp, cooldown_time}` | the cooldown time | memory + Redis, memory re-checked against Redis every 1s |
| `deployment:{id}:allowed_fails` | `Health::record_failure` | the same | fail count | the cooldown time | memory + Redis |
| `deployment:{id}:allowed_fails:{suffix}` | the same, with a deployment-level policy | the same | fail count per error type | the cooldown time | memory + Redis |
| `{id}:successes`, `{id}:fails` | `Health::record_success`, `Health::record_failure` | the default error-rate cooldown rule, `health().failures_this_minute()` | count | 60s | memory only |
| `global_router:{id}:{model}:tpm:{minute}`, `:rpm:` | `Health::record_success` after each response | pre-call rate limit checks, `health().usage()` | count | 60s | Redis increment, memory |
| `global_router:{id}:{model}:itpm:{minute}`, `:otpm:` | the input and output token rate limit check | the same check | input and output token counts | 60s | memory + Redis |
| `{id}:{model}:tpm:{minute}`, `{id}:{model}:rpm:{minute}` | the `usage-based-routing-v2` strategy | the same | count | 60s | memory, pushed to Redis every 1s |
| `{model_group}_map` | the `latency-based-routing` strategy | the same | latency and ttft samples per deployment | `routing_strategy_args.ttl`, 1 hour | memory + Redis |
| `{model_group}_request_count:{id}` | the `least-busy` strategy | the same | requests in flight | 1 hour | memory + Redis |
Strategies and plugins keep their own keys (the strategy counters above, the
budget limiter, prompt cache pins, deployment and session affinity).
## Process-local state
Some state is per router and is never shared.
- the registry, with its indexes and `registry().version`, and the settings
view, each held as one snapshot that `apply()` swaps
- provider clients, cached per deployment for `client_ttl` seconds
- the background health check results in `health().health_checks`
This state is rebuilt from config in every process, which is why deployment ids
must be stable across processes.

187
docs/architecture/router/tutorial/route-your-first-requests.md Normal file
View file

@ -0,0 +1,187 @@
# Route your first requests
In this tutorial you build a Router with three mock deployments and watch it
balance load, retry, cool a deployment down and fall back to another model
group. Every deployment uses `mock_response`, so nothing leaves your machine and
you need no API keys.
You need a checkout of this repo with its environment installed (`uv sync`). Run
each step from the repo root with `uv run python`, or paste the steps one after
another into a single file.
![The five tutorial steps in order: build a Router, send requests, look inside,
break a deployment, break the group](../img/route-your-first-requests-1.svg)
## 1. Build a Router
Create `tutorial.py` with a model list. Two deployments, `a` and `b`, serve the
model group `chat`. A third, `c`, serves the model group `backup`. The router
retries once inside a group, then falls back from `chat` to `backup`.
```python
import asyncio
import random
from collections import Counter
from litellm import Router
from litellm.router_api import ChangeSet
from litellm.types.router import Deployment
random.seed(0)
def mock(id: str, group: str, response: str) -> dict:
return {
"model_name": group,
"litellm_params": {"model": "openai/gpt-4o-mini", "api_key": "sk-fake", "mock_response": response},
"model_info": {"id": id},
}
router = Router(
model_list=[
mock("a", "chat", "hi from a"),
mock("b", "chat", "hi from b"),
mock("c", "backup", "hi from backup"),
],
fallbacks=[{"chat": ["backup"]}],
num_retries=1,
)
async def ask() -> tuple[str, str]:
response = await router.acompletion(model="chat", messages=[{"role": "user", "content": "hello"}])
return response._hidden_params["model_id"], response.choices[0].message.content
```
`random.seed(0)` makes the router's random picks repeat, so your output matches
this page. `model_name` is the name callers send. `litellm_params` is what the
router passes to `litellm.acompletion` once it picks the deployment.
`model_info.id` names the deployment. When you leave it out, the router hashes
the group and params into an id.
`Router(...)` freezes these arguments into a `RouterConfig`, and settings stay
fixed after that.
## 2. Send requests
Add a `main` coroutine and send twenty requests.
```python
async def main() -> None:
print("picked:", Counter([(await ask())[0] for _ in range(20)]))
asyncio.run(main())
```
Run it. You see both deployments picked, in a split close to even.
```
picked: Counter({'b': 12, 'a': 8})
```
No strategy was configured, so the router used `simple-shuffle`, the default.
Neither deployment sets `weight`, `rpm` or `tpm`, so each pick is a uniform
random choice among the healthy deployments of the group. Each attempt still
went through `litellm.acompletion`, so `mock_response` worked as it does without
a router.
## 3. Look inside
The router answers questions about itself through its entry points. Add these
lines to `main`.
```python
registry = router.registry()
print("groups:", sorted(registry.names(kind="group_set")))
print("chat deployments:", [d["model_info"]["id"] for d in registry.deployments("chat")])
print("strategy:", router.settings().routing_strategy)
picked = await router.select("chat")
print("select picked:", picked["model_info"]["id"])
```
```
groups: ['backup', 'chat']
chat deployments: ['a', 'b']
strategy: simple-shuffle
select picked: b
```
`registry()` holds the configured deployments, `settings()` is a frozen view of
the configuration, and `select()` runs the same pick a real request runs,
without calling the provider. [Entry points](../reference/entry-points.md) lists
everything they offer.
## 4. Break a deployment
Replace deployment `a` with one that always answers 429. Router settings are
static after construction, so changes go through `apply()` with a `ChangeSet`.
The router replaces the deployment and bumps the registry version. The exact
number depends on how many deployments were added before. What matters is that
it goes up.
```python
broken_a = Deployment(**mock("a", "chat", "litellm.RateLimitError"))
print("version after apply:", router.apply(ChangeSet(upsert=[broken_a])).version)
print("picked:", Counter([(await ask())[0] for _ in range(10)]))
print("cooldowns:", await router.health().cooldowns.async_get_active_cooldowns(
model_ids=["a", "b", "c"], parent_otel_span=None
))
```
```
version after apply: 7
picked: Counter({'b': 10})
cooldowns: [('a', {'exception_received': 'litellm.RateLimitError: ...', 'status_code': '429', 'timestamp': ..., 'cooldown_time': 5})]
```
Every request succeeded. The first one that landed on `a` failed with a 429, and
two things happened.
![Sequence in which the first attempt on deployment a gets a 429, the router
sets a cooldown on a, retries on b and gets a 200, and later requests only see
b](../img/route-your-first-requests-2.svg)
First, the router retried in the same group and skipped `a`, so the caller got
`b`'s answer. Second, it recorded the failure and put `a` into cooldown, because
a 429 in a group with more than one deployment cools that deployment down at
once. For the next five seconds, the default cooldown time, the cooldown filter
removes `a` before the strategy ever sees it. [Retries, fallbacks and
cooldowns](../explanation/reliability.md) explains the rules.
## 5. Break the group
Break `b` too and send one more request.
```python
broken_b = Deployment(**mock("b", "chat", "litellm.RateLimitError"))
router.apply(ChangeSet(upsert=[broken_b]))
print("answer:", await ask())
```
```
answer: ('c', 'hi from backup')
```
With both `chat` deployments failing, the retry has nowhere left to go in the
group. Once retries are spent, the router's fallback walk looks up `chat` in
`fallbacks`, finds `backup`, and runs the whole retry loop again against that
group.
![A request for chat retries a and b, both return 429, then falls back to
backup, where c returns 200 with 'hi from
backup'](../img/route-your-first-requests-3.svg)
## What you saw
A Router owns a list of deployments grouped by `model_name` into model groups.
Each request is resolved to a model group, the group's deployments are filtered,
and a strategy picks one. Failures feed back into shared state (the cooldown you
saw) and into the retry and fallback loop around the attempt. Code outside the
router reads and changes all of this through a handful of entry points.
Next, read [Router concepts](../explanation/concepts.md) for the ideas behind
each step, [The life of a request](../explanation/request-lifecycle.md) for the
full path, or [Configuration and defaults](../reference/configuration.md) for
the knobs you skipped here.