Adds the litellm-router-core crate, its step-driven PyO3 bridge
(RouterCoreHandle / RouterCall) and RustCore (litellm/router_rust/),
served behind LITELLM_RUST through the ROUTER route rule. The host
call-out is the only boundary: every attempt goes through Host::invoke,
there is no native executor path, and RustCore never builds, wraps or
forwards to PythonCore, so router_python_core.py stays deletable.
Attempt preparation lives in litellm/router_attempt.py so it outlives
router_python_core.py.
RustCore matches PythonCore's behavior, including its known bugs (fixed
in follow-up PRs). Routing decisions (strategy picks, routing groups,
PTU, pre-routing, pre-call checks, fallback walk, health routing, access
groups, weights, cooldowns, allowed fails, usage) live in the crate;
work that needs live Python objects (request shaping, endpoints,
streaming, model info facts, callbacks, alerting, prompt management)
lives in the host-side litellm/router_rust/ package (steps/, invoke/,
surface/) driven by Rust steps.
The crate owns the registry. It runs the deployment build loop (secret
references, ids, organization fan-out, install order) and asks the host
for what needs Python, one deployment at a time so errors surface where
Python raised them: secrets, validate, provider and added steps. It
holds the model list, the index maps and the pattern routers with the
Python core's history rules (host_index.rs), and answers the registry
lookups (lookup.rs). The Python views of that state in
litellm/router_rust/surface/ (model_list, index(), names(), patterns())
read the handle, and every write to them becomes a changeset. Model
info, deployment model info, configured limits and model group info are
merged in the crate (model_info.rs, group_info.rs) from catalog rows and
facts the host answers on a model_info step asked at query time, so
price reloads and register_model stay visible. The usage cache key
format is defined once, in the crate.
The crate records each attempt's outcome itself rather than through
litellm callbacks. A streamed attempt is deferred: it counts its
request when the stream opens, and the host's stream wrappers report
the success with the final usage, or a failure after the first chunk,
when the stream ends, as the Python core's callbacks do.
tests/unit/test_router/core characterizes PythonCore's public behavior
and runs every test on both cores, with nothing pending; only the names
of the Python core's own outcome callbacks are asserted on Python alone.
litellm.Router is now final: subclassing it raises TypeError at the class statement, and the
machinery that kept subclass overrides working is gone. The Rust core never honoured subclass
overrides, so they would have silently stopped running there. Custom routing goes through
register_plugin(), settings through apply() and outcomes through observe(). This is a breaking
change for anyone subclassing Router.
Fuzz parity fixes on top of the rebase: the Rust core keeps num_retries,
max_fallbacks, allowed_fails, retry_after, timeout, the tag settings,
routing_strategy_args and the fallback lists as given (the crate reads
them as numbers, update_settings casts as Python does); a deprecated
`model_list = [...]` write reaches the Rust core; a failed set_model_list
is all or nothing on both cores (the deployments served before the call
are put back and the error raised); config_deployments() is a generator and `del` of a plain
attribute works on the Rust core. Least-busy pick reservations from
main now work on the Rust core: its selectors share one reservation
store, routed calls reserve and free their unstarted picks, and
select() and batch cancel do not reserve. On the Python core a dropped
upsert takes its name back off deployment_names, and
get_deployment_order reads every numeric spelling as one int so mixed
orders no longer raise TypeError.
Rulings on the remaining differences, the Python core changed where it
was the odd one out: sync picks and picks by deployment id apply the
team filter as async picks do, so another team's deployment is refused;
the fallback lists and default_litellm_params in RouterSettings are
copies, so an in-place edit is ignored and settings change only through
apply(); one health record with an unreadable timestamp drops out alone;
an empty fallback entry is skipped; a bool cooldown_time is refused at
construction, a bool or string cooldown_time on a deployment or a call
falls back to the router default, and a failure status that no HTTP
status can be is recorded as 500; an infinite or NaN cost and a whole
number wider than 64 bits (ptu_shares) are refused on both cores with
the same error (litellm/router_utils/deployment_numbers.py); and a
/v1/messages stream whose provider iterator raises cools the deployment
and retries like a non-stream call, with the Rust core waiting for that
cooldown before its retry picks.
Differences the proxy parity fuzz and its regression tests found on the
Rust core: a file upload or a batch on an uploaded file falls back only
within its own model group, as on the Python core, since the file lives
with one provider (the host sends same_group_fallbacks_only and the walk
skips targets in other groups); Decisions calls drop deployments whose
provider has no Decisions config, and deployments that refuse the audio
or videos a call carries, unless that would leave none, as the Python
core does, while the retry count still reads the whole group; a file
upload that fails on one order level is created on the next level's
deployments rather than the first level again; and a response that
cannot be converted (a text_completion body without a message) reports
only its own traceback, so a rate limit handled by the attempt before it
no longer turns the error into a 429, and both cores return 500.
* refactor(router): move litellm/router.py to litellm/router_python_core.py
A pure rename with no content change, so git and review tools track the router body as a move.
The next commit turns it into PythonCore and adds the Router facade back at litellm/router.py
* refactor(router): extract PythonCore behind RouterCore and pass the router its config
The routing logic moves unchanged from litellm/router.py into PythonCore in
litellm/router_python_core.py. Router becomes a facade that builds a RouterConfig, picks a
RouterCore with make_core and delegates every member to it. The core no longer reads or writes
litellm.* directly: process settings come from RouterDynamicConfig, and inference calls, sleeps,
callback lists and the global cache go through PythonHost. A CI check bans litellm.* in the router
modules, and the proxy's construction sites build the config outside the router
Settings are static: the constructor arguments and changeset values are copied when the router is
built or the change applied, and a write to a setting attribute becomes apply(ChangeSet(assign=...)).
A Router subclass's core class also subclasses the facade, so an override's super(Sub, self) works,
and an instance-level patch of a core method is kept on the facade so nested patches unwind.
tests/unit/test_router/core tests the core in isolation: ScriptedHost serves every host operation
(scripted inference answers, recorded sleeps, settings, callbacks, cache), and make_core runs each
test once per core in CORE_BUILDERS
Code outside the router now reaches it only through the entry points
(registry, settings, health, select, apply, observe, register_plugin, retire)
or the endpoint methods; the outside-member allowlist shrinks from 139 to 45.
litellm/router.py no longer imports litellm.proxy: it asks the installed
RouterHost (litellm/router_host.py) for vertex pass-through credentials,
member auto-router authorization and auto-router prompt compression, and the
proxy installs its host on import. Call sites where a model name can be
missing keep the old answer instead of reaching an entry point with None.
Router gains eight entry points (registry, settings, health, select, apply/aapply,
observe, register_plugin, retire) in litellm/router_api.py. The 123 public members
outside code used become DeprecatedRouterMembers (litellm/router_deprecated.py):
same signatures and results, a DeprecationWarning naming the replacement and a use
counter for callers outside the router. Router internals move to _core_ names;
subclass overrides of old members are still called by the router.
Add rustdoc overviews for a new litellm-router-core crate and for
litellm-model-catalog, a module docstring on litellm/router.py, and a
shrink-only allowlist of the Router members and router_utils modules used
outside the router, checked in CI by router_surface_check.py.
Move the router-free helpers out of litellm/router_utils: batch_utils to
litellm_core_utils, and the auto-router naming contract, tuning baseline
and baseline request next to the auto-router plugin. No behaviour change.
