mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-10 08:28:25 +00:00
d554f619d8
litellm-router-core implements the router core in Rust: a versioned registry with Python's deployment ids, selection with native simple-shuffle, the retry, fallback and cooldown loop with Python's cache keys, and per-attempt executors with a native seam for Messages to Anthropic. The python bridge exposes it as RouterCoreHandle, whose call loop is step driven so Rust never awaits Python. RustCore drives that loop for completions, messages, embeddings and responses and serves everything else from a PythonCore built from the same config. A new router rule in the route catalog opts in with LITELLM_RUST, and any config the Rust core cannot serve gets PythonCore at construction The call loop follows Python's fallback walk (per-target depth, a level raising its own error, lookup by the current and requested group, "*" after specific chains), stamps retries the way an exhausted retry loop does, and leaves health writes to the Python attempt, which records them. RustCore builds its handle from the constructor arguments and the applied changesets and rebuilds it when either version moves. The core unit suite runs on RustCore too |
||
|---|---|---|
| .. | ||
| core | ||
| __init__.py | ||
| test_enforce_model_rate_limits.py | ||
| test_io_token_rate_limits.py | ||
| test_router.py | ||
| test_router_callback_hook_sequence.py | ||
| test_router_endpoints.py | ||
| test_router_entry_points.py | ||
| test_router_fallbacks.py | ||
| test_router_get_deployments.py | ||
| test_router_helper_utils.py | ||
| test_router_host.py | ||
| test_router_max_parallel_requests.py | ||
| test_router_provider_endpoints.py | ||
| test_router_retries.py | ||
| test_router_surface_check.py | ||
| test_router_timeout.py | ||
| test_router_vector_store_endpoints.py | ||