mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-04 07:27:51 +00:00
4a57012560
Adds the litellm-router-core crate, its step-driven PyO3 bridge (RouterCoreHandle / RouterCall) and RustCore, served behind LITELLM_RUST through the ROUTER route rule. The host call-out is the only boundary: every attempt goes through Host::invoke, there is no native executor path, and RustCore never builds, wraps or forwards to PythonCore, so router_python_core.py stays deletable. Attempt preparation lives in litellm/router_attempt.py so it outlives router_python_core.py. RustCore matches PythonCore's behavior, including its known bugs (fixed in follow-up PRs). Routing decisions (strategy picks, routing groups, PTU, pre-routing, pre-call checks, fallback walk, health routing, access groups, weights, cooldowns, allowed fails, usage) live in the crate; work that needs live Python objects (request shaping, endpoints, streaming, model info facts, callbacks, alerting, prompt management) lives in host-side litellm/router_rust_*.py modules driven by Rust steps. The crate owns the registry. It runs the deployment build loop (secret references, ids, organization fan-out, install order) and asks the host for what needs Python, one deployment at a time so errors surface where Python raised them: secrets, validate, provider and added steps. It holds the model list, the index maps and the pattern routers with the Python core's history rules (host_index.rs), and answers the registry lookups (lookup.rs). The Python views of that state (model_list, index(), names(), patterns()) read the handle, and every write to them becomes a changeset. Model info, deployment model info, configured limits and model group info are merged in the crate (model_info.rs, group_info.rs) from catalog rows and facts the host answers on a model_info step asked at query time, so price reloads and register_model stay visible. The usage cache key format is defined once, in the crate. tests/unit/test_router/core characterizes PythonCore's public behavior and runs every test on both cores; the remaining Rust differences (the router's own outcome callbacks) are listed in rust_parity_pending.txt. |
||
|---|---|---|
| .. | ||
| auth | ||
| auth-aws | ||
| auth-azure | ||
| auth-gcp | ||
| auth-types | ||
| cache | ||
| cache-azure-blob | ||
| cache-disk | ||
| cache-gcs | ||
| cache-memory | ||
| cache-qdrant-semantic | ||
| cache-redis | ||
| cache-redis-semantic | ||
| cache-response | ||
| cache-s3 | ||
| cache-testing | ||
| cache-valkey-semantic | ||
| callbacks-legacy-python | ||
| circuit-breaker | ||
| config | ||
| core-utils | ||
| coroutine | ||
| cost | ||
| db | ||
| db-testing | ||
| error | ||
| framer | ||
| gateway | ||
| gateway-auth | ||
| gateway-inference | ||
| gateway-management | ||
| gateway-mcp | ||
| gateway-ui | ||
| host | ||
| host-http | ||
| host-native | ||
| host-python | ||
| http | ||
| inference | ||
| inference-chat | ||
| inference-messages | ||
| inference-ocr | ||
| inference-responses | ||
| inference-testing | ||
| inference-transcription | ||
| litellm | ||
| llms | ||
| llms-types | ||
| model-catalog | ||
| model-catalog-refresh | ||
| python-bridge | ||
| python-compat | ||
| redis-client | ||
| redis-testing | ||
| router | ||
| router-core | ||
| router-types | ||
| secret-value | ||
| secrets | ||
| secrets-aws | ||
| secrets-azure | ||
| secrets-cyberark | ||
| secrets-google | ||
| secrets-hashicorp | ||
| secrets-types | ||
| spend-clickhouse | ||
| storage-clickhouse | ||
| testkit | ||
| token-counter | ||
| token-counter-fast | ||
| token-counter-huggingface | ||
| token-counter-tiktoken | ||
| tracing | ||