mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
1d5ab42e14
* add CoreError::Routing variant for deployment selection failures * add minimal Rust Router (simple-shuffle) mirroring router.py spec * add litellm-router crate manifest * add ai-gateway POST /v1/realtime handler calling router.realtime * add ai-gateway health routes * wire ai-gateway routes into the axum app * add ai-gateway AppState holding the shared router * add ai-gateway axum server entrypoint * add litellm-ai-gateway binary crate manifest * docs: add ai-gateway folder-architecture AGENTS.md * register router + ai-gateway crates and axum/rand deps in workspace * update Cargo.lock for router + ai-gateway crates * split router: extract model_list types into deployment module * split router: extract routing policy into strategy module * split router: move Router orchestration into router module * router lib: wire submodules and re-export public API * add read_model_list helper reusing ProxyConfig env/secret resolution * add GIL-activity tracker (records acquisitions, 30s window) * add GET /health/gil endpoint for polling GIL activity * add pyo3 load_router_from_config bridge (feature-gated, load-time only) * register /health/gil route in ai-gateway * wire build_router: load from python config when feature enabled * add optional pyo3 dep + python-config feature to ai-gateway * update Cargo.lock for optional pyo3 dependency * fix: satisfy strict ruff budget (FA100) in read_model_list * test: cover read_model_list env resolution + empty config * ai-gateway: bind localhost by default, warn on bad PORT/missing keys, wire gateway key * ai-gateway: add gateway_key to AppState for realtime auth * ai-gateway: require bearer auth + map unknown model to 404 on /v1/realtime * ai-gateway: move python interop into python/ with load-time-only AGENTS.md * ai-gateway: document auth, gil, and python folder in AGENTS.md * core: add router module (model_list types + simple-shuffle selection) * ai-gateway: dispatch realtime via core router + providers (drop router crate dep) * update Cargo.lock: fold router into core * workspace: drop crates/router member and litellm-router dep * read_model_list: reuse ProxyConfig.get_config (includes + os.environ + DB) instead of thin yaml read * ai-gateway: constant-time bearer compare + 500 (not 503) for unconfigured key * ai-gateway: trim stored gateway key to match trimmed bearer token * ai-gateway: add subtle dep for constant-time comparison * workspace: add subtle dependency * update Cargo.lock for subtle * core router: make strategy a folder (one module per strategy, simple_shuffle) * providers: make realtime() a streaming splice (client stream <-> OpenAI) instead of collect * providers: add futures-channel dev-dep for the streaming live test * ai-gateway: make /v1/realtime a WebSocket (auth before upgrade, splice typed events) * ai-gateway: dispatch realtime as a stream splice * ai-gateway: route /v1/realtime via GET (WebSocket), drop POST * ai-gateway: enable axum ws feature + futures-util * update Cargo.lock for ws feature + futures-channel * core router: add has_deployment() for pre-flight model checks * ai-gateway: extract auth into auth/ module (single master key, LITELLM_MASTER_KEY) * ai-gateway routes: adopt router()-per-module template + merge in app() * ai-gateway: document auth/ + routes template in AGENTS.md * ai-gateway: realtime route as thin handler + service + transport * ai-gateway: auth as a RequireMasterKey extractor (idiomatic axum FromRequestParts) * ai-gateway: docs for auth extractor + simplified route template * ai-gateway: collapse realtime route to mod.rs + service.rs; docs for extractor/template * providers realtime: enforce idle timeout around the splice (reap stalled sessions) * ai-gateway: rename realtime service timeout param to idle_timeout --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
50 lines
2.7 KiB
Markdown
50 lines
2.7 KiB
Markdown
# ai-gateway — folder architecture
|
|
|
|
The Axum server that fronts the Rust gateway. It owns transport + config + auth
|
|
only; deployment selection lives in `core::router`, transforms in `core`/`providers`.
|
|
|
|
```
|
|
src/
|
|
main.rs # entrypoint: build AppState (router + master key), bind, serve
|
|
state.rs # AppState — shared Arc<Router> + master_key
|
|
gil.rs # GIL-activity tracker (records Python acquisitions)
|
|
auth/ # authentication as an axum extractor — added to handler args
|
|
mod.rs # RequireMasterKey: FromRequestParts, single master key (LITELLM_MASTER_KEY)
|
|
routes/ # one module per route, all matching the same template
|
|
AGENTS.md # ← the route template (read this before adding a route)
|
|
mod.rs # app(): merges every module's router()
|
|
health.rs # simple route (one file): router() + liveness/readiness
|
|
gil.rs # simple route (one file): router() + GET /health/gil
|
|
realtime/ # route with logic → axum surface + a no-axum service:
|
|
mod.rs # router() + handler + WS<->events adapter (the axum surface)
|
|
service.rs # business logic (select deployment, call provider) — no axum, testable
|
|
python/ # Python interop (feature: python-config) — load-time only
|
|
mod.rs, config.rs, AGENTS.md
|
|
```
|
|
|
|
## Rules
|
|
|
|
- **Routes follow one template.** Each route module exposes
|
|
`pub fn router() -> Router<AppState>`; `routes/mod.rs` only merges them. Simple
|
|
routes are one file; non-trivial routes are a folder (`handler`/`service`/
|
|
`transport`). See `routes/AGENTS.md`.
|
|
- **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's
|
|
args; it runs during extraction. Never re-implement the check per route.
|
|
- **Handlers are thin.** A handler validates and delegates to its `service`. No
|
|
business logic, no provider calls, no transforms in handlers.
|
|
- **State is shared and cheap to clone.** Long-lived handles live behind `Arc` in
|
|
`state.rs`; read env/config only in `main.rs` when building state.
|
|
|
|
## Auth (interim)
|
|
|
|
A single **master key** (`LITELLM_MASTER_KEY`), enforced by the
|
|
`auth::RequireMasterKey` extractor: any caller presenting it as
|
|
`Authorization: Bearer <key>` may invoke the gateway. Fails closed (500) when
|
|
unset; constant-time compare. The server binds `127.0.0.1` by default (`HOST` to
|
|
override). Full per-key auth + budgets/rate-limits are delegated to the Python
|
|
proxy in a later phase. Health routes don't add the extractor (unauthenticated).
|
|
|
|
## Python interop
|
|
|
|
Anything that calls into Python lives in `python/` and is **load-time only** — see
|
|
`python/AGENTS.md`. The realtime data path never takes the GIL.
|