mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-06 08:16:43 +00:00
refactor(rust): split gateway server crate
This commit is contained in:
Yujong Lee
parent
f59021f0e0
commit
53bf32b33f
43 changed files with 515 additions and 456 deletions
9
.github/workflows/test-rust.yml
vendored
9
.github/workflows/test-rust.yml
vendored
|
|
@ -74,9 +74,12 @@ jobs:
|
|||
- name: Run Clippy with Bedrock auth
|
||||
run: cargo clippy -p litellm-core --all-targets --features bedrock-auth --locked -- -D warnings
|
||||
|
||||
- name: Run Clippy with all gateway features
|
||||
- name: Run Clippy with all gateway runtime features
|
||||
run: cargo clippy -p litellm-ai-gateway --all-targets --all-features --locked -- -D warnings
|
||||
|
||||
- name: Run Clippy for the gateway server
|
||||
run: cargo clippy -p litellm-gateway-server --all-targets --locked -- -D warnings
|
||||
|
||||
- name: Run Rust tests
|
||||
run: cargo test --workspace --locked
|
||||
|
||||
|
|
@ -84,8 +87,8 @@ jobs:
|
|||
run: cargo test -p litellm-core --features bedrock-auth --locked
|
||||
|
||||
# Not --all-features: python-config links libpython, which this job does not install.
|
||||
- name: Run gateway tests with the server feature
|
||||
run: cargo test -p litellm-ai-gateway --features server --locked
|
||||
- name: Run gateway server tests
|
||||
run: cargo test -p litellm-gateway-server --locked
|
||||
|
||||
release-wheel:
|
||||
name: release wheel
|
||||
|
|
|
|||
|
|
@ -26,4 +26,4 @@ variants of it. The test for a good abstraction is that adding the next provider
|
|||
is a few declarative lines, not a new file of duplicated flow. Only diverge from
|
||||
the base when behavior is genuinely different, and say so explicitly in the PR.
|
||||
|
||||
**Calling:** hosts invoke the core entrypoint — the Python bridge and the `ai-gateway` route service both call `litellm_core::messages::messages`. Never add a provider handler to `ai-gateway`. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md).
|
||||
**Calling:** hosts invoke the core entrypoint. The Python bridge and the reusable `ai-gateway` runtime both call `litellm_core::messages::messages`, while `gateway-server` adapts HTTP requests to that runtime. Never add a provider handler to either host layer. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md).
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# AGENTS.md
|
||||
|
||||
litellm-rust has five crates. A crate is a layer or shared foundation, not a route. Routes (ocr, realtime, chat) and providers (mistral, openai) are modules inside the layers.
|
||||
litellm-rust has six crates. A crate is a layer, shared foundation, or separately built host, not a route. Routes (ocr, realtime, chat) and providers (mistral, openai) are modules inside the layers.
|
||||
|
||||
## Crates
|
||||
|
||||
|
|
@ -8,11 +8,12 @@ litellm-rust has five crates. A crate is a layer or shared foundation, not a rou
|
|||
|-------|------|
|
||||
| litellm-core | The LiteLLM SDK in Rust. One public entrypoint per top-level call (`messages::messages()`), owning types, transforms, provider resolution, auth, and the provider HTTP call. Call it, get a typed response. |
|
||||
| litellm-config | Config-loading boundary. Returns resolved core deployment data and optionally delegates loading to Python. |
|
||||
| litellm-ai-gateway | The axum server (behind the `server` feature) plus the WebSocket hosts. Translates HTTP/WS to core entrypoints; owns no provider logic and no handlers. |
|
||||
| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by the server and Python bridge. Owns transport-neutral orchestration and legacy call modules, but no Axum or Tower dependencies. |
|
||||
| litellm-gateway-server | The Axum binary and HTTP/WebSocket host. Owns routes, auth extractors, application state, startup config, and HTTP-only Tower dependencies. |
|
||||
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
|
||||
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
|
||||
|
||||
Dependency direction is acyclic: `litellm-config` depends on `litellm-core`, the gateway depends on both, and `litellm-python-bridge` depends on the domain layers and `litellm-python-interop`. The interop foundation depends on no LiteLLM domain crate.
|
||||
Dependency direction is acyclic: `litellm-config` depends on `litellm-core`; `litellm-ai-gateway` depends on core; `litellm-gateway-server` depends on gateway, core, and config; and `litellm-python-bridge` depends on the reusable domain layers and `litellm-python-interop`. Reusable crates must not depend on `litellm-gateway-server`. The interop foundation depends on no LiteLLM domain crate.
|
||||
|
||||
## Where a route lives
|
||||
|
||||
|
|
@ -28,7 +29,7 @@ core/src/messages/
|
|||
client.rs # the shared reqwest client
|
||||
```
|
||||
|
||||
Handlers never live in `ai-gateway`. `ocr`, `audio_transcription`, and `realtime` are still hosted there from before this rule; they move to `core` as they are touched.
|
||||
Provider handlers never live in `gateway-server`. `ocr`, `audio_transcription`, and realtime provider I/O are still hosted in `ai-gateway` from before this rule; they move to `core` as they are touched.
|
||||
|
||||
Adding a crate: default to a module. A new crate requires a real trigger: separate artifact (binary/cdylib), proc-macro, shared foundation, or publishable standalone. A new provider or route is none of these.
|
||||
|
||||
|
|
|
|||
|
|
@ -25,10 +25,12 @@ the base when behavior is genuinely different, and say so explicitly in the PR.
|
|||
|
||||
`litellm-core` **is** the LiteLLM SDK in Rust: it makes the LLM call.
|
||||
`litellm-config` is the config-loading boundary and returns resolved core types.
|
||||
`litellm-ai-gateway` is an HTTP/WebSocket server in front of it, and
|
||||
`litellm-python-bridge` exposes it to the Python SDK. `litellm-python-interop`
|
||||
holds domain-neutral PyO3 primitives shared by Python-facing Rust code. A crate
|
||||
is a layer or shared foundation, not a route; add modules, not crates.
|
||||
`litellm-ai-gateway` is the framework-independent gateway runtime and integration
|
||||
layer. `litellm-gateway-server` is the HTTP/WebSocket server in front of it, and
|
||||
`litellm-python-bridge` exposes reusable Rust APIs to the Python SDK.
|
||||
`litellm-python-interop` holds domain-neutral PyO3 primitives shared by
|
||||
Python-facing Rust code. A crate is a layer, shared foundation, or separate host,
|
||||
not a route; add modules, not crates.
|
||||
|
||||
## Core Boundary
|
||||
|
||||
|
|
@ -45,7 +47,7 @@ Route-level Rust structure mirrors LiteLLM's Python responsibilities:
|
|||
- `core/src/providers/<provider>/<route>/transformation.rs` owns the
|
||||
provider-specific transform. For Anthropic Messages, this means
|
||||
`core/src/providers/anthropic/messages/transformation.rs`.
|
||||
- Handlers live in `core`, never in a host. `ai-gateway` must not contain a
|
||||
- Provider handlers live in `core`, never in a host. `gateway-server` must not contain a
|
||||
route handler that talks to a provider; its axum route reads the HTTP request,
|
||||
picks a deployment, and calls the `core` entrypoint. `python-bridge` marshals
|
||||
Python objects and calls the same entrypoint.
|
||||
|
|
@ -57,7 +59,7 @@ its own caller; the host still owns no provider logic.
|
|||
Call-hook and lifecycle instrumentation, including phase timing, usage
|
||||
accumulation, and callback payload construction, always lives in `core`.
|
||||
Hosts feed observed events into core and dispatch the completed payloads through
|
||||
their I/O logger; hosts must not own callback orchestration.
|
||||
the reusable gateway integrations; HTTP hosts must not own callback orchestration.
|
||||
|
||||
Allowed in `core`:
|
||||
- The public entrypoint for a top-level LiteLLM call
|
||||
|
|
@ -81,9 +83,11 @@ Env reads in `core` are limited to credential fallback inside a route's
|
|||
no key is passed. Everything else config-shaped is resolved by the host and
|
||||
passed in.
|
||||
|
||||
Routes still hosted in `ai-gateway` (`ocr`, `audio_transcription`, `realtime`)
|
||||
predate this rule and are being moved into `core` route modules; do not add new
|
||||
ones there, and prefer moving one when you touch it.
|
||||
Legacy call runtimes still hosted in `ai-gateway` (`ocr`,
|
||||
`audio_transcription`, realtime provider I/O) predate this rule and are being
|
||||
moved into `core` route modules; do not add new ones there, and prefer moving
|
||||
one when you touch it. Axum routes, auth extractors, application state, and
|
||||
startup belong only in `gateway-server`.
|
||||
|
||||
Python owns rollout state and fallback while Rust is being introduced. Rust
|
||||
paths must be off by default until parity tests prove equivalence with Python.
|
||||
|
|
@ -121,7 +125,7 @@ the first PR:
|
|||
## Network I/O Rules
|
||||
|
||||
These rules apply to every module that executes network I/O, whether it is a
|
||||
`core` route handler or a host such as `ai-gateway`:
|
||||
`core` route handler or a host such as `gateway-server`:
|
||||
|
||||
- Set connect and full-request timeouts. No unbounded waits.
|
||||
- Reuse HTTP clients; do not construct clients per request.
|
||||
|
|
@ -177,12 +181,11 @@ cd litellm-rust
|
|||
cargo fmt --check
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
cargo clippy -p litellm-core --all-targets --features bedrock-auth -- -D warnings
|
||||
# the ai-gateway binary + server code is behind the `server` feature
|
||||
cargo clippy -p litellm-ai-gateway --all-targets --all-features -- -D warnings
|
||||
cargo clippy -p litellm-gateway-server --all-targets -- -D warnings
|
||||
cargo test --workspace
|
||||
cargo test -p litellm-core --features bedrock-auth
|
||||
# the `auth`, `routes`, `state` and `realtime` tests only exist under `server`
|
||||
cargo test -p litellm-ai-gateway --features server
|
||||
cargo test -p litellm-gateway-server
|
||||
```
|
||||
|
||||
When a Rust path is exposed through Python, add Python parity tests that compare
|
||||
|
|
|
|||
24
litellm-rust/Cargo.lock
generated
24
litellm-rust/Cargo.lock
generated
|
|
@ -1408,20 +1408,15 @@ checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
|
|||
name = "litellm-ai-gateway"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"axum",
|
||||
"base64",
|
||||
"futures-channel",
|
||||
"futures-util",
|
||||
"litellm-config",
|
||||
"litellm-core",
|
||||
"reqwest",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2 0.10.9",
|
||||
"subtle",
|
||||
"tokio",
|
||||
"tokio-tungstenite",
|
||||
"tower",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
|
|
@ -1458,6 +1453,25 @@ dependencies = [
|
|||
"tracing-subscriber",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "litellm-gateway-server"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"axum",
|
||||
"futures-util",
|
||||
"litellm-ai-gateway",
|
||||
"litellm-config",
|
||||
"litellm-core",
|
||||
"reqwest",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2 0.10.9",
|
||||
"subtle",
|
||||
"tokio",
|
||||
"tower",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "litellm-python-bridge"
|
||||
version = "0.1.0"
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ members = [
|
|||
"crates/core",
|
||||
"crates/config",
|
||||
"crates/ai-gateway",
|
||||
"crates/gateway-server",
|
||||
"crates/python-interop",
|
||||
"crates/python-bridge",
|
||||
]
|
||||
|
|
@ -20,6 +21,7 @@ tracing-subscriber = { version = "0.3", default-features = false, features = ["r
|
|||
litellm-core = { path = "crates/core" }
|
||||
litellm-config = { path = "crates/config" }
|
||||
litellm-ai-gateway = { path = "crates/ai-gateway", default-features = false }
|
||||
litellm-gateway-server = { path = "crates/gateway-server" }
|
||||
litellm-python-interop = { path = "crates/python-interop" }
|
||||
axum = "0.7"
|
||||
pyo3 = "0.29.2"
|
||||
|
|
|
|||
|
|
@ -26,11 +26,12 @@ coverage and production evidence.
|
|||
|-------|------|
|
||||
| litellm-core | The SDK. Per-route entrypoints (`messages::messages()`), types, provider transforms (modules under `providers/`), provider resolution, auth, the provider HTTP call, and the router. |
|
||||
| litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. |
|
||||
| litellm-ai-gateway | The axum server (behind the `server` feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. |
|
||||
| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by server and Python hosts. |
|
||||
| litellm-gateway-server | Axum binary, HTTP/WebSocket routes, auth extractors, application state, and HTTP-only dependencies. |
|
||||
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
|
||||
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. |
|
||||
|
||||
Dependency direction is acyclic: config depends on core, the gateway depends on config and core, and the Python bridge depends on the domain layers and Python interop.
|
||||
Dependency direction is acyclic: config and the gateway runtime depend on core; the gateway server depends on gateway, config, and core; and the Python bridge depends only on reusable domain layers and Python interop.
|
||||
|
||||
## Layout
|
||||
|
||||
|
|
@ -40,7 +41,8 @@ crates/
|
|||
src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client
|
||||
src/providers/anthropic/messages/transformation.rs
|
||||
config/ Config loading and resolved deployments.
|
||||
ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints.
|
||||
ai-gateway/ Framework-independent gateway runtime and integrations.
|
||||
gateway-server/ Axum binary and HTTP/WebSocket host.
|
||||
python-interop/ Domain-neutral PyO3 conversion and GIL primitives.
|
||||
python-bridge/ PyO3 API adapter for Python LiteLLM.
|
||||
```
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ Rules for adding or changing an LLM provider/route in `litellm-rust`. `messages`
|
|||
|
||||
## Boundaries
|
||||
|
||||
7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `ai-gateway` = serving HTTP/WS (routing, extractors, auth of *our* callers, streaming to the client); `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint; they never build a provider request.
|
||||
7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `ai-gateway` = framework-independent gateway runtime and integrations; `gateway-server` = HTTP/WS routing, extractors, auth of *our* callers, application state, and streaming to the client; `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint or reusable gateway runtime; they never build a provider request.
|
||||
8. Generic/route files contain zero provider-specific branches. A provider is one module under `core/src/providers/<provider>/<route>/`; a route is a module, never a new crate.
|
||||
9. Route entry point stays thin: `core::<route>::<route>()` -> `prepare_*` -> handler (or `CallLifecycle::run_request`, which owns the pre_call -> during_call -> provider call -> success/failure order and phase timing). Axum handlers validate and delegate to a service that calls the entrypoint; no business logic in them.
|
||||
10. Constants (URLs, env-var names, API versions, error messages) live in a crate `constants.rs`, never inline. Config-shaped env reads happen at the host/config layer with the `DEFAULT_*` fallback defined in `constants.rs`; the only env read in `core` is the credential fallback in a route's `prepare.rs`.
|
||||
|
|
|
|||
|
|
@ -1,54 +1,11 @@
|
|||
# ai-gateway — folder architecture
|
||||
# ai-gateway folder architecture
|
||||
|
||||
The Axum server that fronts the Rust gateway. It owns transport + config + auth
|
||||
only; deployment selection lives in `core::router`, and the LLM call itself
|
||||
(transforms, auth headers, provider HTTP) lives behind a `core` route entrypoint
|
||||
such as `litellm_core::messages::messages`. No provider handler lives here.
|
||||
This crate is a reusable, framework-independent gateway runtime and integration
|
||||
library. It is used by `litellm-gateway-server` and `litellm-python-bridge`
|
||||
|
||||
```
|
||||
src/
|
||||
main.rs # entrypoint: build AppState (router + master key), bind, serve
|
||||
state.rs # AppState — shared Arc<Router> + master_key
|
||||
auth/ # authentication as an axum extractor — added to handler args
|
||||
mod.rs # RequireMasterKey: FromRequestParts, single master key (LITELLM_MASTER_KEY)
|
||||
routes/ # one module per route, all matching the same template
|
||||
AGENTS.md # ← the route template (read this before adding a route)
|
||||
mod.rs # app(): merges every module's router()
|
||||
health.rs # simple route (one file): router() + liveness/readiness
|
||||
realtime/ # route with logic → axum surface + a no-axum service:
|
||||
mod.rs # router() + handler + WS<->events adapter (the axum surface)
|
||||
service.rs # business logic (select deployment, call provider) — no axum, testable
|
||||
```
|
||||
Axum routes, auth extractors, application state, startup, and Tower dependencies
|
||||
belong in `litellm-gateway-server`. This crate must not depend on the server
|
||||
|
||||
## Rules
|
||||
|
||||
- **Routes follow one template.** Each route module exposes
|
||||
`pub fn router() -> Router<AppState>`; `routes/mod.rs` only merges them. Simple
|
||||
routes are one file; non-trivial routes are a folder (`handler`/`service`/
|
||||
`transport`). See `routes/AGENTS.md`.
|
||||
- **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's
|
||||
args; it runs during extraction. Never re-implement the check per route.
|
||||
- **Handlers are thin.** A handler validates and delegates to its `service`. No
|
||||
business logic, no provider calls, no transforms in handlers.
|
||||
- **Services call `core`, they don't reimplement it.** A `service` picks the
|
||||
deployment and calls the `core` route entrypoint. Provider resolution, auth
|
||||
headers, URL building, and the HTTP call are `core`'s job; a service that
|
||||
builds a provider request itself is a bug (`routes/messages/service.rs` is
|
||||
the reference).
|
||||
- **State is shared and cheap to clone.** Long-lived handles live behind `Arc` in
|
||||
`state.rs`; read env/config only in `main.rs` when building state.
|
||||
|
||||
## Auth (interim)
|
||||
|
||||
A single **master key** (`LITELLM_MASTER_KEY`), enforced by the
|
||||
`auth::RequireMasterKey` extractor: any caller presenting it as
|
||||
`Authorization: Bearer <key>` may invoke the gateway. Fails closed (500) when
|
||||
unset; constant-time compare. The server binds `127.0.0.1` by default (`HOST` to
|
||||
override). Full per-key auth + budgets/rate-limits are delegated to the Python
|
||||
proxy in a later phase. Health routes don't add the extractor (unauthenticated).
|
||||
|
||||
## Python interop
|
||||
|
||||
Python-backed loading lives in `litellm-config` and is **load-time only**. The
|
||||
gateway's `python-config` feature forwards to that crate. The realtime data path
|
||||
never takes the GIL.
|
||||
Transport-neutral route orchestration lives under `src/runtime/`. Existing OCR,
|
||||
audio transcription, realtime I/O, and callback integrations remain here as
|
||||
migration seams until their provider call paths move into `litellm-core`
|
||||
|
|
|
|||
|
|
@ -1,14 +1,17 @@
|
|||
# ai-gateway architecture
|
||||
|
||||
The Rust ai-gateway does LLM inference (realtime WebSocket). Spend tracking is an
|
||||
API callback: it POSTs each finished session to the LiteLLM proxy, which records
|
||||
spend and runs the usual callbacks.
|
||||
`litellm-ai-gateway` is between hosts and `litellm-core`. It exposes
|
||||
framework-independent runtime services and callback integrations without
|
||||
depending on an HTTP framework
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C[client] <--> G[Rust ai-gateway<br/>LLM inference]
|
||||
G <--> O[OpenAI realtime]
|
||||
G -. spend tracking callback .-> P[litellm proxy]
|
||||
F[litellm-config<br/>load-time only] --> G
|
||||
F -. Python backend .-> P
|
||||
S[litellm-gateway-server] --> G[litellm-ai-gateway runtime]
|
||||
B[litellm-python-bridge] --> G
|
||||
G --> C[litellm-core]
|
||||
S --> C
|
||||
S --> F[litellm-config]
|
||||
```
|
||||
|
||||
The server may depend on the runtime, core, and config crates. Reusable crates
|
||||
must not depend on the server
|
||||
|
|
|
|||
|
|
@ -8,15 +8,9 @@ repository.workspace = true
|
|||
[lib]
|
||||
name = "litellm_ai_gateway"
|
||||
|
||||
[[bin]]
|
||||
name = "litellm-ai-gateway"
|
||||
path = "src/main.rs"
|
||||
required-features = ["server"]
|
||||
|
||||
[dependencies]
|
||||
tracing.workspace = true
|
||||
litellm-core = { workspace = true, features = ["bedrock-auth"] }
|
||||
litellm-config.workspace = true
|
||||
# reqwest (rustls + json) is used by io/ocr and ships realtime logs to the
|
||||
# Python proxy callbacks API.
|
||||
reqwest.workspace = true
|
||||
|
|
@ -26,22 +20,11 @@ tokio-tungstenite.workspace = true
|
|||
futures-util.workspace = true
|
||||
serde_json.workspace = true
|
||||
base64.workspace = true
|
||||
axum = { workspace = true, features = ["ws"], optional = true }
|
||||
serde.workspace = true
|
||||
subtle = { workspace = true, optional = true }
|
||||
# sha2 hashes the master key into user_api_key_hash (matches the proxy's
|
||||
# SHA-256 hash_token) so the plaintext credential never enters a log payload.
|
||||
sha2 = { workspace = true, optional = true }
|
||||
tower = { version = "0.5.3", features = ["util"], optional = true }
|
||||
|
||||
[features]
|
||||
default = []
|
||||
server = ["dep:axum", "dep:subtle", "dep:sha2"]
|
||||
# Build the gateway's config from the proxy YAML via an embedded Python
|
||||
# interpreter (links libpython; requires `litellm` importable at runtime).
|
||||
python-config = ["litellm-config/python"]
|
||||
trace-parity = ["server", "dep:tower", "litellm-core/observability"]
|
||||
trace-parity = ["litellm-core/observability"]
|
||||
|
||||
[dev-dependencies]
|
||||
futures-channel = "0.3"
|
||||
tower = { version = "0.5.3", features = ["util"] }
|
||||
|
|
|
|||
|
|
@ -1,205 +1,12 @@
|
|||
# LiteLLM Rust AI Gateway
|
||||
# LiteLLM AI Gateway Runtime
|
||||
|
||||
A minimal Axum service that fronts OpenAI's realtime API. Clients open a
|
||||
WebSocket to `GET /v1/realtime`; the gateway authenticates, selects a deployment,
|
||||
dials OpenAI upstream, and splices the two sockets frame-by-frame.
|
||||
`litellm-ai-gateway` is the reusable, framework-independent runtime used by the
|
||||
Rust gateway server and Python bridge
|
||||
|
||||
## Crates
|
||||
It owns gateway integrations, transport-neutral route orchestration, and legacy
|
||||
OCR, audio transcription, and realtime I/O that have not yet moved into
|
||||
`litellm-core`. It has no Axum or Tower dependency
|
||||
|
||||
`litellm-rust` has five crates. A crate is a layer or shared foundation, not a route:
|
||||
|
||||
| Crate | Role |
|
||||
|-------|------|
|
||||
| litellm-core | The LiteLLM SDK in Rust — per-route entrypoints (`messages::messages()`) that resolve the provider, transform, and make the call; plus types, provider transforms, and the router. |
|
||||
| litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. |
|
||||
| litellm-ai-gateway | The Axum server (behind the `server` feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. |
|
||||
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
|
||||
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. |
|
||||
|
||||
Dependency direction is acyclic: config depends on core, the gateway depends on config and core, and the Python bridge depends on the domain layers and Python interop.
|
||||
|
||||
- **Client endpoint:** `wss://<host>/v1/realtime?model=<model>` (WebSocket)
|
||||
- **Auth:** `Authorization: Bearer $LITELLM_MASTER_KEY` (fails closed if unset)
|
||||
- **Health:** `GET /health/readiness`, `GET /health/liveness`
|
||||
- **Request logs:** POSTed to a LiteLLM proxy at `/v1/rust_control_plane/logs` (see [Request logging](#request-logging))
|
||||
|
||||
> **Realtime serving is pure Rust.** Python is used at **load time only** — to
|
||||
> read the config once at boot. The realtime hot path never touches Python.
|
||||
|
||||
The former `/health/gil` route and its acquisition counter were removed. They
|
||||
only observed the single startup config load and did not prove that every GIL
|
||||
acquisition was instrumented
|
||||
|
||||
## Configuration (config.yaml)
|
||||
|
||||
The gateway loads its `model_list` from a **config.yaml**, the same as the
|
||||
LiteLLM proxy. Point `LITELLM_CONFIG_PATH` at the file:
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
model_list:
|
||||
- model_name: gpt-realtime
|
||||
litellm_params:
|
||||
model: openai/gpt-realtime
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
```
|
||||
|
||||
```bash
|
||||
LITELLM_CONFIG_PATH=./config.yaml ./litellm-ai-gateway
|
||||
```
|
||||
|
||||
At boot `litellm-config` calls into `litellm.proxy.read_model_list` and returns
|
||||
resolved deployments to the gateway, which constructs the router. The Python
|
||||
backend still reuses the **real proxy config reader** (`ProxyConfig.get_config`),
|
||||
so everything the proxy supports in config.yaml works here too:
|
||||
|
||||
- `include:` to merge in other config files,
|
||||
- `os.environ/VAR` secret references (resolved via the secret manager, never
|
||||
inlined),
|
||||
- DB-stored models (when a database is configured).
|
||||
|
||||
Secrets stay out of the config — reference them with `os.environ/...` and set
|
||||
the env var at deploy time. The shipped Docker image is built with the
|
||||
`python-config` feature and **bundles litellm**, so config loading works out of
|
||||
the box; the default baked config lives at `/app/config.yaml` and can be
|
||||
overridden at deploy time (e.g. a Render secret file mounted at the same path).
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Var | Required | Default | Purpose |
|
||||
|---|---|---|---|
|
||||
| `LITELLM_CONFIG_PATH` | yes (config mode) | — | Path to the config.yaml the gateway loads its `model_list` from. The Docker image defaults this to `/app/config.yaml`. |
|
||||
| `LITELLM_MASTER_KEY` | yes | — | Bearer token clients must send. Unset ⇒ all `/v1/realtime` requests are rejected (fail closed). |
|
||||
| `OPENAI_API_KEY` | yes | — | Upstream OpenAI key. Referenced by config.yaml as `os.environ/OPENAI_API_KEY` for the gateway→OpenAI dial. |
|
||||
| `HOST` | no | `127.0.0.1` | **Set to `0.0.0.0` in any container/deploy** or external traffic is refused. |
|
||||
| `PORT` | no | `4001` | Listen port. Render and most PaaS inject this automatically. |
|
||||
| `LITELLM_PROXY_BASE_URL` | no | `http://localhost:4000` | LiteLLM proxy that request logs are POSTed to. See [Request logging](#request-logging). |
|
||||
|
||||
> Secrets (`LITELLM_MASTER_KEY`, `OPENAI_API_KEY`) are never baked into the image
|
||||
> or `render.yaml` — inject them at deploy time only.
|
||||
|
||||
### Lean env stand-in (fallback)
|
||||
|
||||
If the binary is built **without** `python-config` (default features), or
|
||||
`LITELLM_CONFIG_PATH` is unset, the gateway falls back to a single-deployment
|
||||
stand-in built from the environment:
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `OPENAI_REALTIME_MODEL` | `gpt-realtime` | The single deployment's model name (also the `?model=` clients pass). |
|
||||
|
||||
The default workspace build links no libpython and needs no config file. This
|
||||
fallback mode only supports one hard-coded OpenAI deployment. **config.yaml is the recommended path** — use the
|
||||
stand-in only for the leanest possible build.
|
||||
|
||||
## Request logging
|
||||
|
||||
The gateway runs no spend logic. When a session ends it builds one
|
||||
`StandardLoggingPayload` and POSTs it to `{LITELLM_PROXY_BASE_URL}/v1/rust_control_plane/logs`
|
||||
(admin-only, bearer = `LITELLM_MASTER_KEY`), and the proxy replays it through its
|
||||
normal callbacks (spend logs, Langfuse, etc.). The POST is non-blocking: a bounded
|
||||
channel drained by a background worker, dropping with a counter if the proxy is
|
||||
down. It sends one payload per session. Both env vars are in the table above.
|
||||
|
||||
Worker tuning, rarely needed: `LITELLM_LOG_CHANNEL_CAPACITY` (4096),
|
||||
`LITELLM_LOG_BATCH_SIZE` (256), `LITELLM_LOG_FLUSH_INTERVAL_MS` (500).
|
||||
|
||||
## Build & run with Docker
|
||||
|
||||
The image is built `--features server,python-config` and installs litellm **from this
|
||||
repo's source** (the config reader is newer than any PyPI release), so the build
|
||||
**context is the repo root**:
|
||||
|
||||
```bash
|
||||
# from the repo root
|
||||
docker build -f litellm-rust/crates/ai-gateway/Dockerfile -t litellm-ai-gateway .
|
||||
|
||||
docker run --rm -p 4001:4001 \
|
||||
-e HOST=0.0.0.0 -e PORT=4001 \
|
||||
-e LITELLM_MASTER_KEY=sk-local \
|
||||
-e OPENAI_API_KEY=$OPENAI_API_KEY \
|
||||
litellm-ai-gateway # LITELLM_CONFIG_PATH defaults to /app/config.yaml
|
||||
|
||||
# smoke test
|
||||
curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/health/readiness # -> 200
|
||||
curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/v1/realtime # -> 401 (auth fails closed)
|
||||
```
|
||||
|
||||
On boot you should see `loaded model_list from /app/config.yaml via python
|
||||
config reader` — that confirms the config path (not the env stand-in fallback).
|
||||
To use your own config, mount it over the default:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 4001:4001 \
|
||||
-e HOST=0.0.0.0 -e LITELLM_MASTER_KEY=sk-local -e OPENAI_API_KEY=$OPENAI_API_KEY \
|
||||
-v $(pwd)/my-config.yaml:/app/config.yaml:ro \
|
||||
litellm-ai-gateway
|
||||
```
|
||||
|
||||
### Cargo-only (no Docker)
|
||||
|
||||
```bash
|
||||
# config.yaml mode — needs litellm importable in the active python env
|
||||
LITELLM_CONFIG_PATH=./crates/ai-gateway/config.yaml \
|
||||
cargo run --release -p litellm-ai-gateway --features server,python-config
|
||||
|
||||
# env stand-in mode — no python, no config
|
||||
cargo run --release -p litellm-ai-gateway --features server
|
||||
```
|
||||
|
||||
## Deploy on Render
|
||||
|
||||
The service is a Docker **web service**; Render terminates TLS and supports
|
||||
WebSockets, so the public endpoint is `wss://<service>.onrender.com/v1/realtime`.
|
||||
|
||||
### Option A — Blueprint (`render.yaml`)
|
||||
|
||||
`crates/ai-gateway/render.yaml` describes the service (Docker runtime,
|
||||
`healthCheckPath: /health/readiness`, repo-root `dockerContext: .`,
|
||||
`dockerfilePath: ./litellm-rust/crates/ai-gateway/Dockerfile`,
|
||||
`LITELLM_CONFIG_PATH: /app/config.yaml`). `LITELLM_MASTER_KEY` and
|
||||
`OPENAI_API_KEY` are `sync: false` — set them in the dashboard after the first
|
||||
deploy. To use a non-default model_list, mount a **Render Secret File** at
|
||||
`/app/config.yaml`. Point a Render Blueprint at this repo/branch and apply.
|
||||
|
||||
### Option B — Render API
|
||||
|
||||
```bash
|
||||
# create a Docker web service from this repo+branch, then set env vars:
|
||||
curl -X POST https://api.render.com/v1/services \
|
||||
-H "Authorization: Bearer $RENDER_API_KEY" -H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"type": "web_service", "name": "litellm-rust-ai-gateway",
|
||||
"ownerId": "<owner-id>", "repo": "https://github.com/BerriAI/litellm",
|
||||
"branch": "<branch-with-this-dockerfile>",
|
||||
"serviceDetails": {
|
||||
"env": "docker",
|
||||
"envSpecificDetails": {
|
||||
"dockerfilePath": "./litellm-rust/crates/ai-gateway/Dockerfile",
|
||||
"dockerContext": "."
|
||||
},
|
||||
"healthCheckPath": "/health/readiness"
|
||||
}
|
||||
}'
|
||||
# then set env vars LITELLM_MASTER_KEY, OPENAI_API_KEY, HOST=0.0.0.0,
|
||||
# LITELLM_CONFIG_PATH=/app/config.yaml
|
||||
```
|
||||
|
||||
Health check path **must** be `/health/readiness`. `autoDeploy` is off by default
|
||||
in the blueprint — trigger deploys manually (or flip it on) to pick up new commits.
|
||||
|
||||
## Scaling
|
||||
|
||||
Concurrency is what matters, not total connections: each in-flight session holds
|
||||
one client socket + one upstream socket. To scale, raise the instance count /
|
||||
enable autoscaling on the Render service (e.g. baseline 10, max 100). Each
|
||||
instance needs file descriptors for `2 × peak_concurrent_sessions` — raise
|
||||
`ulimit -n` if you push very high concurrency.
|
||||
|
||||
## Latency note
|
||||
|
||||
The gateway adds the cost of one extra hop: client→gateway, then a fresh
|
||||
gateway→OpenAI realtime handshake (TLS + WS upgrade + `session.created`). In
|
||||
benchmarks this is ~100–150 ms of added session-establishment time; first-audio
|
||||
and steady-state streaming add no measurable overhead. To minimize it, deploy the
|
||||
gateway in the Render region with the lowest RTT to OpenAI's realtime endpoint.
|
||||
The executable server, HTTP/WebSocket routes, auth extractors, application
|
||||
state, config startup, Docker image, and deployment files live in
|
||||
[`../gateway-server`](../gateway-server)
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Measures what the gateway adds over talking to OpenAI's realtime WebSocket
|
||||
directly, and what the pre-warmed connection pool removes. See
|
||||
`../../src/routes/realtime/README.md` for how the pool works.
|
||||
`../../../gateway-server/src/routes/realtime/README.md` for how the pool works.
|
||||
|
||||
## Results
|
||||
|
||||
|
|
|
|||
|
|
@ -25,18 +25,7 @@ pub(crate) const DEFAULT_MAX_BATCH_SIZE: usize = 256;
|
|||
/// Override: `LITELLM_LOG_FLUSH_INTERVAL_MS`.
|
||||
pub(crate) const DEFAULT_FLUSH_INTERVAL_MS: u64 = 500;
|
||||
|
||||
/// Provider attributed to realtime sessions in the logging payload.
|
||||
#[cfg(feature = "server")]
|
||||
pub(crate) const DEFAULT_PROVIDER: &str = "openai";
|
||||
|
||||
pub(crate) const DEFAULT_RESPONSES_WS_CONNECT_TIMEOUT_SECS: u64 = 10;
|
||||
pub(crate) const DEFAULT_RESPONSES_WS_IDLE_TIMEOUT_SECS: u64 = 300;
|
||||
|
||||
/// HTTP path for the non-streaming Anthropic Messages route.
|
||||
#[cfg(feature = "server")]
|
||||
pub(crate) const MESSAGES_ROUTE_PATH: &str = "/v1/messages";
|
||||
|
||||
/// Request headers owned by the gateway and never forwarded upstream.
|
||||
#[cfg(feature = "server")]
|
||||
pub(crate) const MESSAGES_HEADERS_NOT_FORWARDED: &[&str] =
|
||||
&["authorization", "connection", "content-length", "host"];
|
||||
|
|
|
|||
|
|
@ -212,7 +212,7 @@ where
|
|||
/// `transform_realtime_response`). Returns when either side closes.
|
||||
///
|
||||
/// Generic over the client transport (typed events) so this crate stays
|
||||
/// framework-agnostic; the gateway adapts its axum socket to these. This is the
|
||||
/// framework-agnostic; the gateway server adapts its Axum socket to these. This is the
|
||||
/// fresh-dial path: dial, then splice. The pool's warm-handoff path skips the dial
|
||||
/// and calls [`splice`] directly with a buffered `session.created`.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
|
|
@ -295,7 +295,7 @@ mod tests {
|
|||
|
||||
/// Live end-to-end check against OpenAI. Ignored by default (CI never runs
|
||||
/// it); run explicitly with `OPENAI_API_KEY` set:
|
||||
/// `cargo test -p litellm-ai-gateway --features server realtime_invokes_openai -- --ignored --nocapture`
|
||||
/// `cargo test -p litellm-ai-gateway realtime_invokes_openai -- --ignored --nocapture`
|
||||
#[tokio::test]
|
||||
#[ignore = "hits the live OpenAI realtime API; needs OPENAI_API_KEY"]
|
||||
async fn realtime_invokes_openai_and_responds() {
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
//! reuses. The gateway holds an `Arc<RealtimePool>` in its state and asks for a
|
||||
//! warm socket per connect; on a miss it fresh-dials exactly as before. The pool
|
||||
//! is a latency optimization, never a correctness dependency — see the gateway's
|
||||
//! `src/routes/realtime/README.md`.
|
||||
//! `crates/gateway-server/src/routes/realtime/README.md`.
|
||||
//!
|
||||
//! ## Caveats (enforced here)
|
||||
//! - One warm socket serves exactly one session (realtime isn't multiplexed), so
|
||||
|
|
|
|||
|
|
@ -1,32 +1,24 @@
|
|||
//! LiteLLM AI Gateway library.
|
||||
//!
|
||||
//! Two layers, split by feature so the Python `cdylib` can depend on the I/O
|
||||
//! without pulling in the HTTP server:
|
||||
//!
|
||||
//! - Call-type modules such as [`ocr`]: provider transforms, lifecycle hooks,
|
||||
//! and provider I/O. Always available — no feature required. These predate the
|
||||
//! rule that a route's entrypoint and handler live in `litellm-core` (see
|
||||
//! `litellm_core::messages`) and move there as they are touched.
|
||||
//! - [`io`]: compatibility exports and realtime WebSocket splice helpers.
|
||||
//! - The server modules ([`auth`], [`routes`], [`state`]) and anything pulling
|
||||
//! `axum` are gated behind the `server` feature, which the `litellm-ai-gateway`
|
||||
//! binary turns on.
|
||||
//! - [`runtime`]: framework-independent orchestration used by HTTP and Python
|
||||
//! hosts.
|
||||
//!
|
||||
//! The Axum host lives in the separate `litellm-gateway-server` crate.
|
||||
|
||||
pub mod audio_transcription;
|
||||
mod client;
|
||||
pub mod io;
|
||||
pub mod ocr;
|
||||
|
||||
#[cfg(feature = "server")]
|
||||
pub mod auth;
|
||||
#[cfg(feature = "server")]
|
||||
pub mod routes;
|
||||
#[cfg(feature = "server")]
|
||||
pub mod state;
|
||||
pub mod realtime;
|
||||
pub mod runtime;
|
||||
#[cfg(feature = "trace-parity")]
|
||||
pub mod trace_parity;
|
||||
|
||||
mod constants;
|
||||
pub mod integrations;
|
||||
#[cfg(feature = "server")]
|
||||
mod realtime;
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ use litellm_core::messages::{messages, messages_stream};
|
|||
use litellm_core::router::Router;
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
pub(crate) enum MessagesResponse {
|
||||
pub enum MessagesResponse {
|
||||
Json(Value),
|
||||
Stream(reqwest::Response),
|
||||
}
|
||||
3
litellm-rust/crates/ai-gateway/src/runtime/mod.rs
Normal file
3
litellm-rust/crates/ai-gateway/src/runtime/mod.rs
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
pub mod messages;
|
||||
pub mod realtime;
|
||||
pub mod responses;
|
||||
|
|
@ -1,19 +1,13 @@
|
|||
//! Harness-only in-process adapters. Never mounted as production routes.
|
||||
//! Harness-only adapters. Never mounted as production routes.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use axum::body::{Body, to_bytes};
|
||||
use axum::http::header::{AUTHORIZATION, CONTENT_TYPE};
|
||||
use axum::http::{Request, StatusCode};
|
||||
use litellm_core::Error;
|
||||
use litellm_core::router::{Deployment, LiteLLMParams, Router as ModelRouter};
|
||||
use litellm_core::router::{Deployment, LiteLLMParams, Router};
|
||||
use serde::Serialize;
|
||||
use serde_json::Value;
|
||||
use tower::ServiceExt;
|
||||
|
||||
use crate::io::realtime_pool::RealtimePool;
|
||||
use crate::routes;
|
||||
use crate::state::AppState;
|
||||
use crate::runtime::messages::{MessagesResponse, run};
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct GatewayResponse {
|
||||
|
|
@ -21,45 +15,53 @@ pub struct GatewayResponse {
|
|||
pub body: Value,
|
||||
}
|
||||
|
||||
#[tracing::instrument(
|
||||
name = "messages_gateway_route",
|
||||
target = "litellm::function_trace",
|
||||
level = "trace",
|
||||
skip_all
|
||||
)]
|
||||
pub async fn messages_request(
|
||||
model_alias: String,
|
||||
provider_model: String,
|
||||
api_base: String,
|
||||
body: Value,
|
||||
) -> Result<GatewayResponse, Error> {
|
||||
let state = AppState {
|
||||
router: Arc::new(ModelRouter::new(vec![Deployment {
|
||||
model_name: model_alias,
|
||||
litellm_params: LiteLLMParams {
|
||||
model: provider_model,
|
||||
api_key: Some("trace-provider-key".to_string()),
|
||||
api_base: Some(api_base),
|
||||
},
|
||||
}])),
|
||||
master_key: Some(Arc::from("trace-master-key")),
|
||||
loggers: Arc::new(Vec::new()),
|
||||
realtime_pool: RealtimePool::disabled(),
|
||||
};
|
||||
let request = Request::builder()
|
||||
.method("POST")
|
||||
.uri("/v1/messages")
|
||||
.header(AUTHORIZATION, "Bearer trace-master-key")
|
||||
.header(CONTENT_TYPE, "application/json")
|
||||
.body(Body::from(body.to_string()))
|
||||
.map_err(|error| Error::InvalidRequest(error.to_string()))?;
|
||||
let response = routes::app(state)
|
||||
.oneshot(request)
|
||||
.await
|
||||
.map_err(|error| match error {})?;
|
||||
let status: StatusCode = response.status();
|
||||
let bytes = to_bytes(response.into_body(), usize::MAX)
|
||||
.await
|
||||
.map_err(|error| Error::InvalidResponse(error.to_string()))?;
|
||||
let body = serde_json::from_slice(&bytes).map_err(|error| {
|
||||
Error::InvalidResponse(format!("gateway returned invalid JSON: {error}"))
|
||||
})?;
|
||||
Ok(GatewayResponse {
|
||||
status: status.as_u16(),
|
||||
body,
|
||||
})
|
||||
let router = Arc::new(Router::new(vec![Deployment {
|
||||
model_name: model_alias,
|
||||
litellm_params: LiteLLMParams {
|
||||
model: provider_model,
|
||||
api_key: Some("trace-provider-key".to_string()),
|
||||
api_base: Some(api_base),
|
||||
},
|
||||
}]));
|
||||
match run(&router, body, None).await {
|
||||
Ok(MessagesResponse::Json(body)) => Ok(GatewayResponse { status: 200, body }),
|
||||
Ok(MessagesResponse::Stream(_)) => Err(Error::InvalidResponse(
|
||||
"gateway returned a streaming trace response".to_string(),
|
||||
)),
|
||||
Err(error) => Ok(error_response(error)),
|
||||
}
|
||||
}
|
||||
|
||||
fn error_response(error: Error) -> GatewayResponse {
|
||||
let (status, message) = match error {
|
||||
Error::InvalidRequest(message) => (400, message),
|
||||
Error::InvalidProvider(_) | Error::Routing(_) => (
|
||||
404,
|
||||
"no messages deployment is configured for this model".to_string(),
|
||||
),
|
||||
Error::Auth(_) => (502, "messages provider authentication failed".to_string()),
|
||||
Error::Http { .. }
|
||||
| Error::Network(_)
|
||||
| Error::Connect(_)
|
||||
| Error::InvalidResponse(_)
|
||||
| Error::InvalidType { .. }
|
||||
| Error::MissingField(_) => (502, "messages provider request failed".to_string()),
|
||||
Error::Unsupported(reason) => (400, format!("messages request is not supported: {reason}")),
|
||||
};
|
||||
GatewayResponse {
|
||||
status,
|
||||
body: serde_json::json!({"error": {"message": message}}),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
//! Enforcement: the litellm-rust workspace has exactly five crates.
|
||||
//! Enforcement: the litellm-rust workspace has exactly six crates.
|
||||
//!
|
||||
//! `core` (the Rust SDK), `config` (the config-loading boundary),
|
||||
//! `ai-gateway` (the HTTP/WebSocket host),
|
||||
//! `ai-gateway` (framework-independent gateway runtime),
|
||||
//! `gateway-server` (the HTTP/WebSocket host),
|
||||
//! `python-interop` (domain-neutral PyO3 primitives), and `python-bridge` (the
|
||||
//! PyO3 cdylib). Adding or removing a crate must be a
|
||||
//! deliberate act: this test fails until the allowlist here is updated, forcing
|
||||
|
|
@ -22,6 +23,7 @@ const EXPECTED_MEMBERS: &[&str] = &[
|
|||
"crates/core",
|
||||
"crates/config",
|
||||
"crates/ai-gateway",
|
||||
"crates/gateway-server",
|
||||
"crates/python-interop",
|
||||
"crates/python-bridge",
|
||||
];
|
||||
|
|
@ -31,6 +33,7 @@ const EXPECTED_CRATE_DIRS: &[&str] = &[
|
|||
"core",
|
||||
"config",
|
||||
"ai-gateway",
|
||||
"gateway-server",
|
||||
"python-interop",
|
||||
"python-bridge",
|
||||
];
|
||||
|
|
|
|||
52
litellm-rust/crates/gateway-server/AGENTS.md
Normal file
52
litellm-rust/crates/gateway-server/AGENTS.md
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# gateway-server folder architecture
|
||||
|
||||
The Axum server that fronts the reusable Rust gateway runtime. It owns HTTP/WS
|
||||
transport, startup config, auth extractors, and application state only.
|
||||
Deployment selection and transport-neutral orchestration live in
|
||||
`litellm-ai-gateway` or `core::router`, and provider calls live behind core route
|
||||
entrypoints such as `litellm_core::messages::messages`. No provider handler
|
||||
lives here.
|
||||
|
||||
```
|
||||
src/
|
||||
main.rs # entrypoint: build AppState (router + master key), bind, serve
|
||||
state.rs # AppState — shared Arc<Router> + master_key
|
||||
auth/ # authentication as an axum extractor — added to handler args
|
||||
mod.rs # RequireMasterKey: FromRequestParts, single master key (LITELLM_MASTER_KEY)
|
||||
routes/ # one module per route, all matching the same template
|
||||
AGENTS.md # ← the route template (read this before adding a route)
|
||||
mod.rs # app(): merges every module's router()
|
||||
health.rs # simple route (one file): router() + liveness/readiness
|
||||
realtime/ # router() + handler + WS<->events adapter
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- **Routes follow one template.** Each route module exposes
|
||||
`pub fn router() -> Router<AppState>`; `routes/mod.rs` only merges them. Simple
|
||||
routes are one file; non-trivial routes are a folder (`handler`/`service`/
|
||||
`transport`). See `routes/AGENTS.md`.
|
||||
- **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's
|
||||
args; it runs during extraction. Never re-implement the check per route.
|
||||
- **Handlers are thin.** A handler validates and delegates to `litellm-ai-gateway::runtime`. No
|
||||
business logic, no provider calls, no transforms in handlers.
|
||||
- **Runtime services call `core`, they don't reimplement it.** The reusable
|
||||
service picks the deployment and calls the `core` route entrypoint. Provider
|
||||
resolution, auth headers, URL building, and the HTTP call are `core`'s job.
|
||||
- **State is shared and cheap to clone.** Long-lived handles live behind `Arc` in
|
||||
`state.rs`; read env/config only in `main.rs` when building state.
|
||||
|
||||
## Auth (interim)
|
||||
|
||||
A single **master key** (`LITELLM_MASTER_KEY`), enforced by the
|
||||
`auth::RequireMasterKey` extractor: any caller presenting it as
|
||||
`Authorization: Bearer <key>` may invoke the gateway. Fails closed (500) when
|
||||
unset; constant-time compare. The server binds `127.0.0.1` by default (`HOST` to
|
||||
override). Full per-key auth + budgets/rate-limits are delegated to the Python
|
||||
proxy in a later phase. Health routes don't add the extractor (unauthenticated).
|
||||
|
||||
## Python interop
|
||||
|
||||
Python-backed loading lives in `litellm-config` and is **load-time only**. The
|
||||
server's `python-config` feature forwards to that crate. The realtime data path
|
||||
never takes the GIL.
|
||||
16
litellm-rust/crates/gateway-server/ARCHITECTURE.md
Normal file
16
litellm-rust/crates/gateway-server/ARCHITECTURE.md
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# gateway-server architecture
|
||||
|
||||
The Rust gateway server owns the Axum binary, HTTP/WebSocket routes, auth
|
||||
extractors, application state, and startup config. It delegates transport-neutral
|
||||
orchestration and callback integrations to `litellm-ai-gateway`
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C[client] <--> S[litellm-gateway-server<br/>Axum host]
|
||||
S --> G[litellm-ai-gateway<br/>runtime and integrations]
|
||||
G --> K[litellm-core]
|
||||
G <--> O[OpenAI realtime]
|
||||
G -. spend tracking callback .-> P[litellm proxy]
|
||||
F[litellm-config<br/>load-time only] --> S
|
||||
F -. Python backend .-> P
|
||||
```
|
||||
34
litellm-rust/crates/gateway-server/Cargo.toml
Normal file
34
litellm-rust/crates/gateway-server/Cargo.toml
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
[package]
|
||||
name = "litellm-gateway-server"
|
||||
version = "0.1.0"
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
repository.workspace = true
|
||||
|
||||
[lib]
|
||||
name = "litellm_gateway_server"
|
||||
|
||||
[[bin]]
|
||||
name = "litellm-ai-gateway"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
axum = { workspace = true, features = ["ws"] }
|
||||
futures-util.workspace = true
|
||||
litellm-ai-gateway.workspace = true
|
||||
litellm-config.workspace = true
|
||||
litellm-core = { workspace = true, features = ["bedrock-auth"] }
|
||||
reqwest.workspace = true
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
sha2.workspace = true
|
||||
subtle.workspace = true
|
||||
tokio = { workspace = true, features = ["rt-multi-thread", "macros", "net", "time"] }
|
||||
tracing.workspace = true
|
||||
|
||||
[features]
|
||||
default = []
|
||||
python-config = ["litellm-config/python"]
|
||||
|
||||
[dev-dependencies]
|
||||
tower = { version = "0.5.3", features = ["util"] }
|
||||
|
|
@ -5,7 +5,7 @@
|
|||
# which is not in any PyPI release yet) AND build the rust workspace under
|
||||
# litellm-rust/.
|
||||
#
|
||||
# docker build -f litellm-rust/crates/ai-gateway/Dockerfile -t litellm-ai-gateway .
|
||||
# docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-ai-gateway .
|
||||
#
|
||||
# No secrets live in this file. Runtime config (LITELLM_MASTER_KEY,
|
||||
# OPENAI_API_KEY referenced by config.yaml, etc.) is injected as environment
|
||||
|
|
@ -36,12 +36,12 @@ FROM chef AS builder
|
|||
# whenever only gateway source changes.
|
||||
COPY --from=planner /build/litellm-rust/recipe.json recipe.json
|
||||
RUN cargo chef cook --locked --release \
|
||||
-p litellm-ai-gateway --features server,python-config \
|
||||
-p litellm-gateway-server --features python-config \
|
||||
--recipe-path recipe.json
|
||||
# Now copy the real sources and build the gateway binary. Deps are already cooked
|
||||
# above, so this step only recompiles the gateway crate.
|
||||
COPY litellm-rust/ .
|
||||
RUN cargo build --locked --release -p litellm-ai-gateway --bin litellm-ai-gateway --features server,python-config
|
||||
RUN cargo build --locked --release -p litellm-gateway-server --bin litellm-ai-gateway --features python-config
|
||||
|
||||
# ---- Runtime ----------------------------------------------------------------
|
||||
# python:3.11-slim-bookworm ships libpython3.11, matching the builder's PyO3
|
||||
|
|
@ -68,7 +68,7 @@ COPY --from=builder /build/litellm-rust/target/release/litellm-ai-gateway /usr/l
|
|||
|
||||
# Default config.yaml. A real deploy can override this (e.g. mount a Render
|
||||
# secret file at the same path) — never bake secrets into the image.
|
||||
COPY litellm-rust/crates/ai-gateway/config.yaml /app/config.yaml
|
||||
COPY litellm-rust/crates/gateway-server/config.yaml /app/config.yaml
|
||||
|
||||
# Bind to all interfaces (Render routes to 0.0.0.0:$PORT) and load the model_list
|
||||
# from config.yaml via the embedded python config reader.
|
||||
206
litellm-rust/crates/gateway-server/README.md
Normal file
206
litellm-rust/crates/gateway-server/README.md
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
# LiteLLM Rust Gateway Server
|
||||
|
||||
A minimal Axum service that fronts OpenAI's realtime API. Clients open a
|
||||
WebSocket to `GET /v1/realtime`; the gateway authenticates, selects a deployment,
|
||||
dials OpenAI upstream, and splices the two sockets frame-by-frame.
|
||||
|
||||
## Crates
|
||||
|
||||
`litellm-rust` has six crates. A crate is a layer, shared foundation, or separate host, not a route:
|
||||
|
||||
| Crate | Role |
|
||||
|-------|------|
|
||||
| litellm-core | The LiteLLM SDK in Rust — per-route entrypoints (`messages::messages()`) that resolve the provider, transform, and make the call; plus types, provider transforms, and the router. |
|
||||
| litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. |
|
||||
| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by the server and Python bridge. |
|
||||
| litellm-gateway-server | Axum binary, HTTP/WebSocket routes, auth extractors, application state, startup config, and HTTP-only dependencies. |
|
||||
| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. |
|
||||
| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. |
|
||||
|
||||
Dependency direction is acyclic: config and the gateway runtime depend on core, the server depends on gateway, config, and core, and the Python bridge depends only on reusable domain layers and Python interop.
|
||||
|
||||
- **Client endpoint:** `wss://<host>/v1/realtime?model=<model>` (WebSocket)
|
||||
- **Auth:** `Authorization: Bearer $LITELLM_MASTER_KEY` (fails closed if unset)
|
||||
- **Health:** `GET /health/readiness`, `GET /health/liveness`
|
||||
- **Request logs:** POSTed to a LiteLLM proxy at `/v1/rust_control_plane/logs` (see [Request logging](#request-logging))
|
||||
|
||||
> **Realtime serving is pure Rust.** Python is used at **load time only** — to
|
||||
> read the config once at boot. The realtime hot path never touches Python.
|
||||
|
||||
The former `/health/gil` route and its acquisition counter were removed. They
|
||||
only observed the single startup config load and did not prove that every GIL
|
||||
acquisition was instrumented
|
||||
|
||||
## Configuration (config.yaml)
|
||||
|
||||
The gateway loads its `model_list` from a **config.yaml**, the same as the
|
||||
LiteLLM proxy. Point `LITELLM_CONFIG_PATH` at the file:
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
model_list:
|
||||
- model_name: gpt-realtime
|
||||
litellm_params:
|
||||
model: openai/gpt-realtime
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
```
|
||||
|
||||
```bash
|
||||
LITELLM_CONFIG_PATH=./config.yaml ./litellm-ai-gateway
|
||||
```
|
||||
|
||||
At boot `litellm-config` calls into `litellm.proxy.read_model_list` and returns
|
||||
resolved deployments to the gateway, which constructs the router. The Python
|
||||
backend still reuses the **real proxy config reader** (`ProxyConfig.get_config`),
|
||||
so everything the proxy supports in config.yaml works here too:
|
||||
|
||||
- `include:` to merge in other config files,
|
||||
- `os.environ/VAR` secret references (resolved via the secret manager, never
|
||||
inlined),
|
||||
- DB-stored models (when a database is configured).
|
||||
|
||||
Secrets stay out of the config — reference them with `os.environ/...` and set
|
||||
the env var at deploy time. The shipped Docker image is built with the
|
||||
`python-config` feature and **bundles litellm**, so config loading works out of
|
||||
the box; the default baked config lives at `/app/config.yaml` and can be
|
||||
overridden at deploy time (e.g. a Render secret file mounted at the same path).
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Var | Required | Default | Purpose |
|
||||
|---|---|---|---|
|
||||
| `LITELLM_CONFIG_PATH` | yes (config mode) | — | Path to the config.yaml the gateway loads its `model_list` from. The Docker image defaults this to `/app/config.yaml`. |
|
||||
| `LITELLM_MASTER_KEY` | yes | — | Bearer token clients must send. Unset ⇒ all `/v1/realtime` requests are rejected (fail closed). |
|
||||
| `OPENAI_API_KEY` | yes | — | Upstream OpenAI key. Referenced by config.yaml as `os.environ/OPENAI_API_KEY` for the gateway→OpenAI dial. |
|
||||
| `HOST` | no | `127.0.0.1` | **Set to `0.0.0.0` in any container/deploy** or external traffic is refused. |
|
||||
| `PORT` | no | `4001` | Listen port. Render and most PaaS inject this automatically. |
|
||||
| `LITELLM_PROXY_BASE_URL` | no | `http://localhost:4000` | LiteLLM proxy that request logs are POSTed to. See [Request logging](#request-logging). |
|
||||
|
||||
> Secrets (`LITELLM_MASTER_KEY`, `OPENAI_API_KEY`) are never baked into the image
|
||||
> or `render.yaml` — inject them at deploy time only.
|
||||
|
||||
### Lean env stand-in (fallback)
|
||||
|
||||
If the binary is built **without** `python-config` (default features), or
|
||||
`LITELLM_CONFIG_PATH` is unset, the gateway falls back to a single-deployment
|
||||
stand-in built from the environment:
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `OPENAI_REALTIME_MODEL` | `gpt-realtime` | The single deployment's model name (also the `?model=` clients pass). |
|
||||
|
||||
The default workspace build links no libpython and needs no config file. This
|
||||
fallback mode only supports one hard-coded OpenAI deployment. **config.yaml is the recommended path** — use the
|
||||
stand-in only for the leanest possible build.
|
||||
|
||||
## Request logging
|
||||
|
||||
The gateway runs no spend logic. When a session ends it builds one
|
||||
`StandardLoggingPayload` and POSTs it to `{LITELLM_PROXY_BASE_URL}/v1/rust_control_plane/logs`
|
||||
(admin-only, bearer = `LITELLM_MASTER_KEY`), and the proxy replays it through its
|
||||
normal callbacks (spend logs, Langfuse, etc.). The POST is non-blocking: a bounded
|
||||
channel drained by a background worker, dropping with a counter if the proxy is
|
||||
down. It sends one payload per session. Both env vars are in the table above.
|
||||
|
||||
Worker tuning, rarely needed: `LITELLM_LOG_CHANNEL_CAPACITY` (4096),
|
||||
`LITELLM_LOG_BATCH_SIZE` (256), `LITELLM_LOG_FLUSH_INTERVAL_MS` (500).
|
||||
|
||||
## Build & run with Docker
|
||||
|
||||
The image is built with `--features python-config` and installs litellm **from this
|
||||
repo's source** (the config reader is newer than any PyPI release), so the build
|
||||
**context is the repo root**:
|
||||
|
||||
```bash
|
||||
# from the repo root
|
||||
docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-ai-gateway .
|
||||
|
||||
docker run --rm -p 4001:4001 \
|
||||
-e HOST=0.0.0.0 -e PORT=4001 \
|
||||
-e LITELLM_MASTER_KEY=sk-local \
|
||||
-e OPENAI_API_KEY=$OPENAI_API_KEY \
|
||||
litellm-ai-gateway # LITELLM_CONFIG_PATH defaults to /app/config.yaml
|
||||
|
||||
# smoke test
|
||||
curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/health/readiness # -> 200
|
||||
curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/v1/realtime # -> 401 (auth fails closed)
|
||||
```
|
||||
|
||||
On boot you should see `loaded model_list from /app/config.yaml via python
|
||||
config reader` — that confirms the config path (not the env stand-in fallback).
|
||||
To use your own config, mount it over the default:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 4001:4001 \
|
||||
-e HOST=0.0.0.0 -e LITELLM_MASTER_KEY=sk-local -e OPENAI_API_KEY=$OPENAI_API_KEY \
|
||||
-v $(pwd)/my-config.yaml:/app/config.yaml:ro \
|
||||
litellm-ai-gateway
|
||||
```
|
||||
|
||||
### Cargo-only (no Docker)
|
||||
|
||||
```bash
|
||||
# config.yaml mode — needs litellm importable in the active python env
|
||||
LITELLM_CONFIG_PATH=./crates/gateway-server/config.yaml \
|
||||
cargo run --release -p litellm-gateway-server --features python-config
|
||||
|
||||
# env stand-in mode — no python, no config
|
||||
cargo run --release -p litellm-gateway-server
|
||||
```
|
||||
|
||||
## Deploy on Render
|
||||
|
||||
The service is a Docker **web service**; Render terminates TLS and supports
|
||||
WebSockets, so the public endpoint is `wss://<service>.onrender.com/v1/realtime`.
|
||||
|
||||
### Option A — Blueprint (`render.yaml`)
|
||||
|
||||
`crates/gateway-server/render.yaml` describes the service (Docker runtime,
|
||||
`healthCheckPath: /health/readiness`, repo-root `dockerContext: .`,
|
||||
`dockerfilePath: ./litellm-rust/crates/gateway-server/Dockerfile`,
|
||||
`LITELLM_CONFIG_PATH: /app/config.yaml`). `LITELLM_MASTER_KEY` and
|
||||
`OPENAI_API_KEY` are `sync: false` — set them in the dashboard after the first
|
||||
deploy. To use a non-default model_list, mount a **Render Secret File** at
|
||||
`/app/config.yaml`. Point a Render Blueprint at this repo/branch and apply.
|
||||
|
||||
### Option B — Render API
|
||||
|
||||
```bash
|
||||
# create a Docker web service from this repo+branch, then set env vars:
|
||||
curl -X POST https://api.render.com/v1/services \
|
||||
-H "Authorization: Bearer $RENDER_API_KEY" -H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"type": "web_service", "name": "litellm-rust-ai-gateway",
|
||||
"ownerId": "<owner-id>", "repo": "https://github.com/BerriAI/litellm",
|
||||
"branch": "<branch-with-this-dockerfile>",
|
||||
"serviceDetails": {
|
||||
"env": "docker",
|
||||
"envSpecificDetails": {
|
||||
"dockerfilePath": "./litellm-rust/crates/gateway-server/Dockerfile",
|
||||
"dockerContext": "."
|
||||
},
|
||||
"healthCheckPath": "/health/readiness"
|
||||
}
|
||||
}'
|
||||
# then set env vars LITELLM_MASTER_KEY, OPENAI_API_KEY, HOST=0.0.0.0,
|
||||
# LITELLM_CONFIG_PATH=/app/config.yaml
|
||||
```
|
||||
|
||||
Health check path **must** be `/health/readiness`. `autoDeploy` is off by default
|
||||
in the blueprint — trigger deploys manually (or flip it on) to pick up new commits.
|
||||
|
||||
## Scaling
|
||||
|
||||
Concurrency is what matters, not total connections: each in-flight session holds
|
||||
one client socket + one upstream socket. To scale, raise the instance count /
|
||||
enable autoscaling on the Render service (e.g. baseline 10, max 100). Each
|
||||
instance needs file descriptors for `2 × peak_concurrent_sessions` — raise
|
||||
`ulimit -n` if you push very high concurrency.
|
||||
|
||||
## Latency note
|
||||
|
||||
The gateway adds the cost of one extra hop: client→gateway, then a fresh
|
||||
gateway→OpenAI realtime handshake (TLS + WS upgrade + `session.created`). In
|
||||
benchmarks this is ~100–150 ms of added session-establishment time; first-audio
|
||||
and steady-state streaming add no measurable overhead. To minimize it, deploy the
|
||||
gateway in the Render region with the lowest RTT to OpenAI's realtime endpoint.
|
||||
|
|
@ -14,7 +14,7 @@ services:
|
|||
name: litellm-rust-ai-gateway
|
||||
runtime: docker
|
||||
plan: standard
|
||||
dockerfilePath: ./litellm-rust/crates/ai-gateway/Dockerfile
|
||||
dockerfilePath: ./litellm-rust/crates/gateway-server/Dockerfile
|
||||
dockerContext: .
|
||||
healthCheckPath: /health/readiness
|
||||
numInstances: 1
|
||||
3
litellm-rust/crates/gateway-server/src/constants.rs
Normal file
3
litellm-rust/crates/gateway-server/src/constants.rs
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
pub(crate) const MESSAGES_ROUTE_PATH: &str = "/v1/messages";
|
||||
pub(crate) const MESSAGES_HEADERS_NOT_FORWARDED: &[&str] =
|
||||
&["authorization", "connection", "content-length", "host"];
|
||||
4
litellm-rust/crates/gateway-server/src/lib.rs
Normal file
4
litellm-rust/crates/gateway-server/src/lib.rs
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
pub mod auth;
|
||||
mod constants;
|
||||
pub mod routes;
|
||||
pub mod state;
|
||||
|
|
@ -1,28 +1,20 @@
|
|||
//! LiteLLM AI Gateway — a minimal Axum server fronting the Rust router.
|
||||
//! LiteLLM AI Gateway server.
|
||||
//!
|
||||
//! Flow: client → `POST /v1/realtime` → `router.realtime()` selects a deployment
|
||||
//! (simple-shuffle) → `io::realtime::realtime()` invokes OpenAI. The
|
||||
//! server owns transport + config; routing lives in the `router` crate.
|
||||
//!
|
||||
//! The binary requires the `server` feature (declared in `Cargo.toml` via
|
||||
//! `required-features`), so cargo skips it unless that feature is on. Everything
|
||||
//! the binary needs lives in the library (`litellm_ai_gateway`); `main` just
|
||||
//! wires startup.
|
||||
//! The binary owns startup and config, then mounts the Axum routes from
|
||||
//! `litellm_gateway_server`. Transport-neutral runtime and integrations come
|
||||
//! from `litellm_ai_gateway`.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
|
||||
use litellm_ai_gateway::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger;
|
||||
use litellm_ai_gateway::io::realtime_pool::{PoolConfig, RealtimePool, upstream_key};
|
||||
use litellm_ai_gateway::routes;
|
||||
use litellm_ai_gateway::state::AppState;
|
||||
#[cfg(feature = "python-config")]
|
||||
use litellm_config::load_model_list;
|
||||
use litellm_core::router::{Deployment, LiteLLMParams, Router};
|
||||
use litellm_gateway_server::routes;
|
||||
use litellm_gateway_server::state::AppState;
|
||||
|
||||
use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
|
||||
use litellm_ai_gateway::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger;
|
||||
|
||||
/// Bind to localhost by default so the gateway is not a public, unauthenticated
|
||||
/// provider proxy out of the box. Override with `HOST` (e.g. `0.0.0.0`).
|
||||
const DEFAULT_HOST: &str = "127.0.0.1";
|
||||
const DEFAULT_PORT: u16 = 4001;
|
||||
|
||||
|
|
@ -87,9 +79,10 @@ async fn main() {
|
|||
}
|
||||
|
||||
/// Register every deployment's upstream key with the pool so the replenisher
|
||||
/// pre-warms it. Mirrors `service::run`'s key derivation (strip `openai/`, resolve
|
||||
/// api_key); deployments whose key can't be resolved are skipped (they fresh-dial
|
||||
/// and surface the auth error on the request path, as before).
|
||||
/// pre-warms it. Mirrors `runtime::realtime::run`'s key derivation (strip
|
||||
/// `openai/`, resolve api_key); deployments whose key can't be resolved are
|
||||
/// skipped (they fresh-dial and surface the auth error on the request path, as
|
||||
/// before).
|
||||
fn register_deployments(router: &Router, pool: &RealtimePool) {
|
||||
for deployment in router.deployments() {
|
||||
let params = &deployment.litellm_params;
|
||||
|
|
@ -15,27 +15,17 @@ async fn handle(...) -> impl IntoResponse { ... }
|
|||
```
|
||||
`health.rs` is the example.
|
||||
|
||||
## Split out `service` when there's real logic
|
||||
When a route has business logic worth testing without axum, put it in a sibling
|
||||
`service` (a file, or a folder if the route grows). The route file stays the
|
||||
**axum surface** (router + handler + any socket/SSE adapter); `service` is plain
|
||||
Rust with **no axum types**, and its job is to pick the deployment and call the
|
||||
`core` route entrypoint (see `messages/service.rs` calling
|
||||
`litellm_core::messages::messages`). Never build a provider request, resolve a
|
||||
key, or perform the provider call here. `realtime/` is the older example:
|
||||
```
|
||||
realtime/
|
||||
mod.rs # axum surface: router() + handler + the WS<->events adapter
|
||||
service.rs # pure logic: select deployment + call provider (no axum) — testable
|
||||
```
|
||||
Split `service` further (or add `transport`, `repo`, …) only once a single file
|
||||
genuinely gets hard to read.
|
||||
## Runtime boundary
|
||||
When a route has transport-neutral orchestration, put it under
|
||||
`litellm-ai-gateway::runtime` and test it there. The route file stays the Axum
|
||||
surface: router, handler, and socket or SSE adapter. Never build a provider
|
||||
request, resolve a provider key, or perform the provider call in this crate.
|
||||
|
||||
## Invariants
|
||||
- **Auth is an extractor, not a manual call.** A handler requires auth by adding
|
||||
`crate::auth::RequireMasterKey` to its arguments; it runs during extraction.
|
||||
Never re-implement the check per route.
|
||||
- **Handlers contain no business logic; `service` contains no axum types.**
|
||||
- **Handlers contain no business logic; `litellm-ai-gateway` contains no Axum types.**
|
||||
- **No provider handlers in this crate.** Transforms, auth headers, and the
|
||||
provider HTTP call live in `core/src/<route>/`.
|
||||
- A route owns its paths in its own `router()`; `mod.rs` only merges.
|
||||
|
|
@ -1,7 +1,5 @@
|
|||
//! `POST /v1/messages`, the Anthropic Messages HTTP surface.
|
||||
|
||||
mod service;
|
||||
|
||||
use axum::Router;
|
||||
use axum::body::Body;
|
||||
use axum::extract::{Json, State};
|
||||
|
|
@ -9,6 +7,7 @@ use axum::http::StatusCode;
|
|||
use axum::http::header::{CACHE_CONTROL, CONTENT_TYPE, HeaderMap, HeaderValue};
|
||||
use axum::response::{IntoResponse, Response};
|
||||
use axum::routing::post;
|
||||
use litellm_ai_gateway::runtime::messages::{MessagesResponse, run};
|
||||
use litellm_core::Error;
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
|
|
@ -34,12 +33,12 @@ async fn handle(
|
|||
Json(body): Json<Value>,
|
||||
) -> Result<Response, MessagesRouteError> {
|
||||
let extra_headers = forwarded_headers(&headers)?;
|
||||
match service::run(&state.router, body, extra_headers)
|
||||
match run(&state.router, body, extra_headers)
|
||||
.await
|
||||
.map_err(MessagesRouteError::from)?
|
||||
{
|
||||
service::MessagesResponse::Json(body) => Ok(Json(body).into_response()),
|
||||
service::MessagesResponse::Stream(upstream) => stream_response(upstream),
|
||||
MessagesResponse::Json(body) => Ok(Json(body).into_response()),
|
||||
MessagesResponse::Stream(upstream) => stream_response(upstream),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -149,8 +148,8 @@ mod tests {
|
|||
use tower::ServiceExt;
|
||||
|
||||
use super::super::app;
|
||||
use crate::io::realtime_pool::RealtimePool;
|
||||
use crate::state::AppState;
|
||||
use litellm_ai_gateway::io::realtime_pool::RealtimePool;
|
||||
|
||||
fn state(model: &str, api_base: String, master_key: Option<&str>) -> AppState {
|
||||
state_with_provider(model, model, api_base, master_key)
|
||||
|
|
@ -1,9 +1,9 @@
|
|||
# Realtime route (`GET /v1/realtime`)
|
||||
|
||||
Proxies OpenAI's realtime WebSocket. `mod.rs` is the axum surface (handler +
|
||||
socket↔events adapter); `service.rs` is the pure logic (select a deployment, then
|
||||
splice client ↔ upstream). The pool itself lives in
|
||||
`crates/providers/src/realtime_pool.rs`.
|
||||
Proxies OpenAI's realtime WebSocket. `mod.rs` is the Axum surface (handler and
|
||||
socket-to-events adapter); the pure logic lives in
|
||||
`litellm-ai-gateway::runtime::realtime`. The pool lives in
|
||||
`litellm-ai-gateway::io::realtime_pool`.
|
||||
|
||||
## Connection pooling
|
||||
|
||||
|
|
@ -27,7 +27,7 @@ never a correctness dependency.
|
|||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
client connect ──────► │ routes/realtime → service::run │
|
||||
client connect ──────► │ routes/realtime -> runtime::run │
|
||||
│ pool.take(key) │
|
||||
│ hit → relay buffered │
|
||||
│ session.created, then splice │
|
||||
|
|
@ -84,4 +84,4 @@ upstream sockets, which is why warm sockets are short-lived
|
|||
attempts against a broken key so it can't exhaust upstream rate limits and degrade
|
||||
valid cold-path traffic; the backoff resets the moment a dial succeeds.
|
||||
|
||||
Benchmarks and repro: `../../benchmarks/realtime/README.md`.
|
||||
Benchmarks and repro: `../../../../ai-gateway/benchmarks/realtime/README.md`.
|
||||
|
|
@ -1,16 +1,14 @@
|
|||
//! `GET /v1/realtime` (WebSocket).
|
||||
//!
|
||||
//! This file is the **axum surface**: `router()`, the handler, and the small
|
||||
//! socket↔events adapter. The pure logic (no axum) lives in [`service`]. Auth is
|
||||
//! the `RequireMasterKey` extractor, so the handler stays thin.
|
||||
|
||||
mod service;
|
||||
//! socket↔events adapter. The pure logic lives in
|
||||
//! [`litellm_ai_gateway::runtime::realtime`]. Auth is the `RequireMasterKey`
|
||||
//! extractor, so the handler stays thin.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
use crate::io::realtime_pool::RealtimePool;
|
||||
use axum::Router;
|
||||
use axum::extract::ws::{Message, WebSocket, WebSocketUpgrade};
|
||||
use axum::extract::{Query, State};
|
||||
|
|
@ -18,14 +16,16 @@ use axum::http::StatusCode;
|
|||
use axum::response::Response;
|
||||
use axum::routing::get;
|
||||
use futures_util::{SinkExt, StreamExt};
|
||||
use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
|
||||
use litellm_ai_gateway::integrations::types::RequestMetadata;
|
||||
use litellm_ai_gateway::io::realtime_pool::RealtimePool;
|
||||
use litellm_ai_gateway::realtime::streaming::{RealTimeStreaming, SessionStatus};
|
||||
use litellm_ai_gateway::runtime::realtime;
|
||||
use litellm_core::realtime::types::RealtimeEvent;
|
||||
use litellm_core::router::Router as ModelRouter;
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::auth::RequireMasterKey;
|
||||
use crate::integrations::custom_logger::CustomLogger;
|
||||
use crate::integrations::types::RequestMetadata;
|
||||
use crate::realtime::streaming::{RealTimeStreaming, SessionStatus};
|
||||
use crate::state::AppState;
|
||||
|
||||
/// Process-local monotonic counter, mixed into the per-session call id so two
|
||||
|
|
@ -84,7 +84,7 @@ async fn handle(
|
|||
}
|
||||
|
||||
/// Adapt the axum socket (text frames) to the typed-event `Stream`/`Sink` the
|
||||
/// service wants, keeping axum types out of `service`.
|
||||
/// runtime wants, keeping axum types out of `litellm-ai-gateway`.
|
||||
///
|
||||
/// This is also the realtime-logging seam: every upstream→client event (the
|
||||
/// direction carrying `session.created` and `response.done` with usage) is fed
|
||||
|
|
@ -146,7 +146,7 @@ async fn bridge(
|
|||
// splice; the borrow ends when `run` returns, freeing the collector for the
|
||||
// single post-session `log_messages` flush. `run` picks a pooled (warm) or
|
||||
// fresh upstream — observe fires on the upstream arm either way.
|
||||
let result = service::run(
|
||||
let result = realtime::run(
|
||||
&router,
|
||||
&pool,
|
||||
&model,
|
||||
|
|
@ -1,5 +1,3 @@
|
|||
mod service;
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
|
@ -11,13 +9,14 @@ use axum::http::StatusCode;
|
|||
use axum::response::Response;
|
||||
use axum::routing::get;
|
||||
use futures_util::{Sink, SinkExt, StreamExt};
|
||||
use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
|
||||
use litellm_ai_gateway::integrations::types::RequestMetadata;
|
||||
use litellm_ai_gateway::runtime::responses;
|
||||
use litellm_core::responses::types::{ResponsesErrorFrame, ResponsesWsEvent, ResponsesWsEventType};
|
||||
use litellm_core::router::Router as ModelRouter;
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::auth::RequireMasterKey;
|
||||
use crate::integrations::custom_logger::CustomLogger;
|
||||
use crate::integrations::types::RequestMetadata;
|
||||
use crate::state::AppState;
|
||||
|
||||
static CALL_SEQ: AtomicU64 = AtomicU64::new(0);
|
||||
|
|
@ -217,7 +216,7 @@ async fn bridge(
|
|||
}
|
||||
}));
|
||||
let mut client_out = ResponseClientSink { sink: ws_sink };
|
||||
let result = service::run(
|
||||
let result = responses::run(
|
||||
&router,
|
||||
&model,
|
||||
first_frame,
|
||||
|
|
@ -239,10 +238,10 @@ async fn bridge(
|
|||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::io::realtime_pool::RealtimePool;
|
||||
use crate::state::AppState;
|
||||
use axum::body::Body;
|
||||
use axum::http::Request;
|
||||
use litellm_ai_gateway::io::realtime_pool::RealtimePool;
|
||||
use litellm_core::router::Router as ModelRouter;
|
||||
use serde_json::json;
|
||||
use std::pin::Pin;
|
||||
|
|
@ -1,10 +1,9 @@
|
|||
use std::sync::Arc;
|
||||
|
||||
use crate::io::realtime_pool::RealtimePool;
|
||||
use litellm_ai_gateway::integrations::custom_logger::CustomLogger;
|
||||
use litellm_ai_gateway::io::realtime_pool::RealtimePool;
|
||||
use litellm_core::router::Router;
|
||||
|
||||
use crate::integrations::custom_logger::CustomLogger;
|
||||
|
||||
/// Shared application state handed to every route handler.
|
||||
#[derive(Clone)]
|
||||
pub struct AppState {
|
||||
Loading…
Add table
Reference in a new issue