diff --git a/.github/workflows/test-rust.yml b/.github/workflows/test-rust.yml index 9b8b132df62..beeb9c46151 100644 --- a/.github/workflows/test-rust.yml +++ b/.github/workflows/test-rust.yml @@ -74,9 +74,12 @@ jobs: - name: Run Clippy with Bedrock auth run: cargo clippy -p litellm-core --all-targets --features bedrock-auth --locked -- -D warnings - - name: Run Clippy with all gateway features + - name: Run Clippy with all gateway runtime features run: cargo clippy -p litellm-ai-gateway --all-targets --all-features --locked -- -D warnings + - name: Run Clippy for the gateway server + run: cargo clippy -p litellm-gateway-server --all-targets --locked -- -D warnings + - name: Run Rust tests run: cargo test --workspace --locked @@ -84,8 +87,8 @@ jobs: run: cargo test -p litellm-core --features bedrock-auth --locked # Not --all-features: python-config links libpython, which this job does not install. - - name: Run gateway tests with the server feature - run: cargo test -p litellm-ai-gateway --features server --locked + - name: Run gateway server tests + run: cargo test -p litellm-gateway-server --locked release-wheel: name: release wheel diff --git a/litellm-rust/ADDING_A_PROVIDER.md b/litellm-rust/ADDING_A_PROVIDER.md index ae8ae5a6870..d7198cfcecf 100644 --- a/litellm-rust/ADDING_A_PROVIDER.md +++ b/litellm-rust/ADDING_A_PROVIDER.md @@ -26,4 +26,4 @@ variants of it. The test for a good abstraction is that adding the next provider is a few declarative lines, not a new file of duplicated flow. Only diverge from the base when behavior is genuinely different, and say so explicitly in the PR. -**Calling:** hosts invoke the core entrypoint — the Python bridge and the `ai-gateway` route service both call `litellm_core::messages::messages`. Never add a provider handler to `ai-gateway`. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md). +**Calling:** hosts invoke the core entrypoint. The Python bridge and the reusable `ai-gateway` runtime both call `litellm_core::messages::messages`, while `gateway-server` adapts HTTP requests to that runtime. Never add a provider handler to either host layer. Register new modules in `lib.rs` / `mod.rs`, then run the commands under "Checks" in [CLAUDE.md](CLAUDE.md). diff --git a/litellm-rust/AGENTS.md b/litellm-rust/AGENTS.md index b8d1f2db4d7..066239be46c 100644 --- a/litellm-rust/AGENTS.md +++ b/litellm-rust/AGENTS.md @@ -1,6 +1,6 @@ # AGENTS.md -litellm-rust has five crates. A crate is a layer or shared foundation, not a route. Routes (ocr, realtime, chat) and providers (mistral, openai) are modules inside the layers. +litellm-rust has six crates. A crate is a layer, shared foundation, or separately built host, not a route. Routes (ocr, realtime, chat) and providers (mistral, openai) are modules inside the layers. ## Crates @@ -8,11 +8,12 @@ litellm-rust has five crates. A crate is a layer or shared foundation, not a rou |-------|------| | litellm-core | The LiteLLM SDK in Rust. One public entrypoint per top-level call (`messages::messages()`), owning types, transforms, provider resolution, auth, and the provider HTTP call. Call it, get a typed response. | | litellm-config | Config-loading boundary. Returns resolved core deployment data and optionally delegates loading to Python. | -| litellm-ai-gateway | The axum server (behind the `server` feature) plus the WebSocket hosts. Translates HTTP/WS to core entrypoints; owns no provider logic and no handlers. | +| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by the server and Python bridge. Owns transport-neutral orchestration and legacy call modules, but no Axum or Tower dependencies. | +| litellm-gateway-server | The Axum binary and HTTP/WebSocket host. Owns routes, auth extractors, application state, startup config, and HTTP-only Tower dependencies. | | litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. | | litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. | -Dependency direction is acyclic: `litellm-config` depends on `litellm-core`, the gateway depends on both, and `litellm-python-bridge` depends on the domain layers and `litellm-python-interop`. The interop foundation depends on no LiteLLM domain crate. +Dependency direction is acyclic: `litellm-config` depends on `litellm-core`; `litellm-ai-gateway` depends on core; `litellm-gateway-server` depends on gateway, core, and config; and `litellm-python-bridge` depends on the reusable domain layers and `litellm-python-interop`. Reusable crates must not depend on `litellm-gateway-server`. The interop foundation depends on no LiteLLM domain crate. ## Where a route lives @@ -28,7 +29,7 @@ core/src/messages/ client.rs # the shared reqwest client ``` -Handlers never live in `ai-gateway`. `ocr`, `audio_transcription`, and `realtime` are still hosted there from before this rule; they move to `core` as they are touched. +Provider handlers never live in `gateway-server`. `ocr`, `audio_transcription`, and realtime provider I/O are still hosted in `ai-gateway` from before this rule; they move to `core` as they are touched. Adding a crate: default to a module. A new crate requires a real trigger: separate artifact (binary/cdylib), proc-macro, shared foundation, or publishable standalone. A new provider or route is none of these. diff --git a/litellm-rust/CLAUDE.md b/litellm-rust/CLAUDE.md index dfacf37b6cd..e594818a810 100644 --- a/litellm-rust/CLAUDE.md +++ b/litellm-rust/CLAUDE.md @@ -25,10 +25,12 @@ the base when behavior is genuinely different, and say so explicitly in the PR. `litellm-core` **is** the LiteLLM SDK in Rust: it makes the LLM call. `litellm-config` is the config-loading boundary and returns resolved core types. -`litellm-ai-gateway` is an HTTP/WebSocket server in front of it, and -`litellm-python-bridge` exposes it to the Python SDK. `litellm-python-interop` -holds domain-neutral PyO3 primitives shared by Python-facing Rust code. A crate -is a layer or shared foundation, not a route; add modules, not crates. +`litellm-ai-gateway` is the framework-independent gateway runtime and integration +layer. `litellm-gateway-server` is the HTTP/WebSocket server in front of it, and +`litellm-python-bridge` exposes reusable Rust APIs to the Python SDK. +`litellm-python-interop` holds domain-neutral PyO3 primitives shared by +Python-facing Rust code. A crate is a layer, shared foundation, or separate host, +not a route; add modules, not crates. ## Core Boundary @@ -45,7 +47,7 @@ Route-level Rust structure mirrors LiteLLM's Python responsibilities: - `core/src/providers///transformation.rs` owns the provider-specific transform. For Anthropic Messages, this means `core/src/providers/anthropic/messages/transformation.rs`. -- Handlers live in `core`, never in a host. `ai-gateway` must not contain a +- Provider handlers live in `core`, never in a host. `gateway-server` must not contain a route handler that talks to a provider; its axum route reads the HTTP request, picks a deployment, and calls the `core` entrypoint. `python-bridge` marshals Python objects and calls the same entrypoint. @@ -57,7 +59,7 @@ its own caller; the host still owns no provider logic. Call-hook and lifecycle instrumentation, including phase timing, usage accumulation, and callback payload construction, always lives in `core`. Hosts feed observed events into core and dispatch the completed payloads through -their I/O logger; hosts must not own callback orchestration. +the reusable gateway integrations; HTTP hosts must not own callback orchestration. Allowed in `core`: - The public entrypoint for a top-level LiteLLM call @@ -81,9 +83,11 @@ Env reads in `core` are limited to credential fallback inside a route's no key is passed. Everything else config-shaped is resolved by the host and passed in. -Routes still hosted in `ai-gateway` (`ocr`, `audio_transcription`, `realtime`) -predate this rule and are being moved into `core` route modules; do not add new -ones there, and prefer moving one when you touch it. +Legacy call runtimes still hosted in `ai-gateway` (`ocr`, +`audio_transcription`, realtime provider I/O) predate this rule and are being +moved into `core` route modules; do not add new ones there, and prefer moving +one when you touch it. Axum routes, auth extractors, application state, and +startup belong only in `gateway-server`. Python owns rollout state and fallback while Rust is being introduced. Rust paths must be off by default until parity tests prove equivalence with Python. @@ -121,7 +125,7 @@ the first PR: ## Network I/O Rules These rules apply to every module that executes network I/O, whether it is a -`core` route handler or a host such as `ai-gateway`: +`core` route handler or a host such as `gateway-server`: - Set connect and full-request timeouts. No unbounded waits. - Reuse HTTP clients; do not construct clients per request. @@ -177,12 +181,11 @@ cd litellm-rust cargo fmt --check cargo clippy --workspace --all-targets -- -D warnings cargo clippy -p litellm-core --all-targets --features bedrock-auth -- -D warnings -# the ai-gateway binary + server code is behind the `server` feature cargo clippy -p litellm-ai-gateway --all-targets --all-features -- -D warnings +cargo clippy -p litellm-gateway-server --all-targets -- -D warnings cargo test --workspace cargo test -p litellm-core --features bedrock-auth -# the `auth`, `routes`, `state` and `realtime` tests only exist under `server` -cargo test -p litellm-ai-gateway --features server +cargo test -p litellm-gateway-server ``` When a Rust path is exposed through Python, add Python parity tests that compare diff --git a/litellm-rust/Cargo.lock b/litellm-rust/Cargo.lock index 62e943d0f42..6ead3ea2ce1 100644 --- a/litellm-rust/Cargo.lock +++ b/litellm-rust/Cargo.lock @@ -1408,20 +1408,15 @@ checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" name = "litellm-ai-gateway" version = "0.1.0" dependencies = [ - "axum", "base64", "futures-channel", "futures-util", - "litellm-config", "litellm-core", "reqwest", "serde", "serde_json", - "sha2 0.10.9", - "subtle", "tokio", "tokio-tungstenite", - "tower", "tracing", ] @@ -1458,6 +1453,25 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "litellm-gateway-server" +version = "0.1.0" +dependencies = [ + "axum", + "futures-util", + "litellm-ai-gateway", + "litellm-config", + "litellm-core", + "reqwest", + "serde", + "serde_json", + "sha2 0.10.9", + "subtle", + "tokio", + "tower", + "tracing", +] + [[package]] name = "litellm-python-bridge" version = "0.1.0" diff --git a/litellm-rust/Cargo.toml b/litellm-rust/Cargo.toml index 720c4545181..5cf3a5178b4 100644 --- a/litellm-rust/Cargo.toml +++ b/litellm-rust/Cargo.toml @@ -3,6 +3,7 @@ members = [ "crates/core", "crates/config", "crates/ai-gateway", + "crates/gateway-server", "crates/python-interop", "crates/python-bridge", ] @@ -20,6 +21,7 @@ tracing-subscriber = { version = "0.3", default-features = false, features = ["r litellm-core = { path = "crates/core" } litellm-config = { path = "crates/config" } litellm-ai-gateway = { path = "crates/ai-gateway", default-features = false } +litellm-gateway-server = { path = "crates/gateway-server" } litellm-python-interop = { path = "crates/python-interop" } axum = "0.7" pyo3 = "0.29.2" diff --git a/litellm-rust/README.md b/litellm-rust/README.md index 650d38753e7..7a756903c30 100644 --- a/litellm-rust/README.md +++ b/litellm-rust/README.md @@ -26,11 +26,12 @@ coverage and production evidence. |-------|------| | litellm-core | The SDK. Per-route entrypoints (`messages::messages()`), types, provider transforms (modules under `providers/`), provider resolution, auth, the provider HTTP call, and the router. | | litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. | -| litellm-ai-gateway | The axum server (behind the `server` feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. | +| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by server and Python hosts. | +| litellm-gateway-server | Axum binary, HTTP/WebSocket routes, auth extractors, application state, and HTTP-only dependencies. | | litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. | | litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. Owns API registration, domain wiring, and Python exception mapping. | -Dependency direction is acyclic: config depends on core, the gateway depends on config and core, and the Python bridge depends on the domain layers and Python interop. +Dependency direction is acyclic: config and the gateway runtime depend on core; the gateway server depends on gateway, config, and core; and the Python bridge depends only on reusable domain layers and Python interop. ## Layout @@ -40,7 +41,8 @@ crates/ src/messages/ mod.rs (entrypoint), types, transformation, prepare, handler, client src/providers/anthropic/messages/transformation.rs config/ Config loading and resolved deployments. - ai-gateway/ Axum server + WebSocket hosts; calls core entrypoints. + ai-gateway/ Framework-independent gateway runtime and integrations. + gateway-server/ Axum binary and HTTP/WebSocket host. python-interop/ Domain-neutral PyO3 conversion and GIL primitives. python-bridge/ PyO3 API adapter for Python LiteLLM. ``` diff --git a/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md b/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md index 952bbc38b43..a529a71f803 100644 --- a/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md +++ b/litellm-rust/crates/CODING_STANDARDS/PROVIDER_CODING_STANDARDS.md @@ -16,7 +16,7 @@ Rules for adding or changing an LLM provider/route in `litellm-rust`. `messages` ## Boundaries -7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `ai-gateway` = serving HTTP/WS (routing, extractors, auth of *our* callers, streaming to the client); `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint; they never build a provider request. +7. Layers never cross: `core` = the call itself (entrypoint, types, transforms, provider resolution, auth headers, provider HTTP, lifecycle hooks); `ai-gateway` = framework-independent gateway runtime and integrations; `gateway-server` = HTTP/WS routing, extractors, auth of *our* callers, application state, and streaming to the client; `python-bridge` = thin PyO3 adapter. Hosts call the core entrypoint or reusable gateway runtime; they never build a provider request. 8. Generic/route files contain zero provider-specific branches. A provider is one module under `core/src/providers///`; a route is a module, never a new crate. 9. Route entry point stays thin: `core::::()` -> `prepare_*` -> handler (or `CallLifecycle::run_request`, which owns the pre_call -> during_call -> provider call -> success/failure order and phase timing). Axum handlers validate and delegate to a service that calls the entrypoint; no business logic in them. 10. Constants (URLs, env-var names, API versions, error messages) live in a crate `constants.rs`, never inline. Config-shaped env reads happen at the host/config layer with the `DEFAULT_*` fallback defined in `constants.rs`; the only env read in `core` is the credential fallback in a route's `prepare.rs`. diff --git a/litellm-rust/crates/ai-gateway/AGENTS.md b/litellm-rust/crates/ai-gateway/AGENTS.md index b2fd583316b..91812cc3385 100644 --- a/litellm-rust/crates/ai-gateway/AGENTS.md +++ b/litellm-rust/crates/ai-gateway/AGENTS.md @@ -1,54 +1,11 @@ -# ai-gateway — folder architecture +# ai-gateway folder architecture -The Axum server that fronts the Rust gateway. It owns transport + config + auth -only; deployment selection lives in `core::router`, and the LLM call itself -(transforms, auth headers, provider HTTP) lives behind a `core` route entrypoint -such as `litellm_core::messages::messages`. No provider handler lives here. +This crate is a reusable, framework-independent gateway runtime and integration +library. It is used by `litellm-gateway-server` and `litellm-python-bridge` -``` -src/ - main.rs # entrypoint: build AppState (router + master key), bind, serve - state.rs # AppState — shared Arc + master_key - auth/ # authentication as an axum extractor — added to handler args - mod.rs # RequireMasterKey: FromRequestParts, single master key (LITELLM_MASTER_KEY) - routes/ # one module per route, all matching the same template - AGENTS.md # ← the route template (read this before adding a route) - mod.rs # app(): merges every module's router() - health.rs # simple route (one file): router() + liveness/readiness - realtime/ # route with logic → axum surface + a no-axum service: - mod.rs # router() + handler + WS<->events adapter (the axum surface) - service.rs # business logic (select deployment, call provider) — no axum, testable -``` +Axum routes, auth extractors, application state, startup, and Tower dependencies +belong in `litellm-gateway-server`. This crate must not depend on the server -## Rules - -- **Routes follow one template.** Each route module exposes - `pub fn router() -> Router`; `routes/mod.rs` only merges them. Simple - routes are one file; non-trivial routes are a folder (`handler`/`service`/ - `transport`). See `routes/AGENTS.md`. -- **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's - args; it runs during extraction. Never re-implement the check per route. -- **Handlers are thin.** A handler validates and delegates to its `service`. No - business logic, no provider calls, no transforms in handlers. -- **Services call `core`, they don't reimplement it.** A `service` picks the - deployment and calls the `core` route entrypoint. Provider resolution, auth - headers, URL building, and the HTTP call are `core`'s job; a service that - builds a provider request itself is a bug (`routes/messages/service.rs` is - the reference). -- **State is shared and cheap to clone.** Long-lived handles live behind `Arc` in - `state.rs`; read env/config only in `main.rs` when building state. - -## Auth (interim) - -A single **master key** (`LITELLM_MASTER_KEY`), enforced by the -`auth::RequireMasterKey` extractor: any caller presenting it as -`Authorization: Bearer ` may invoke the gateway. Fails closed (500) when -unset; constant-time compare. The server binds `127.0.0.1` by default (`HOST` to -override). Full per-key auth + budgets/rate-limits are delegated to the Python -proxy in a later phase. Health routes don't add the extractor (unauthenticated). - -## Python interop - -Python-backed loading lives in `litellm-config` and is **load-time only**. The -gateway's `python-config` feature forwards to that crate. The realtime data path -never takes the GIL. +Transport-neutral route orchestration lives under `src/runtime/`. Existing OCR, +audio transcription, realtime I/O, and callback integrations remain here as +migration seams until their provider call paths move into `litellm-core` diff --git a/litellm-rust/crates/ai-gateway/ARCHITECTURE.md b/litellm-rust/crates/ai-gateway/ARCHITECTURE.md index 6d090cf4c8e..796a3b6506f 100644 --- a/litellm-rust/crates/ai-gateway/ARCHITECTURE.md +++ b/litellm-rust/crates/ai-gateway/ARCHITECTURE.md @@ -1,14 +1,17 @@ # ai-gateway architecture -The Rust ai-gateway does LLM inference (realtime WebSocket). Spend tracking is an -API callback: it POSTs each finished session to the LiteLLM proxy, which records -spend and runs the usual callbacks. +`litellm-ai-gateway` is between hosts and `litellm-core`. It exposes +framework-independent runtime services and callback integrations without +depending on an HTTP framework ```mermaid flowchart LR - C[client] <--> G[Rust ai-gateway
LLM inference] - G <--> O[OpenAI realtime] - G -. spend tracking callback .-> P[litellm proxy] - F[litellm-config
load-time only] --> G - F -. Python backend .-> P + S[litellm-gateway-server] --> G[litellm-ai-gateway runtime] + B[litellm-python-bridge] --> G + G --> C[litellm-core] + S --> C + S --> F[litellm-config] ``` + +The server may depend on the runtime, core, and config crates. Reusable crates +must not depend on the server diff --git a/litellm-rust/crates/ai-gateway/Cargo.toml b/litellm-rust/crates/ai-gateway/Cargo.toml index 10369fa3bfd..6c838110a20 100644 --- a/litellm-rust/crates/ai-gateway/Cargo.toml +++ b/litellm-rust/crates/ai-gateway/Cargo.toml @@ -8,15 +8,9 @@ repository.workspace = true [lib] name = "litellm_ai_gateway" -[[bin]] -name = "litellm-ai-gateway" -path = "src/main.rs" -required-features = ["server"] - [dependencies] tracing.workspace = true litellm-core = { workspace = true, features = ["bedrock-auth"] } -litellm-config.workspace = true # reqwest (rustls + json) is used by io/ocr and ships realtime logs to the # Python proxy callbacks API. reqwest.workspace = true @@ -26,22 +20,11 @@ tokio-tungstenite.workspace = true futures-util.workspace = true serde_json.workspace = true base64.workspace = true -axum = { workspace = true, features = ["ws"], optional = true } serde.workspace = true -subtle = { workspace = true, optional = true } -# sha2 hashes the master key into user_api_key_hash (matches the proxy's -# SHA-256 hash_token) so the plaintext credential never enters a log payload. -sha2 = { workspace = true, optional = true } -tower = { version = "0.5.3", features = ["util"], optional = true } [features] default = [] -server = ["dep:axum", "dep:subtle", "dep:sha2"] -# Build the gateway's config from the proxy YAML via an embedded Python -# interpreter (links libpython; requires `litellm` importable at runtime). -python-config = ["litellm-config/python"] -trace-parity = ["server", "dep:tower", "litellm-core/observability"] +trace-parity = ["litellm-core/observability"] [dev-dependencies] futures-channel = "0.3" -tower = { version = "0.5.3", features = ["util"] } diff --git a/litellm-rust/crates/ai-gateway/README.md b/litellm-rust/crates/ai-gateway/README.md index 9fef59a277d..dfd3af80164 100644 --- a/litellm-rust/crates/ai-gateway/README.md +++ b/litellm-rust/crates/ai-gateway/README.md @@ -1,205 +1,12 @@ -# LiteLLM Rust AI Gateway +# LiteLLM AI Gateway Runtime -A minimal Axum service that fronts OpenAI's realtime API. Clients open a -WebSocket to `GET /v1/realtime`; the gateway authenticates, selects a deployment, -dials OpenAI upstream, and splices the two sockets frame-by-frame. +`litellm-ai-gateway` is the reusable, framework-independent runtime used by the +Rust gateway server and Python bridge -## Crates +It owns gateway integrations, transport-neutral route orchestration, and legacy +OCR, audio transcription, and realtime I/O that have not yet moved into +`litellm-core`. It has no Axum or Tower dependency -`litellm-rust` has five crates. A crate is a layer or shared foundation, not a route: - -| Crate | Role | -|-------|------| -| litellm-core | The LiteLLM SDK in Rust — per-route entrypoints (`messages::messages()`) that resolve the provider, transform, and make the call; plus types, provider transforms, and the router. | -| litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. | -| litellm-ai-gateway | The Axum server (behind the `server` feature) and WebSocket hosts. Translates HTTP/WS to core entrypoints; no provider handlers. | -| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. | -| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. | - -Dependency direction is acyclic: config depends on core, the gateway depends on config and core, and the Python bridge depends on the domain layers and Python interop. - -- **Client endpoint:** `wss:///v1/realtime?model=` (WebSocket) -- **Auth:** `Authorization: Bearer $LITELLM_MASTER_KEY` (fails closed if unset) -- **Health:** `GET /health/readiness`, `GET /health/liveness` -- **Request logs:** POSTed to a LiteLLM proxy at `/v1/rust_control_plane/logs` (see [Request logging](#request-logging)) - -> **Realtime serving is pure Rust.** Python is used at **load time only** — to -> read the config once at boot. The realtime hot path never touches Python. - -The former `/health/gil` route and its acquisition counter were removed. They -only observed the single startup config load and did not prove that every GIL -acquisition was instrumented - -## Configuration (config.yaml) - -The gateway loads its `model_list` from a **config.yaml**, the same as the -LiteLLM proxy. Point `LITELLM_CONFIG_PATH` at the file: - -```yaml -# config.yaml -model_list: - - model_name: gpt-realtime - litellm_params: - model: openai/gpt-realtime - api_key: os.environ/OPENAI_API_KEY -``` - -```bash -LITELLM_CONFIG_PATH=./config.yaml ./litellm-ai-gateway -``` - -At boot `litellm-config` calls into `litellm.proxy.read_model_list` and returns -resolved deployments to the gateway, which constructs the router. The Python -backend still reuses the **real proxy config reader** (`ProxyConfig.get_config`), -so everything the proxy supports in config.yaml works here too: - -- `include:` to merge in other config files, -- `os.environ/VAR` secret references (resolved via the secret manager, never - inlined), -- DB-stored models (when a database is configured). - -Secrets stay out of the config — reference them with `os.environ/...` and set -the env var at deploy time. The shipped Docker image is built with the -`python-config` feature and **bundles litellm**, so config loading works out of -the box; the default baked config lives at `/app/config.yaml` and can be -overridden at deploy time (e.g. a Render secret file mounted at the same path). - -### Environment variables - -| Var | Required | Default | Purpose | -|---|---|---|---| -| `LITELLM_CONFIG_PATH` | yes (config mode) | — | Path to the config.yaml the gateway loads its `model_list` from. The Docker image defaults this to `/app/config.yaml`. | -| `LITELLM_MASTER_KEY` | yes | — | Bearer token clients must send. Unset ⇒ all `/v1/realtime` requests are rejected (fail closed). | -| `OPENAI_API_KEY` | yes | — | Upstream OpenAI key. Referenced by config.yaml as `os.environ/OPENAI_API_KEY` for the gateway→OpenAI dial. | -| `HOST` | no | `127.0.0.1` | **Set to `0.0.0.0` in any container/deploy** or external traffic is refused. | -| `PORT` | no | `4001` | Listen port. Render and most PaaS inject this automatically. | -| `LITELLM_PROXY_BASE_URL` | no | `http://localhost:4000` | LiteLLM proxy that request logs are POSTed to. See [Request logging](#request-logging). | - -> Secrets (`LITELLM_MASTER_KEY`, `OPENAI_API_KEY`) are never baked into the image -> or `render.yaml` — inject them at deploy time only. - -### Lean env stand-in (fallback) - -If the binary is built **without** `python-config` (default features), or -`LITELLM_CONFIG_PATH` is unset, the gateway falls back to a single-deployment -stand-in built from the environment: - -| Var | Default | Purpose | -|---|---|---| -| `OPENAI_REALTIME_MODEL` | `gpt-realtime` | The single deployment's model name (also the `?model=` clients pass). | - -The default workspace build links no libpython and needs no config file. This -fallback mode only supports one hard-coded OpenAI deployment. **config.yaml is the recommended path** — use the -stand-in only for the leanest possible build. - -## Request logging - -The gateway runs no spend logic. When a session ends it builds one -`StandardLoggingPayload` and POSTs it to `{LITELLM_PROXY_BASE_URL}/v1/rust_control_plane/logs` -(admin-only, bearer = `LITELLM_MASTER_KEY`), and the proxy replays it through its -normal callbacks (spend logs, Langfuse, etc.). The POST is non-blocking: a bounded -channel drained by a background worker, dropping with a counter if the proxy is -down. It sends one payload per session. Both env vars are in the table above. - -Worker tuning, rarely needed: `LITELLM_LOG_CHANNEL_CAPACITY` (4096), -`LITELLM_LOG_BATCH_SIZE` (256), `LITELLM_LOG_FLUSH_INTERVAL_MS` (500). - -## Build & run with Docker - -The image is built `--features server,python-config` and installs litellm **from this -repo's source** (the config reader is newer than any PyPI release), so the build -**context is the repo root**: - -```bash -# from the repo root -docker build -f litellm-rust/crates/ai-gateway/Dockerfile -t litellm-ai-gateway . - -docker run --rm -p 4001:4001 \ - -e HOST=0.0.0.0 -e PORT=4001 \ - -e LITELLM_MASTER_KEY=sk-local \ - -e OPENAI_API_KEY=$OPENAI_API_KEY \ - litellm-ai-gateway # LITELLM_CONFIG_PATH defaults to /app/config.yaml - -# smoke test -curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/health/readiness # -> 200 -curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/v1/realtime # -> 401 (auth fails closed) -``` - -On boot you should see `loaded model_list from /app/config.yaml via python -config reader` — that confirms the config path (not the env stand-in fallback). -To use your own config, mount it over the default: - -```bash -docker run --rm -p 4001:4001 \ - -e HOST=0.0.0.0 -e LITELLM_MASTER_KEY=sk-local -e OPENAI_API_KEY=$OPENAI_API_KEY \ - -v $(pwd)/my-config.yaml:/app/config.yaml:ro \ - litellm-ai-gateway -``` - -### Cargo-only (no Docker) - -```bash -# config.yaml mode — needs litellm importable in the active python env -LITELLM_CONFIG_PATH=./crates/ai-gateway/config.yaml \ - cargo run --release -p litellm-ai-gateway --features server,python-config - -# env stand-in mode — no python, no config -cargo run --release -p litellm-ai-gateway --features server -``` - -## Deploy on Render - -The service is a Docker **web service**; Render terminates TLS and supports -WebSockets, so the public endpoint is `wss://.onrender.com/v1/realtime`. - -### Option A — Blueprint (`render.yaml`) - -`crates/ai-gateway/render.yaml` describes the service (Docker runtime, -`healthCheckPath: /health/readiness`, repo-root `dockerContext: .`, -`dockerfilePath: ./litellm-rust/crates/ai-gateway/Dockerfile`, -`LITELLM_CONFIG_PATH: /app/config.yaml`). `LITELLM_MASTER_KEY` and -`OPENAI_API_KEY` are `sync: false` — set them in the dashboard after the first -deploy. To use a non-default model_list, mount a **Render Secret File** at -`/app/config.yaml`. Point a Render Blueprint at this repo/branch and apply. - -### Option B — Render API - -```bash -# create a Docker web service from this repo+branch, then set env vars: -curl -X POST https://api.render.com/v1/services \ - -H "Authorization: Bearer $RENDER_API_KEY" -H "Content-Type: application/json" \ - -d '{ - "type": "web_service", "name": "litellm-rust-ai-gateway", - "ownerId": "", "repo": "https://github.com/BerriAI/litellm", - "branch": "", - "serviceDetails": { - "env": "docker", - "envSpecificDetails": { - "dockerfilePath": "./litellm-rust/crates/ai-gateway/Dockerfile", - "dockerContext": "." - }, - "healthCheckPath": "/health/readiness" - } - }' -# then set env vars LITELLM_MASTER_KEY, OPENAI_API_KEY, HOST=0.0.0.0, -# LITELLM_CONFIG_PATH=/app/config.yaml -``` - -Health check path **must** be `/health/readiness`. `autoDeploy` is off by default -in the blueprint — trigger deploys manually (or flip it on) to pick up new commits. - -## Scaling - -Concurrency is what matters, not total connections: each in-flight session holds -one client socket + one upstream socket. To scale, raise the instance count / -enable autoscaling on the Render service (e.g. baseline 10, max 100). Each -instance needs file descriptors for `2 × peak_concurrent_sessions` — raise -`ulimit -n` if you push very high concurrency. - -## Latency note - -The gateway adds the cost of one extra hop: client→gateway, then a fresh -gateway→OpenAI realtime handshake (TLS + WS upgrade + `session.created`). In -benchmarks this is ~100–150 ms of added session-establishment time; first-audio -and steady-state streaming add no measurable overhead. To minimize it, deploy the -gateway in the Render region with the lowest RTT to OpenAI's realtime endpoint. +The executable server, HTTP/WebSocket routes, auth extractors, application +state, config startup, Docker image, and deployment files live in +[`../gateway-server`](../gateway-server) diff --git a/litellm-rust/crates/ai-gateway/benchmarks/realtime/README.md b/litellm-rust/crates/ai-gateway/benchmarks/realtime/README.md index 84e926af243..1f6ba08beaf 100644 --- a/litellm-rust/crates/ai-gateway/benchmarks/realtime/README.md +++ b/litellm-rust/crates/ai-gateway/benchmarks/realtime/README.md @@ -2,7 +2,7 @@ Measures what the gateway adds over talking to OpenAI's realtime WebSocket directly, and what the pre-warmed connection pool removes. See -`../../src/routes/realtime/README.md` for how the pool works. +`../../../gateway-server/src/routes/realtime/README.md` for how the pool works. ## Results diff --git a/litellm-rust/crates/ai-gateway/src/constants.rs b/litellm-rust/crates/ai-gateway/src/constants.rs index 78af374bf70..1258cb4eb3b 100644 --- a/litellm-rust/crates/ai-gateway/src/constants.rs +++ b/litellm-rust/crates/ai-gateway/src/constants.rs @@ -25,18 +25,7 @@ pub(crate) const DEFAULT_MAX_BATCH_SIZE: usize = 256; /// Override: `LITELLM_LOG_FLUSH_INTERVAL_MS`. pub(crate) const DEFAULT_FLUSH_INTERVAL_MS: u64 = 500; -/// Provider attributed to realtime sessions in the logging payload. -#[cfg(feature = "server")] pub(crate) const DEFAULT_PROVIDER: &str = "openai"; pub(crate) const DEFAULT_RESPONSES_WS_CONNECT_TIMEOUT_SECS: u64 = 10; pub(crate) const DEFAULT_RESPONSES_WS_IDLE_TIMEOUT_SECS: u64 = 300; - -/// HTTP path for the non-streaming Anthropic Messages route. -#[cfg(feature = "server")] -pub(crate) const MESSAGES_ROUTE_PATH: &str = "/v1/messages"; - -/// Request headers owned by the gateway and never forwarded upstream. -#[cfg(feature = "server")] -pub(crate) const MESSAGES_HEADERS_NOT_FORWARDED: &[&str] = - &["authorization", "connection", "content-length", "host"]; diff --git a/litellm-rust/crates/ai-gateway/src/io/realtime.rs b/litellm-rust/crates/ai-gateway/src/io/realtime.rs index 662f7328982..b2e4635fbe7 100644 --- a/litellm-rust/crates/ai-gateway/src/io/realtime.rs +++ b/litellm-rust/crates/ai-gateway/src/io/realtime.rs @@ -212,7 +212,7 @@ where /// `transform_realtime_response`). Returns when either side closes. /// /// Generic over the client transport (typed events) so this crate stays -/// framework-agnostic; the gateway adapts its axum socket to these. This is the +/// framework-agnostic; the gateway server adapts its Axum socket to these. This is the /// fresh-dial path: dial, then splice. The pool's warm-handoff path skips the dial /// and calls [`splice`] directly with a buffered `session.created`. #[allow(clippy::too_many_arguments)] @@ -295,7 +295,7 @@ mod tests { /// Live end-to-end check against OpenAI. Ignored by default (CI never runs /// it); run explicitly with `OPENAI_API_KEY` set: - /// `cargo test -p litellm-ai-gateway --features server realtime_invokes_openai -- --ignored --nocapture` + /// `cargo test -p litellm-ai-gateway realtime_invokes_openai -- --ignored --nocapture` #[tokio::test] #[ignore = "hits the live OpenAI realtime API; needs OPENAI_API_KEY"] async fn realtime_invokes_openai_and_responds() { diff --git a/litellm-rust/crates/ai-gateway/src/io/realtime_pool.rs b/litellm-rust/crates/ai-gateway/src/io/realtime_pool.rs index 49e9c459a88..8bae0285eca 100644 --- a/litellm-rust/crates/ai-gateway/src/io/realtime_pool.rs +++ b/litellm-rust/crates/ai-gateway/src/io/realtime_pool.rs @@ -10,7 +10,7 @@ //! reuses. The gateway holds an `Arc` in its state and asks for a //! warm socket per connect; on a miss it fresh-dials exactly as before. The pool //! is a latency optimization, never a correctness dependency — see the gateway's -//! `src/routes/realtime/README.md`. +//! `crates/gateway-server/src/routes/realtime/README.md`. //! //! ## Caveats (enforced here) //! - One warm socket serves exactly one session (realtime isn't multiplexed), so diff --git a/litellm-rust/crates/ai-gateway/src/lib.rs b/litellm-rust/crates/ai-gateway/src/lib.rs index 08fbde564ed..296d3ac149f 100644 --- a/litellm-rust/crates/ai-gateway/src/lib.rs +++ b/litellm-rust/crates/ai-gateway/src/lib.rs @@ -1,32 +1,24 @@ //! LiteLLM AI Gateway library. //! -//! Two layers, split by feature so the Python `cdylib` can depend on the I/O -//! without pulling in the HTTP server: -//! //! - Call-type modules such as [`ocr`]: provider transforms, lifecycle hooks, //! and provider I/O. Always available — no feature required. These predate the //! rule that a route's entrypoint and handler live in `litellm-core` (see //! `litellm_core::messages`) and move there as they are touched. //! - [`io`]: compatibility exports and realtime WebSocket splice helpers. -//! - The server modules ([`auth`], [`routes`], [`state`]) and anything pulling -//! `axum` are gated behind the `server` feature, which the `litellm-ai-gateway` -//! binary turns on. +//! - [`runtime`]: framework-independent orchestration used by HTTP and Python +//! hosts. +//! +//! The Axum host lives in the separate `litellm-gateway-server` crate. pub mod audio_transcription; mod client; pub mod io; pub mod ocr; -#[cfg(feature = "server")] -pub mod auth; -#[cfg(feature = "server")] -pub mod routes; -#[cfg(feature = "server")] -pub mod state; +pub mod realtime; +pub mod runtime; #[cfg(feature = "trace-parity")] pub mod trace_parity; mod constants; pub mod integrations; -#[cfg(feature = "server")] -mod realtime; diff --git a/litellm-rust/crates/ai-gateway/src/routes/messages/service.rs b/litellm-rust/crates/ai-gateway/src/runtime/messages.rs similarity index 98% rename from litellm-rust/crates/ai-gateway/src/routes/messages/service.rs rename to litellm-rust/crates/ai-gateway/src/runtime/messages.rs index 5434719987b..8a87d30b8b9 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/messages/service.rs +++ b/litellm-rust/crates/ai-gateway/src/runtime/messages.rs @@ -7,7 +7,7 @@ use litellm_core::messages::{messages, messages_stream}; use litellm_core::router::Router; use serde_json::{Map, Value}; -pub(crate) enum MessagesResponse { +pub enum MessagesResponse { Json(Value), Stream(reqwest::Response), } diff --git a/litellm-rust/crates/ai-gateway/src/runtime/mod.rs b/litellm-rust/crates/ai-gateway/src/runtime/mod.rs new file mode 100644 index 00000000000..bc4cab4d38d --- /dev/null +++ b/litellm-rust/crates/ai-gateway/src/runtime/mod.rs @@ -0,0 +1,3 @@ +pub mod messages; +pub mod realtime; +pub mod responses; diff --git a/litellm-rust/crates/ai-gateway/src/routes/realtime/service.rs b/litellm-rust/crates/ai-gateway/src/runtime/realtime.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/routes/realtime/service.rs rename to litellm-rust/crates/ai-gateway/src/runtime/realtime.rs diff --git a/litellm-rust/crates/ai-gateway/src/routes/responses/service.rs b/litellm-rust/crates/ai-gateway/src/runtime/responses.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/routes/responses/service.rs rename to litellm-rust/crates/ai-gateway/src/runtime/responses.rs diff --git a/litellm-rust/crates/ai-gateway/src/trace_parity.rs b/litellm-rust/crates/ai-gateway/src/trace_parity.rs index 614852c541d..93a8a8089d1 100644 --- a/litellm-rust/crates/ai-gateway/src/trace_parity.rs +++ b/litellm-rust/crates/ai-gateway/src/trace_parity.rs @@ -1,19 +1,13 @@ -//! Harness-only in-process adapters. Never mounted as production routes. +//! Harness-only adapters. Never mounted as production routes. use std::sync::Arc; -use axum::body::{Body, to_bytes}; -use axum::http::header::{AUTHORIZATION, CONTENT_TYPE}; -use axum::http::{Request, StatusCode}; use litellm_core::Error; -use litellm_core::router::{Deployment, LiteLLMParams, Router as ModelRouter}; +use litellm_core::router::{Deployment, LiteLLMParams, Router}; use serde::Serialize; use serde_json::Value; -use tower::ServiceExt; -use crate::io::realtime_pool::RealtimePool; -use crate::routes; -use crate::state::AppState; +use crate::runtime::messages::{MessagesResponse, run}; #[derive(Debug, Serialize)] pub struct GatewayResponse { @@ -21,45 +15,53 @@ pub struct GatewayResponse { pub body: Value, } +#[tracing::instrument( + name = "messages_gateway_route", + target = "litellm::function_trace", + level = "trace", + skip_all +)] pub async fn messages_request( model_alias: String, provider_model: String, api_base: String, body: Value, ) -> Result { - let state = AppState { - router: Arc::new(ModelRouter::new(vec![Deployment { - model_name: model_alias, - litellm_params: LiteLLMParams { - model: provider_model, - api_key: Some("trace-provider-key".to_string()), - api_base: Some(api_base), - }, - }])), - master_key: Some(Arc::from("trace-master-key")), - loggers: Arc::new(Vec::new()), - realtime_pool: RealtimePool::disabled(), - }; - let request = Request::builder() - .method("POST") - .uri("/v1/messages") - .header(AUTHORIZATION, "Bearer trace-master-key") - .header(CONTENT_TYPE, "application/json") - .body(Body::from(body.to_string())) - .map_err(|error| Error::InvalidRequest(error.to_string()))?; - let response = routes::app(state) - .oneshot(request) - .await - .map_err(|error| match error {})?; - let status: StatusCode = response.status(); - let bytes = to_bytes(response.into_body(), usize::MAX) - .await - .map_err(|error| Error::InvalidResponse(error.to_string()))?; - let body = serde_json::from_slice(&bytes).map_err(|error| { - Error::InvalidResponse(format!("gateway returned invalid JSON: {error}")) - })?; - Ok(GatewayResponse { - status: status.as_u16(), - body, - }) + let router = Arc::new(Router::new(vec![Deployment { + model_name: model_alias, + litellm_params: LiteLLMParams { + model: provider_model, + api_key: Some("trace-provider-key".to_string()), + api_base: Some(api_base), + }, + }])); + match run(&router, body, None).await { + Ok(MessagesResponse::Json(body)) => Ok(GatewayResponse { status: 200, body }), + Ok(MessagesResponse::Stream(_)) => Err(Error::InvalidResponse( + "gateway returned a streaming trace response".to_string(), + )), + Err(error) => Ok(error_response(error)), + } +} + +fn error_response(error: Error) -> GatewayResponse { + let (status, message) = match error { + Error::InvalidRequest(message) => (400, message), + Error::InvalidProvider(_) | Error::Routing(_) => ( + 404, + "no messages deployment is configured for this model".to_string(), + ), + Error::Auth(_) => (502, "messages provider authentication failed".to_string()), + Error::Http { .. } + | Error::Network(_) + | Error::Connect(_) + | Error::InvalidResponse(_) + | Error::InvalidType { .. } + | Error::MissingField(_) => (502, "messages provider request failed".to_string()), + Error::Unsupported(reason) => (400, format!("messages request is not supported: {reason}")), + }; + GatewayResponse { + status, + body: serde_json::json!({"error": {"message": message}}), + } } diff --git a/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs b/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs index fc0ab2b62a3..2d82905882d 100644 --- a/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs +++ b/litellm-rust/crates/core/tests/workspace_crate_allowlist.rs @@ -1,7 +1,8 @@ -//! Enforcement: the litellm-rust workspace has exactly five crates. +//! Enforcement: the litellm-rust workspace has exactly six crates. //! //! `core` (the Rust SDK), `config` (the config-loading boundary), -//! `ai-gateway` (the HTTP/WebSocket host), +//! `ai-gateway` (framework-independent gateway runtime), +//! `gateway-server` (the HTTP/WebSocket host), //! `python-interop` (domain-neutral PyO3 primitives), and `python-bridge` (the //! PyO3 cdylib). Adding or removing a crate must be a //! deliberate act: this test fails until the allowlist here is updated, forcing @@ -22,6 +23,7 @@ const EXPECTED_MEMBERS: &[&str] = &[ "crates/core", "crates/config", "crates/ai-gateway", + "crates/gateway-server", "crates/python-interop", "crates/python-bridge", ]; @@ -31,6 +33,7 @@ const EXPECTED_CRATE_DIRS: &[&str] = &[ "core", "config", "ai-gateway", + "gateway-server", "python-interop", "python-bridge", ]; diff --git a/litellm-rust/crates/gateway-server/AGENTS.md b/litellm-rust/crates/gateway-server/AGENTS.md new file mode 100644 index 00000000000..8eec0bf9ac6 --- /dev/null +++ b/litellm-rust/crates/gateway-server/AGENTS.md @@ -0,0 +1,52 @@ +# gateway-server folder architecture + +The Axum server that fronts the reusable Rust gateway runtime. It owns HTTP/WS +transport, startup config, auth extractors, and application state only. +Deployment selection and transport-neutral orchestration live in +`litellm-ai-gateway` or `core::router`, and provider calls live behind core route +entrypoints such as `litellm_core::messages::messages`. No provider handler +lives here. + +``` +src/ + main.rs # entrypoint: build AppState (router + master key), bind, serve + state.rs # AppState — shared Arc + master_key + auth/ # authentication as an axum extractor — added to handler args + mod.rs # RequireMasterKey: FromRequestParts, single master key (LITELLM_MASTER_KEY) + routes/ # one module per route, all matching the same template + AGENTS.md # ← the route template (read this before adding a route) + mod.rs # app(): merges every module's router() + health.rs # simple route (one file): router() + liveness/readiness + realtime/ # router() + handler + WS<->events adapter +``` + +## Rules + +- **Routes follow one template.** Each route module exposes + `pub fn router() -> Router`; `routes/mod.rs` only merges them. Simple + routes are one file; non-trivial routes are a folder (`handler`/`service`/ + `transport`). See `routes/AGENTS.md`. +- **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's + args; it runs during extraction. Never re-implement the check per route. +- **Handlers are thin.** A handler validates and delegates to `litellm-ai-gateway::runtime`. No + business logic, no provider calls, no transforms in handlers. +- **Runtime services call `core`, they don't reimplement it.** The reusable + service picks the deployment and calls the `core` route entrypoint. Provider + resolution, auth headers, URL building, and the HTTP call are `core`'s job. +- **State is shared and cheap to clone.** Long-lived handles live behind `Arc` in + `state.rs`; read env/config only in `main.rs` when building state. + +## Auth (interim) + +A single **master key** (`LITELLM_MASTER_KEY`), enforced by the +`auth::RequireMasterKey` extractor: any caller presenting it as +`Authorization: Bearer ` may invoke the gateway. Fails closed (500) when +unset; constant-time compare. The server binds `127.0.0.1` by default (`HOST` to +override). Full per-key auth + budgets/rate-limits are delegated to the Python +proxy in a later phase. Health routes don't add the extractor (unauthenticated). + +## Python interop + +Python-backed loading lives in `litellm-config` and is **load-time only**. The +server's `python-config` feature forwards to that crate. The realtime data path +never takes the GIL. diff --git a/litellm-rust/crates/gateway-server/ARCHITECTURE.md b/litellm-rust/crates/gateway-server/ARCHITECTURE.md new file mode 100644 index 00000000000..9902368bb2d --- /dev/null +++ b/litellm-rust/crates/gateway-server/ARCHITECTURE.md @@ -0,0 +1,16 @@ +# gateway-server architecture + +The Rust gateway server owns the Axum binary, HTTP/WebSocket routes, auth +extractors, application state, and startup config. It delegates transport-neutral +orchestration and callback integrations to `litellm-ai-gateway` + +```mermaid +flowchart LR + C[client] <--> S[litellm-gateway-server
Axum host] + S --> G[litellm-ai-gateway
runtime and integrations] + G --> K[litellm-core] + G <--> O[OpenAI realtime] + G -. spend tracking callback .-> P[litellm proxy] + F[litellm-config
load-time only] --> S + F -. Python backend .-> P +``` diff --git a/litellm-rust/crates/gateway-server/Cargo.toml b/litellm-rust/crates/gateway-server/Cargo.toml new file mode 100644 index 00000000000..0ed6b747971 --- /dev/null +++ b/litellm-rust/crates/gateway-server/Cargo.toml @@ -0,0 +1,34 @@ +[package] +name = "litellm-gateway-server" +version = "0.1.0" +edition.workspace = true +license.workspace = true +repository.workspace = true + +[lib] +name = "litellm_gateway_server" + +[[bin]] +name = "litellm-ai-gateway" +path = "src/main.rs" + +[dependencies] +axum = { workspace = true, features = ["ws"] } +futures-util.workspace = true +litellm-ai-gateway.workspace = true +litellm-config.workspace = true +litellm-core = { workspace = true, features = ["bedrock-auth"] } +reqwest.workspace = true +serde.workspace = true +serde_json.workspace = true +sha2.workspace = true +subtle.workspace = true +tokio = { workspace = true, features = ["rt-multi-thread", "macros", "net", "time"] } +tracing.workspace = true + +[features] +default = [] +python-config = ["litellm-config/python"] + +[dev-dependencies] +tower = { version = "0.5.3", features = ["util"] } diff --git a/litellm-rust/crates/ai-gateway/Dockerfile b/litellm-rust/crates/gateway-server/Dockerfile similarity index 91% rename from litellm-rust/crates/ai-gateway/Dockerfile rename to litellm-rust/crates/gateway-server/Dockerfile index 2bc3c05ad7e..3cbaeae5875 100644 --- a/litellm-rust/crates/ai-gateway/Dockerfile +++ b/litellm-rust/crates/gateway-server/Dockerfile @@ -5,7 +5,7 @@ # which is not in any PyPI release yet) AND build the rust workspace under # litellm-rust/. # -# docker build -f litellm-rust/crates/ai-gateway/Dockerfile -t litellm-ai-gateway . +# docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-ai-gateway . # # No secrets live in this file. Runtime config (LITELLM_MASTER_KEY, # OPENAI_API_KEY referenced by config.yaml, etc.) is injected as environment @@ -36,12 +36,12 @@ FROM chef AS builder # whenever only gateway source changes. COPY --from=planner /build/litellm-rust/recipe.json recipe.json RUN cargo chef cook --locked --release \ - -p litellm-ai-gateway --features server,python-config \ + -p litellm-gateway-server --features python-config \ --recipe-path recipe.json # Now copy the real sources and build the gateway binary. Deps are already cooked # above, so this step only recompiles the gateway crate. COPY litellm-rust/ . -RUN cargo build --locked --release -p litellm-ai-gateway --bin litellm-ai-gateway --features server,python-config +RUN cargo build --locked --release -p litellm-gateway-server --bin litellm-ai-gateway --features python-config # ---- Runtime ---------------------------------------------------------------- # python:3.11-slim-bookworm ships libpython3.11, matching the builder's PyO3 @@ -68,7 +68,7 @@ COPY --from=builder /build/litellm-rust/target/release/litellm-ai-gateway /usr/l # Default config.yaml. A real deploy can override this (e.g. mount a Render # secret file at the same path) — never bake secrets into the image. -COPY litellm-rust/crates/ai-gateway/config.yaml /app/config.yaml +COPY litellm-rust/crates/gateway-server/config.yaml /app/config.yaml # Bind to all interfaces (Render routes to 0.0.0.0:$PORT) and load the model_list # from config.yaml via the embedded python config reader. diff --git a/litellm-rust/crates/ai-gateway/Dockerfile.dockerignore b/litellm-rust/crates/gateway-server/Dockerfile.dockerignore similarity index 100% rename from litellm-rust/crates/ai-gateway/Dockerfile.dockerignore rename to litellm-rust/crates/gateway-server/Dockerfile.dockerignore diff --git a/litellm-rust/crates/gateway-server/README.md b/litellm-rust/crates/gateway-server/README.md new file mode 100644 index 00000000000..b6ede1ae47a --- /dev/null +++ b/litellm-rust/crates/gateway-server/README.md @@ -0,0 +1,206 @@ +# LiteLLM Rust Gateway Server + +A minimal Axum service that fronts OpenAI's realtime API. Clients open a +WebSocket to `GET /v1/realtime`; the gateway authenticates, selects a deployment, +dials OpenAI upstream, and splices the two sockets frame-by-frame. + +## Crates + +`litellm-rust` has six crates. A crate is a layer, shared foundation, or separate host, not a route: + +| Crate | Role | +|-------|------| +| litellm-core | The LiteLLM SDK in Rust — per-route entrypoints (`messages::messages()`) that resolve the provider, transform, and make the call; plus types, provider transforms, and the router. | +| litellm-config | Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python. | +| litellm-ai-gateway | Framework-independent gateway runtime and integrations shared by the server and Python bridge. | +| litellm-gateway-server | Axum binary, HTTP/WebSocket routes, auth extractors, application state, startup config, and HTTP-only dependencies. | +| litellm-python-interop | Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion. | +| litellm-python-bridge | PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK. | + +Dependency direction is acyclic: config and the gateway runtime depend on core, the server depends on gateway, config, and core, and the Python bridge depends only on reusable domain layers and Python interop. + +- **Client endpoint:** `wss:///v1/realtime?model=` (WebSocket) +- **Auth:** `Authorization: Bearer $LITELLM_MASTER_KEY` (fails closed if unset) +- **Health:** `GET /health/readiness`, `GET /health/liveness` +- **Request logs:** POSTed to a LiteLLM proxy at `/v1/rust_control_plane/logs` (see [Request logging](#request-logging)) + +> **Realtime serving is pure Rust.** Python is used at **load time only** — to +> read the config once at boot. The realtime hot path never touches Python. + +The former `/health/gil` route and its acquisition counter were removed. They +only observed the single startup config load and did not prove that every GIL +acquisition was instrumented + +## Configuration (config.yaml) + +The gateway loads its `model_list` from a **config.yaml**, the same as the +LiteLLM proxy. Point `LITELLM_CONFIG_PATH` at the file: + +```yaml +# config.yaml +model_list: + - model_name: gpt-realtime + litellm_params: + model: openai/gpt-realtime + api_key: os.environ/OPENAI_API_KEY +``` + +```bash +LITELLM_CONFIG_PATH=./config.yaml ./litellm-ai-gateway +``` + +At boot `litellm-config` calls into `litellm.proxy.read_model_list` and returns +resolved deployments to the gateway, which constructs the router. The Python +backend still reuses the **real proxy config reader** (`ProxyConfig.get_config`), +so everything the proxy supports in config.yaml works here too: + +- `include:` to merge in other config files, +- `os.environ/VAR` secret references (resolved via the secret manager, never + inlined), +- DB-stored models (when a database is configured). + +Secrets stay out of the config — reference them with `os.environ/...` and set +the env var at deploy time. The shipped Docker image is built with the +`python-config` feature and **bundles litellm**, so config loading works out of +the box; the default baked config lives at `/app/config.yaml` and can be +overridden at deploy time (e.g. a Render secret file mounted at the same path). + +### Environment variables + +| Var | Required | Default | Purpose | +|---|---|---|---| +| `LITELLM_CONFIG_PATH` | yes (config mode) | — | Path to the config.yaml the gateway loads its `model_list` from. The Docker image defaults this to `/app/config.yaml`. | +| `LITELLM_MASTER_KEY` | yes | — | Bearer token clients must send. Unset ⇒ all `/v1/realtime` requests are rejected (fail closed). | +| `OPENAI_API_KEY` | yes | — | Upstream OpenAI key. Referenced by config.yaml as `os.environ/OPENAI_API_KEY` for the gateway→OpenAI dial. | +| `HOST` | no | `127.0.0.1` | **Set to `0.0.0.0` in any container/deploy** or external traffic is refused. | +| `PORT` | no | `4001` | Listen port. Render and most PaaS inject this automatically. | +| `LITELLM_PROXY_BASE_URL` | no | `http://localhost:4000` | LiteLLM proxy that request logs are POSTed to. See [Request logging](#request-logging). | + +> Secrets (`LITELLM_MASTER_KEY`, `OPENAI_API_KEY`) are never baked into the image +> or `render.yaml` — inject them at deploy time only. + +### Lean env stand-in (fallback) + +If the binary is built **without** `python-config` (default features), or +`LITELLM_CONFIG_PATH` is unset, the gateway falls back to a single-deployment +stand-in built from the environment: + +| Var | Default | Purpose | +|---|---|---| +| `OPENAI_REALTIME_MODEL` | `gpt-realtime` | The single deployment's model name (also the `?model=` clients pass). | + +The default workspace build links no libpython and needs no config file. This +fallback mode only supports one hard-coded OpenAI deployment. **config.yaml is the recommended path** — use the +stand-in only for the leanest possible build. + +## Request logging + +The gateway runs no spend logic. When a session ends it builds one +`StandardLoggingPayload` and POSTs it to `{LITELLM_PROXY_BASE_URL}/v1/rust_control_plane/logs` +(admin-only, bearer = `LITELLM_MASTER_KEY`), and the proxy replays it through its +normal callbacks (spend logs, Langfuse, etc.). The POST is non-blocking: a bounded +channel drained by a background worker, dropping with a counter if the proxy is +down. It sends one payload per session. Both env vars are in the table above. + +Worker tuning, rarely needed: `LITELLM_LOG_CHANNEL_CAPACITY` (4096), +`LITELLM_LOG_BATCH_SIZE` (256), `LITELLM_LOG_FLUSH_INTERVAL_MS` (500). + +## Build & run with Docker + +The image is built with `--features python-config` and installs litellm **from this +repo's source** (the config reader is newer than any PyPI release), so the build +**context is the repo root**: + +```bash +# from the repo root +docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-ai-gateway . + +docker run --rm -p 4001:4001 \ + -e HOST=0.0.0.0 -e PORT=4001 \ + -e LITELLM_MASTER_KEY=sk-local \ + -e OPENAI_API_KEY=$OPENAI_API_KEY \ + litellm-ai-gateway # LITELLM_CONFIG_PATH defaults to /app/config.yaml + +# smoke test +curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/health/readiness # -> 200 +curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/v1/realtime # -> 401 (auth fails closed) +``` + +On boot you should see `loaded model_list from /app/config.yaml via python +config reader` — that confirms the config path (not the env stand-in fallback). +To use your own config, mount it over the default: + +```bash +docker run --rm -p 4001:4001 \ + -e HOST=0.0.0.0 -e LITELLM_MASTER_KEY=sk-local -e OPENAI_API_KEY=$OPENAI_API_KEY \ + -v $(pwd)/my-config.yaml:/app/config.yaml:ro \ + litellm-ai-gateway +``` + +### Cargo-only (no Docker) + +```bash +# config.yaml mode — needs litellm importable in the active python env +LITELLM_CONFIG_PATH=./crates/gateway-server/config.yaml \ + cargo run --release -p litellm-gateway-server --features python-config + +# env stand-in mode — no python, no config +cargo run --release -p litellm-gateway-server +``` + +## Deploy on Render + +The service is a Docker **web service**; Render terminates TLS and supports +WebSockets, so the public endpoint is `wss://.onrender.com/v1/realtime`. + +### Option A — Blueprint (`render.yaml`) + +`crates/gateway-server/render.yaml` describes the service (Docker runtime, +`healthCheckPath: /health/readiness`, repo-root `dockerContext: .`, +`dockerfilePath: ./litellm-rust/crates/gateway-server/Dockerfile`, +`LITELLM_CONFIG_PATH: /app/config.yaml`). `LITELLM_MASTER_KEY` and +`OPENAI_API_KEY` are `sync: false` — set them in the dashboard after the first +deploy. To use a non-default model_list, mount a **Render Secret File** at +`/app/config.yaml`. Point a Render Blueprint at this repo/branch and apply. + +### Option B — Render API + +```bash +# create a Docker web service from this repo+branch, then set env vars: +curl -X POST https://api.render.com/v1/services \ + -H "Authorization: Bearer $RENDER_API_KEY" -H "Content-Type: application/json" \ + -d '{ + "type": "web_service", "name": "litellm-rust-ai-gateway", + "ownerId": "", "repo": "https://github.com/BerriAI/litellm", + "branch": "", + "serviceDetails": { + "env": "docker", + "envSpecificDetails": { + "dockerfilePath": "./litellm-rust/crates/gateway-server/Dockerfile", + "dockerContext": "." + }, + "healthCheckPath": "/health/readiness" + } + }' +# then set env vars LITELLM_MASTER_KEY, OPENAI_API_KEY, HOST=0.0.0.0, +# LITELLM_CONFIG_PATH=/app/config.yaml +``` + +Health check path **must** be `/health/readiness`. `autoDeploy` is off by default +in the blueprint — trigger deploys manually (or flip it on) to pick up new commits. + +## Scaling + +Concurrency is what matters, not total connections: each in-flight session holds +one client socket + one upstream socket. To scale, raise the instance count / +enable autoscaling on the Render service (e.g. baseline 10, max 100). Each +instance needs file descriptors for `2 × peak_concurrent_sessions` — raise +`ulimit -n` if you push very high concurrency. + +## Latency note + +The gateway adds the cost of one extra hop: client→gateway, then a fresh +gateway→OpenAI realtime handshake (TLS + WS upgrade + `session.created`). In +benchmarks this is ~100–150 ms of added session-establishment time; first-audio +and steady-state streaming add no measurable overhead. To minimize it, deploy the +gateway in the Render region with the lowest RTT to OpenAI's realtime endpoint. diff --git a/litellm-rust/crates/ai-gateway/config.yaml b/litellm-rust/crates/gateway-server/config.yaml similarity index 100% rename from litellm-rust/crates/ai-gateway/config.yaml rename to litellm-rust/crates/gateway-server/config.yaml diff --git a/litellm-rust/crates/ai-gateway/render.yaml b/litellm-rust/crates/gateway-server/render.yaml similarity index 95% rename from litellm-rust/crates/ai-gateway/render.yaml rename to litellm-rust/crates/gateway-server/render.yaml index 4170849f65d..295049fbcde 100644 --- a/litellm-rust/crates/ai-gateway/render.yaml +++ b/litellm-rust/crates/gateway-server/render.yaml @@ -14,7 +14,7 @@ services: name: litellm-rust-ai-gateway runtime: docker plan: standard - dockerfilePath: ./litellm-rust/crates/ai-gateway/Dockerfile + dockerfilePath: ./litellm-rust/crates/gateway-server/Dockerfile dockerContext: . healthCheckPath: /health/readiness numInstances: 1 diff --git a/litellm-rust/crates/ai-gateway/src/auth/mod.rs b/litellm-rust/crates/gateway-server/src/auth/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/auth/mod.rs rename to litellm-rust/crates/gateway-server/src/auth/mod.rs diff --git a/litellm-rust/crates/gateway-server/src/constants.rs b/litellm-rust/crates/gateway-server/src/constants.rs new file mode 100644 index 00000000000..4370b03e42f --- /dev/null +++ b/litellm-rust/crates/gateway-server/src/constants.rs @@ -0,0 +1,3 @@ +pub(crate) const MESSAGES_ROUTE_PATH: &str = "/v1/messages"; +pub(crate) const MESSAGES_HEADERS_NOT_FORWARDED: &[&str] = + &["authorization", "connection", "content-length", "host"]; diff --git a/litellm-rust/crates/gateway-server/src/lib.rs b/litellm-rust/crates/gateway-server/src/lib.rs new file mode 100644 index 00000000000..59a568f0877 --- /dev/null +++ b/litellm-rust/crates/gateway-server/src/lib.rs @@ -0,0 +1,4 @@ +pub mod auth; +mod constants; +pub mod routes; +pub mod state; diff --git a/litellm-rust/crates/ai-gateway/src/main.rs b/litellm-rust/crates/gateway-server/src/main.rs similarity index 83% rename from litellm-rust/crates/ai-gateway/src/main.rs rename to litellm-rust/crates/gateway-server/src/main.rs index 88d7b1dbcf8..aac266be0d7 100644 --- a/litellm-rust/crates/ai-gateway/src/main.rs +++ b/litellm-rust/crates/gateway-server/src/main.rs @@ -1,28 +1,20 @@ -//! LiteLLM AI Gateway — a minimal Axum server fronting the Rust router. +//! LiteLLM AI Gateway server. //! -//! Flow: client → `POST /v1/realtime` → `router.realtime()` selects a deployment -//! (simple-shuffle) → `io::realtime::realtime()` invokes OpenAI. The -//! server owns transport + config; routing lives in the `router` crate. -//! -//! The binary requires the `server` feature (declared in `Cargo.toml` via -//! `required-features`), so cargo skips it unless that feature is on. Everything -//! the binary needs lives in the library (`litellm_ai_gateway`); `main` just -//! wires startup. +//! The binary owns startup and config, then mounts the Axum routes from +//! `litellm_gateway_server`. Transport-neutral runtime and integrations come +//! from `litellm_ai_gateway`. use std::sync::Arc; +use litellm_ai_gateway::integrations::custom_logger::CustomLogger; +use litellm_ai_gateway::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger; use litellm_ai_gateway::io::realtime_pool::{PoolConfig, RealtimePool, upstream_key}; -use litellm_ai_gateway::routes; -use litellm_ai_gateway::state::AppState; #[cfg(feature = "python-config")] use litellm_config::load_model_list; use litellm_core::router::{Deployment, LiteLLMParams, Router}; +use litellm_gateway_server::routes; +use litellm_gateway_server::state::AppState; -use litellm_ai_gateway::integrations::custom_logger::CustomLogger; -use litellm_ai_gateway::integrations::litellm_python_proxy_api::LiteLLMPythonProxyAPILogger; - -/// Bind to localhost by default so the gateway is not a public, unauthenticated -/// provider proxy out of the box. Override with `HOST` (e.g. `0.0.0.0`). const DEFAULT_HOST: &str = "127.0.0.1"; const DEFAULT_PORT: u16 = 4001; @@ -87,9 +79,10 @@ async fn main() { } /// Register every deployment's upstream key with the pool so the replenisher -/// pre-warms it. Mirrors `service::run`'s key derivation (strip `openai/`, resolve -/// api_key); deployments whose key can't be resolved are skipped (they fresh-dial -/// and surface the auth error on the request path, as before). +/// pre-warms it. Mirrors `runtime::realtime::run`'s key derivation (strip +/// `openai/`, resolve api_key); deployments whose key can't be resolved are +/// skipped (they fresh-dial and surface the auth error on the request path, as +/// before). fn register_deployments(router: &Router, pool: &RealtimePool) { for deployment in router.deployments() { let params = &deployment.litellm_params; diff --git a/litellm-rust/crates/ai-gateway/src/routes/AGENTS.md b/litellm-rust/crates/gateway-server/src/routes/AGENTS.md similarity index 55% rename from litellm-rust/crates/ai-gateway/src/routes/AGENTS.md rename to litellm-rust/crates/gateway-server/src/routes/AGENTS.md index c675916f71a..bdb74929049 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/AGENTS.md +++ b/litellm-rust/crates/gateway-server/src/routes/AGENTS.md @@ -15,27 +15,17 @@ async fn handle(...) -> impl IntoResponse { ... } ``` `health.rs` is the example. -## Split out `service` when there's real logic -When a route has business logic worth testing without axum, put it in a sibling -`service` (a file, or a folder if the route grows). The route file stays the -**axum surface** (router + handler + any socket/SSE adapter); `service` is plain -Rust with **no axum types**, and its job is to pick the deployment and call the -`core` route entrypoint (see `messages/service.rs` calling -`litellm_core::messages::messages`). Never build a provider request, resolve a -key, or perform the provider call here. `realtime/` is the older example: -``` -realtime/ - mod.rs # axum surface: router() + handler + the WS<->events adapter - service.rs # pure logic: select deployment + call provider (no axum) — testable -``` -Split `service` further (or add `transport`, `repo`, …) only once a single file -genuinely gets hard to read. +## Runtime boundary +When a route has transport-neutral orchestration, put it under +`litellm-ai-gateway::runtime` and test it there. The route file stays the Axum +surface: router, handler, and socket or SSE adapter. Never build a provider +request, resolve a provider key, or perform the provider call in this crate. ## Invariants - **Auth is an extractor, not a manual call.** A handler requires auth by adding `crate::auth::RequireMasterKey` to its arguments; it runs during extraction. Never re-implement the check per route. -- **Handlers contain no business logic; `service` contains no axum types.** +- **Handlers contain no business logic; `litellm-ai-gateway` contains no Axum types.** - **No provider handlers in this crate.** Transforms, auth headers, and the provider HTTP call live in `core/src//`. - A route owns its paths in its own `router()`; `mod.rs` only merges. diff --git a/litellm-rust/crates/ai-gateway/src/routes/health.rs b/litellm-rust/crates/gateway-server/src/routes/health.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/routes/health.rs rename to litellm-rust/crates/gateway-server/src/routes/health.rs diff --git a/litellm-rust/crates/ai-gateway/src/routes/messages/mod.rs b/litellm-rust/crates/gateway-server/src/routes/messages/mod.rs similarity index 98% rename from litellm-rust/crates/ai-gateway/src/routes/messages/mod.rs rename to litellm-rust/crates/gateway-server/src/routes/messages/mod.rs index bb9f3851a77..f057e3a9411 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/messages/mod.rs +++ b/litellm-rust/crates/gateway-server/src/routes/messages/mod.rs @@ -1,7 +1,5 @@ //! `POST /v1/messages`, the Anthropic Messages HTTP surface. -mod service; - use axum::Router; use axum::body::Body; use axum::extract::{Json, State}; @@ -9,6 +7,7 @@ use axum::http::StatusCode; use axum::http::header::{CACHE_CONTROL, CONTENT_TYPE, HeaderMap, HeaderValue}; use axum::response::{IntoResponse, Response}; use axum::routing::post; +use litellm_ai_gateway::runtime::messages::{MessagesResponse, run}; use litellm_core::Error; use serde_json::{Map, Value}; @@ -34,12 +33,12 @@ async fn handle( Json(body): Json, ) -> Result { let extra_headers = forwarded_headers(&headers)?; - match service::run(&state.router, body, extra_headers) + match run(&state.router, body, extra_headers) .await .map_err(MessagesRouteError::from)? { - service::MessagesResponse::Json(body) => Ok(Json(body).into_response()), - service::MessagesResponse::Stream(upstream) => stream_response(upstream), + MessagesResponse::Json(body) => Ok(Json(body).into_response()), + MessagesResponse::Stream(upstream) => stream_response(upstream), } } @@ -149,8 +148,8 @@ mod tests { use tower::ServiceExt; use super::super::app; - use crate::io::realtime_pool::RealtimePool; use crate::state::AppState; + use litellm_ai_gateway::io::realtime_pool::RealtimePool; fn state(model: &str, api_base: String, master_key: Option<&str>) -> AppState { state_with_provider(model, model, api_base, master_key) diff --git a/litellm-rust/crates/ai-gateway/src/routes/mod.rs b/litellm-rust/crates/gateway-server/src/routes/mod.rs similarity index 100% rename from litellm-rust/crates/ai-gateway/src/routes/mod.rs rename to litellm-rust/crates/gateway-server/src/routes/mod.rs diff --git a/litellm-rust/crates/ai-gateway/src/routes/realtime/README.md b/litellm-rust/crates/gateway-server/src/routes/realtime/README.md similarity index 91% rename from litellm-rust/crates/ai-gateway/src/routes/realtime/README.md rename to litellm-rust/crates/gateway-server/src/routes/realtime/README.md index 3301576bb85..358490b2939 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/realtime/README.md +++ b/litellm-rust/crates/gateway-server/src/routes/realtime/README.md @@ -1,9 +1,9 @@ # Realtime route (`GET /v1/realtime`) -Proxies OpenAI's realtime WebSocket. `mod.rs` is the axum surface (handler + -socket↔events adapter); `service.rs` is the pure logic (select a deployment, then -splice client ↔ upstream). The pool itself lives in -`crates/providers/src/realtime_pool.rs`. +Proxies OpenAI's realtime WebSocket. `mod.rs` is the Axum surface (handler and +socket-to-events adapter); the pure logic lives in +`litellm-ai-gateway::runtime::realtime`. The pool lives in +`litellm-ai-gateway::io::realtime_pool`. ## Connection pooling @@ -27,7 +27,7 @@ never a correctness dependency. ``` ┌───────────────────────────────────────┐ - client connect ──────► │ routes/realtime → service::run │ + client connect ──────► │ routes/realtime -> runtime::run │ │ pool.take(key) │ │ hit → relay buffered │ │ session.created, then splice │ @@ -84,4 +84,4 @@ upstream sockets, which is why warm sockets are short-lived attempts against a broken key so it can't exhaust upstream rate limits and degrade valid cold-path traffic; the backoff resets the moment a dial succeeds. -Benchmarks and repro: `../../benchmarks/realtime/README.md`. +Benchmarks and repro: `../../../../ai-gateway/benchmarks/realtime/README.md`. diff --git a/litellm-rust/crates/ai-gateway/src/routes/realtime/mod.rs b/litellm-rust/crates/gateway-server/src/routes/realtime/mod.rs similarity index 90% rename from litellm-rust/crates/ai-gateway/src/routes/realtime/mod.rs rename to litellm-rust/crates/gateway-server/src/routes/realtime/mod.rs index f9144ad1fdb..e8f8df4913a 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/realtime/mod.rs +++ b/litellm-rust/crates/gateway-server/src/routes/realtime/mod.rs @@ -1,16 +1,14 @@ //! `GET /v1/realtime` (WebSocket). //! //! This file is the **axum surface**: `router()`, the handler, and the small -//! socket↔events adapter. The pure logic (no axum) lives in [`service`]. Auth is -//! the `RequireMasterKey` extractor, so the handler stays thin. - -mod service; +//! socket↔events adapter. The pure logic lives in +//! [`litellm_ai_gateway::runtime::realtime`]. Auth is the `RequireMasterKey` +//! extractor, so the handler stays thin. use std::sync::Arc; use std::sync::atomic::{AtomicU64, Ordering}; use std::time::{SystemTime, UNIX_EPOCH}; -use crate::io::realtime_pool::RealtimePool; use axum::Router; use axum::extract::ws::{Message, WebSocket, WebSocketUpgrade}; use axum::extract::{Query, State}; @@ -18,14 +16,16 @@ use axum::http::StatusCode; use axum::response::Response; use axum::routing::get; use futures_util::{SinkExt, StreamExt}; +use litellm_ai_gateway::integrations::custom_logger::CustomLogger; +use litellm_ai_gateway::integrations::types::RequestMetadata; +use litellm_ai_gateway::io::realtime_pool::RealtimePool; +use litellm_ai_gateway::realtime::streaming::{RealTimeStreaming, SessionStatus}; +use litellm_ai_gateway::runtime::realtime; use litellm_core::realtime::types::RealtimeEvent; use litellm_core::router::Router as ModelRouter; use serde::Deserialize; use crate::auth::RequireMasterKey; -use crate::integrations::custom_logger::CustomLogger; -use crate::integrations::types::RequestMetadata; -use crate::realtime::streaming::{RealTimeStreaming, SessionStatus}; use crate::state::AppState; /// Process-local monotonic counter, mixed into the per-session call id so two @@ -84,7 +84,7 @@ async fn handle( } /// Adapt the axum socket (text frames) to the typed-event `Stream`/`Sink` the -/// service wants, keeping axum types out of `service`. +/// runtime wants, keeping axum types out of `litellm-ai-gateway`. /// /// This is also the realtime-logging seam: every upstream→client event (the /// direction carrying `session.created` and `response.done` with usage) is fed @@ -146,7 +146,7 @@ async fn bridge( // splice; the borrow ends when `run` returns, freeing the collector for the // single post-session `log_messages` flush. `run` picks a pooled (warm) or // fresh upstream — observe fires on the upstream arm either way. - let result = service::run( + let result = realtime::run( &router, &pool, &model, diff --git a/litellm-rust/crates/ai-gateway/src/routes/responses/mod.rs b/litellm-rust/crates/gateway-server/src/routes/responses/mod.rs similarity index 97% rename from litellm-rust/crates/ai-gateway/src/routes/responses/mod.rs rename to litellm-rust/crates/gateway-server/src/routes/responses/mod.rs index a94853e106d..0f708d0897d 100644 --- a/litellm-rust/crates/ai-gateway/src/routes/responses/mod.rs +++ b/litellm-rust/crates/gateway-server/src/routes/responses/mod.rs @@ -1,5 +1,3 @@ -mod service; - use std::sync::Arc; use std::sync::atomic::{AtomicU64, Ordering}; use std::time::{SystemTime, UNIX_EPOCH}; @@ -11,13 +9,14 @@ use axum::http::StatusCode; use axum::response::Response; use axum::routing::get; use futures_util::{Sink, SinkExt, StreamExt}; +use litellm_ai_gateway::integrations::custom_logger::CustomLogger; +use litellm_ai_gateway::integrations::types::RequestMetadata; +use litellm_ai_gateway::runtime::responses; use litellm_core::responses::types::{ResponsesErrorFrame, ResponsesWsEvent, ResponsesWsEventType}; use litellm_core::router::Router as ModelRouter; use serde::Deserialize; use crate::auth::RequireMasterKey; -use crate::integrations::custom_logger::CustomLogger; -use crate::integrations::types::RequestMetadata; use crate::state::AppState; static CALL_SEQ: AtomicU64 = AtomicU64::new(0); @@ -217,7 +216,7 @@ async fn bridge( } })); let mut client_out = ResponseClientSink { sink: ws_sink }; - let result = service::run( + let result = responses::run( &router, &model, first_frame, @@ -239,10 +238,10 @@ async fn bridge( #[cfg(test)] mod tests { use super::*; - use crate::io::realtime_pool::RealtimePool; use crate::state::AppState; use axum::body::Body; use axum::http::Request; + use litellm_ai_gateway::io::realtime_pool::RealtimePool; use litellm_core::router::Router as ModelRouter; use serde_json::json; use std::pin::Pin; diff --git a/litellm-rust/crates/ai-gateway/src/state.rs b/litellm-rust/crates/gateway-server/src/state.rs similarity index 85% rename from litellm-rust/crates/ai-gateway/src/state.rs rename to litellm-rust/crates/gateway-server/src/state.rs index 3b61d8309ea..cd4a5e5deed 100644 --- a/litellm-rust/crates/ai-gateway/src/state.rs +++ b/litellm-rust/crates/gateway-server/src/state.rs @@ -1,10 +1,9 @@ use std::sync::Arc; -use crate::io::realtime_pool::RealtimePool; +use litellm_ai_gateway::integrations::custom_logger::CustomLogger; +use litellm_ai_gateway::io::realtime_pool::RealtimePool; use litellm_core::router::Router; -use crate::integrations::custom_logger::CustomLogger; - /// Shared application state handed to every route handler. #[derive(Clone)] pub struct AppState {