litellm/litellm-rust/crates/gateway-server

LiteLLM Rust Gateway Server

A minimal Axum service that fronts OpenAI's realtime API. Clients open a WebSocket to GET /v1/realtime; the gateway authenticates, selects a deployment, dials OpenAI upstream, and splices the two sockets frame-by-frame.

Crates

litellm-rust has six crates. A crate is a layer, shared foundation, or separate host, not a route:

Crate	Role
litellm-core	The LiteLLM SDK in Rust — per-route entrypoints (messages::messages()) that resolve the provider, transform, and make the call; plus types, provider transforms, and the router.
litellm-config	Config-loading boundary. Returns resolved deployments and optionally delegates loading to Python.
litellm-gateway-inference	Framework-independent gateway runtime and integrations shared by the server and Python bridge.
litellm-gateway-server	Root Axum binary and composition crate. Owns routes, auth, state, startup config, and HTTP-only dependencies.
litellm-python-interop	Domain-neutral PyO3 foundation for GIL handling and typed Python/Serde conversion.
litellm-python-bridge	PyO3 cdylib exposing LiteLLM Rust APIs to the Python SDK.

Dependency direction is acyclic: config and the gateway runtime depend on core, the server depends on gateway, config, and core, and the Python bridge depends only on reusable domain layers and Python interop.

• Client endpoint: wss://<host>/v1/realtime?model=<model> (WebSocket)
• Auth: Authorization: Bearer $LITELLM_MASTER_KEY (fails closed if unset)
• Health: GET /health/readiness, GET /health/liveness
• Request logs: POSTed to a LiteLLM proxy at /v1/rust_control_plane/logs (see Request logging)

Realtime serving is pure Rust. Python is used at load time only — to read the config once at boot. The realtime hot path never touches Python.

The former /health/gil route and its acquisition counter were removed. They only observed the single startup config load and did not prove that every GIL acquisition was instrumented

Configuration (config.yaml)

The gateway loads its model_list from a config.yaml, the same as the LiteLLM proxy. Point LITELLM_CONFIG_PATH at the file:

# config.yaml
model_list:
  - model_name: gpt-realtime
    litellm_params:
      model: openai/gpt-realtime
      api_key: os.environ/OPENAI_API_KEY

LITELLM_CONFIG_PATH=./config.yaml ./litellm-gateway-server

At boot litellm-config calls into litellm.proxy.read_model_list and returns resolved deployments to the gateway, which constructs the router. The Python backend still reuses the real proxy config reader (ProxyConfig.get_config), so everything the proxy supports in config.yaml works here too:

• include: to merge in other config files,
• os.environ/VAR secret references (resolved via the secret manager, never inlined),
• DB-stored models (when a database is configured).

Secrets stay out of the config — reference them with os.environ/... and set the env var at deploy time. The shipped Docker image is built with the python-config feature and bundles litellm, so config loading works out of the box; the default baked config lives at /app/config.yaml and can be overridden at deploy time (e.g. a Render secret file mounted at the same path).

Environment variables

Var	Required	Default	Purpose
LITELLM_CONFIG_PATH	yes (config mode)	—	Path to the config.yaml the gateway loads its model_list from. The Docker image defaults this to /app/config.yaml.
LITELLM_MASTER_KEY	yes	—	Bearer token clients must send. Unset ⇒ all /v1/realtime requests are rejected (fail closed).
OPENAI_API_KEY	yes	—	Upstream OpenAI key. Referenced by config.yaml as os.environ/OPENAI_API_KEY for the gateway→OpenAI dial.
HOST	no	127.0.0.1	Set to 0.0.0.0 in any container/deploy or external traffic is refused.
PORT	no	4001	Listen port. Render and most PaaS inject this automatically.
LITELLM_PROXY_BASE_URL	no	http://localhost:4000	LiteLLM proxy that request logs are POSTed to. See Request logging.

Secrets (LITELLM_MASTER_KEY, OPENAI_API_KEY) are never baked into the image or render.yaml — inject them at deploy time only.

Lean env stand-in (fallback)

If the binary is built without python-config (default features), or LITELLM_CONFIG_PATH is unset, the gateway falls back to a single-deployment stand-in built from the environment:

Var	Default	Purpose
OPENAI_REALTIME_MODEL	gpt-realtime	The single deployment's model name (also the ?model= clients pass).

The default workspace build links no libpython and needs no config file. This fallback mode only supports one hard-coded OpenAI deployment. config.yaml is the recommended path — use the stand-in only for the leanest possible build.

Request logging

The gateway runs no spend logic. When a session ends it builds one StandardLoggingPayload and POSTs it to {LITELLM_PROXY_BASE_URL}/v1/rust_control_plane/logs (admin-only, bearer = LITELLM_MASTER_KEY), and the proxy replays it through its normal callbacks (spend logs, Langfuse, etc.). The POST is non-blocking: a bounded channel drained by a background worker, dropping with a counter if the proxy is down. It sends one payload per session. Both env vars are in the table above.

Worker tuning, rarely needed: LITELLM_LOG_CHANNEL_CAPACITY (4096), LITELLM_LOG_BATCH_SIZE (256), LITELLM_LOG_FLUSH_INTERVAL_MS (500).

Build & run with Docker

The image is built with --features python-config and installs litellm from this repo's source (the config reader is newer than any PyPI release), so the build context is the repo root:

# from the repo root
docker build -f litellm-rust/crates/gateway-server/Dockerfile -t litellm-gateway-server .

docker run --rm -p 4001:4001 \
  -e HOST=0.0.0.0 -e PORT=4001 \
  -e LITELLM_MASTER_KEY=sk-local \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  litellm-gateway-server          # LITELLM_CONFIG_PATH defaults to /app/config.yaml

# smoke test
curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/health/readiness   # -> 200
curl -s -o /dev/null -w '%{http_code}\n' localhost:4001/v1/realtime         # -> 401 (auth fails closed)

On boot you should see loaded model_list from /app/config.yaml via python config reader — that confirms the config path (not the env stand-in fallback). To use your own config, mount it over the default:

docker run --rm -p 4001:4001 \
  -e HOST=0.0.0.0 -e LITELLM_MASTER_KEY=sk-local -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -v $(pwd)/my-config.yaml:/app/config.yaml:ro \
  litellm-gateway-server

Cargo-only (no Docker)

# config.yaml mode — needs litellm importable in the active python env
LITELLM_CONFIG_PATH=./crates/gateway-server/config.yaml \
  cargo run --release -p litellm-gateway-server --features python-config

# env stand-in mode — no python, no config
cargo run --release -p litellm-gateway-server

Deploy on Render

The service is a Docker web service; Render terminates TLS and supports WebSockets, so the public endpoint is wss://<service>.onrender.com/v1/realtime.

Option A — Blueprint (render.yaml)

crates/gateway-server/render.yaml describes the service (Docker runtime, healthCheckPath: /health/readiness, repo-root dockerContext: ., dockerfilePath: ./litellm-rust/crates/gateway-server/Dockerfile, LITELLM_CONFIG_PATH: /app/config.yaml). LITELLM_MASTER_KEY and OPENAI_API_KEY are sync: false — set them in the dashboard after the first deploy. To use a non-default model_list, mount a Render Secret File at /app/config.yaml. Point a Render Blueprint at this repo/branch and apply.

Option B — Render API

# create a Docker web service from this repo+branch, then set env vars:
curl -X POST https://api.render.com/v1/services \
  -H "Authorization: Bearer $RENDER_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "type": "web_service", "name": "litellm-rust-gateway-server",
    "ownerId": "<owner-id>", "repo": "https://github.com/BerriAI/litellm",
    "branch": "<branch-with-this-dockerfile>",
    "serviceDetails": {
      "env": "docker",
      "envSpecificDetails": {
        "dockerfilePath": "./litellm-rust/crates/gateway-server/Dockerfile",
        "dockerContext": "."
      },
      "healthCheckPath": "/health/readiness"
    }
  }'
# then set env vars LITELLM_MASTER_KEY, OPENAI_API_KEY, HOST=0.0.0.0,
# LITELLM_CONFIG_PATH=/app/config.yaml

Health check path must be /health/readiness. autoDeploy is off by default in the blueprint — trigger deploys manually (or flip it on) to pick up new commits.

Scaling

Concurrency is what matters, not total connections: each in-flight session holds one client socket + one upstream socket. To scale, raise the instance count / enable autoscaling on the Render service (e.g. baseline 10, max 100). Each instance needs file descriptors for 2 × peak_concurrent_sessions — raise ulimit -n if you push very high concurrency.

Latency note

The gateway adds the cost of one extra hop: client→gateway, then a fresh gateway→OpenAI realtime handshake (TLS + WS upgrade + session.created). In benchmarks this is ~100–150 ms of added session-establishment time; first-audio and steady-state streaming add no measurable overhead. To minimize it, deploy the gateway in the Render region with the lowest RTT to OpenAI's realtime endpoint.