diff --git a/litellm-rust/crates/ai-gateway/README.md b/litellm-rust/crates/ai-gateway/README.md index 649b9b07942..9e304f2c1a8 100644 --- a/litellm-rust/crates/ai-gateway/README.md +++ b/litellm-rust/crates/ai-gateway/README.md @@ -17,12 +17,60 @@ dials OpenAI upstream, and splices the two sockets frame-by-frame. Dependency direction (acyclic): litellm-core ← litellm-ai-gateway ← litellm-python-bridge. - **Client endpoint:** `wss:///v1/realtime?model=` (WebSocket) -- **Auth:** `Authorization: Bearer $LITELLM_MASTER_KEY` (fails closed if unset) +- **Auth:** `Authorization: Bearer ` — the master key (admin) or a virtual key validated by the LiteLLM proxy (see below). Fails closed. - **Health:** `GET /health/readiness`, `GET /health/liveness`, `GET /health/gil` > **Realtime serving is pure Rust.** Python is used at **load time only** — to > read the config once at boot. The realtime hot path never touches Python. +## Authentication — delegated to the LiteLLM proxy + +The gateway does **not** re-implement key validation. It delegates virtual-key auth +to the LiteLLM proxy's FastAPI **control plane**, so keys, budgets, rate limits, and +route/model permissions are enforced in exactly one place. A client presents +`Authorization: Bearer ` on the `/v1/realtime` upgrade: + +- **Master key** (`LITELLM_MASTER_KEY`) → treated as proxy admin, checked locally + (constant-time compare). No network call. +- **Virtual key** (`sk-…`) → the gateway POSTs `{api_key, route, model}` to the proxy's + **`POST /internal/v1/auth/verify`**, which runs the proxy's real `user_api_key_auth` + (key lookup, expiry, budget, rate-limit, and route **+ model** permissions via + `can_key_call_model`) and returns the resolved identity. The answer is cached + in-memory (bounded ≤200 entries, short TTL), so it is not a round-trip per request. + +Data plane → control plane is itself authenticated with a **dedicated data-plane key** +(NOT the master key — least privilege): the gateway sends +`X-LiteLLM-Data-Plane-Key: $LITELLM_DATA_PLANE_KEY` and the proxy rejects any call +without it. Set the **same** secret on both sides. + +```text +client ──Bearer sk-…──▶ ai-gateway ──POST /internal/v1/auth/verify──▶ LiteLLM proxy + (X-LiteLLM-Data-Plane-Key) user_api_key_auth() + cache ≤200 / TTL → UserAPIKeyAuth | 401 +``` + +### Wiring it up + +On the **LiteLLM proxy** (control plane) — expose the verify endpoint: + +```bash +export LITELLM_DATA_PLANE_KEY= # a NEW secret, not the master key +litellm --config proxy_config.yaml # serves POST /internal/v1/auth/verify +``` + +On the **gateway** (data plane) — point it at the proxy: + +```bash +export LITELLM_DATA_PLANE_KEY= +export LITELLM_AUTH_VERIFY_URL=https:///internal/v1/auth/verify +./litellm-ai-gateway +``` + +Fails closed: a missing/wrong data-plane key, an unreachable proxy, or a rejected key +all yield `401`. Revocation and budget changes take effect within the cache TTL +(`LITELLM_AUTH_CACHE_TTL_SECS`, default 60s). Keep the proxy on a private network — +the verify endpoint is internal-only and excluded from the public OpenAPI spec. + ## Configuration (config.yaml) The gateway loads its `model_list` from a **config.yaml**, the same as the @@ -61,13 +109,16 @@ overridden at deploy time (e.g. a Render secret file mounted at the same path). | Var | Required | Default | Purpose | |---|---|---|---| | `LITELLM_CONFIG_PATH` | yes (config mode) | — | Path to the config.yaml the gateway loads its `model_list` from. The Docker image defaults this to `/app/config.yaml`. | -| `LITELLM_MASTER_KEY` | yes | — | Bearer token clients must send. Unset ⇒ all `/v1/realtime` requests are rejected (fail closed). | +| `LITELLM_MASTER_KEY` | yes | — | Admin bearer token (checked locally, no proxy call). Unset ⇒ the master-key path is disabled. | +| `LITELLM_DATA_PLANE_KEY` | for virtual keys | — | Dedicated secret the gateway sends as `X-LiteLLM-Data-Plane-Key` to authenticate itself to the proxy's verify endpoint. **Must match the proxy's `LITELLM_DATA_PLANE_KEY`.** Not the master key. | +| `LITELLM_AUTH_VERIFY_URL` | for virtual keys | `http://localhost:4000/internal/v1/auth/verify` | The proxy's verify endpoint the gateway delegates virtual-key auth to. | +| `LITELLM_AUTH_CACHE_TTL_SECS` | no | `60` | TTL for the gateway's in-memory verified-key cache. Lower = faster revocation, more proxy calls. | | `OPENAI_API_KEY` | yes | — | Upstream OpenAI key. Referenced by config.yaml as `os.environ/OPENAI_API_KEY` for the gateway→OpenAI dial. | | `HOST` | no | `127.0.0.1` | **Set to `0.0.0.0` in any container/deploy** or external traffic is refused. | | `PORT` | no | `4001` | Listen port. Render and most PaaS inject this automatically. | -> Secrets (`LITELLM_MASTER_KEY`, `OPENAI_API_KEY`) are never baked into the image -> or `render.yaml` — inject them at deploy time only. +> Secrets (`LITELLM_MASTER_KEY`, `LITELLM_DATA_PLANE_KEY`, `OPENAI_API_KEY`) are never +> baked into the image or `render.yaml` — inject them at deploy time only. ### Lean env stand-in (fallback) @@ -136,9 +187,11 @@ WebSockets, so the public endpoint is `wss://.onrender.com/v1/realtime` `crates/ai-gateway/render.yaml` describes the service (Docker runtime, `healthCheckPath: /health/readiness`, repo-root `dockerContext: .`, `dockerfilePath: ./litellm-rust/crates/ai-gateway/Dockerfile`, -`LITELLM_CONFIG_PATH: /app/config.yaml`). `LITELLM_MASTER_KEY` and -`OPENAI_API_KEY` are `sync: false` — set them in the dashboard after the first -deploy. To use a non-default model_list, mount a **Render Secret File** at +`LITELLM_CONFIG_PATH: /app/config.yaml`). `LITELLM_MASTER_KEY`, +`LITELLM_DATA_PLANE_KEY`, `LITELLM_AUTH_VERIFY_URL`, and `OPENAI_API_KEY` are +`sync: false` — set them in the dashboard after the first deploy (the data-plane +key + verify URL wire virtual-key auth to the proxy; see **Authentication** +above). To use a non-default model_list, mount a **Render Secret File** at `/app/config.yaml`. Point a Render Blueprint at this repo/branch and apply. ### Option B — Render API @@ -161,7 +214,8 @@ curl -X POST https://api.render.com/v1/services \ } }' # then set env vars LITELLM_MASTER_KEY, OPENAI_API_KEY, HOST=0.0.0.0, -# LITELLM_CONFIG_PATH=/app/config.yaml +# LITELLM_CONFIG_PATH=/app/config.yaml, and (for virtual-key auth) +# LITELLM_DATA_PLANE_KEY + LITELLM_AUTH_VERIFY_URL ``` Health check path **must** be `/health/readiness`. `autoDeploy` is off by default