diff --git a/litellm-rust/crates/ai-gateway/README.md b/litellm-rust/crates/ai-gateway/README.md index 15c68795807..c1ed69d103b 100644 --- a/litellm-rust/crates/ai-gateway/README.md +++ b/litellm-rust/crates/ai-gateway/README.md @@ -35,14 +35,14 @@ route/model permissions are enforced in exactly one place. A client presents - **Virtual key** (`sk-…`) → the gateway POSTs `{api_key, route, model}` to the proxy's **`POST /internal/v1/auth/verify`**, which runs the proxy's real `user_api_key_auth` (key lookup, expiry, budget, rate-limit, and route **+ model** permissions via - `can_key_call_model`) and returns the resolved identity. The result is cached - in-memory (bounded ≤200 entries) **for performance** so the verify endpoint - isn't a per-request bottleneck under load — the same reason the LiteLLM proxy - caches key-auth in-process. The TTL is the perf/freshness knob - (`LITELLM_AUTH_CACHE_TTL_SECS`, default 60s): budget/block/rate-limit changes - take effect within the TTL window. For **strict per-connection enforcement** on - billable routes (every connection re-verified, zero staleness), set the TTL to - `0`. + `can_key_call_model`) and returns the resolved identity. **By default every + connection re-verifies** (no caching), so budget/block/rate-limit are enforced + per connection. This is cheap here because realtime auth is **per long-lived + connection, not per request** — one verify per voice session, not a 10k-RPS + load. A bounded ≤200-entry cache is available for future high-RPS *per-request* + routes via `LITELLM_AUTH_CACHE_TTL_SECS > 0`, which trades budget/rate-limit + freshness (bounded by the TTL) for fewer control-plane calls — like the proxy's + own ~60s auth cache. Data plane → control plane is itself authenticated with a **dedicated data-plane key** (NOT the master key — least privilege): the gateway sends @@ -52,7 +52,7 @@ without it. Set the **same** secret on both sides. ```text client ──Bearer sk-…──▶ ai-gateway ──POST /internal/v1/auth/verify──▶ LiteLLM proxy (X-LiteLLM-Data-Plane-Key) user_api_key_auth() - cache on (TTL, default 60s) → UserAPIKeyAuth | 401 + re-verify per connection (cache opt-in) → UserAPIKeyAuth | 401 ``` ### Wiring it up @@ -74,7 +74,7 @@ export LITELLM_AUTH_VERIFY_URL=https:///internal/v1/auth/verify Fails closed: a missing/wrong data-plane key, an unreachable proxy, or a rejected key all yield `401`. Revocation and budget changes take effect within the cache TTL -(`LITELLM_AUTH_CACHE_TTL_SECS`, default 60s; set `0` for strict per-connection verification). Keep the proxy on a private network — +(if caching is enabled via `LITELLM_AUTH_CACHE_TTL_SECS`; off by default → every connection re-verifies). Keep the proxy on a private network — the verify endpoint is internal-only and excluded from the public OpenAPI spec. ## Configuration (config.yaml) @@ -118,7 +118,7 @@ overridden at deploy time (e.g. a Render secret file mounted at the same path). | `LITELLM_MASTER_KEY` | yes | — | Admin bearer token (checked locally, no proxy call). Unset ⇒ the master-key path is disabled. | | `LITELLM_DATA_PLANE_KEY` | for virtual keys | — | Dedicated secret the gateway sends as `X-LiteLLM-Data-Plane-Key` to authenticate itself to the proxy's verify endpoint. **Must match the proxy's `LITELLM_DATA_PLANE_KEY`.** Not the master key. | | `LITELLM_AUTH_VERIFY_URL` | for virtual keys | `http://localhost:4000/internal/v1/auth/verify` | The proxy's verify endpoint the gateway delegates virtual-key auth to. | -| `LITELLM_AUTH_CACHE_TTL_SECS` | no | `60` | TTL for the gateway's verified-key cache (on by default for performance — keeps the verify endpoint from being a per-request bottleneck). Budget/block/rate-limit changes apply within the TTL. Set `0` for strict per-connection verification (zero staleness). | +| `LITELLM_AUTH_CACHE_TTL_SECS` | no | `0` | Verified-key cache TTL. **`0` = off (default)** → every connection re-verifies (budget/rate-limit enforced each time); cheap for realtime since auth is per-connection. Set `> 0` only for high-RPS per-request routes, trading budget/rate-limit freshness for fewer proxy calls. | | `OPENAI_API_KEY` | yes | — | Upstream OpenAI key. Referenced by config.yaml as `os.environ/OPENAI_API_KEY` for the gateway→OpenAI dial. | | `HOST` | no | `127.0.0.1` | **Set to `0.0.0.0` in any container/deploy** or external traffic is refused. | | `PORT` | no | `4001` | Listen port. Render and most PaaS inject this automatically. |