docs(gateway): consolidate Rust guidance in AGENTS.md

This commit is contained in:
Devin AI 2026-10-24 02:09:44 +00:00
parent ae77ac3307
commit 4b57d19672
2 changed files with 17 additions and 68 deletions

17
gateway/AGENTS.md Normal file
View file

@ -0,0 +1,17 @@
# Gateway
## Rust gateway
- `LITELLM_RUST_GATEWAY=true` enables the public Rust listener. Unset or exact `false` preserves Python-only startup
- Keep `LITELLM_RUST` independent. It controls the in-process Rust bridge
- Rust forwards every request to Python, which owns authentication, routing, policy and provider calls. Do not add native request selection or database initialization here
- `launch.py` owns startup and pooling. `supervisor.py` coordinates container processes, not application requests. A Python or Rust child exit fails the container rather than restarting that child
- Start Python on loopback `127.0.0.1:4002` before Rust on public port 4000. Reserve 4001 for metrics and never expose the private Python port in container or Service ports
- With a writer `DATABASE_URL`, start one managed PgBouncer even when `LITELLM_PGBOUNCER_ENABLED=false`. Both children receive the same pooled URL and marker, and Python must not start another pooler
- Keep database-free mode database-free. Read replicas remain outside the shared writer connection cap
- Drain Rust, stop Python, then stop PgBouncer and reap remaining children within the shutdown deadline. Defaults allow 15 seconds for Rust, 5 for Python and a 10-second cleanup margin
- Docker stop timeout and Helm termination grace must cover `LITELLM_RUST_GATEWAY_TERMINATION_GRACE_SECONDS`, which defaults to 30
- Configure `LITELLM_RUST_GATEWAY_TRUSTED_PROXIES` with explicit ingress IPs or CIDRs. Empty trusts no external proxy, wildcard trust is rejected, and `Host` and `Origin` are preserved
- Terminate TLS at the edge. Enabled mode rejects Uvicorn TLS, reload, Unix-socket and file-descriptor listeners, env-file overrides and disabled proxy headers
- The image packages `/usr/local/bin/litellm-gateway`. Local runs may override `LITELLM_RUST_GATEWAY_BINARY` and the private port through `LITELLM_RUST_GATEWAY_PYTHON_PORT`
- Roll back by removing the Rust gateway opt-in or setting it to `false` and redeploying

View file

@ -1,68 +0,0 @@
# Opt-in Rust gateway
The Rust gateway is an optional public HTTP listener. The existing Python gateway remains the default. Set
`LITELLM_RUST_GATEWAY=true` to start the Rust gateway on the public listener and supervise the existing Python
gateway on loopback port 4002. Set the flag to exactly `false`, or leave it unset, to keep the existing startup
path. No separate enable flag is required for the Python upstream.
The Rust binary defaults to `/usr/local/bin/litellm-gateway`; local deployments can set
`LITELLM_RUST_GATEWAY_BINARY` to an executable binary. `LITELLM_RUST_GATEWAY_PYTHON_PORT` changes the private
Python port. The private listener must remain on loopback and must not be added to a container or Kubernetes
Service port.
When the Rust gateway is enabled with a writer `DATABASE_URL`, the supervisor starts one managed PgBouncer for
both gateway children, even when `LITELLM_PGBOUNCER_ENABLED=false`. That flag continues to control optional
pooling in Python-only mode. A database-free Rust gateway stays database-free, and Rust only forwards requests
without initializing a database client. The managed writer pool uses `LITELLM_PGBOUNCER_MAX_DB_CONNECTIONS`;
read-replica connections keep their existing behavior and are outside that writer cap
```sh
docker build -f gateway/Dockerfile -t litellm-gateway:local .
docker run --stop-timeout 30 --mount type=bind,src="$(pwd)/config.yaml",dst=/app/config/config.yaml,readonly \
-p 4000:4000 \
-e LITELLM_RUST_GATEWAY=true \
-e LITELLM_RUST_GATEWAY_TRUSTED_PROXIES=10.0.0.0/8 \
-e LITELLM_RUST_GATEWAY_TERMINATION_GRACE_SECONDS=30 \
-e LITELLM_GATEWAY_METRICS_PORT=4001 \
-e CONFIG_FILE_PATH=/app/config/config.yaml \
litellm-gateway:local
```
The supervisor waits up to 120 seconds for startup, drains Rust for up to 15 seconds, then gives Python 5 seconds
to stop. It also allows a 10-second cleanup margin. `LITELLM_RUST_GATEWAY_STARTUP_TIMEOUT_SECONDS`,
`LITELLM_RUST_GATEWAY_DRAIN_TIMEOUT_SECONDS`, and `LITELLM_RUST_GATEWAY_PYTHON_GRACE_SECONDS` configure those
timeouts. `LITELLM_RUST_GATEWAY_TERMINATION_GRACE_SECONDS` must be at least the drain timeout plus Python grace
plus the cleanup margin. Container orchestrators must grant at least this value before sending SIGKILL. For
Docker, set `--stop-timeout` to the same or a larger number; Docker's default stop timeout is shorter
The trusted-proxy setting is an optional comma-separated list of IP addresses or CIDRs. An unset or empty list
trusts no external proxy. Wildcard trust is rejected. The Rust gateway canonicalizes forwarded client and scheme
headers using the configured trust list. It preserves the original `Host` and `Origin` headers. Configure only
proxy addresses that are controlled by your deployment
The Rust gateway does not support TLS termination or Uvicorn TLS settings. Terminate TLS at the load balancer or
another trusted edge, then forward HTTP to the gateway. Uvicorn reload, Unix-domain and file-descriptor listeners,
environment-file overrides, and explicit `--no-proxy-headers` are rejected in enabled mode
The Helm chart keeps the Service on port 4000. Configure `LITELLM_RUST_GATEWAY` through the gateway container's
environment or `envFrom`, and set `gateway.terminationGracePeriodSeconds` to match the supervisor grace setting.
When the metrics sidecar is enabled, the chart exports its reserved port to the supervisor even when the feature
flag comes from `envFrom`, so port collisions are rejected during enabled startup
```yaml
gateway:
envConfigMaps:
- gateway-runtime
metricsServer:
enabled: true
port: 4001
terminationGracePeriodSeconds: 30
```
The `gateway-runtime` ConfigMap can contain `LITELLM_RUST_GATEWAY: "true"` and
`LITELLM_RUST_GATEWAY_TRUSTED_PROXIES: "10.0.0.0/8"`. The chart does not add a private Python port to the pod
ports or Service
To roll back, set `LITELLM_RUST_GATEWAY=false` in the same environment source and redeploy, or roll back the
Deployment revision. The feature flag is strict and disabled by default, so removing the opt-in also restores the
existing Python startup path