From 4b57d19672aab355c0f07882fc2d2ea38dfb05b9 Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Sat, 24 Oct 2026 02:09:44 +0000 Subject: [PATCH] docs(gateway): consolidate Rust guidance in AGENTS.md --- gateway/AGENTS.md | 17 +++++++++++ gateway/RUST_GATEWAY.md | 68 ----------------------------------------- 2 files changed, 17 insertions(+), 68 deletions(-) create mode 100644 gateway/AGENTS.md delete mode 100644 gateway/RUST_GATEWAY.md diff --git a/gateway/AGENTS.md b/gateway/AGENTS.md new file mode 100644 index 00000000000..951abe49e59 --- /dev/null +++ b/gateway/AGENTS.md @@ -0,0 +1,17 @@ +# Gateway + +## Rust gateway + +- `LITELLM_RUST_GATEWAY=true` enables the public Rust listener. Unset or exact `false` preserves Python-only startup +- Keep `LITELLM_RUST` independent. It controls the in-process Rust bridge +- Rust forwards every request to Python, which owns authentication, routing, policy and provider calls. Do not add native request selection or database initialization here +- `launch.py` owns startup and pooling. `supervisor.py` coordinates container processes, not application requests. A Python or Rust child exit fails the container rather than restarting that child +- Start Python on loopback `127.0.0.1:4002` before Rust on public port 4000. Reserve 4001 for metrics and never expose the private Python port in container or Service ports +- With a writer `DATABASE_URL`, start one managed PgBouncer even when `LITELLM_PGBOUNCER_ENABLED=false`. Both children receive the same pooled URL and marker, and Python must not start another pooler +- Keep database-free mode database-free. Read replicas remain outside the shared writer connection cap +- Drain Rust, stop Python, then stop PgBouncer and reap remaining children within the shutdown deadline. Defaults allow 15 seconds for Rust, 5 for Python and a 10-second cleanup margin +- Docker stop timeout and Helm termination grace must cover `LITELLM_RUST_GATEWAY_TERMINATION_GRACE_SECONDS`, which defaults to 30 +- Configure `LITELLM_RUST_GATEWAY_TRUSTED_PROXIES` with explicit ingress IPs or CIDRs. Empty trusts no external proxy, wildcard trust is rejected, and `Host` and `Origin` are preserved +- Terminate TLS at the edge. Enabled mode rejects Uvicorn TLS, reload, Unix-socket and file-descriptor listeners, env-file overrides and disabled proxy headers +- The image packages `/usr/local/bin/litellm-gateway`. Local runs may override `LITELLM_RUST_GATEWAY_BINARY` and the private port through `LITELLM_RUST_GATEWAY_PYTHON_PORT` +- Roll back by removing the Rust gateway opt-in or setting it to `false` and redeploying diff --git a/gateway/RUST_GATEWAY.md b/gateway/RUST_GATEWAY.md deleted file mode 100644 index ab0e212e6cb..00000000000 --- a/gateway/RUST_GATEWAY.md +++ /dev/null @@ -1,68 +0,0 @@ -# Opt-in Rust gateway - -The Rust gateway is an optional public HTTP listener. The existing Python gateway remains the default. Set -`LITELLM_RUST_GATEWAY=true` to start the Rust gateway on the public listener and supervise the existing Python -gateway on loopback port 4002. Set the flag to exactly `false`, or leave it unset, to keep the existing startup -path. No separate enable flag is required for the Python upstream. - -The Rust binary defaults to `/usr/local/bin/litellm-gateway`; local deployments can set -`LITELLM_RUST_GATEWAY_BINARY` to an executable binary. `LITELLM_RUST_GATEWAY_PYTHON_PORT` changes the private -Python port. The private listener must remain on loopback and must not be added to a container or Kubernetes -Service port. - -When the Rust gateway is enabled with a writer `DATABASE_URL`, the supervisor starts one managed PgBouncer for -both gateway children, even when `LITELLM_PGBOUNCER_ENABLED=false`. That flag continues to control optional -pooling in Python-only mode. A database-free Rust gateway stays database-free, and Rust only forwards requests -without initializing a database client. The managed writer pool uses `LITELLM_PGBOUNCER_MAX_DB_CONNECTIONS`; -read-replica connections keep their existing behavior and are outside that writer cap - -```sh -docker build -f gateway/Dockerfile -t litellm-gateway:local . -docker run --stop-timeout 30 --mount type=bind,src="$(pwd)/config.yaml",dst=/app/config/config.yaml,readonly \ - -p 4000:4000 \ - -e LITELLM_RUST_GATEWAY=true \ - -e LITELLM_RUST_GATEWAY_TRUSTED_PROXIES=10.0.0.0/8 \ - -e LITELLM_RUST_GATEWAY_TERMINATION_GRACE_SECONDS=30 \ - -e LITELLM_GATEWAY_METRICS_PORT=4001 \ - -e CONFIG_FILE_PATH=/app/config/config.yaml \ - litellm-gateway:local -``` - -The supervisor waits up to 120 seconds for startup, drains Rust for up to 15 seconds, then gives Python 5 seconds -to stop. It also allows a 10-second cleanup margin. `LITELLM_RUST_GATEWAY_STARTUP_TIMEOUT_SECONDS`, -`LITELLM_RUST_GATEWAY_DRAIN_TIMEOUT_SECONDS`, and `LITELLM_RUST_GATEWAY_PYTHON_GRACE_SECONDS` configure those -timeouts. `LITELLM_RUST_GATEWAY_TERMINATION_GRACE_SECONDS` must be at least the drain timeout plus Python grace -plus the cleanup margin. Container orchestrators must grant at least this value before sending SIGKILL. For -Docker, set `--stop-timeout` to the same or a larger number; Docker's default stop timeout is shorter - -The trusted-proxy setting is an optional comma-separated list of IP addresses or CIDRs. An unset or empty list -trusts no external proxy. Wildcard trust is rejected. The Rust gateway canonicalizes forwarded client and scheme -headers using the configured trust list. It preserves the original `Host` and `Origin` headers. Configure only -proxy addresses that are controlled by your deployment - -The Rust gateway does not support TLS termination or Uvicorn TLS settings. Terminate TLS at the load balancer or -another trusted edge, then forward HTTP to the gateway. Uvicorn reload, Unix-domain and file-descriptor listeners, -environment-file overrides, and explicit `--no-proxy-headers` are rejected in enabled mode - -The Helm chart keeps the Service on port 4000. Configure `LITELLM_RUST_GATEWAY` through the gateway container's -environment or `envFrom`, and set `gateway.terminationGracePeriodSeconds` to match the supervisor grace setting. -When the metrics sidecar is enabled, the chart exports its reserved port to the supervisor even when the feature -flag comes from `envFrom`, so port collisions are rejected during enabled startup - -```yaml -gateway: - envConfigMaps: - - gateway-runtime - metricsServer: - enabled: true - port: 4001 - terminationGracePeriodSeconds: 30 -``` - -The `gateway-runtime` ConfigMap can contain `LITELLM_RUST_GATEWAY: "true"` and -`LITELLM_RUST_GATEWAY_TRUSTED_PROXIES: "10.0.0.0/8"`. The chart does not add a private Python port to the pod -ports or Service - -To roll back, set `LITELLM_RUST_GATEWAY=false` in the same environment source and redeploy, or roll back the -Deployment revision. The feature flag is strict and disabled by default, so removing the opt-in also restores the -existing Python startup path