mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-17 09:34:48 +00:00
docs(gateway): consolidate Rust guidance in AGENTS.md
This commit is contained in:
parent
ae77ac3307
commit
4b57d19672
2 changed files with 17 additions and 68 deletions
17
gateway/AGENTS.md
Normal file
17
gateway/AGENTS.md
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# Gateway
|
||||
|
||||
## Rust gateway
|
||||
|
||||
- `LITELLM_RUST_GATEWAY=true` enables the public Rust listener. Unset or exact `false` preserves Python-only startup
|
||||
- Keep `LITELLM_RUST` independent. It controls the in-process Rust bridge
|
||||
- Rust forwards every request to Python, which owns authentication, routing, policy and provider calls. Do not add native request selection or database initialization here
|
||||
- `launch.py` owns startup and pooling. `supervisor.py` coordinates container processes, not application requests. A Python or Rust child exit fails the container rather than restarting that child
|
||||
- Start Python on loopback `127.0.0.1:4002` before Rust on public port 4000. Reserve 4001 for metrics and never expose the private Python port in container or Service ports
|
||||
- With a writer `DATABASE_URL`, start one managed PgBouncer even when `LITELLM_PGBOUNCER_ENABLED=false`. Both children receive the same pooled URL and marker, and Python must not start another pooler
|
||||
- Keep database-free mode database-free. Read replicas remain outside the shared writer connection cap
|
||||
- Drain Rust, stop Python, then stop PgBouncer and reap remaining children within the shutdown deadline. Defaults allow 15 seconds for Rust, 5 for Python and a 10-second cleanup margin
|
||||
- Docker stop timeout and Helm termination grace must cover `LITELLM_RUST_GATEWAY_TERMINATION_GRACE_SECONDS`, which defaults to 30
|
||||
- Configure `LITELLM_RUST_GATEWAY_TRUSTED_PROXIES` with explicit ingress IPs or CIDRs. Empty trusts no external proxy, wildcard trust is rejected, and `Host` and `Origin` are preserved
|
||||
- Terminate TLS at the edge. Enabled mode rejects Uvicorn TLS, reload, Unix-socket and file-descriptor listeners, env-file overrides and disabled proxy headers
|
||||
- The image packages `/usr/local/bin/litellm-gateway`. Local runs may override `LITELLM_RUST_GATEWAY_BINARY` and the private port through `LITELLM_RUST_GATEWAY_PYTHON_PORT`
|
||||
- Roll back by removing the Rust gateway opt-in or setting it to `false` and redeploying
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
# Opt-in Rust gateway
|
||||
|
||||
The Rust gateway is an optional public HTTP listener. The existing Python gateway remains the default. Set
|
||||
`LITELLM_RUST_GATEWAY=true` to start the Rust gateway on the public listener and supervise the existing Python
|
||||
gateway on loopback port 4002. Set the flag to exactly `false`, or leave it unset, to keep the existing startup
|
||||
path. No separate enable flag is required for the Python upstream.
|
||||
|
||||
The Rust binary defaults to `/usr/local/bin/litellm-gateway`; local deployments can set
|
||||
`LITELLM_RUST_GATEWAY_BINARY` to an executable binary. `LITELLM_RUST_GATEWAY_PYTHON_PORT` changes the private
|
||||
Python port. The private listener must remain on loopback and must not be added to a container or Kubernetes
|
||||
Service port.
|
||||
|
||||
When the Rust gateway is enabled with a writer `DATABASE_URL`, the supervisor starts one managed PgBouncer for
|
||||
both gateway children, even when `LITELLM_PGBOUNCER_ENABLED=false`. That flag continues to control optional
|
||||
pooling in Python-only mode. A database-free Rust gateway stays database-free, and Rust only forwards requests
|
||||
without initializing a database client. The managed writer pool uses `LITELLM_PGBOUNCER_MAX_DB_CONNECTIONS`;
|
||||
read-replica connections keep their existing behavior and are outside that writer cap
|
||||
|
||||
```sh
|
||||
docker build -f gateway/Dockerfile -t litellm-gateway:local .
|
||||
docker run --stop-timeout 30 --mount type=bind,src="$(pwd)/config.yaml",dst=/app/config/config.yaml,readonly \
|
||||
-p 4000:4000 \
|
||||
-e LITELLM_RUST_GATEWAY=true \
|
||||
-e LITELLM_RUST_GATEWAY_TRUSTED_PROXIES=10.0.0.0/8 \
|
||||
-e LITELLM_RUST_GATEWAY_TERMINATION_GRACE_SECONDS=30 \
|
||||
-e LITELLM_GATEWAY_METRICS_PORT=4001 \
|
||||
-e CONFIG_FILE_PATH=/app/config/config.yaml \
|
||||
litellm-gateway:local
|
||||
```
|
||||
|
||||
The supervisor waits up to 120 seconds for startup, drains Rust for up to 15 seconds, then gives Python 5 seconds
|
||||
to stop. It also allows a 10-second cleanup margin. `LITELLM_RUST_GATEWAY_STARTUP_TIMEOUT_SECONDS`,
|
||||
`LITELLM_RUST_GATEWAY_DRAIN_TIMEOUT_SECONDS`, and `LITELLM_RUST_GATEWAY_PYTHON_GRACE_SECONDS` configure those
|
||||
timeouts. `LITELLM_RUST_GATEWAY_TERMINATION_GRACE_SECONDS` must be at least the drain timeout plus Python grace
|
||||
plus the cleanup margin. Container orchestrators must grant at least this value before sending SIGKILL. For
|
||||
Docker, set `--stop-timeout` to the same or a larger number; Docker's default stop timeout is shorter
|
||||
|
||||
The trusted-proxy setting is an optional comma-separated list of IP addresses or CIDRs. An unset or empty list
|
||||
trusts no external proxy. Wildcard trust is rejected. The Rust gateway canonicalizes forwarded client and scheme
|
||||
headers using the configured trust list. It preserves the original `Host` and `Origin` headers. Configure only
|
||||
proxy addresses that are controlled by your deployment
|
||||
|
||||
The Rust gateway does not support TLS termination or Uvicorn TLS settings. Terminate TLS at the load balancer or
|
||||
another trusted edge, then forward HTTP to the gateway. Uvicorn reload, Unix-domain and file-descriptor listeners,
|
||||
environment-file overrides, and explicit `--no-proxy-headers` are rejected in enabled mode
|
||||
|
||||
The Helm chart keeps the Service on port 4000. Configure `LITELLM_RUST_GATEWAY` through the gateway container's
|
||||
environment or `envFrom`, and set `gateway.terminationGracePeriodSeconds` to match the supervisor grace setting.
|
||||
When the metrics sidecar is enabled, the chart exports its reserved port to the supervisor even when the feature
|
||||
flag comes from `envFrom`, so port collisions are rejected during enabled startup
|
||||
|
||||
```yaml
|
||||
gateway:
|
||||
envConfigMaps:
|
||||
- gateway-runtime
|
||||
metricsServer:
|
||||
enabled: true
|
||||
port: 4001
|
||||
terminationGracePeriodSeconds: 30
|
||||
```
|
||||
|
||||
The `gateway-runtime` ConfigMap can contain `LITELLM_RUST_GATEWAY: "true"` and
|
||||
`LITELLM_RUST_GATEWAY_TRUSTED_PROXIES: "10.0.0.0/8"`. The chart does not add a private Python port to the pod
|
||||
ports or Service
|
||||
|
||||
To roll back, set `LITELLM_RUST_GATEWAY=false` in the same environment source and redeploy, or roll back the
|
||||
Deployment revision. The feature flag is strict and disabled by default, so removing the opt-in also restores the
|
||||
existing Python startup path
|
||||
Loading…
Add table
Reference in a new issue