litellm/docker-compose/README.md

LiteLLM Hardened Production Stack Documentation

This document outlines the production-grade deployment guidelines for the LiteLLM Proxy infrastructure. The system is engineered as a single, unified architecture offering two configurable egress topologies: Direct Egress and Proxy-Filtered Egress (Squid). The underlying design emphasizes container isolation, OWASP-compliant secrets management, and strict network segmentation through a Zero-Trust internal model.

---

Network Architecture and Traffic Flows

In both execution modes, the architecture enforces strict isolation for sensitive infrastructure components. The PostgreSQL database is placed within a dedicated internal network lacking a default gateway (backend-internal), permanently preventing outbound connection attempts to the external internet.

Topology Option 1: Direct Egress

This configuration optimizes for minimal latency. LiteLLM establishes outbound connections directly to external AI provider endpoints via the host's bridge interface.

                  [ Host Firewall / UFW ]
                            │
                     (127.0.0.1:4000)
                            │
┌───────────────────────────┼───────────────────────────────────────────┐
│ DOCKER HOST               ▼                                           │
│                                                                       │
│  [ public-egress Network ]                                            │
│  ┌─────────────────────────────────┐                                  │
│  │         litellm Proxy           │                                  │
│  │    (Reads Secrets via tmpfs)    │                                  │
│  └────────────────┬────────────────┘                                  │
│                   │                                                   │
│  [ backend-internal Network (NO GATEWAY) ]                            │
│  ┌────────────────┴────────────────┐                                  │
│  │           litellm_db            │                                  │
│  │   (PostgreSQL - Isolated)       │                                  │
│  └─────────────────────────────────┘                                  │
└───────────────────┬───────────────────────────────────────────────────┘
                    │ Direct egress via host bridge
                    ▼
          [ Internet / Cloud APIs ]
          ├── api.openai.com
          ├── api.anthropic.com
          └── bedrock.us-east-1.amazonaws.com

Topology Option 2: Proxy-Filtered Egress (Squid)

This configuration introduces outbound traffic control to mitigate data exfiltration risks. All egress requests initiated by LiteLLM are forcibly directed through an internal Squid proxy acting as an egress gateway.

                  [ Host Firewall / UFW ]
                            │
                     (127.0.0.1:4000)
                            │
┌───────────────────────────┼───────────────────────────────────────────┐
│ DOCKER HOST               ▼                                           │
│                                                                       │
│  [ public-egress Network ]                                            │
│  ┌─────────────────────────────────┐                                  │
│  │         litellm Proxy           │                                  │
│  │  HTTP_PROXY=http://squid:3128   ├────────┐                         │
│  └────────────────┬────────────────┘        │ HTTP/HTTPS via Proxy    │
│                   │                         ▼                         │
│                   │              ┌────────────────────┐               │
│                   │              │    litellm_squid   │               │
│                   │              │   (Egress Proxy)   │               │
│                   │              └──────────┬─────────┘               │
│  [ backend-internal Network ]               │                         │
│  ┌────────────────┴────────────────┐        │                         │
│  │           litellm_db            │        │ Evaluates Egress Rules  │
│  │   (PostgreSQL - Isolated)       │        │                         │
│  └─────────────────────────────────┘        │                         │
└─────────────────────────────────────────────┼─────────────────────────┘
                                              │
                                              ▼
                                  [ Internet / Cloud APIs ]

---

System Prerequisites

• Operating System: Linux (Ubuntu LTS or Debian recommended).
• Access Privileges: Administrative rights (root or sudo access) required for systemd unit installation, filesystem permission management, and firewall rule modification.
• Software Dependencies:
  • git (for repository cloning and sparse-checkout)
  • docker (Engine 20.10+)
  • docker compose (V2 plugin)
  • openssl
  • envsubst (provided by gettext-base)

---

Deployment Procedures

Step 1: Prepare Destination Directory in /opt/

Create the production installation path under /opt/litellm and navigate into it:

sudo mkdir -p /opt/litellm
cd /opt/litellm

Step 2: Download Deployment Files via Git Sparse-Checkout

Fetch strictly the target deployment directory without cloning the entire repository history:

sudo git clone --no-checkout --depth 1 git@github.com:BerriAI/litellm.git src
cd src
sudo git sparse-checkout init --cone
sudo git sparse-checkout set docker-compose
sudo git checkout main
cd docker-compose

Step 3: Validate Provisioned Deployment Files

Verify that the sparse checkout successfully populated the working directory with the required artifacts:

• deploy.sh
• compose.yaml.template

Step 4: Set Execution Permissions

Grant execution rights to the deployment script:

sudo chmod +x deploy.sh

Step 5: Execute Deployment Script

Select the desired topology option using the runtime flags. Append --install-systemd if system boot persistence is required:

Option 1: Direct Egress Deployment

# Standard container execution
sudo ./deploy.sh --force-open-firewall

# With Systemd boot persistence
sudo ./deploy.sh --force-open-firewall --install-systemd

Option 2: Proxy-Filtered Egress Deployment

# Standard container execution
sudo ./deploy.sh --force-open-firewall --enable-egress-proxy

# With Systemd boot persistence
sudo ./deploy.sh --force-open-firewall --enable-egress-proxy --install-systemd

Automated Execution Lifecycle

Upon invocation, the deployment script executes the following operations:

1. Validates host dependencies and configures system firewall rules (UFW).
2. Generates cryptographically secure secrets in ./secrets/ with restricted 600 permissions mapped to UID/GID 101:101.
3. Renders the final compose.yaml configuration using compose.yaml.template.
4. Registers and enables the litellm.service systemd unit only if --install-systemd is passed.
5. Initializes the container stack via docker compose up -d (or systemctl restart litellm.service when systemd is enabled).

---

Post-Installation Operations

Environment Tuning & Operational Flags

To maintain a clean Admin UI and prevent false-positive operational warnings in single-worker deployments, the stack sets specific environment flags:

• LITELLM_DISABLE_NO_REDIS_WARNING=true: Suppresses the persistent Redis configuration warning banner in the Admin UI. Since this deployment runs as a single, isolated container instance, an in-memory/PostgreSQL setup is sufficient for rate limiting and state persistence without introducing additional Redis infrastructure overhead.

  • Official Reference: LiteLLM Redis Requirements & Single-Worker Configuration

• Secrets Ingestion Wrapper (entrypoint): To comply with Docker Secrets security standards while working around LiteLLM's lack of native *_FILE variable parsing, secrets are injected into environment variables at container startup using an inline shell wrapper:

  export DATABASE_URL=$(cat /run/secrets/db_url)
  export LITELLM_MASTER_KEY=$(cat /run/secrets/master_key)
  export LITELLM_SALT_KEY=$(cat /run/secrets/salt_key)
  exec docker/prod_entrypoint.sh
  

### Service Endpoints

- **Service Base URL:** `http://127.0.0.1:4000`
- **Health Verification Endpoint:** `http://127.0.0.1:4000/health`

### Secrets and Credential Retrieval

Generated credentials are stored in the `./secrets/` directory on the host and are readable exclusively by the `root` user.

#### Retrieve Master API Key (Admin / UI Access)

```bash
sudo cat ./secrets/master_key.txt

Retrieve Application Salt Key

sudo cat ./secrets/salt_key.txt

Retrieve Database Connection String

sudo cat ./secrets/db_url.txt

Retrieve PostgreSQL Database Password

sudo cat ./secrets/db_password.txt

---

Teardown and Cleanup Procedures

To completely remove the container stack, associated systemd units, networks, persistent data volumes, and installation files:

Step 1: Stop and Remove Systemd Service (If Installed)

sudo systemctl stop litellm.service 2>/dev/null || true
sudo systemctl disable litellm.service 2>/dev/null || true
sudo rm -f /etc/systemd/system/litellm.service
sudo systemctl daemon-reload

Step 2: Purge Docker Resources and Volumes

Run from /opt/litellm/src/docker-compose:

sudo docker compose down -v --remove-orphans

Step 3: Remove Production Installation Directory

sudo rm -rf /opt/litellm