diff --git a/docs/my-website/docs/proxy/deploy.md b/docs/my-website/docs/proxy/deploy.md index 8ee1c00558f..b79de2f3072 100644 --- a/docs/my-website/docs/proxy/deploy.md +++ b/docs/my-website/docs/proxy/deploy.md @@ -3,13 +3,10 @@ import TabItem from '@theme/TabItem'; # 🐳 Docker, Deploying LiteLLM Proxy -## Dockerfile - You can find the Dockerfile to build litellm proxy [here](https://github.com/BerriAI/litellm/blob/main/Dockerfile) -## Quick Start Docker Image: Github Container Registry +## Quick Start -### Pull the litellm ghcr docker image See the latest available ghcr docker image here: https://github.com/berriai/litellm/pkgs/container/litellm @@ -17,12 +14,11 @@ https://github.com/berriai/litellm/pkgs/container/litellm docker pull ghcr.io/berriai/litellm:main-latest ``` -### Run the Docker Image ```shell docker run ghcr.io/berriai/litellm:main-latest ``` -#### Run the Docker Image with LiteLLM CLI args +### Run with LiteLLM CLI args See all supported CLI args [here](https://docs.litellm.ai/docs/proxy/cli): @@ -35,8 +31,145 @@ Here's how you can run the docker image and start litellm on port 8002 with `num ```shell docker run ghcr.io/berriai/litellm:main-latest --port 8002 --num_workers 8 ``` - -#### Run the Docker Image using docker compose + +## Deploy with Database + +We maintain a [seperate Dockerfile](https://github.com/BerriAI/litellm/pkgs/container/litellm-database) for reducing build time when running LiteLLM proxy with a connected Postgres Database + + + + +``` +docker pull docker pull ghcr.io/berriai/litellm-database:main-v1.16.20 +``` + +``` +docker run --name litellm-proxy \ +-e DATABASE_URL=postgresql://:@:/ \ +-p 4000:4000 \ +ghcr.io/berriai/litellm-database:main-v1.16.20 +``` + +Your OpenAI proxy server is now running on `http://0.0.0.0:4000`. + + + + +### Step 1. Create deployment.yaml + +```yaml + apiVersion: apps/v1 + kind: Deployment + metadata: + name: litellm-deployment + spec: + replicas: 1 + selector: + matchLabels: + app: litellm + template: + metadata: + labels: + app: litellm + spec: + containers: + - name: litellm-container + image: ghcr.io/berriai/litellm-database:main-v1.16.20 + env: + - name: DATABASE_URL + value: postgresql://:@:/ +``` + +```bash +kubectl apply -f /path/to/deployment.yaml +``` + +### Step 2. Create service.yaml + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: litellm-service +spec: + selector: + app: litellm + ports: + - protocol: TCP + port: 4000 + targetPort: 4000 + type: NodePort +``` + +```bash +kubectl apply -f /path/to/service.yaml +``` + +### Step 3. Start server + +``` +kubectl port-forward service/litellm-service 4000:4000 +``` + +Your OpenAI proxy server is now running on `http://0.0.0.0:4000`. + + + + +## Platform-specific Guide + + + + + +### Deploy on Google Cloud Run +**Click the button** to deploy to Google Cloud Run + +[![Deploy](https://deploy.cloud.run/button.svg)](https://deploy.cloud.run/?git_repo=https://github.com/BerriAI/litellm) + +#### Testing your deployed proxy +**Assuming the required keys are set as Environment Variables** + +https://litellm-7yjrj3ha2q-uc.a.run.app is our example proxy, substitute it with your deployed cloud run app + +```shell +curl https://litellm-7yjrj3ha2q-uc.a.run.app/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-3.5-turbo", + "messages": [{"role": "user", "content": "Say this is a test!"}], + "temperature": 0.7 + }' +``` + + + + + +### Deploy on Render https://render.com/ + + + + + + + + +### Deploy on Railway https://railway.app + +**Step 1: Click the button** to deploy to Railway + +[![Deploy on Railway](https://railway.app/button.svg)](https://railway.app/template/S7P9sn?referralCode=t3ukrU) + +**Step 2:** Set `PORT` = 4000 on Railway Environment Variables + + + + + +## Extras + +### Run with docker compose **Step 1** @@ -80,64 +213,6 @@ Run the command `docker-compose up` or `docker compose up` as per your docker in Your LiteLLM container should be running now on the defined port e.g. `8000`. -## Deploy with Database - -#### Step 1. Save the database url in your environment -.env example: https://github.com/BerriAI/litellm/blob/main/docker/.env.example - - -```env -DATABASE_URL = "my-postgres-db-url" -``` - -#### Step 2. Build docker image with build-args - -Set `with_database=true` in the docker build, to trigger the prisma logic to be run - -Example build command: -```bash -docker build -t my-docker-build --build-arg with_database=true . -``` - -#### Step 3. Run docker image - -```bash -docker run -it -p 8000:4000 my-docker-build -``` - - -## Deploy on Render https://render.com/ - - - - -## Deploy on Google Cloud Run -**Click the button** to deploy to Google Cloud Run - -[![Deploy](https://deploy.cloud.run/button.svg)](https://deploy.cloud.run/?git_repo=https://github.com/BerriAI/litellm) - -#### Testing your deployed proxy -**Assuming the required keys are set as Environment Variables** - -https://litellm-7yjrj3ha2q-uc.a.run.app is our example proxy, substitute it with your deployed cloud run app - -```shell -curl https://litellm-7yjrj3ha2q-uc.a.run.app/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "gpt-3.5-turbo", - "messages": [{"role": "user", "content": "Say this is a test!"}], - "temperature": 0.7 - }' -``` - -## Deploy on Railway https://railway.app - -**Step 1: Click the button** to deploy to Railway - -[![Deploy on Railway](https://railway.app/button.svg)](https://railway.app/template/S7P9sn?referralCode=t3ukrU) - -**Step 2:** Set `PORT` = 4000 on Railway Environment Variables ## LiteLLM Proxy Performance