mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
docs(deploy.md): update docs for deploying proxy with connected db
This commit is contained in:
Krrish Dholakia
parent
5da88809b7
commit
d6361e155a
1 changed files with 141 additions and 66 deletions
|
|
@ -3,13 +3,10 @@ import TabItem from '@theme/TabItem';
|
|||
|
||||
# 🐳 Docker, Deploying LiteLLM Proxy
|
||||
|
||||
## Dockerfile
|
||||
|
||||
You can find the Dockerfile to build litellm proxy [here](https://github.com/BerriAI/litellm/blob/main/Dockerfile)
|
||||
|
||||
## Quick Start Docker Image: Github Container Registry
|
||||
## Quick Start
|
||||
|
||||
### Pull the litellm ghcr docker image
|
||||
See the latest available ghcr docker image here:
|
||||
https://github.com/berriai/litellm/pkgs/container/litellm
|
||||
|
||||
|
|
@ -17,12 +14,11 @@ https://github.com/berriai/litellm/pkgs/container/litellm
|
|||
docker pull ghcr.io/berriai/litellm:main-latest
|
||||
```
|
||||
|
||||
### Run the Docker Image
|
||||
```shell
|
||||
docker run ghcr.io/berriai/litellm:main-latest
|
||||
```
|
||||
|
||||
#### Run the Docker Image with LiteLLM CLI args
|
||||
### Run with LiteLLM CLI args
|
||||
|
||||
See all supported CLI args [here](https://docs.litellm.ai/docs/proxy/cli):
|
||||
|
||||
|
|
@ -35,8 +31,145 @@ Here's how you can run the docker image and start litellm on port 8002 with `num
|
|||
```shell
|
||||
docker run ghcr.io/berriai/litellm:main-latest --port 8002 --num_workers 8
|
||||
```
|
||||
|
||||
#### Run the Docker Image using docker compose
|
||||
|
||||
## Deploy with Database
|
||||
|
||||
We maintain a [seperate Dockerfile](https://github.com/BerriAI/litellm/pkgs/container/litellm-database) for reducing build time when running LiteLLM proxy with a connected Postgres Database
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="docker-deploy" label="Dockerfile">
|
||||
|
||||
```
|
||||
docker pull docker pull ghcr.io/berriai/litellm-database:main-v1.16.20
|
||||
```
|
||||
|
||||
```
|
||||
docker run --name litellm-proxy \
|
||||
-e DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname> \
|
||||
-p 4000:4000 \
|
||||
ghcr.io/berriai/litellm-database:main-v1.16.20
|
||||
```
|
||||
|
||||
Your OpenAI proxy server is now running on `http://0.0.0.0:4000`.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="kubernetes-deploy" label="Kubernetes">
|
||||
|
||||
### Step 1. Create deployment.yaml
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: litellm-deployment
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: litellm
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: litellm
|
||||
spec:
|
||||
containers:
|
||||
- name: litellm-container
|
||||
image: ghcr.io/berriai/litellm-database:main-v1.16.20
|
||||
env:
|
||||
- name: DATABASE_URL
|
||||
value: postgresql://<user>:<password>@<host>:<port>/<dbname>
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f /path/to/deployment.yaml
|
||||
```
|
||||
|
||||
### Step 2. Create service.yaml
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: litellm-service
|
||||
spec:
|
||||
selector:
|
||||
app: litellm
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 4000
|
||||
targetPort: 4000
|
||||
type: NodePort
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f /path/to/service.yaml
|
||||
```
|
||||
|
||||
### Step 3. Start server
|
||||
|
||||
```
|
||||
kubectl port-forward service/litellm-service 4000:4000
|
||||
```
|
||||
|
||||
Your OpenAI proxy server is now running on `http://0.0.0.0:4000`.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Platform-specific Guide
|
||||
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="google-cloud-run" label="Google Cloud Run">
|
||||
|
||||
### Deploy on Google Cloud Run
|
||||
**Click the button** to deploy to Google Cloud Run
|
||||
|
||||
[](https://deploy.cloud.run/?git_repo=https://github.com/BerriAI/litellm)
|
||||
|
||||
#### Testing your deployed proxy
|
||||
**Assuming the required keys are set as Environment Variables**
|
||||
|
||||
https://litellm-7yjrj3ha2q-uc.a.run.app is our example proxy, substitute it with your deployed cloud run app
|
||||
|
||||
```shell
|
||||
curl https://litellm-7yjrj3ha2q-uc.a.run.app/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [{"role": "user", "content": "Say this is a test!"}],
|
||||
"temperature": 0.7
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="render" label="Render deploy">
|
||||
|
||||
### Deploy on Render https://render.com/
|
||||
|
||||
<iframe width="840" height="500" src="https://www.loom.com/embed/805964b3c8384b41be180a61442389a3" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
|
||||
|
||||
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="railway" label="Railway">
|
||||
|
||||
### Deploy on Railway https://railway.app
|
||||
|
||||
**Step 1: Click the button** to deploy to Railway
|
||||
|
||||
[](https://railway.app/template/S7P9sn?referralCode=t3ukrU)
|
||||
|
||||
**Step 2:** Set `PORT` = 4000 on Railway Environment Variables
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Extras
|
||||
|
||||
### Run with docker compose
|
||||
|
||||
**Step 1**
|
||||
|
||||
|
|
@ -80,64 +213,6 @@ Run the command `docker-compose up` or `docker compose up` as per your docker in
|
|||
Your LiteLLM container should be running now on the defined port e.g. `8000`.
|
||||
|
||||
|
||||
## Deploy with Database
|
||||
|
||||
#### Step 1. Save the database url in your environment
|
||||
.env example: https://github.com/BerriAI/litellm/blob/main/docker/.env.example
|
||||
|
||||
|
||||
```env
|
||||
DATABASE_URL = "my-postgres-db-url"
|
||||
```
|
||||
|
||||
#### Step 2. Build docker image with build-args
|
||||
|
||||
Set `with_database=true` in the docker build, to trigger the prisma logic to be run
|
||||
|
||||
Example build command:
|
||||
```bash
|
||||
docker build -t my-docker-build --build-arg with_database=true .
|
||||
```
|
||||
|
||||
#### Step 3. Run docker image
|
||||
|
||||
```bash
|
||||
docker run -it -p 8000:4000 my-docker-build
|
||||
```
|
||||
|
||||
|
||||
## Deploy on Render https://render.com/
|
||||
|
||||
<iframe width="840" height="500" src="https://www.loom.com/embed/805964b3c8384b41be180a61442389a3" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
|
||||
|
||||
|
||||
## Deploy on Google Cloud Run
|
||||
**Click the button** to deploy to Google Cloud Run
|
||||
|
||||
[](https://deploy.cloud.run/?git_repo=https://github.com/BerriAI/litellm)
|
||||
|
||||
#### Testing your deployed proxy
|
||||
**Assuming the required keys are set as Environment Variables**
|
||||
|
||||
https://litellm-7yjrj3ha2q-uc.a.run.app is our example proxy, substitute it with your deployed cloud run app
|
||||
|
||||
```shell
|
||||
curl https://litellm-7yjrj3ha2q-uc.a.run.app/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [{"role": "user", "content": "Say this is a test!"}],
|
||||
"temperature": 0.7
|
||||
}'
|
||||
```
|
||||
|
||||
## Deploy on Railway https://railway.app
|
||||
|
||||
**Step 1: Click the button** to deploy to Railway
|
||||
|
||||
[](https://railway.app/template/S7P9sn?referralCode=t3ukrU)
|
||||
|
||||
**Step 2:** Set `PORT` = 4000 on Railway Environment Variables
|
||||
|
||||
## LiteLLM Proxy Performance
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue