mirror of https://github.com/BerriAI/litellm.git synced 2026-09-09 22:31:41 +00:00

Julio Quinteros Pro 445c67cfec Add AWS ECS deployment template matching benchmark specifications

This commit adds a complete 1-click deployment solution for LiteLLM on AWS ECS,
configured to match the benchmark specifications from https://docs.litellm.ai/docs/benchmarks

## What's Added

### Infrastructure (1 file)
- cloudformation-ecs.yaml: AWS CloudFormation template for ECS deployment
  - 4 ECS Fargate tasks (4 vCPU, 8 GB RAM each)
  - 4 workers per task (16 total workers)
  - RDS PostgreSQL database (db.t3.medium)
  - Application Load Balancer
  - VPC with public/private subnets across 2 AZs
  - Security groups, NAT Gateway, monitoring

### Deployment Tools (3 files)
- deploy.sh: Automated deployment script with interactive prompts
- test-deployment.sh: Deployment validation and health check script
- cost-calculator.sh: Interactive cost estimation tool

### Documentation (6 files)
- 00-START-HERE.md: Quick start guide and overview
- QUICKSTART.md: 5-minute deployment guide
- README.md: Complete deployment documentation
- ARCHITECTURE.md: Detailed architecture deep-dive with diagrams
- INDEX.md: Master index of all files
- .summary.md: Internal summary document

### Testing & Configuration (2 files)
- locustfile.py: Load testing script to replicate benchmark tests
- example-config.yaml: LiteLLM configuration example

## Configuration

- 4 instances with 4 vCPU and 8 GB RAM each
- 4 workers per instance
- Expected performance:
  - Median latency: ~100 ms
  - P95 latency: ~150 ms
  - Throughput: ~1,170 RPS
  - LiteLLM overhead: ~2 ms

## Usage

```bash
cd deploy/aws
./deploy.sh
```

## Monthly Cost

~$440-460 (pay-as-you-go) or ~$270-370 (with reserved capacity)

## Features

- ✅ CloudFormation template validated with AWS
- ✅ Production-ready with high availability
- ✅ Secure by default (private subnets, security groups, encrypted secrets)
- ✅ Well-documented with comprehensive guides
- ✅ Includes validation and load testing tools
- ✅ Cost-optimized configuration

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

2026-02-16 13:03:05 -03:00

5 KiB

Raw Permalink Blame History

Quick Start Guide - AWS Deployment

Deploy LiteLLM on AWS in under 5 minutes with the benchmark configuration.

Prerequisites

AWS account with CLI configured
Bash shell (Linux, macOS, or WSL on Windows)

1-Click Deployment

Run the deployment script:

cd deploy/aws
./deploy.sh

The script will:

Prompt you for a database password and master key
Create all necessary AWS resources (VPC, ECS, RDS, ALB)
Deploy 4 LiteLLM instances with 4 workers each
Wait for deployment to complete (~10-15 minutes)
Display your API endpoint and credentials

What Gets Deployed

┌─────────────────────────────────────────┐
│         Application Load Balancer        │
│              (Public)                    │
└────────────┬────────────────────────────┘
             │
    ┌────────┴────────┐
    │                 │
┌───▼───┐         ┌───▼───┐
│ ECS   │         │ ECS   │
│ Task  │  ...    │ Task  │
│ (4    │         │ (4    │
│ vCPU) │         │ vCPU) │
│ 4     │         │ 4     │
│ workers)        │ workers)
└───┬───┘         └───┬───┘
    │                 │
    └────────┬────────┘
             │
    ┌────────▼────────┐
    │  RDS PostgreSQL │
    │   (db.t3.medium)│
    │      100 GB      │
    └─────────────────┘

Configuration:

4 ECS Fargate tasks (4 vCPU, 8 GB RAM each)
4 workers per task = 16 total workers
PostgreSQL database (db.t3.medium)
Application Load Balancer
Private VPC with NAT Gateway

Using Your Deployment

Make Your First API Call

# Set your credentials (from deployment output)
export LITELLM_URL="http://your-alb-url"
export LITELLM_KEY="your-master-key"

# Test the API
curl -X POST "$LITELLM_URL/v1/chat/completions" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "fake-openai-endpoint",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Add Real LLM Providers

Update your configuration to use real providers like OpenAI, Anthropic, etc:

# Get your ECS cluster and service names
CLUSTER=$(aws cloudformation describe-stacks \
  --stack-name litellm-benchmark \
  --query 'Stacks[0].Outputs[?OutputKey==`ECSClusterName`].OutputValue' \
  --output text)

SERVICE=$(aws cloudformation describe-stacks \
  --stack-name litellm-benchmark \
  --query 'Stacks[0].Outputs[?OutputKey==`ECSServiceName`].OutputValue' \
  --output text)

# Update task definition environment variables
# (See README.md for detailed instructions)

Benchmark Your Deployment

Install Locust and run the benchmark test:

# Install Locust
pip install locust

# Run benchmark (1000 users, 500 spawn rate, 5 minutes)
export LITELLM_MASTER_KEY="your-master-key"
locust -f locustfile.py \
  --host=$LITELLM_URL \
  --users=1000 \
  --spawn-rate=500 \
  --run-time=5m \
  --headless

Expected Results:

Median latency: ~100 ms
P95 latency: ~150 ms
Throughput: ~1,170 RPS

Monitoring

View real-time logs:

aws logs tail /ecs/litellm-benchmark-litellm --follow

Monitor key metrics in CloudWatch:

ECS CPU/Memory utilization
ALB request count and latency
RDS connections and CPU

Cleanup

Delete all resources when done:

aws cloudformation delete-stack --stack-name litellm-benchmark

This will remove all AWS resources and stop charges.

Cost

Estimated monthly cost: ~$440-460

Breakdown:

ECS Fargate: ~$350
RDS PostgreSQL: ~$60
Application Load Balancer: ~$20
Data Transfer & NAT Gateway: ~$10-30

Customization

Scale to 8 instances

DESIRED_TASKS=8 ./deploy.sh

Use different instance sizes

TASK_CPU=8192 TASK_MEMORY=16384 ./deploy.sh

Deploy to a different region

AWS_REGION=us-west-2 ./deploy.sh

Troubleshooting

Tasks not starting

Check ECS service events:

aws ecs describe-services \
  --cluster litellm-benchmark-LiteLLM-Cluster \
  --services litellm-benchmark-litellm-service

Health checks failing

The tasks may take 2-3 minutes to become healthy after deployment. Check logs:

aws logs tail /ecs/litellm-benchmark-litellm --follow

High latency

Check if all tasks are running
Verify you have the right number of workers
Consider scaling up task count or instance size

Next Steps

Full README - Complete documentation
Benchmark Guide - Performance details
LiteLLM Docs - Configuration and features

Support

GitHub Issues: https://github.com/BerriAI/litellm/issues
Documentation: https://docs.litellm.ai

5 KiB Raw Permalink Blame History