mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
da2fe9c462
After thorough review of https://docs.litellm.ai/docs/benchmarks, fixed several discrepancies to achieve full benchmark compliance. ## Critical Fixes ### 1. Database Upgraded (Most Important) - **Before:** db.t3.medium (2 vCPU, 4 GB RAM, 100 GB) - **After:** db.r6g.xlarge (4 vCPU, 32 GB RAM, 200 GB) - **Guide requires:** 4-8 cores, 16GB RAM, 200GB SSD for 1-2K RPS - **Impact:** +$162/month, but necessary for benchmark performance ### 2. Added proxy_batch_write_at Setting - **Before:** Not configured - **After:** `proxy_batch_write_at: 60` - **Purpose:** Batch writes every 60 seconds to reduce DB load - **Guide specifies:** Required for 1-2K RPS workloads ### 3. Fixed Model Parameter - **Before:** `model: openai/fake` - **After:** `model: openai/any` - **Guide specifies:** Must use `openai/any` ### 4. Fixed Locust Wait Time - **Before:** `between(0.1, 0.5)` seconds - **After:** `between(0.5, 1)` seconds - **Guide specifies:** 0.5-1 second wait between requests - **Impact:** More realistic load generation matching benchmark ### 5. Storage Configuration - **Before:** 100 GB - **After:** 200 GB gp3 with 3000 IOPS - **Guide requires:** 200 GB SSD ## Additional Changes - Made DBInstanceClass configurable via parameter - Added BENCHMARK_COMPLIANCE.md with detailed verification - Updated cost estimates in documentation - Added parameter for choosing db instance size ## Compliance Status ✅ **FULLY COMPLIANT** with official benchmark guide All specifications now match: - Hardware: 4 instances × 4 vCPU × 8 GB RAM ✅ - Workers: 4 per instance (16 total) ✅ - Database: 4 vCPU, 32 GB RAM, 200 GB ✅ - Config: proxy_batch_write_at=60 ✅ - Model: openai/any at fake endpoint ✅ - Load test: 1000 users, 0.5-1s wait ✅ ## Cost Impact Monthly cost increased from ~$440-460 to ~$600-620 due to: - Database upgrade: +$150/month - Additional storage: +$12/month Users can override DBInstanceClass parameter for cost savings in non-benchmark scenarios. ## Expected Performance With these fixes, deployment should achieve benchmark targets: - Median latency: ~100 ms - P95 latency: ~150 ms - P99 latency: ~240 ms - Throughput: ~1,170 RPS - LiteLLM overhead: ~2 ms ## Files Changed - cloudformation-ecs.yaml: DB upgrade, config fixes, new parameter - locustfile.py: Fixed wait_time to 0.5-1 seconds - BENCHMARK_COMPLIANCE.md: New comprehensive compliance check - cost-calculator.sh: Updated for new DB pricing (future) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| aws | ||
| azure_resource_manager | ||
| charts/litellm-helm | ||
| kubernetes | ||
| Dockerfile.ghcr_base | ||