After testing the AWS ECS deployment, discovered that the CloudFormation
template does not configure the fake-openai-endpoint model required for
running benchmark tests as documented in https://docs.litellm.ai/docs/benchmarks
## Changes
- Added KNOWN_ISSUES.md documenting the limitation in detail
- Updated README.md with prominent warning about the gap
- Updated 00-START-HERE.md with limitation notice
- Added simple-loadtest.py for infrastructure-only testing
## What Was Tested
✅ Successfully deployed:
- 4 ECS Fargate tasks (4 vCPU, 8 GB RAM each)
- RDS PostgreSQL database
- Application Load Balancer
- Full VPC with security groups
✅ Verified working:
- All tasks running and healthy
- Database connections
- Health endpoints responding
- Load balancer routing
❌ Cannot test (missing model config):
- API /v1/chat/completions requests
- Locust benchmark with 1000 users
- LiteLLM overhead measurement
- Performance metrics (latency, RPS)
## Root Cause
The CloudFormation template only sets environment variables but does not:
- Mount a config.yaml file
- Configure model_list with fake-openai-endpoint
- Set up the test endpoint needed for benchmarking
## Impact
Users can deploy the infrastructure matching benchmark specs, but cannot
run the actual benchmark without manually configuring models via API or
updating the template to mount a config file.
## Stack Cleanup
The test deployment was deleted to avoid ongoing costs (~$440/month).
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
