Resolves the model configuration gap identified in previous commit.
The deployment now supports running the full benchmark test as documented
in https://docs.litellm.ai/docs/benchmarks
## Changes
### CloudFormation Template (cloudformation-ecs.yaml)
1. **Added SSM Parameter for Config**
- New resource: LiteLLMConfigParameter
- Stores LiteLLM configuration YAML in SSM Parameter Store
- Includes fake-openai-endpoint model configuration
- Path: /${StackName}/litellm-config
2. **Updated IAM Permissions**
- TaskRole now includes SSMConfigAccess policy
- Allows tasks to read SSM parameters
- Scoped to specific config parameter
3. **Modified Container Startup**
- Added CONFIG_SSM_PARAMETER environment variable
- Added PROXY_MASTER_KEY environment variable
- New entrypoint script fetches config from SSM using boto3
- Writes config to /tmp/config.yaml
- Starts LiteLLM with --config flag
### Configuration Included
```yaml
model_list:
- model_name: fake-openai-endpoint
litellm_params:
model: openai/fake
api_key: fake-key
api_base: https://exampleopenaiendpoint-production.up.railway.app/
general_settings:
master_key: os.environ/PROXY_MASTER_KEY
database_url: os.environ/DATABASE_URL
store_model_in_db: true
```
### Documentation Updates
- KNOWN_ISSUES.md: Marked limitation as RESOLVED
- README.md: Updated to reflect full benchmark support
- 00-START-HERE.md: Updated to show ready for benchmark testing
## How It Works
1. CloudFormation creates SSM Parameter with config YAML
2. ECS task starts with entrypoint script
3. Script uses boto3 to fetch config from SSM
4. Config written to /tmp/config.yaml
5. LiteLLM starts with: --config /tmp/config.yaml
6. fake-openai-endpoint model now available for API calls
## Testing
Template validated with: aws cloudformation validate-template
✅ Syntax valid
✅ Parameters correct
✅ IAM permissions scoped properly
## Impact
Users can now:
- ✅ Deploy and immediately run benchmark tests
- ✅ Use Locust with 1,000 users as documented
- ✅ Measure API latency (P50, P95, P99)
- ✅ Measure LiteLLM overhead via x-litellm-overhead-duration-ms header
- ✅ Compare results with official benchmark guide
Expected results:
- Median latency: ~100 ms
- P95 latency: ~150 ms
- Throughput: ~1,170 RPS
- LiteLLM overhead: ~2 ms
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
After testing the AWS ECS deployment, discovered that the CloudFormation
template does not configure the fake-openai-endpoint model required for
running benchmark tests as documented in https://docs.litellm.ai/docs/benchmarks
## Changes
- Added KNOWN_ISSUES.md documenting the limitation in detail
- Updated README.md with prominent warning about the gap
- Updated 00-START-HERE.md with limitation notice
- Added simple-loadtest.py for infrastructure-only testing
## What Was Tested
✅ Successfully deployed:
- 4 ECS Fargate tasks (4 vCPU, 8 GB RAM each)
- RDS PostgreSQL database
- Application Load Balancer
- Full VPC with security groups
✅ Verified working:
- All tasks running and healthy
- Database connections
- Health endpoints responding
- Load balancer routing
❌ Cannot test (missing model config):
- API /v1/chat/completions requests
- Locust benchmark with 1000 users
- LiteLLM overhead measurement
- Performance metrics (latency, RPS)
## Root Cause
The CloudFormation template only sets environment variables but does not:
- Mount a config.yaml file
- Configure model_list with fake-openai-endpoint
- Set up the test endpoint needed for benchmarking
## Impact
Users can deploy the infrastructure matching benchmark specs, but cannot
run the actual benchmark without manually configuring models via API or
updating the template to mount a config file.
## Stack Cleanup
The test deployment was deleted to avoid ongoing costs (~$440/month).
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The Helm chart on GHCR displays a `docker pull` command instead of
the correct `helm pull oci://` command. This is because the OCI artifact
is missing the `org.opencontainers.image.source` annotation that GHCR
uses to identify and properly display Helm charts.
Changes:
- Add OCI annotations to Chart.yaml (source + url) which Helm 3.10+
propagates to the OCI manifest on push
- Install explicit Helm v3.20.0 via azure/setup-helm@v4 for reproducible
builds and proper OCI annotation support
- Remove deprecated HELM_EXPERIMENTAL_OCI env var (OCI is GA since Helm 3.8)
* feat: add support for keda in helm chart
Signed-off-by: R.Sicart <roger.sicart@gmail.com>
* chore: bump chart version
---------
Signed-off-by: R.Sicart <roger.sicart@gmail.com>
* fix: sync Helm chart versioning with production standards and Docker versions
- Update Chart.yaml version from 0.4.10 to 1.0.0 (SemVer 0.x is for development, 1.0+ for production)
- Update appVersion from v1.50.2 to v1.80.12 to match current Docker image version
- Update workflow defaults from 0.1.0 to 1.0.0 for new chart version scheme
- Maintain independent chart versioning per Helm best practices
This ensures:
- Helm chart follows SemVer production standards (1.x instead of 0.x)
- appVersion stays synchronized with Docker/application version
- Chart version remains independent for flexibility (can update chart without waiting for app releases)
* fix: sync Helm chart appVersion with Docker image tags in release workflow
Updates the GitHub workflow to ensure Helm chart appVersion matches the
Docker image tags that are actually published:
- For stable/rc releases: Uses the workflow input tag (e.g., v1.80.12)
- For latest/dev releases: Uses the release_type to match main-{type} tags
- Makes 'tag' input required to prevent accidental releases with wrong versions
- Simplifies fallback logic by removing git-describe dependency
This ensures the chart's appVersion correctly references Docker images
that exist, preventing deployment failures from missing image tags.
* Update ghcr_deploy.yml
- Add support for envVars (simple key-value pairs) in migrations job
- Add support for extraEnvVars (complex environment variable configurations)
- Include comprehensive test coverage for both envVars and extraEnvVars
- Ensure backward compatibility with existing configurations
- Tests verify proper rendering of environment variables in container spec
* feat(helm): Add loadBalancerClass support for LoadBalancer services
Adds the ability to specify a loadBalancerClass when using LoadBalancer service type.
This enables integration with custom load balancer implementations like Tailscale.
* fixup! feat(helm): Add loadBalancerClass support for LoadBalancer services
* build(README.md): initial commit adding a separate folder for additional proxy files. Meant to reduce size of core package
* build(litellm-proxy-extras/): new pip package for storing migration files
allows litellm proxy to use migration files, without adding them to core repo
* build(litellm-proxy-extras/): cleanup pyproject.toml
* build: move prisma migration files inside new proxy extras package
* build(run_migration.py): update script to write to correct folder
* build(proxy_cli.py): load in migration files from litellm-proxy-extras
Closes https://github.com/BerriAI/litellm/issues/9558
* build: add MIT license to litellm-proxy-extras
* test: update test
* fix: fix schema
* bump: version 0.1.0 → 0.1.1
* build(publish-proxy-extras.sh): add script for publishing new proxy-extras version
* build(liccheck.ini): add litellm-proxy-extras to authorized packages
* fix(litellm-proxy-extras/utils.py): move prisma migrate logic inside extra proxy pkg
easier since migrations folder already there
* build(pre-commit-config.yaml): add litellm_proxy_extras to ci tests
* docs(config_settings.md): document new env var
* build(pyproject.toml): bump relevant files when litellm-proxy-extras version changed
* build(pre-commit-config.yaml): run poetry check on litellm-proxy-extras as well
* feat(spend_management_endpoints.py): expose new endpoint for querying user's usage at 1m+ spend logs
Allows user to view their spend at 1m+ spend logs
* build(schema.prisma): add api_requests to dailyuserspend table
* build(migration.sql): add migration file for new column to daily user spend table
* build(prisma_client.py): add logic for copying over migration folder, if deploy/migrations present in expected location
enables easier testing of prisma migration flow
* build(ui/): initial commit successfully using the dailyuserspend table on the UI
* refactor(internal_user_endpoints.py): refactor `/user/daily/activity` to give breakdowns by provider/model/key
* feat: feature parity (cost page) with existing 'usage' page
* build(ui/): add activity tab to new_usage.tsx
gets to feature parity on 'All Up' page of 'usage.tsx'
* fix(proxy/utils.py): count number of api requests in daily user spend table
allows us to see activity by model on new usage tab
* style(new_usage.tsx): fix y-axis to be in ascending order of date
* fix: fix linting errors
* fix: fix ruff check errors
