scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-10 03:28:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 85

docs fix

Browse source
This commit is contained in:
Ishaan Jaffer 2025-09-20 17:30:03 -07:00
parent fce28a4c84
commit 41ce31bd1b
1 changed files with 8 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
docs/my-website/docs/proxy/dynamic_rate_limit.md
View file

@ -1,10 +1,12 @@
# Dynamic TPM/RPM Allocation
Prevent projects from gobbling too much tpm/rpm. You should use this feature when you want to reserve tpm/rpm capacity for specific projects. For example, a realtime use case should get higher priority than a different use case.
Prevent projects from gobbling too much tpm/rpm.
Dynamically allocate TPM/RPM quota to api keys, based on active keys in that minute. [**See Code**](https://github.com/BerriAI/litellm/blob/9bffa9a48e610cc6886fc2dce5c1815aeae2ad46/litellm/proxy/hooks/dynamic_rate_limiter.py#L125)
## Quick Start Usage
1. Setup config.yaml
```yaml showLineNumbers title="config.yaml"
@ -97,15 +99,17 @@ This was rate limited b/c - Error code: 429 - {'error': {'message': {'error': 'K
```
#### ✨ [BETA] Set Priority / Reserve Quota
## [BETA] Set Priority / Reserve Quota
Reserve tpm/rpm capacity for projects in prod. You should use this feature when you want to reserve tpm/rpm capacity for specific projects. For example, a realtime use case should get higher priority than a different use case.
Reserve tpm/rpm capacity for projects in prod.
:::tip
Reserving tpm/rpm on keys based on priority is a premium feature. Please [get an enterprise license](./enterprise.md) for it.
:::
### Usage
1. Setup config.yaml
@ -118,7 +122,7 @@ model_list:
rpm: 100
litellm_settings:
callbacks: [""]
callbacks: ["dynamic_rate_limiter_v3"]
priority_reservation: {"dev": 0, "prod": 1}
general_settings: