scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-11 03:38:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 84

docs

Browse source
This commit is contained in:
ishaan-jaff 2023-10-04 16:14:09 -07:00
parent e487c33adc
commit 5c55011888
1 changed files with 7 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

11
docs/my-website/docs/rate_limit_manager.md
View file

@ -2,21 +2,23 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Rate Limit Manager
`RateLimitManager` allows you to maximize throughput while staying under rate limits
`RateLimitManager` allows you to maximize throughput while staying under rate limits. You can use RateLimitManager to submit a batch of completion jobs to execute
## Quick start - using
## Quick start
```python
import asyncio
from litellm import RateLimitManager batch_completion
# init RateLimitManager with your rate limits
handler = RateLimitManager(
max_requests_per_minute = 60,
max_tokens_per_minute = 20000
)
##### USAGE ################
# make batch completion requests
# define jobs, list of kwargs to go to the completion() call
jobs = [
{"model": "gpt-3.5-turbo-16k", "messages": [{"content": "Please provide a summary of the latest scientific discoveries.", "role": "user"}]},
{"model": "gpt-3.5-turbo-16k", "messages": [{"content": "Please provide a summary of the latest scientific discoveries.", "role": "user"}]},
@ -25,7 +27,8 @@ jobs = [
{"model": "gpt-3.5-turbo-16k", "messages": [{"content": "Please provide a summary of the latest scientific discoveries.", "role": "user"}]}
]
# use RateLimitManager.batch_completion to execute several jobs in parallel
# output stored in litellm_results.jsonl
try:
asyncio.run(
handler.batch_completion(