scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-10 03:28:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 85

docs(routing.md): redis cache

Browse source 
n
This commit is contained in:
Krrish Dholakia 2023-10-18 14:39:01 -07:00
parent 64eeb0d421
commit 1ba4297d09
1 changed files with 18 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
docs/my-website/docs/routing.md
View file

@ -1,7 +1,12 @@
# API Load-Balancing
# LLM API Load-Balancing
Use this if you're trying to load-balance across multiple deployments.
Use this if you're trying to load-balance across multiple Azure/OpenAI deployments.
`Router` prevents failed requests, by picking the deployment which is below rate-limit and has the least amount of tokens used.
In production, [Router connects to a Redis Cache](#redis-queue) to track usage across multiple deployments.
## Quick Start
```python
pip install litellm
@ -46,5 +51,16 @@ router = Router(model_list=model_list)
response = router.completion(model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hey, how's it going?"}]
print(response)
```
## Redis Queue
```python
router = Router(model_list=model_list,
redis_host=os.getenv("REDIS_HOST"),
redis_password=os.getenv("REDIS_PASSWORD"),
redis_port=os.getenv("REDIS_PORT"))
print(response)
```