From 1ba4297d09f9bb9e32d80d111c7090f26f9b20c7 Mon Sep 17 00:00:00 2001 From: Krrish Dholakia Date: Wed, 18 Oct 2023 14:39:01 -0700 Subject: [PATCH] docs(routing.md): redis cache n --- docs/my-website/docs/routing.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/docs/my-website/docs/routing.md b/docs/my-website/docs/routing.md index 8105df4c323..ded25905bcd 100644 --- a/docs/my-website/docs/routing.md +++ b/docs/my-website/docs/routing.md @@ -1,7 +1,12 @@ -# API Load-Balancing +# LLM API Load-Balancing -Use this if you're trying to load-balance across multiple deployments. +Use this if you're trying to load-balance across multiple Azure/OpenAI deployments. +`Router` prevents failed requests, by picking the deployment which is below rate-limit and has the least amount of tokens used. + +In production, [Router connects to a Redis Cache](#redis-queue) to track usage across multiple deployments. + +## Quick Start ```python pip install litellm @@ -46,5 +51,16 @@ router = Router(model_list=model_list) response = router.completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}] +print(response) +``` + +## Redis Queue + +```python +router = Router(model_list=model_list, + redis_host=os.getenv("REDIS_HOST"), + redis_password=os.getenv("REDIS_PASSWORD"), + redis_port=os.getenv("REDIS_PORT")) + print(response) ``` \ No newline at end of file