scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-07 08:26:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 89

docs tag based routing per team

Browse source
This commit is contained in:
Ishaan Jaff 2024-08-29 14:23:55 -07:00
parent 242f66054d
commit 308377fbe2
1 changed files with 8 additions and 38 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

46
docs/my-website/docs/proxy/tag_routing.md
View file

@ -2,8 +2,8 @@
Route requests based on tags.
This is useful for
- implementing free / paid tiers for users
- controlling model access per team, example Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B
- Implementing free / paid tiers for users
- Controlling model access per team, example Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B
## Quick Start
@ -223,39 +223,7 @@ Here's how to set up and use team-based tag routing using curl commands:
Replace `team_a_id_here` and `team_b_id_here` with the actual team IDs received from step 2.
4. **Make requests with team-specific keys:**
When making requests to the `/chat/completions` endpoint, use the team-specific keys. The proxy will automatically route the request to the appropriate model based on the team's tags:
```shell
# Request using Team A's key
curl -X POST http://0.0.0.0:4000/chat/completions \
-H "Authorization: Bearer team_a_key_here" \
-H "Content-Type: application/json" \
-d '{
"model": "fake-openai-endpoint",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
```
```shell
# Request using Team B's key
curl -X POST http://0.0.0.0:4000/chat/completions \
-H "Authorization: Bearer team_b_key_here" \
-H "Content-Type: application/json" \
-d '{
"model": "fake-openai-endpoint",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
```
Replace `team_a_key_here` and `team_b_key_here` with the actual keys generated in step 3.
5. **Verify routing:**
4. **Verify routing:**
Check the `x-litellm-model-id` header in the response to confirm that the request was routed to the correct model based on the team's tags. You can use the `-i` flag with curl to include the response headers:
@ -274,15 +242,17 @@ Here's how to set up and use team-based tag routing using curl commands:
In the response headers, you should see:
```
x-litellm-model-id: teama
x-litellm-model-id: team-a-model
```
Similarly, when using Team B's key, you should see:
```
x-litellm-model-id: teamb
x-litellm-model-id: team-b-model
```
By following these steps and using these curl commands, you can implement and test team-based tag routing in your LiteLLM Proxy setup, ensuring that different teams are routed to the appropriate models or deployments based on their assigned tags.
## Other Tag Based Features
- [Track spend per tag](cost_tracking#-custom-tags)
- [Setup Budgets per Virtual Key, Team](users)