From 308377fbe27a092526213a006f70ec7dc94dc5dc Mon Sep 17 00:00:00 2001 From: Ishaan Jaff Date: Thu, 29 Aug 2024 14:23:55 -0700 Subject: [PATCH] docs tag based routing per team --- docs/my-website/docs/proxy/tag_routing.md | 46 ++++------------------- 1 file changed, 8 insertions(+), 38 deletions(-) diff --git a/docs/my-website/docs/proxy/tag_routing.md b/docs/my-website/docs/proxy/tag_routing.md index 81012c90084..e63a35deee7 100644 --- a/docs/my-website/docs/proxy/tag_routing.md +++ b/docs/my-website/docs/proxy/tag_routing.md @@ -2,8 +2,8 @@ Route requests based on tags. This is useful for -- implementing free / paid tiers for users -- controlling model access per team, example Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B +- Implementing free / paid tiers for users +- Controlling model access per team, example Team A can access gpt-4 deployment A, Team B can access gpt-4 deployment B ## Quick Start @@ -223,39 +223,7 @@ Here's how to set up and use team-based tag routing using curl commands: Replace `team_a_id_here` and `team_b_id_here` with the actual team IDs received from step 2. -4. **Make requests with team-specific keys:** - - When making requests to the `/chat/completions` endpoint, use the team-specific keys. The proxy will automatically route the request to the appropriate model based on the team's tags: - - ```shell - # Request using Team A's key - curl -X POST http://0.0.0.0:4000/chat/completions \ - -H "Authorization: Bearer team_a_key_here" \ - -H "Content-Type: application/json" \ - -d '{ - "model": "fake-openai-endpoint", - "messages": [ - {"role": "user", "content": "Hello!"} - ] - }' - ``` - - ```shell - # Request using Team B's key - curl -X POST http://0.0.0.0:4000/chat/completions \ - -H "Authorization: Bearer team_b_key_here" \ - -H "Content-Type: application/json" \ - -d '{ - "model": "fake-openai-endpoint", - "messages": [ - {"role": "user", "content": "Hello!"} - ] - }' - ``` - - Replace `team_a_key_here` and `team_b_key_here` with the actual keys generated in step 3. - -5. **Verify routing:** +4. **Verify routing:** Check the `x-litellm-model-id` header in the response to confirm that the request was routed to the correct model based on the team's tags. You can use the `-i` flag with curl to include the response headers: @@ -274,15 +242,17 @@ Here's how to set up and use team-based tag routing using curl commands: In the response headers, you should see: ``` - x-litellm-model-id: teama + x-litellm-model-id: team-a-model ``` Similarly, when using Team B's key, you should see: ``` - x-litellm-model-id: teamb + x-litellm-model-id: team-b-model ``` By following these steps and using these curl commands, you can implement and test team-based tag routing in your LiteLLM Proxy setup, ensuring that different teams are routed to the appropriate models or deployments based on their assigned tags. - +## Other Tag Based Features +- [Track spend per tag](cost_tracking#-custom-tags) +- [Setup Budgets per Virtual Key, Team](users)