scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-05 08:07:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 137

address greptile review round 3: feature flag, provider hint, fallback logging, docs

Browse source
This commit is contained in:
michelligabriele 2026-03-24 14:24:40 +01:00
parent 4ce7a644f6
commit be153a0874
No known key found for this signature in database
5 changed files with 375 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

270
docs/my-website/docs/proxy/credential_routing.md Normal file
View file

@ -0,0 +1,270 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Per-Team/Project Credential Routing
Route the same model to different LLM provider endpoints (e.g. different Azure instances) based on which team or project makes the request.
## Overview
In multi-tenant deployments, different teams often need the same model name (e.g., `gpt-4`) to hit different provider endpoints — for example, separate Azure OpenAI instances per business unit for cost isolation, data residency, or rate limit separation.
**Credential routing** lets you configure this in team/project metadata using the existing [credentials table](./ui_credentials.md), without duplicating model definitions or creating separate model groups per team.
```
Hotel Team → gpt-4 → https://hotel-eastus.openai.azure.com/
Flight Team → gpt-4 → https://flight-centralus.openai.azure.com/
```
### Precedence Chain
When a request comes in, the system walks this precedence chain (first match wins):
1. **Clientside credentials** — `api_base`/`api_key` passed in the request body ([docs](./clientside_auth.md))
2. **Project model-specific** — override for this exact model in the project's `model_config`
3. **Project default** — `defaultconfig` in the project's `model_config`
4. **Team model-specific** — override for this exact model in the team's `model_config`
5. **Team default** — `defaultconfig` in the team's `model_config`
6. **Deployment default** — the model's `litellm_params` as configured in `config.yaml`
## Quick Start
### Step 1: Create Credentials
Store your Azure endpoint credentials in the credentials table. You can do this via the [UI](./ui_credentials.md) or API:
```bash showLineNumbers
# Create credential for Hotel team's Azure endpoint
curl -X POST 'http://0.0.0.0:4000/credentials' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"credential_name": "hotel-azure-eastus",
"credential_values": {
"api_base": "https://hotel-eastus.openai.azure.com/",
"api_key": "sk-azure-hotel-key-xxx"
}
}'
```
```bash showLineNumbers
# Create credential for Flight team's Azure endpoint
curl -X POST 'http://0.0.0.0:4000/credentials' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"credential_name": "flight-azure-centralus",
"credential_values": {
"api_base": "https://flight-centralus.openai.azure.com/",
"api_key": "sk-azure-flight-key-xxx"
}
}'
```
### Step 2: Set `model_config` on Teams
Add a `model_config` key to the team's metadata referencing the credential by name:
```bash showLineNumbers
# Hotel team — default Azure endpoint for all models
curl -X PATCH 'http://0.0.0.0:4000/team/update' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"team_id": "hotel-team-id",
"metadata": {
"model_config": {
"defaultconfig": {
"azure": {
"litellm_credentials": "hotel-azure-eastus"
}
}
}
}
}'
```
```bash showLineNumbers
# Flight team — default Azure endpoint for all models
curl -X PATCH 'http://0.0.0.0:4000/team/update' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"team_id": "flight-team-id",
"metadata": {
"model_config": {
"defaultconfig": {
"azure": {
"litellm_credentials": "flight-azure-centralus"
}
}
}
}
}'
```
### Step 3: Make Requests
Requests are automatically routed to the correct Azure endpoint based on the API key's team:
```bash showLineNumbers
# Request using Hotel team's API key → routes to hotel-eastus.openai.azure.com
curl http://localhost:4000/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-hotel-team-key' \
-d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}'
# Request using Flight team's API key → routes to flight-centralus.openai.azure.com
curl http://localhost:4000/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-flight-team-key' \
-d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Per-Model Overrides
You can set different credentials for specific models while keeping a default for everything else:
```bash showLineNumbers
curl -X PATCH 'http://0.0.0.0:4000/team/update' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"team_id": "hotel-team-id",
"metadata": {
"model_config": {
"defaultconfig": {
"azure": {
"litellm_credentials": "hotel-azure-eastus"
}
},
"gpt-4": {
"azure": {
"litellm_credentials": "hotel-azure-westus"
}
}
}
}
}'
```
With this config:
- `gpt-4` requests → `hotel-azure-westus` credential (model-specific)
- All other models → `hotel-azure-eastus` credential (default)
## Project-Level Overrides
Projects inherit their team's `model_config` but can override at the project level. Project overrides take precedence over team overrides.
```bash showLineNumbers
# Project overrides the team default for all models
curl -X PATCH 'http://0.0.0.0:4000/project/update' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
"project_id": "hotel-rec-app-id",
"metadata": {
"model_config": {
"defaultconfig": {
"azure": {
"litellm_credentials": "hotel-rec-azure"
}
},
"gpt-4-vision": {
"azure": {
"litellm_credentials": "hotel-rec-vision"
}
}
}
}
}'
```
### Full Example: Hotel Team with Two Projects
**Setup:**
- **Hotel Team**: default `hotel-azure-eastus`, GPT-4 override to `hotel-azure-westus`
- **Hotel Rec App** (project): default `hotel-rec-azure`, GPT-4-Vision override to `hotel-rec-vision`
- **Hotel Review App** (project): no overrides — inherits team config
**Resolution:**
| Request | Resolved Credential | Why |
|---|---|---|
| Hotel Rec App → `gpt-4` | `hotel-rec-azure` | Project default (no project model-specific match for gpt-4) |
| Hotel Rec App → `gpt-4-vision` | `hotel-rec-vision` | Project model-specific |
| Hotel Review App → `gpt-3.5` | `hotel-azure-eastus` | Team default (no project config) |
| Hotel Review App → `gpt-4` | `hotel-azure-westus` | Team model-specific |
## `model_config` Schema
The `model_config` key is a JSON object in team/project `metadata`:
```json
{
"model_config": {
"defaultconfig": {
"<provider>": {
"litellm_credentials": "<credential-name>"
}
},
"<model-name>": {
"<provider>": {
"litellm_credentials": "<credential-name>"
}
}
}
}
```
| Field | Description |
|---|---|
| `defaultconfig` | Fallback credential for any model not explicitly listed |
| `<model-name>` | Model-specific override — must match the LiteLLM model group name |
| `<provider>` | Provider key (e.g. `azure`, `openai`, `bedrock`). When the model name includes a provider prefix (e.g. `azure/gpt-4`), the system prefers the matching provider key |
| `litellm_credentials` | Name of a credential in the [credentials table](./ui_credentials.md) |
### Credential Values
The referenced credential can contain any combination of:
| Key | Description |
|---|---|
| `api_base` | Provider endpoint URL |
| `api_key` | API key for the provider |
| `api_version` | API version (e.g. for Azure) |
Only keys present in the credential are applied. Keys already in the request (e.g. clientside `api_version`) are never overwritten.
## Disabling the Feature
The feature is enabled by default. To disable it globally:
<Tabs>
<TabItem value="config" label="config.yaml">
```yaml
litellm_settings:
enable_model_config_credential_overrides: false
```
</TabItem>
<TabItem value="env" label="Environment Variable">
```bash
export LITELLM_ENABLE_MODEL_CONFIG_CREDENTIAL_OVERRIDES=false
```
</TabItem>
</Tabs>
## Related Documentation
- [Adding LLM Credentials](./ui_credentials.md) — Create and manage reusable credentials
- [Project Management](./project_management.md) — Project hierarchy and API
- [Team Budgets](./team_budgets.md) — Team-level budget management
- [Clientside LLM Credentials](./clientside_auth.md) — Passing credentials in the request body
- [Credential Usage Tracking](./credential_usage_tracking.md) — Track spend by credential

3
docs/my-website/sidebars.js
View file

@ -561,7 +561,8 @@ const sidebars = {
"proxy/model_access",
"proxy/model_access_groups",
"proxy/access_groups",
"proxy/team_model_add"
"proxy/team_model_add",
"proxy/credential_routing"
]
},
{

1
litellm/__init__.py
View file

@ -318,6 +318,7 @@ return_response_headers: bool = (
False # get response headers from LLM Api providers - example x-remaining-requests,
)
enable_json_schema_validation: bool = False
enable_model_config_credential_overrides: bool = True
enable_key_alias_format_validation: bool = (
False # opt-in validation of key_alias format on /key/generate and /key/update
)

47
litellm/proxy/litellm_pre_call_utils.py
View file

@ -1363,6 +1363,10 @@ def _apply_credential_overrides_from_model_config(
5. Team default override (defaultconfig)
6. Deployment default (no action needed)
"""
# Feature flag gate — operators can disable this with litellm.enable_model_config_credential_overrides = False
if not litellm.enable_model_config_credential_overrides:
return
# Respect clientside credentials — highest precedence
if data.get("api_base") is not None or data.get("api_key") is not None:
return
@ -1380,11 +1384,17 @@ def _apply_credential_overrides_from_model_config(
if not project_model_config and not team_model_config:
return
# Extract provider hint from model name (e.g. "azure/gpt-4" -> "azure")
provider: Optional[str] = None
if "/" in model_name:
provider = model_name.split("/", 1)[0]
credential_name = _resolve_credential_from_model_config(
model_name=model_name,
project_model_config=project_model_config,
team_model_config=team_model_config,
pre_alias_model_name=pre_alias_model_name,
provider=provider,
)
if not credential_name:
@ -1415,6 +1425,7 @@ def _resolve_credential_from_model_config(
project_model_config: Optional[dict],
team_model_config: Optional[dict],
pre_alias_model_name: Optional[str] = None,
provider: Optional[str] = None,
) -> Optional[str]:
"""
Walk the precedence chain and return the first matching credential name.
@ -1426,6 +1437,11 @@ def _resolve_credential_from_model_config(
4. team_model_config[model_name][provider] — team model-specific
5. team_model_config[pre_alias_model_name][provider] — team pre-alias
6. team_model_config["defaultconfig"][provider] — team default
When a model-specific entry exists but contains no litellm_credentials,
the function falls through to defaultconfig. This is intentional —
an entry without litellm_credentials is treated as incomplete config,
not as an explicit "no override" signal.
"""
# Build the list of model names to try (post-alias first, then pre-alias)
model_names_to_try = [model_name]
@ -1440,29 +1456,52 @@ def _resolve_credential_from_model_config(
for name in model_names_to_try:
model_entry = model_config.get(name)
if model_entry:
credential_name = _extract_credential_from_entry(model_entry)
credential_name = _extract_credential_from_entry(
model_entry, provider=provider
)
if credential_name:
return credential_name
verbose_proxy_logger.debug(
"model_config entry '%s' found but has no litellm_credentials, "
"falling through to defaultconfig",
name,
)
# Default check
default_entry = model_config.get("defaultconfig")
if default_entry:
credential_name = _extract_credential_from_entry(default_entry)
credential_name = _extract_credential_from_entry(
default_entry, provider=provider
)
if credential_name:
return credential_name
return None
def _extract_credential_from_entry(entry: dict) -> Optional[str]:
def _extract_credential_from_entry(
entry: dict, provider: Optional[str] = None
) -> Optional[str]:
"""
Extract litellm_credentials from a model_config entry.
Entry structure: {"azure": {"litellm_credentials": "name"}, ...}
Returns the first credential name found across all provider keys.
When provider is given (e.g. "azure"), tries an exact provider match first.
Falls back to the first credential found across all provider keys.
"""
if not isinstance(entry, dict):
return None
# Prefer exact provider match when provider hint is available
if provider and provider in entry:
provider_config = entry[provider]
if isinstance(provider_config, dict):
credential_name = provider_config.get("litellm_credentials")
if credential_name:
return credential_name
# Fall back to first available provider
for provider_config in entry.values():
if isinstance(provider_config, dict):
credential_name = provider_config.get("litellm_credentials")

59
tests/test_litellm/proxy/test_litellm_pre_call_utils.py
View file

@ -2302,3 +2302,62 @@ def test_apply_overrides_with_alias(setup_test_credentials):
)
assert data["api_base"] == "https://hotel-eastus.openai.azure.com/"
assert data["api_key"] == "key-hotel-eastus"
def test_apply_overrides_feature_flag_disabled(setup_test_credentials):
"""Feature flag litellm.enable_model_config_credential_overrides disables the feature."""
data = {"model": "gpt-4"}
user_api_key_dict = UserAPIKeyAuth(
api_key="test-key",
team_metadata={
"model_config": {
"gpt-4": {"azure": {"litellm_credentials": "hotel-azure-eastus"}}
}
},
)
original = litellm.enable_model_config_credential_overrides
try:
litellm.enable_model_config_credential_overrides = False
_apply_credential_overrides_from_model_config(
data=data, user_api_key_dict=user_api_key_dict
)
assert "api_base" not in data
assert "api_key" not in data
finally:
litellm.enable_model_config_credential_overrides = original
def test_extract_credential_provider_hint_prefers_exact_match():
"""Provider hint selects the correct provider in a multi-provider entry."""
entry = {
"openai": {"litellm_credentials": "openai-cred"},
"azure": {"litellm_credentials": "azure-cred"},
}
# With provider hint, should pick the exact match
assert _extract_credential_from_entry(entry, provider="azure") == "azure-cred"
assert _extract_credential_from_entry(entry, provider="openai") == "openai-cred"
# Without provider hint, falls back to first key (insertion order)
result = _extract_credential_from_entry(entry)
assert result in ("openai-cred", "azure-cred")
# Unknown provider falls back to first available
result = _extract_credential_from_entry(entry, provider="bedrock")
assert result in ("openai-cred", "azure-cred")
def test_resolve_provider_hint_from_model_name():
"""Provider prefix in model name (e.g. azure/gpt-4) threads through to entry extraction."""
config = {
"gpt-4": {
"openai": {"litellm_credentials": "openai-cred"},
"azure": {"litellm_credentials": "azure-cred"},
},
}
# Model name "azure/gpt-4" -> provider="azure" -> should prefer azure-cred
# But _resolve_credential_from_model_config tries "azure/gpt-4" first (no match),
# then falls to defaultconfig (no match). So we need to use pre_alias_model_name.
result = _resolve_credential_from_model_config(
"azure/gpt-4", config, None, pre_alias_model_name="gpt-4", provider="azure"
)
assert result == "azure-cred"