mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-17 23:51:30 +00:00
feat(router): add order-based fallback so higher order deployments are tried on failure
When a request to an order=1 deployment fails, the router now automatically tries order=2, order=3, etc. before falling through to external fallbacks. Works for all error types (429, 404, connection errors). Requires enable_pre_call_checks=True. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
cd0a141115
commit
6295e6b3d8
2 changed files with 16 additions and 1 deletions
|
|
@ -325,8 +325,15 @@ model_list:
|
|||
model: azure/gpt-4-fallback
|
||||
api_key: os.environ/AZURE_API_KEY_2
|
||||
order: 2 # 👈 Used when order=1 fails
|
||||
|
||||
router_settings:
|
||||
enable_pre_call_checks: true # 👈 Required for 'order' to work
|
||||
```
|
||||
|
||||
:::important
|
||||
The `order` parameter requires `enable_pre_call_checks: true` in `router_settings`.
|
||||
:::
|
||||
|
||||
### How order-based fallback works
|
||||
|
||||
When a request to an `order=1` deployment fails (connection error, 404, 429, etc.), the router automatically tries `order=2` deployments, then `order=3`, and so on. Each order level gets its own set of retries before escalating to the next.
|
||||
|
|
@ -353,6 +360,7 @@ model_list:
|
|||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
router_settings:
|
||||
enable_pre_call_checks: true
|
||||
fallbacks:
|
||||
- gpt-4:
|
||||
- gpt-4-fallback # tried after all order levels fail
|
||||
|
|
|
|||
|
|
@ -869,9 +869,13 @@ model_list = [
|
|||
},
|
||||
]
|
||||
|
||||
router = Router(model_list=model_list)
|
||||
router = Router(model_list=model_list, enable_pre_call_checks=True) # 👈 Required for 'order' to work
|
||||
```
|
||||
|
||||
:::important
|
||||
The `order` parameter requires `enable_pre_call_checks=True` to be set on the Router.
|
||||
:::
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="proxy" label="PROXY">
|
||||
|
||||
|
|
@ -888,6 +892,9 @@ model_list:
|
|||
model: azure/gpt-4-fallback
|
||||
api_key: os.environ/AZURE_API_KEY_2
|
||||
order: 2 # 👈 Tried when order=1 fails
|
||||
|
||||
router_settings:
|
||||
enable_pre_call_checks: true # 👈 Required for 'order' to work
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue