feat(router): add order-based fallback so higher order deployments are tried on failure

When a request to an order=1 deployment fails, the router now
automatically tries order=2, order=3, etc. before falling through to
external fallbacks. Works for all error types (429, 404, connection
errors). Requires enable_pre_call_checks=True.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Sameer Kankute 2026-03-26 12:07:37 +05:30
parent cd0a141115
commit 6295e6b3d8
No known key found for this signature in database
2 changed files with 16 additions and 1 deletions

View file

@ -325,8 +325,15 @@ model_list:
model: azure/gpt-4-fallback
api_key: os.environ/AZURE_API_KEY_2
order: 2 # 👈 Used when order=1 fails
router_settings:
enable_pre_call_checks: true # 👈 Required for 'order' to work
```
:::important
The `order` parameter requires `enable_pre_call_checks: true` in `router_settings`.
:::
### How order-based fallback works
When a request to an `order=1` deployment fails (connection error, 404, 429, etc.), the router automatically tries `order=2` deployments, then `order=3`, and so on. Each order level gets its own set of retries before escalating to the next.
@ -353,6 +360,7 @@ model_list:
api_key: os.environ/OPENAI_API_KEY
router_settings:
enable_pre_call_checks: true
fallbacks:
- gpt-4:
- gpt-4-fallback # tried after all order levels fail

View file

@ -869,9 +869,13 @@ model_list = [
},
]
router = Router(model_list=model_list)
router = Router(model_list=model_list, enable_pre_call_checks=True) # 👈 Required for 'order' to work
```
:::important
The `order` parameter requires `enable_pre_call_checks=True` to be set on the Router.
:::
</TabItem>
<TabItem value="proxy" label="PROXY">
@ -888,6 +892,9 @@ model_list:
model: azure/gpt-4-fallback
api_key: os.environ/AZURE_API_KEY_2
order: 2 # 👈 Tried when order=1 fails
router_settings:
enable_pre_call_checks: true # 👈 Required for 'order' to work
```
</TabItem>