scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-05 02:41:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 48

docs(reliability.md): add doc on mock testing fallbacks

Browse source
This commit is contained in:
Krrish Dholakia 2024-06-25 11:07:07 -07:00 • committed by Ishaan Jaff
parent 9b47ba72cb
commit 81fd42258c
2 changed files with 66 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

61
docs/my-website/docs/proxy/reliability.md
View file

@ -431,6 +431,67 @@ litellm_settings:
content_policy_fallbacks: [{"gpt-3.5-turbo-small": ["claude-opus"]}]
```
### Test Fallbacks!
Check if your fallbacks are working as expected.
#### **Regular Fallbacks**
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-D '{
"model": "my-bad-model",
"messages": [
{
"role": "user",
"content": "ping"
}
],
"mock_testing_fallbacks": true # 👈 KEY CHANGE
}
'
```
#### **Content Policy Fallbacks**
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-D '{
"model": "my-bad-model",
"messages": [
{
"role": "user",
"content": "ping"
}
],
"mock_testing_content_policy_fallbacks": true # 👈 KEY CHANGE
}
'
```
#### **Context Window Fallbacks**
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-D '{
"model": "my-bad-model",
"messages": [
{
"role": "user",
"content": "ping"
}
],
"mock_testing_context_window_fallbacks": true # 👈 KEY CHANGE
}
'
```
### EU-Region Filtering (Pre-Call Checks)
**Before call is made** check if a call is within model context window with **`enable_pre_call_checks: true`**.

10
litellm/model_prices_and_context_window_backup.json
View file

@ -887,7 +887,7 @@
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.00000005,
"output_cost_per_token": 0.00000010,
"output_cost_per_token": 0.00000008,
"litellm_provider": "groq",
"mode": "chat",
"supports_function_calling": true
@ -906,8 +906,8 @@
"max_tokens": 32768,
"max_input_tokens": 32768,
"max_output_tokens": 32768,
"input_cost_per_token": 0.00000027,
"output_cost_per_token": 0.00000027,
"input_cost_per_token": 0.00000024,
"output_cost_per_token": 0.00000024,
"litellm_provider": "groq",
"mode": "chat",
"supports_function_calling": true
@ -916,8 +916,8 @@
"max_tokens": 8192,
"max_input_tokens": 8192,
"max_output_tokens": 8192,
"input_cost_per_token": 0.00000010,
"output_cost_per_token": 0.00000010,
"input_cost_per_token": 0.00000007,
"output_cost_per_token": 0.00000007,
"litellm_provider": "groq",
"mode": "chat",
"supports_function_calling": true