mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-11 03:38:38 +00:00
0220609ab6
This prevents upstream GitHub changes from breaking LLM calls by: 1. Runtime validation in get_model_cost_map.py: - Validates JSON structure before accepting remote data - Checks for required fields (litellm_provider) - Validates mode values against allowed list - Validates numeric cost fields - Falls back to local backup if validation fails - Logs warnings when falling back 2. Enhanced CI validation (.github/workflows/test-model-map.yaml): - JSON syntax validation (fast fail) - Comprehensive schema validation via Python script - Validates both main file and backup file - Checks backup freshness 3. New validation script (scripts/validate_model_cost_map.py): - Validates all entries have required fields - Checks mode values against allowed modes - Validates numeric fields are actually numeric - Sanity checks for suspiciously high costs - Detailed error reporting 4. Unit tests for validation logic The key improvement: if GitHub returns valid JSON but semantically broken data (missing fields, wrong types), we now detect this and fall back to the local backup instead of using the bad data. Customers can also set LITELLM_LOCAL_MODEL_COST_MAP=True to completely disable remote fetching for maximum stability. Co-authored-by: ishaan <ishaan@berri.ai>
60 lines
2.2 KiB
YAML
60 lines
2.2 KiB
YAML
name: Validate model_prices_and_context_window.json
|
|
|
|
on:
|
|
pull_request:
|
|
branches: [ main ]
|
|
push:
|
|
branches: [ main ]
|
|
paths:
|
|
- 'model_prices_and_context_window.json'
|
|
- 'scripts/validate_model_cost_map.py'
|
|
- '.github/workflows/test-model-map.yaml'
|
|
|
|
jobs:
|
|
validate-model-prices-json:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@v4
|
|
with:
|
|
python-version: '3.11'
|
|
|
|
# Step 1: Basic JSON syntax validation (fast fail)
|
|
- name: Validate JSON syntax
|
|
run: |
|
|
echo "Checking JSON syntax..."
|
|
jq empty model_prices_and_context_window.json
|
|
echo "✅ JSON syntax is valid"
|
|
|
|
# Step 2: Comprehensive schema/semantic validation
|
|
- name: Validate model cost map schema
|
|
run: |
|
|
echo "Running comprehensive validation..."
|
|
python scripts/validate_model_cost_map.py model_prices_and_context_window.json
|
|
|
|
# Step 3: Verify backup file is also valid
|
|
- name: Validate backup model cost map
|
|
run: |
|
|
echo "Validating backup file..."
|
|
python scripts/validate_model_cost_map.py litellm/model_prices_and_context_window_backup.json
|
|
|
|
# Step 4: Ensure backup stays in sync (optional check)
|
|
- name: Check backup freshness
|
|
run: |
|
|
echo "Checking if backup matches main file..."
|
|
# Compare entry counts - backup should be reasonably close to main
|
|
MAIN_COUNT=$(jq 'keys | length' model_prices_and_context_window.json)
|
|
BACKUP_COUNT=$(jq 'keys | length' litellm/model_prices_and_context_window_backup.json)
|
|
echo "Main file entries: $MAIN_COUNT"
|
|
echo "Backup file entries: $BACKUP_COUNT"
|
|
|
|
# Warn if backup is significantly out of date (more than 100 entries behind)
|
|
DIFF=$((MAIN_COUNT - BACKUP_COUNT))
|
|
if [ $DIFF -gt 100 ]; then
|
|
echo "⚠️ WARNING: Backup file is $DIFF entries behind main file"
|
|
echo "Consider updating the backup during release"
|
|
else
|
|
echo "✅ Backup file is reasonably up to date"
|
|
fi
|