This prevents upstream GitHub changes from breaking LLM calls by:
1. Runtime validation in get_model_cost_map.py:
- Validates JSON structure before accepting remote data
- Checks for required fields (litellm_provider)
- Validates mode values against allowed list
- Validates numeric cost fields
- Falls back to local backup if validation fails
- Logs warnings when falling back
2. Enhanced CI validation (.github/workflows/test-model-map.yaml):
- JSON syntax validation (fast fail)
- Comprehensive schema validation via Python script
- Validates both main file and backup file
- Checks backup freshness
3. New validation script (scripts/validate_model_cost_map.py):
- Validates all entries have required fields
- Checks mode values against allowed modes
- Validates numeric fields are actually numeric
- Sanity checks for suspiciously high costs
- Detailed error reporting
4. Unit tests for validation logic
The key improvement: if GitHub returns valid JSON but semantically
broken data (missing fields, wrong types), we now detect this and
fall back to the local backup instead of using the bad data.
Customers can also set LITELLM_LOCAL_MODEL_COST_MAP=True to completely
disable remote fetching for maximum stability.
Co-authored-by: ishaan <ishaan@berri.ai>
0220609ab6