diff --git a/docs/external_dependencies_audit.md b/docs/external_dependencies_audit.md index 32108ed9894..069fca5764d 100644 --- a/docs/external_dependencies_audit.md +++ b/docs/external_dependencies_audit.md @@ -177,67 +177,41 @@ export LITELLM_LOCAL_MODEL_COST_MAP=True --- -## Proposed Code Changes for Enhanced Safety +## Implemented Code Changes -### 1. Add Validation to `get_model_cost_map.py` +### 1. Runtime Validation in `get_model_cost_map.py` ✅ IMPLEMENTED -```python -def validate_model_cost_map(data: dict) -> bool: - """Validate that fetched data is a valid model cost map.""" - if not isinstance(data, dict): - return False - if len(data) < 100: # Sanity check - should have many models - return False - # Check a sample of entries have required fields - for key, value in list(data.items())[:10]: - if key == "sample_spec": - continue - if not isinstance(value, dict): - return False - if "litellm_provider" not in value: - return False - return True +The `get_model_cost_map()` function now: +- Validates JSON structure before accepting remote data +- Checks for required fields (`litellm_provider`) +- Validates mode values against allowed list +- Validates numeric cost fields +- Falls back to local backup if validation fails +- Logs warnings when falling back -def get_model_cost_map(url: str) -> dict: - # ... existing code ... - try: - response = httpx.get(url, timeout=5) - response.raise_for_status() - content = response.json() - - # NEW: Validate before accepting - if not validate_model_cost_map(content): - verbose_logger.warning( - "Remote model cost map failed validation, using local backup" - ) - return _load_local_backup() - - return content - except Exception: - return _load_local_backup() -``` +### 2. Enhanced CI Validation ✅ IMPLEMENTED -### 2. Enhanced CI Validation +`.github/workflows/test-model-map.yaml` now: +- Validates JSON syntax (fast fail with `jq`) +- Runs comprehensive schema validation via `scripts/validate_model_cost_map.py` +- Validates both main file and backup file +- Checks backup freshness -Add comprehensive validation to `.github/workflows/test-model-map.yaml`: +### 3. Validation Script ✅ IMPLEMENTED -```yaml -- name: Validate model_prices_and_context_window.json - run: | - python scripts/validate_model_cost_map.py -``` +`scripts/validate_model_cost_map.py` provides: +- Validates all entries have required fields +- Checks mode values against allowed modes +- Validates numeric fields are actually numeric +- Sanity checks for suspiciously high costs +- Detailed error reporting with exit codes for CI -### 3. Add Monitoring/Alerting +### 4. Observability ✅ IMPLEMENTED -Log when falling back to local backup: - -```python -if using_local_backup: - verbose_logger.warning( - "Using local model cost map backup. " - "Remote fetch failed or returned invalid data." - ) -``` +The code now logs warnings when: +- Falling back to local backup due to network errors +- Falling back due to validation failures +- Using local backup mode ---