diff --git a/docs/my-website/blog/model_cost_map_incident/index.md b/docs/my-website/blog/model_cost_map_incident/index.md new file mode 100644 index 00000000000..abd00d2b652 --- /dev/null +++ b/docs/my-website/blog/model_cost_map_incident/index.md @@ -0,0 +1,151 @@ +--- +slug: model-cost-map-incident +title: "Incident Report: Broken Model Cost Map on main" +date: 2026-02-10T10:00:00 +authors: + - name: Ishaan Jaffer + title: "CTO, LiteLLM" + url: https://www.linkedin.com/in/ishaanjaffer/ + image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg +tags: [incident-report, stability, model-cost-map] +--- + +# Incident Report: Broken Model Cost Map on `main` + +## What happened? + +2 weeks ago a contributor PR with changes to the model cost map had a poorly formatted JSON entry. When this was merged into `main` ([commit `562f0a0`](https://github.com/BerriAI/litellm/commit/562f0a028251750e3d75386bee0e630d9796d0df)) it led to the following error message reported from users. + +Users found that their code started erroring out with the following message: + +``` +{"type":"error","error":"This model isn't mapped yet. model=gpt-5.2, custom_llm_provider=azure. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json."} +``` + +The bad commit added an extra `{` bracket at line 24258 of `model_prices_and_context_window.json`, making the entire file invalid JSON: + +``` +json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 24258 column 5 +``` + +--- + +## What caused this type of exception? + +### How the model cost map loading works + +1. At **import time**, `litellm` fetches the model cost map from GitHub `main`: + `https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json` +2. If the fetch fails (network error, invalid JSON, etc.), it **silently** falls back to a local backup file (`model_prices_and_context_window_backup.json`) bundled with the installed package +3. There is **no log or warning** when the fallback occurs -- users have no way to know they're running on stale data + +### What happened in this incident + +1. The bad commit broke the JSON on `main` +2. Every litellm installation (not just a specific version) fetches from `main` at import time +3. `response.json()` threw `JSONDecodeError`, caught by `except Exception` +4. Silently fell back to the local backup, which is pinned to the installed package version +5. A customer on v1.80.5 had a backup missing **661+ newer models** including `azure/gpt-5.2` +6. Any call to `get_model_info("azure/gpt-5.2")` raised `"This model isn't mapped yet"` +7. This affected **all litellm users** (not just those on a specific version), since every installation fetches the remote JSON from `main` at import time + +### Impact matrix + +| Call Path | Worked? | Error | +|---|---|---| +| `litellm.completion(model="azure/gpt-5.2")` | Yes -- model map error caught silently in debug logs, request still sent to Azure | | +| `litellm.completion(model="azure/gpt-5.2", stream=True)` | Yes -- same behavior, streams fine | | +| `litellm.get_model_info("azure/gpt-5.2")` | No | `This model isn't mapped yet. model=gpt-5.2, custom_llm_provider=azure.` | +| Proxy routing (request forwarding) | Yes -- routes based on config `model_list`, not cost map | | +| Proxy cost tracking / spend logging | No | `get_model_info()` fails in cost calculation callbacks, error surfaces in logging | +| Proxy `/model/info` endpoint | Partial | Returns default values (0 cost, null limits) for unmapped models | + +### Key finding + +`litellm.completion()` **never blocks** on a missing model in the cost map. It catches the `get_model_info()` error and proceeds to the API. The customer error surfaced from the **cost tracking / logging callbacks** path, where `get_model_info()` is called for spend calculation. + +--- + +## Root cause analysis + +### Why did this happen? + +- No CI tests validating the JSON structure of `model_prices_and_context_window.json` before merge +- The fallback in `get_model_cost_map()` is **completely silent** -- no log, no metric, no warning +- The backup file can be arbitrarily stale depending on the installed package version + +### Why fetch from GitHub? + +To get live day-0 model updates (new model pricing, context windows) without requiring a package upgrade. This is valuable but creates a hard dependency on the correctness of a file on `main`. + +--- + +## Shippable improvements + +| # | Improvement | Status | Details | +|---|---|---|---| +| 1 | **CI validation for model cost map JSON** | Shipped | [PR #20605](https://github.com/BerriAI/litellm/pull/20605) -- Validates JSON schema + structure on every PR that touches `model_prices_and_context_window.json` | +| 2 | **Warning logging on fallback** | Shipped | `get_model_cost_map()` now logs a `WARNING` when the remote fetch fails and falls back to the local backup, instead of silently swallowing the error. See `litellm/litellm_core_utils/get_model_cost_map.py` | +| 3 | **Fetched JSON integrity validation** | Shipped | New `validate_model_cost_map()` helper in `litellm/litellm_core_utils/get_model_cost_map.py` checks: (a) fetched map is a dict, (b) has minimum model count, (c) hasn't shrunk >50% vs backup. If any check fails, falls back to backup with a warning | +| 4 | **CI/CD test for bad cost map resilience** | Shipped | `tests/llm_translation/test_model_cost_map_resilience.py` -- 13 tests covering: empty map, invalid JSON, network errors, shrinkage detection, `get_model_info()` error messages, and `litellm.completion()` resilience | +| 5 | Keep backup file in sync on every release | Planned | Update `model_prices_and_context_window_backup.json` as part of the release process so fallback data is never more than 1 release behind | +| 6 | `LITELLM_LOCAL_MODEL_COST_MAP=True` as default for production | Planned | Eliminates runtime GitHub dependency. Users who want live updates can opt in | +| 7 | Health check endpoint for external deps | Planned | Proxy endpoint (e.g., `/health/dependencies`) reporting status of all external fetches | + +--- + +## Other upstream dependencies in the codebase + +The model cost map is not the only external dependency that could impact LLM calls. Here is a full audit: + +### Critical (can block LLM calls or auth) + +| Dependency | File | URL | When Fetched | Fallback | Silent? | +|---|---|---|---|---|---| +| **Model cost map** | `litellm/__init__.py` | `raw.githubusercontent.com/.../model_prices_and_context_window.json` | Import time | Local backup file | Yes (now logs warning) | +| **Model cost map reload** | `litellm/proxy/proxy_server.py` | Same as above | Runtime (scheduled / manual) | Keeps existing map | No (logged) | +| **JWT public keys** | `litellm/proxy/auth/handle_jwt.py` | Configurable via `JWT_PUBLIC_KEY_URL` | Runtime (on-demand, cached with TTL) | **None -- raises exception** | No (exception) | +| **OIDC UserInfo** | `litellm/proxy/auth/handle_jwt.py` | Configurable via `oidc_userinfo_endpoint` | Runtime (on-demand, cached 300s) | **None -- raises exception** | No (exception) | + +### Medium impact + +| Dependency | File | URL | When Fetched | Fallback | Silent? | +|---|---|---|---|---|---| +| **HuggingFace provider mapping** | `litellm/llms/huggingface/common_utils.py` | `huggingface.co/api/models/{model}` | Runtime (on-demand, LRU cached) | Raises `HuggingFaceError` | No | + +### Low impact (non-blocking) + +| Dependency | File | URL | When Fetched | Fallback | Silent? | +|---|---|---|---|---|---| +| **Ollama model tags** | `litellm/llms/ollama/common_utils.py` | `{ollama_base}/api/tags` (localhost) | Runtime (on-demand) | Static model list | Warning logged | +| **Together AI model info** | `litellm/litellm_core_utils/prompt_templates/factory.py` | `api.together.xyz/models/info` | Runtime (on-demand) | Returns `None` | Yes (silent) | +| **AssemblyAI transcript polling** | `litellm/proxy/pass_through_endpoints/...` | `api.assemblyai.com/v2/transcript/{id}` | Runtime (on-demand) | Returns `None` | Logged | + +--- + +## How to prevent this type of error again + +### Shipped fixes + +1. **JSON validation CI check** ([PR #20605](https://github.com/BerriAI/litellm/pull/20605)) -- Runs `json.loads()` and schema validation on `model_prices_and_context_window.json` on every PR. This would have caught the bad commit before merge. + +2. **Warning logging on fallback** -- When `get_model_cost_map()` falls back to the backup, it now logs a `WARNING`: + ``` + LiteLLM: Failed to fetch remote model cost map from : . Falling back to local backup. + ``` + +3. **Fetched JSON integrity validation** -- New `validate_model_cost_map()` helper validates the fetched map before using it. Catches: non-dict responses, empty maps, and maps that have shrunk significantly compared to the backup. + +4. **CI/CD resilience tests** -- 13 tests in `tests/llm_translation/test_model_cost_map_resilience.py` that simulate bad upstream, bad backup, and verify `litellm.completion()` and `litellm.get_model_info()` behavior. + +### Planned improvements + +5. **Keep the backup file in sync** -- Update `model_prices_and_context_window_backup.json` more frequently (e.g., on every release) so the fallback has recent models. + +6. **Consider `LITELLM_LOCAL_MODEL_COST_MAP=True` as default for production** -- This eliminates the runtime dependency on GitHub entirely. Users who want live updates can opt in. + +7. **Audit all upstream dependencies** -- Apply the same resilience patterns (fallback, logging, validation) to the other external dependencies listed above, especially: + - JWT public key fetch (no fallback today -- should add retry + caching) + - OIDC UserInfo fetch (no fallback today -- should add graceful degradation) + +8. **Add health check for external dependencies** -- A proxy endpoint (e.g., `/health/dependencies`) that reports the status of all external fetches: whether the model cost map was loaded from remote or backup, whether JWT keys are fresh, etc. diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 2c3dfb2b863..28e1724ef82 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -1085,6 +1085,17 @@ const sidebars = { "troubleshoot/max_callbacks", ], }, + { + type: "category", + label: "Blog", + items: [ + { + type: "link", + label: "Incident: Broken Model Cost Map", + href: "/blog/model-cost-map-incident", + }, + ], + }, ], }; diff --git a/litellm/litellm_core_utils/get_model_cost_map.py b/litellm/litellm_core_utils/get_model_cost_map.py index 9b86f4ca2f0..4e3dc8dd452 100644 --- a/litellm/litellm_core_utils/get_model_cost_map.py +++ b/litellm/litellm_core_utils/get_model_cost_map.py @@ -8,40 +8,135 @@ export LITELLM_LOCAL_MODEL_COST_MAP=True ``` """ +import json +import logging import os +from importlib.resources import files import httpx +logger = logging.getLogger("LiteLLM") + + +class GetModelCostMap: + """ + Handles fetching, validating, and loading the model cost map. + + All methods are static — no instance state is needed. This class groups + the helpers that support `get_model_cost_map()` into a single namespace. + """ + + @staticmethod + def load_local_model_cost_map() -> dict: + """Load the local backup model cost map bundled with the package.""" + content = json.loads( + files("litellm") + .joinpath("model_prices_and_context_window_backup.json") + .read_text(encoding="utf-8") + ) + return content + + @staticmethod + def validate_model_cost_map( + fetched_map: dict, + backup_map: dict, + min_model_count: int = 10, + max_shrink_pct: float = 0.5, + ) -> bool: + """ + Validate the integrity of a fetched model cost map. + + Checks: + 1. The fetched map is a non-empty dict. + 2. It has at least ``min_model_count`` models. + 3. It did not shrink by more than ``max_shrink_pct`` (50%) compared to the backup. + + Returns True if the fetched map looks valid, False otherwise. + """ + if not isinstance(fetched_map, dict): + logger.warning( + "LiteLLM: Fetched model cost map is not a dict (type=%s). " + "Falling back to local backup.", + type(fetched_map).__name__, + ) + return False + + fetched_count = len(fetched_map) + + if fetched_count < min_model_count: + logger.warning( + "LiteLLM: Fetched model cost map has only %d models (minimum=%d). " + "This may indicate a corrupted upstream file. " + "Falling back to local backup.", + fetched_count, + min_model_count, + ) + return False + + backup_count = len(backup_map) if isinstance(backup_map, dict) else 0 + if backup_count > 0 and fetched_count < backup_count * max_shrink_pct: + logger.warning( + "LiteLLM: Fetched model cost map shrank significantly " + "(fetched=%d, backup=%d, threshold=%.0f%%). " + "This may indicate a corrupted upstream file. " + "Falling back to local backup.", + fetched_count, + backup_count, + max_shrink_pct * 100, + ) + return False + + return True + + @staticmethod + def fetch_remote_model_cost_map(url: str, timeout: int = 5) -> dict: + """ + Fetch the model cost map from a remote URL. + + Returns the parsed JSON dict. Raises on network/parse errors + (caller is expected to handle). + """ + response = httpx.get(url, timeout=timeout) + response.raise_for_status() + return response.json() + def get_model_cost_map(url: str) -> dict: + """ + Public entry point — returns the model cost map dict. + + 1. If ``LITELLM_LOCAL_MODEL_COST_MAP`` is set, uses the local backup only. + 2. Otherwise fetches from ``url``, validates integrity, and falls back + to the local backup on any failure. + """ if ( os.getenv("LITELLM_LOCAL_MODEL_COST_MAP", False) or os.getenv("LITELLM_LOCAL_MODEL_COST_MAP", False) == "True" ): - from importlib.resources import files - import json + return GetModelCostMap.load_local_model_cost_map() - content = json.loads( - files("litellm") - .joinpath("model_prices_and_context_window_backup.json") - .read_text(encoding="utf-8") - ) - return content + backup_map = GetModelCostMap.load_local_model_cost_map() try: - response = httpx.get( - url, timeout=5 - ) # set a 5 second timeout for the get request - response.raise_for_status() # Raise an exception if the request is unsuccessful - content = response.json() - return content - except Exception: - from importlib.resources import files - import json + content = GetModelCostMap.fetch_remote_model_cost_map(url) + + # Validate fetched JSON integrity before using it + if not GetModelCostMap.validate_model_cost_map( + fetched_map=content, backup_map=backup_map + ): + logger.warning( + "LiteLLM: Fetched model cost map failed integrity check. " + "Using local backup instead. url=%s", + url, + ) + return backup_map - content = json.loads( - files("litellm") - .joinpath("model_prices_and_context_window_backup.json") - .read_text(encoding="utf-8") - ) return content + except Exception as e: + logger.warning( + "LiteLLM: Failed to fetch remote model cost map from %s: %s. " + "Falling back to local backup.", + url, + str(e), + ) + return backup_map diff --git a/tests/llm_translation/test_model_cost_map_resilience.py b/tests/llm_translation/test_model_cost_map_resilience.py new file mode 100644 index 00000000000..93fe731a796 --- /dev/null +++ b/tests/llm_translation/test_model_cost_map_resilience.py @@ -0,0 +1,218 @@ +""" +Tests for model cost map resilience. + +Simulates: +- A bad (invalid JSON) model cost map upstream +- A bad (empty/missing) backup model cost map +- Verifies litellm.completion() still works even with a broken cost map +- Verifies litellm.get_model_info() raises the expected error for unmapped models +- Verifies the integrity validation helper catches corrupted maps +""" + +import json +import os +import sys +from unittest.mock import MagicMock, patch + +import pytest + +sys.path.insert( + 0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")) +) + +import litellm +from litellm.litellm_core_utils.get_model_cost_map import ( + GetModelCostMap, + get_model_cost_map, +) + + +class TestValidateModelCostMap: + """Unit tests for the validate_model_cost_map helper.""" + + def test_should_reject_non_dict(self): + """Non-dict fetched map should fail validation.""" + assert GetModelCostMap.validate_model_cost_map(fetched_map="not a dict", backup_map={}) is False + + def test_should_reject_empty_map(self): + """Empty fetched map should fail validation.""" + assert GetModelCostMap.validate_model_cost_map(fetched_map={}, backup_map={}) is False + + def test_should_reject_too_few_models(self): + """Fetched map with fewer models than min_model_count should fail.""" + small_map = {f"model-{i}": {} for i in range(5)} + assert ( + GetModelCostMap.validate_model_cost_map( + fetched_map=small_map, backup_map={}, min_model_count=10 + ) + is False + ) + + def test_should_reject_significant_shrinkage(self): + """Fetched map that shrunk >50% vs backup should fail.""" + backup = {f"model-{i}": {} for i in range(100)} + fetched = {f"model-{i}": {} for i in range(40)} # 40% of backup + assert ( + GetModelCostMap.validate_model_cost_map( + fetched_map=fetched, backup_map=backup, min_model_count=10 + ) + is False + ) + + def test_should_accept_valid_map(self): + """A fetched map with enough models that hasn't shrunk should pass.""" + backup = {f"model-{i}": {} for i in range(100)} + fetched = {f"model-{i}": {} for i in range(120)} + assert ( + GetModelCostMap.validate_model_cost_map( + fetched_map=fetched, backup_map=backup, min_model_count=10 + ) + is True + ) + + def test_should_accept_equal_size_map(self): + """A fetched map equal in size to backup should pass.""" + backup = {f"model-{i}": {} for i in range(100)} + fetched = {f"model-{i}": {} for i in range(100)} + assert ( + GetModelCostMap.validate_model_cost_map( + fetched_map=fetched, backup_map=backup, min_model_count=10 + ) + is True + ) + + +class TestGetModelCostMapFallback: + """Tests for get_model_cost_map fallback behavior with bad upstream.""" + + def test_should_fallback_to_backup_on_invalid_json(self): + """When upstream returns invalid JSON, should fall back to local backup.""" + mock_response = MagicMock() + mock_response.raise_for_status = MagicMock() + mock_response.json.side_effect = json.JSONDecodeError("bad json", "", 0) + + with patch("httpx.get", return_value=mock_response): + result = get_model_cost_map("https://fake-url.com/model_prices.json") + + # Should have fallen back to backup — backup always has models + assert isinstance(result, dict) + assert len(result) > 0 + + def test_should_fallback_to_backup_on_network_error(self): + """When upstream is unreachable, should fall back to local backup.""" + with patch("httpx.get", side_effect=Exception("Connection refused")): + result = get_model_cost_map("https://fake-url.com/model_prices.json") + + assert isinstance(result, dict) + assert len(result) > 0 + + def test_should_fallback_when_fetched_map_is_empty(self): + """When upstream returns valid JSON but empty dict, should fall back.""" + mock_response = MagicMock() + mock_response.raise_for_status = MagicMock() + mock_response.json.return_value = {} # empty map + + with patch("httpx.get", return_value=mock_response): + result = get_model_cost_map("https://fake-url.com/model_prices.json") + + # Should have fallen back to backup since empty map fails validation + assert isinstance(result, dict) + assert len(result) > 0 + + def test_should_fallback_when_fetched_map_shrinks_dramatically(self): + """When upstream returns far fewer models than backup, should fall back.""" + tiny_map = {f"model-{i}": {"litellm_provider": "test"} for i in range(11)} + mock_response = MagicMock() + mock_response.raise_for_status = MagicMock() + mock_response.json.return_value = tiny_map + + with patch("httpx.get", return_value=mock_response): + result = get_model_cost_map("https://fake-url.com/model_prices.json") + + # Backup has thousands of models; 11 is a massive shrinkage → fallback + assert len(result) > 11 + + def test_should_use_local_map_when_env_var_set(self): + """LITELLM_LOCAL_MODEL_COST_MAP=True should skip remote fetch entirely.""" + with patch.dict(os.environ, {"LITELLM_LOCAL_MODEL_COST_MAP": "True"}): + with patch("httpx.get") as mock_get: + result = get_model_cost_map( + "https://fake-url.com/model_prices.json" + ) + mock_get.assert_not_called() + + assert isinstance(result, dict) + assert len(result) > 0 + + +class TestCompletionWithBadModelCostMap: + """ + Simulates a bad model cost map and verifies litellm.completion() + still works (it catches cost map errors silently) and + litellm.get_model_info() raises the expected error. + """ + + def test_should_raise_model_not_mapped_with_empty_cost_map(self): + """ + With an empty model cost map, get_model_info should raise + 'This model isn't mapped yet' for any model. + """ + original = litellm.model_cost + litellm.model_cost = {} + try: + with pytest.raises(Exception, match="This model isn't mapped yet"): + litellm.get_model_info("azure/gpt-5.2") + finally: + litellm.model_cost = original + + def test_should_raise_model_not_mapped_for_missing_model(self): + """ + With a cost map that has some models but not the requested one, + get_model_info should raise 'This model isn't mapped yet'. + """ + original = litellm.model_cost + litellm.model_cost = { + "gpt-4": { + "litellm_provider": "openai", + "max_tokens": 8192, + "input_cost_per_token": 0.00003, + "output_cost_per_token": 0.00006, + "mode": "chat", + } + } + try: + # gpt-4 should work + info = litellm.get_model_info("gpt-4") + assert info is not None + + # azure/gpt-5.2 is not in the map — should fail + with pytest.raises(Exception, match="This model isn't mapped yet"): + litellm.get_model_info("azure/gpt-5.2") + finally: + litellm.model_cost = original + + @pytest.mark.asyncio + async def test_should_complete_even_with_bad_cost_map(self): + """ + litellm.completion() should NOT fail even when the model cost map + is empty. It catches get_model_info errors internally and proceeds + to the API call. The cost tracking will be wrong, but the LLM + request itself should succeed. + + This is the key behavior: completion() is resilient to cost map failures. + """ + original = litellm.model_cost + litellm.model_cost = {} + try: + # This should still work — completion catches model_info errors + response = litellm.completion( + model="azure/gpt-4o-mini", + messages=[{"role": "user", "content": "say hi"}], + stream=True, + ) + chunks = [] + for chunk in response: + chunks.append(chunk) + assert len(chunks) > 0, "Expected streaming chunks from completion()" + finally: + litellm.model_cost = original