scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102

init: GetModelCostMap

Browse source
This commit is contained in:
Ishaan Jaffer 2026-02-10 14:05:38 -08:00
parent 9f8878ee17
commit 240eb7583d
4 changed files with 497 additions and 22 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

151
docs/my-website/blog/model_cost_map_incident/index.md Normal file
View file

@ -0,0 +1,151 @@
---
slug: model-cost-map-incident
title: "Incident Report: Broken Model Cost Map on main"
date: 2026-02-10T10:00:00
authors:
- name: Ishaan Jaffer
title: "CTO, LiteLLM"
url: https://www.linkedin.com/in/ishaanjaffer/
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
tags: [incident-report, stability, model-cost-map]
---
# Incident Report: Broken Model Cost Map on `main`
## What happened?
2 weeks ago a contributor PR with changes to the model cost map had a poorly formatted JSON entry. When this was merged into `main` ([commit `562f0a0`](https://github.com/BerriAI/litellm/commit/562f0a028251750e3d75386bee0e630d9796d0df)) it led to the following error message reported from users.
Users found that their code started erroring out with the following message:
```
{"type":"error","error":"This model isn't mapped yet. model=gpt-5.2, custom_llm_provider=azure. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json."}
```
The bad commit added an extra `{` bracket at line 24258 of `model_prices_and_context_window.json`, making the entire file invalid JSON:
```
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 24258 column 5
```
---
## What caused this type of exception?
### How the model cost map loading works
1. At **import time**, `litellm` fetches the model cost map from GitHub `main`:
`https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json`
2. If the fetch fails (network error, invalid JSON, etc.), it **silently** falls back to a local backup file (`model_prices_and_context_window_backup.json`) bundled with the installed package
3. There is **no log or warning** when the fallback occurs -- users have no way to know they're running on stale data
### What happened in this incident
1. The bad commit broke the JSON on `main`
2. Every litellm installation (not just a specific version) fetches from `main` at import time
3. `response.json()` threw `JSONDecodeError`, caught by `except Exception`
4. Silently fell back to the local backup, which is pinned to the installed package version
5. A customer on v1.80.5 had a backup missing **661+ newer models** including `azure/gpt-5.2`
6. Any call to `get_model_info("azure/gpt-5.2")` raised `"This model isn't mapped yet"`
7. This affected **all litellm users** (not just those on a specific version), since every installation fetches the remote JSON from `main` at import time
### Impact matrix
| Call Path | Worked? | Error |
|---|---|---|
| `litellm.completion(model="azure/gpt-5.2")` | Yes -- model map error caught silently in debug logs, request still sent to Azure | |
| `litellm.completion(model="azure/gpt-5.2", stream=True)` | Yes -- same behavior, streams fine | |
| `litellm.get_model_info("azure/gpt-5.2")` | No | `This model isn't mapped yet. model=gpt-5.2, custom_llm_provider=azure.` |
| Proxy routing (request forwarding) | Yes -- routes based on config `model_list`, not cost map | |
| Proxy cost tracking / spend logging | No | `get_model_info()` fails in cost calculation callbacks, error surfaces in logging |
| Proxy `/model/info` endpoint | Partial | Returns default values (0 cost, null limits) for unmapped models |
### Key finding
`litellm.completion()` **never blocks** on a missing model in the cost map. It catches the `get_model_info()` error and proceeds to the API. The customer error surfaced from the **cost tracking / logging callbacks** path, where `get_model_info()` is called for spend calculation.
---
## Root cause analysis
### Why did this happen?
- No CI tests validating the JSON structure of `model_prices_and_context_window.json` before merge
- The fallback in `get_model_cost_map()` is **completely silent** -- no log, no metric, no warning
- The backup file can be arbitrarily stale depending on the installed package version
### Why fetch from GitHub?
To get live day-0 model updates (new model pricing, context windows) without requiring a package upgrade. This is valuable but creates a hard dependency on the correctness of a file on `main`.
---
## Shippable improvements
| # | Improvement | Status | Details |
|---|---|---|---|
| 1 | **CI validation for model cost map JSON** | Shipped | [PR #20605](https://github.com/BerriAI/litellm/pull/20605) -- Validates JSON schema + structure on every PR that touches `model_prices_and_context_window.json` |
| 2 | **Warning logging on fallback** | Shipped | `get_model_cost_map()` now logs a `WARNING` when the remote fetch fails and falls back to the local backup, instead of silently swallowing the error. See `litellm/litellm_core_utils/get_model_cost_map.py` |
| 3 | **Fetched JSON integrity validation** | Shipped | New `validate_model_cost_map()` helper in `litellm/litellm_core_utils/get_model_cost_map.py` checks: (a) fetched map is a dict, (b) has minimum model count, (c) hasn't shrunk >50% vs backup. If any check fails, falls back to backup with a warning |
| 4 | **CI/CD test for bad cost map resilience** | Shipped | `tests/llm_translation/test_model_cost_map_resilience.py` -- 13 tests covering: empty map, invalid JSON, network errors, shrinkage detection, `get_model_info()` error messages, and `litellm.completion()` resilience |
| 5 | Keep backup file in sync on every release | Planned | Update `model_prices_and_context_window_backup.json` as part of the release process so fallback data is never more than 1 release behind |
| 6 | `LITELLM_LOCAL_MODEL_COST_MAP=True` as default for production | Planned | Eliminates runtime GitHub dependency. Users who want live updates can opt in |
| 7 | Health check endpoint for external deps | Planned | Proxy endpoint (e.g., `/health/dependencies`) reporting status of all external fetches |
---
## Other upstream dependencies in the codebase
The model cost map is not the only external dependency that could impact LLM calls. Here is a full audit:
### Critical (can block LLM calls or auth)
| Dependency | File | URL | When Fetched | Fallback | Silent? |
|---|---|---|---|---|---|
| **Model cost map** | `litellm/__init__.py` | `raw.githubusercontent.com/.../model_prices_and_context_window.json` | Import time | Local backup file | Yes (now logs warning) |
| **Model cost map reload** | `litellm/proxy/proxy_server.py` | Same as above | Runtime (scheduled / manual) | Keeps existing map | No (logged) |
| **JWT public keys** | `litellm/proxy/auth/handle_jwt.py` | Configurable via `JWT_PUBLIC_KEY_URL` | Runtime (on-demand, cached with TTL) | **None -- raises exception** | No (exception) |
| **OIDC UserInfo** | `litellm/proxy/auth/handle_jwt.py` | Configurable via `oidc_userinfo_endpoint` | Runtime (on-demand, cached 300s) | **None -- raises exception** | No (exception) |
### Medium impact
| Dependency | File | URL | When Fetched | Fallback | Silent? |
|---|---|---|---|---|---|
| **HuggingFace provider mapping** | `litellm/llms/huggingface/common_utils.py` | `huggingface.co/api/models/{model}` | Runtime (on-demand, LRU cached) | Raises `HuggingFaceError` | No |
### Low impact (non-blocking)
| Dependency | File | URL | When Fetched | Fallback | Silent? |
|---|---|---|---|---|---|
| **Ollama model tags** | `litellm/llms/ollama/common_utils.py` | `{ollama_base}/api/tags` (localhost) | Runtime (on-demand) | Static model list | Warning logged |
| **Together AI model info** | `litellm/litellm_core_utils/prompt_templates/factory.py` | `api.together.xyz/models/info` | Runtime (on-demand) | Returns `None` | Yes (silent) |
| **AssemblyAI transcript polling** | `litellm/proxy/pass_through_endpoints/...` | `api.assemblyai.com/v2/transcript/{id}` | Runtime (on-demand) | Returns `None` | Logged |
---
## How to prevent this type of error again
### Shipped fixes
1. **JSON validation CI check** ([PR #20605](https://github.com/BerriAI/litellm/pull/20605)) -- Runs `json.loads()` and schema validation on `model_prices_and_context_window.json` on every PR. This would have caught the bad commit before merge.
2. **Warning logging on fallback** -- When `get_model_cost_map()` falls back to the backup, it now logs a `WARNING`:
```
LiteLLM: Failed to fetch remote model cost map from <url>: <error>. Falling back to local backup.
```
3. **Fetched JSON integrity validation** -- New `validate_model_cost_map()` helper validates the fetched map before using it. Catches: non-dict responses, empty maps, and maps that have shrunk significantly compared to the backup.
4. **CI/CD resilience tests** -- 13 tests in `tests/llm_translation/test_model_cost_map_resilience.py` that simulate bad upstream, bad backup, and verify `litellm.completion()` and `litellm.get_model_info()` behavior.
### Planned improvements
5. **Keep the backup file in sync** -- Update `model_prices_and_context_window_backup.json` more frequently (e.g., on every release) so the fallback has recent models.
6. **Consider `LITELLM_LOCAL_MODEL_COST_MAP=True` as default for production** -- This eliminates the runtime dependency on GitHub entirely. Users who want live updates can opt in.
7. **Audit all upstream dependencies** -- Apply the same resilience patterns (fallback, logging, validation) to the other external dependencies listed above, especially:
- JWT public key fetch (no fallback today -- should add retry + caching)
- OIDC UserInfo fetch (no fallback today -- should add graceful degradation)
8. **Add health check for external dependencies** -- A proxy endpoint (e.g., `/health/dependencies`) that reports the status of all external fetches: whether the model cost map was loaded from remote or backup, whether JWT keys are fresh, etc.

11
docs/my-website/sidebars.js
View file

@ -1085,6 +1085,17 @@ const sidebars = {
"troubleshoot/max_callbacks",
],
},
{
type: "category",
label: "Blog",
items: [
{
type: "link",
label: "Incident: Broken Model Cost Map",
href: "/blog/model-cost-map-incident",
},
],
},
],
};

139
litellm/litellm_core_utils/get_model_cost_map.py
View file

@ -8,40 +8,135 @@ export LITELLM_LOCAL_MODEL_COST_MAP=True
```
"""
import json
import logging
import os
from importlib.resources import files
import httpx
logger = logging.getLogger("LiteLLM")
class GetModelCostMap:
"""
Handles fetching, validating, and loading the model cost map.
All methods are static — no instance state is needed. This class groups
the helpers that support `get_model_cost_map()` into a single namespace.
"""
@staticmethod
def load_local_model_cost_map() -> dict:
"""Load the local backup model cost map bundled with the package."""
content = json.loads(
files("litellm")
.joinpath("model_prices_and_context_window_backup.json")
.read_text(encoding="utf-8")
)
return content
@staticmethod
def validate_model_cost_map(
fetched_map: dict,
backup_map: dict,
min_model_count: int = 10,
max_shrink_pct: float = 0.5,
) -> bool:
"""
Validate the integrity of a fetched model cost map.
Checks:
1. The fetched map is a non-empty dict.
2. It has at least ``min_model_count`` models.
3. It did not shrink by more than ``max_shrink_pct`` (50%) compared to the backup.
Returns True if the fetched map looks valid, False otherwise.
"""
if not isinstance(fetched_map, dict):
logger.warning(
"LiteLLM: Fetched model cost map is not a dict (type=%s). "
"Falling back to local backup.",
type(fetched_map).__name__,
)
return False
fetched_count = len(fetched_map)
if fetched_count < min_model_count:
logger.warning(
"LiteLLM: Fetched model cost map has only %d models (minimum=%d). "
"This may indicate a corrupted upstream file. "
"Falling back to local backup.",
fetched_count,
min_model_count,
)
return False
backup_count = len(backup_map) if isinstance(backup_map, dict) else 0
if backup_count > 0 and fetched_count < backup_count * max_shrink_pct:
logger.warning(
"LiteLLM: Fetched model cost map shrank significantly "
"(fetched=%d, backup=%d, threshold=%.0f%%). "
"This may indicate a corrupted upstream file. "
"Falling back to local backup.",
fetched_count,
backup_count,
max_shrink_pct * 100,
)
return False
return True
@staticmethod
def fetch_remote_model_cost_map(url: str, timeout: int = 5) -> dict:
"""
Fetch the model cost map from a remote URL.
Returns the parsed JSON dict. Raises on network/parse errors
(caller is expected to handle).
"""
response = httpx.get(url, timeout=timeout)
response.raise_for_status()
return response.json()
def get_model_cost_map(url: str) -> dict:
"""
Public entry point — returns the model cost map dict.
1. If ``LITELLM_LOCAL_MODEL_COST_MAP`` is set, uses the local backup only.
2. Otherwise fetches from ``url``, validates integrity, and falls back
to the local backup on any failure.
"""
if (
os.getenv("LITELLM_LOCAL_MODEL_COST_MAP", False)
or os.getenv("LITELLM_LOCAL_MODEL_COST_MAP", False) == "True"
):
from importlib.resources import files
import json
return GetModelCostMap.load_local_model_cost_map()
content = json.loads(
files("litellm")
.joinpath("model_prices_and_context_window_backup.json")
.read_text(encoding="utf-8")
)
return content
backup_map = GetModelCostMap.load_local_model_cost_map()
try:
response = httpx.get(
url, timeout=5
) # set a 5 second timeout for the get request
response.raise_for_status() # Raise an exception if the request is unsuccessful
content = response.json()
return content
except Exception:
from importlib.resources import files
import json
content = GetModelCostMap.fetch_remote_model_cost_map(url)
# Validate fetched JSON integrity before using it
if not GetModelCostMap.validate_model_cost_map(
fetched_map=content, backup_map=backup_map
):
logger.warning(
"LiteLLM: Fetched model cost map failed integrity check. "
"Using local backup instead. url=%s",
url,
)
return backup_map
content = json.loads(
files("litellm")
.joinpath("model_prices_and_context_window_backup.json")
.read_text(encoding="utf-8")
)
return content
except Exception as e:
logger.warning(
"LiteLLM: Failed to fetch remote model cost map from %s: %s. "
"Falling back to local backup.",
url,
str(e),
)
return backup_map

218
tests/llm_translation/test_model_cost_map_resilience.py Normal file
View file

@ -0,0 +1,218 @@
"""
Tests for model cost map resilience.
Simulates:
- A bad (invalid JSON) model cost map upstream
- A bad (empty/missing) backup model cost map
- Verifies litellm.completion() still works even with a broken cost map
- Verifies litellm.get_model_info() raises the expected error for unmapped models
- Verifies the integrity validation helper catches corrupted maps
"""
import json
import os
import sys
from unittest.mock import MagicMock, patch
import pytest
sys.path.insert(
0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../.."))
)
import litellm
from litellm.litellm_core_utils.get_model_cost_map import (
GetModelCostMap,
get_model_cost_map,
)
class TestValidateModelCostMap:
"""Unit tests for the validate_model_cost_map helper."""
def test_should_reject_non_dict(self):
"""Non-dict fetched map should fail validation."""
assert GetModelCostMap.validate_model_cost_map(fetched_map="not a dict", backup_map={}) is False
def test_should_reject_empty_map(self):
"""Empty fetched map should fail validation."""
assert GetModelCostMap.validate_model_cost_map(fetched_map={}, backup_map={}) is False
def test_should_reject_too_few_models(self):
"""Fetched map with fewer models than min_model_count should fail."""
small_map = {f"model-{i}": {} for i in range(5)}
assert (
GetModelCostMap.validate_model_cost_map(
fetched_map=small_map, backup_map={}, min_model_count=10
)
is False
)
def test_should_reject_significant_shrinkage(self):
"""Fetched map that shrunk >50% vs backup should fail."""
backup = {f"model-{i}": {} for i in range(100)}
fetched = {f"model-{i}": {} for i in range(40)} # 40% of backup
assert (
GetModelCostMap.validate_model_cost_map(
fetched_map=fetched, backup_map=backup, min_model_count=10
)
is False
)
def test_should_accept_valid_map(self):
"""A fetched map with enough models that hasn't shrunk should pass."""
backup = {f"model-{i}": {} for i in range(100)}
fetched = {f"model-{i}": {} for i in range(120)}
assert (
GetModelCostMap.validate_model_cost_map(
fetched_map=fetched, backup_map=backup, min_model_count=10
)
is True
)
def test_should_accept_equal_size_map(self):
"""A fetched map equal in size to backup should pass."""
backup = {f"model-{i}": {} for i in range(100)}
fetched = {f"model-{i}": {} for i in range(100)}
assert (
GetModelCostMap.validate_model_cost_map(
fetched_map=fetched, backup_map=backup, min_model_count=10
)
is True
)
class TestGetModelCostMapFallback:
"""Tests for get_model_cost_map fallback behavior with bad upstream."""
def test_should_fallback_to_backup_on_invalid_json(self):
"""When upstream returns invalid JSON, should fall back to local backup."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_response.json.side_effect = json.JSONDecodeError("bad json", "", 0)
with patch("httpx.get", return_value=mock_response):
result = get_model_cost_map("https://fake-url.com/model_prices.json")
# Should have fallen back to backup — backup always has models
assert isinstance(result, dict)
assert len(result) > 0
def test_should_fallback_to_backup_on_network_error(self):
"""When upstream is unreachable, should fall back to local backup."""
with patch("httpx.get", side_effect=Exception("Connection refused")):
result = get_model_cost_map("https://fake-url.com/model_prices.json")
assert isinstance(result, dict)
assert len(result) > 0
def test_should_fallback_when_fetched_map_is_empty(self):
"""When upstream returns valid JSON but empty dict, should fall back."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_response.json.return_value = {} # empty map
with patch("httpx.get", return_value=mock_response):
result = get_model_cost_map("https://fake-url.com/model_prices.json")
# Should have fallen back to backup since empty map fails validation
assert isinstance(result, dict)
assert len(result) > 0
def test_should_fallback_when_fetched_map_shrinks_dramatically(self):
"""When upstream returns far fewer models than backup, should fall back."""
tiny_map = {f"model-{i}": {"litellm_provider": "test"} for i in range(11)}
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_response.json.return_value = tiny_map
with patch("httpx.get", return_value=mock_response):
result = get_model_cost_map("https://fake-url.com/model_prices.json")
# Backup has thousands of models; 11 is a massive shrinkage → fallback
assert len(result) > 11
def test_should_use_local_map_when_env_var_set(self):
"""LITELLM_LOCAL_MODEL_COST_MAP=True should skip remote fetch entirely."""
with patch.dict(os.environ, {"LITELLM_LOCAL_MODEL_COST_MAP": "True"}):
with patch("httpx.get") as mock_get:
result = get_model_cost_map(
"https://fake-url.com/model_prices.json"
)
mock_get.assert_not_called()
assert isinstance(result, dict)
assert len(result) > 0
class TestCompletionWithBadModelCostMap:
"""
Simulates a bad model cost map and verifies litellm.completion()
still works (it catches cost map errors silently) and
litellm.get_model_info() raises the expected error.
"""
def test_should_raise_model_not_mapped_with_empty_cost_map(self):
"""
With an empty model cost map, get_model_info should raise
'This model isn't mapped yet' for any model.
"""
original = litellm.model_cost
litellm.model_cost = {}
try:
with pytest.raises(Exception, match="This model isn't mapped yet"):
litellm.get_model_info("azure/gpt-5.2")
finally:
litellm.model_cost = original
def test_should_raise_model_not_mapped_for_missing_model(self):
"""
With a cost map that has some models but not the requested one,
get_model_info should raise 'This model isn't mapped yet'.
"""
original = litellm.model_cost
litellm.model_cost = {
"gpt-4": {
"litellm_provider": "openai",
"max_tokens": 8192,
"input_cost_per_token": 0.00003,
"output_cost_per_token": 0.00006,
"mode": "chat",
}
}
try:
# gpt-4 should work
info = litellm.get_model_info("gpt-4")
assert info is not None
# azure/gpt-5.2 is not in the map — should fail
with pytest.raises(Exception, match="This model isn't mapped yet"):
litellm.get_model_info("azure/gpt-5.2")
finally:
litellm.model_cost = original
@pytest.mark.asyncio
async def test_should_complete_even_with_bad_cost_map(self):
"""
litellm.completion() should NOT fail even when the model cost map
is empty. It catches get_model_info errors internally and proceeds
to the API call. The cost tracking will be wrong, but the LLM
request itself should succeed.
This is the key behavior: completion() is resilient to cost map failures.
"""
original = litellm.model_cost
litellm.model_cost = {}
try:
# This should still work — completion catches model_info errors
response = litellm.completion(
model="azure/gpt-4o-mini",
messages=[{"role": "user", "content": "say hi"}],
stream=True,
)
chunks = []
for chunk in response:
chunks.append(chunk)
assert len(chunks) > 0, "Expected streaming chunks from completion()"
finally:
litellm.model_cost = original