ci(proxy): fail the build when the lazy OpenAPI snapshot goes stale

_lazy_openapi_snapshot.json is not just a Swagger nicety. The proxy lazy-loads
heavy feature modules, so their routes are never registered on the app; both
/openapi.json and `npm run gen:api` fall back to this committed file for them.
That makes the snapshot an *input* to the dashboard's schema.d.ts, which is why
the existing "Check UI API Types Sync" job could never catch drift here: it
regenerates the types from the same stale snapshot and finds them in sync.

The result was silent rot. Running the generator on the current tree rewrites
31 of 32 fragments: two features (a2a_registration, gemini_agents) were never
snapshotted at all, one (vector_store_management) no longer exists, and the
rest carry schemas that no longer match their Pydantic models. Every dashboard
type derived from a lazy route has been generated from stale data.

Add `--check`, which regenerates in memory and diffs against the committed
file, and run it from a new workflow. Verification is strict about a feature
that fails to import: a skipped module would otherwise look exactly like a
deleted fragment and quietly weaken the check, so it errors instead.

Regenerate the snapshot so the check passes, and regenerate schema.d.ts, which
the snapshot feeds. The module docstring claimed CI already verified this; it
did not, and now it does.

.github/workflows/check-lazy-openapi-snapshot.yml

@@ -0,0 +1,59 @@
+name: Check Lazy OpenAPI Snapshot Sync
+
+on:
+  pull_request:
+    paths:
+      - "litellm/proxy/**"
+      - "litellm/types/**"
+      - "enterprise/**"
+      - "uv.lock"
+      - ".github/workflows/check-lazy-openapi-snapshot.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  check-sync:
+    name: Verify _lazy_openapi_snapshot.json matches the proxy routes
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        with:
+          version: "0.10.9"
+
+      - name: Cache uv dependencies
+        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+        with:
+          path: |
+            ~/.cache/uv
+            .venv
+          key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-uv-
+
+      - name: Install backend dependencies
+        run: .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
+
+      - name: Generate Prisma client
+        env:
+          PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
+        run: uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
+
+      - name: Fail if the snapshot is stale
+        run: |
+          if ! uv run --no-sync python -m litellm.proxy._lazy_openapi_snapshot --check; then
+            echo "::error file=litellm/proxy/_lazy_openapi_snapshot.json::Lazy OpenAPI snapshot is out of sync with the proxy routes."
+            exit 1
+          fi

litellm/proxy/_lazy_openapi_snapshot.py

@@ -3,15 +3,20 @@ Per-feature OpenAPI snapshot for lazy-loaded routers.
 
 The committed JSON is generated by `python -m litellm.proxy._lazy_openapi_snapshot`
 and consumed at runtime so /openapi.json can show full route info for unloaded
-features without importing them. CI verifies the file is current and surfaces
-any drift as a neutral check.
+features without importing them.
+
+The snapshot is also an *input* to the dashboard's generated types: `npm run gen:api`
+dumps the proxy spec without loading the lazy features, so their paths and schemas
+come from this file verbatim. A stale fragment therefore ships stale `schema.d.ts`
+types, and the UI types-sync check cannot catch it (it regenerates from this same
+file). `--check` closes that loop; the Lazy OpenAPI Snapshot Sync workflow runs it.
 """
 
 import json
 import re
 import sys
 from pathlib import Path
-from typing import Dict, Optional, Set
+from typing import Dict, Optional, Sequence, Set
 
 SNAPSHOT_FILE = Path(__file__).parent / "_lazy_openapi_snapshot.json"
 HTTP_METHOD_SUFFIXES = {
@@ -80,22 +85,42 @@ def _normalize_operation_ids(paths: Dict[str, Dict]) -> None:
                     break
 
 
-def generate_snapshot() -> Dict[str, Dict]:
+def _register_lazy_feature(app, feat) -> tuple[str, str] | None:
+    """Import and register one lazy feature. Returns (name, error) if it could not be loaded."""
     import importlib
 
+    if feat.module_path in sys.modules:
+        return None
+    try:
+        feat.register_fn(app, importlib.import_module(feat.module_path))
+        return None
+    except Exception as exc:
+        return (feat.name, f"{type(exc).__name__}: {exc}")
+
+
+def generate_snapshot(*, strict: bool = False) -> dict[str, dict]:
+    """Build a fragment per lazy feature.
+
+    A feature whose module fails to import is skipped, which yields an incomplete
+    snapshot. That is tolerable when writing the file by hand but not when verifying
+    it, so `strict` turns a skip into an error rather than silent drift.
+    """
     from fastapi.openapi.utils import get_openapi
 
     from litellm.proxy._lazy_features import LAZY_FEATURES
     from litellm.proxy.proxy_server import app, ensure_unique_openapi_operation_ids
 
-    for feat in LAZY_FEATURES:
-        if feat.module_path in sys.modules:
-            continue
-        try:
-            module = importlib.import_module(feat.module_path)
-            feat.register_fn(app, module)
-        except Exception as exc:
-            sys.stderr.write(f"warning: skip {feat.name}: {exc}\n")
+    skipped = tuple(
+        result for result in (_register_lazy_feature(app, feat) for feat in LAZY_FEATURES) if result is not None
+    )
+    for name, error in skipped:
+        sys.stderr.write(f"warning: skip {name}: {error}\n")
+    if strict and skipped:
+        raise RuntimeError(
+            "cannot verify the snapshot: "
+            + ", ".join(f"{name} ({error})" for name, error in skipped)
+            + ". Install the extras these features need, then re-run."
+        )
 
     fragments: Dict[str, Dict] = {}
     used_operation_ids: Set[str] = set()
@@ -126,7 +151,49 @@ def generate_snapshot() -> Dict[str, Dict]:
     return fragments
 
 
-if __name__ == "__main__":
+REGEN_COMMAND = "python -m litellm.proxy._lazy_openapi_snapshot"
+
+
+def serialize_snapshot(fragments: dict[str, dict]) -> str:
+    return json.dumps(fragments, indent=2, sort_keys=True) + "\n"
+
+
+def drifted_features(committed: dict[str, dict], fresh: dict[str, dict]) -> tuple[str, ...]:
+    """Feature names whose fragment was added, removed, or changed."""
+    return tuple(sorted(name for name in set(committed) | set(fresh) if committed.get(name) != fresh.get(name)))
+
+
+def check_snapshot() -> int:
+    """Regenerate in memory and compare against the committed file. 0 when in sync."""
+    fresh = generate_snapshot(strict=True)
+    committed = load_snapshot()
+
+    if committed is not None and serialize_snapshot(committed) == serialize_snapshot(fresh):
+        sys.stdout.write(f"{SNAPSHOT_FILE.name} is in sync ({len(fresh)} feature fragments).\n")
+        return 0
+
+    drifted = drifted_features(committed or {}, fresh)
+    sys.stderr.write(
+        f"{SNAPSHOT_FILE.name} is out of sync with the proxy routes.\n\n"
+        "This file feeds /openapi.json for unloaded features AND the dashboard's generated\n"
+        "types, so stale fragments ship stale schema.d.ts.\n\n"
+        f"Drifted features ({len(drifted)}): {', '.join(drifted) or 'formatting only'}\n\n"
+        f"To fix, run:\n  {REGEN_COMMAND}\n"
+        "then commit the updated litellm/proxy/_lazy_openapi_snapshot.json (and re-run\n"
+        "`npm run gen:api` in ui/litellm-dashboard if schema.d.ts changes).\n"
+    )
+    return 1
+
+
+def _main(argv: Sequence[str]) -> int:
+    if "--check" in argv:
+        return check_snapshot()
+
     fragments = generate_snapshot()
-    SNAPSHOT_FILE.write_text(json.dumps(fragments, indent=2, sort_keys=True) + "\n")
+    SNAPSHOT_FILE.write_text(serialize_snapshot(fragments))
     sys.stdout.write(f"wrote {len(fragments)} feature fragments to {SNAPSHOT_FILE}\n")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(_main(sys.argv[1:]))

ruff-strict-budget.json

@@ -324,7 +324,7 @@
     "limit": 883
   },
   "UP006": {
-    "limit": 12792
+    "limit": 12790
   },
   "UP007": {
     "limit": 2570

tests/test_litellm/proxy/test_lazy_openapi_snapshot.py

@@ -1,7 +1,16 @@
 import sys
 from types import ModuleType, SimpleNamespace
 
-from litellm.proxy._lazy_openapi_snapshot import _normalize_operation_ids
+import pytest
+
+from litellm.proxy import _lazy_openapi_snapshot as snapshot_module
+from litellm.proxy._lazy_openapi_snapshot import (
+    _normalize_operation_ids,
+    _register_lazy_feature,
+    check_snapshot,
+    drifted_features,
+    serialize_snapshot,
+)
 
 
 def test_generate_snapshot_uses_shared_operation_id_reservations(monkeypatch):
@@ -35,9 +44,7 @@ def test_generate_snapshot_uses_shared_operation_id_reservations(monkeypatch):
             register_fn=lambda app, module: None,
         ),
     ]
-    monkeypatch.setitem(
-        sys.modules, "litellm.proxy._lazy_features", fake_lazy_features_module
-    )
+    monkeypatch.setitem(sys.modules, "litellm.proxy._lazy_features", fake_lazy_features_module)
 
     def fake_get_openapi(title, version, routes):
         path = routes[0].path
@@ -58,30 +65,16 @@ def test_generate_snapshot_uses_shared_operation_id_reservations(monkeypatch):
 
     fake_proxy_server_module = ModuleType("litellm.proxy.proxy_server")
     fake_proxy_server_module.app = fake_app
-    fake_proxy_server_module.ensure_unique_openapi_operation_ids = (
-        fake_ensure_unique_openapi_operation_ids
-    )
-    monkeypatch.setitem(
-        sys.modules, "litellm.proxy.proxy_server", fake_proxy_server_module
-    )
+    fake_proxy_server_module.ensure_unique_openapi_operation_ids = fake_ensure_unique_openapi_operation_ids
+    monkeypatch.setitem(sys.modules, "litellm.proxy.proxy_server", fake_proxy_server_module)
     monkeypatch.setattr("fastapi.openapi.utils.get_openapi", fake_get_openapi)
 
     fragments = _lazy_openapi_snapshot.generate_snapshot()
 
-    assert (
-        fragments["feature-a"]["paths"]["/feature-a/items"]["get"]["operationId"]
-        == "shared_operation_id_get"
-    )
-    assert (
-        fragments["feature-b"]["paths"]["/feature-b/items"]["get"]["operationId"]
-        == "shared_operation_id_get_2"
-    )
-    assert fragments["feature-a"]["paths"]["/feature-a/items"]["get"]["tags"] == [
-        "feature-a"
-    ]
-    assert fragments["feature-b"]["paths"]["/feature-b/items"]["get"]["tags"] == [
-        "feature-b"
-    ]
+    assert fragments["feature-a"]["paths"]["/feature-a/items"]["get"]["operationId"] == "shared_operation_id_get"
+    assert fragments["feature-b"]["paths"]["/feature-b/items"]["get"]["operationId"] == "shared_operation_id_get_2"
+    assert fragments["feature-a"]["paths"]["/feature-a/items"]["get"]["tags"] == ["feature-a"]
+    assert fragments["feature-b"]["paths"]["/feature-b/items"]["get"]["tags"] == ["feature-b"]
 
 
 def test_normalize_operation_ids_uses_each_http_method():
@@ -116,3 +109,105 @@ def test_normalize_operation_ids_preserves_custom_ids():
     operations = paths["/proxy/{endpoint}"]
     assert operations["get"]["operationId"] == "custom_operation"
     assert operations["post"]["operationId"] == "custom_operation"
+
+
+class TestDriftedFeatures:
+    def test_reports_changed_added_and_removed_fragments(self):
+        committed = {"kept": {"paths": {}}, "changed": {"paths": {"/a": {}}}, "removed": {"paths": {}}}
+        fresh = {"kept": {"paths": {}}, "changed": {"paths": {"/b": {}}}, "added": {"paths": {}}}
+
+        assert drifted_features(committed, fresh) == ("added", "changed", "removed")
+
+    def test_identical_snapshots_have_no_drift(self):
+        fragments = {"a": {"paths": {"/a": {}}, "components": {"schemas": {}}}}
+
+        assert drifted_features(fragments, dict(fragments)) == ()
+
+
+class TestRegisterLazyFeature:
+    def test_skips_a_module_that_is_already_imported(self):
+        feature = SimpleNamespace(name="ok", module_path="json", register_fn=lambda app, module: None)
+
+        assert _register_lazy_feature(SimpleNamespace(), feature) is None
+
+    def test_registers_a_module_that_imports_cleanly(self, monkeypatch):
+        registered = []
+        feature = SimpleNamespace(
+            name="ok", module_path="json", register_fn=lambda app, module: registered.append(module.__name__)
+        )
+        monkeypatch.delitem(sys.modules, "json", raising=False)
+
+        assert _register_lazy_feature(SimpleNamespace(), feature) is None
+        assert registered == ["json"]
+
+    def test_reports_the_feature_and_error_when_the_module_is_missing(self):
+        feature = SimpleNamespace(
+            name="broken", module_path="litellm_no_such_module", register_fn=lambda app, module: None
+        )
+
+        result = _register_lazy_feature(SimpleNamespace(), feature)
+
+        assert result is not None
+        name, error = result
+        assert name == "broken"
+        assert "ModuleNotFoundError" in error
+
+    def test_reports_the_error_when_registration_raises(self, monkeypatch):
+        def _raise(app, module):
+            raise ValueError("router blew up")
+
+        feature = SimpleNamespace(name="broken", module_path="json", register_fn=_raise)
+        monkeypatch.delitem(sys.modules, "json", raising=False)
+
+        assert _register_lazy_feature(SimpleNamespace(), feature) == ("broken", "ValueError: router blew up")
+
+
+class TestCheckSnapshot:
+    """The check exists to fail when the committed file drifts; prove it does."""
+
+    def test_passes_when_the_committed_file_matches(self, monkeypatch):
+        fragments = {"guardrails": {"paths": {"/g": {}}, "components": {"schemas": {}}}}
+        monkeypatch.setattr(snapshot_module, "generate_snapshot", lambda *, strict=False: fragments)
+        monkeypatch.setattr(snapshot_module, "load_snapshot", lambda: dict(fragments))
+
+        assert check_snapshot() == 0
+
+    def test_fails_when_a_fragment_drifted(self, monkeypatch, capsys):
+        monkeypatch.setattr(
+            snapshot_module,
+            "generate_snapshot",
+            lambda *, strict=False: {"guardrails": {"paths": {"/new": {}}, "components": {"schemas": {}}}},
+        )
+        monkeypatch.setattr(
+            snapshot_module,
+            "load_snapshot",
+            lambda: {"guardrails": {"paths": {"/old": {}}, "components": {"schemas": {}}}},
+        )
+
+        assert check_snapshot() == 1
+        assert "guardrails" in capsys.readouterr().err
+
+    def test_fails_when_the_file_is_absent(self, monkeypatch):
+        monkeypatch.setattr(snapshot_module, "generate_snapshot", lambda *, strict=False: {"a": {"paths": {}}})
+        monkeypatch.setattr(snapshot_module, "load_snapshot", lambda: None)
+
+        assert check_snapshot() == 1
+
+    def test_verification_refuses_to_run_on_an_incomplete_regen(self, monkeypatch):
+        """A feature that fails to import would silently look like a deleted fragment."""
+
+        def _boom(*, strict: bool = False):
+            raise RuntimeError("cannot verify the snapshot: mcp_app (ImportError: no module)")
+
+        monkeypatch.setattr(snapshot_module, "generate_snapshot", _boom)
+
+        with pytest.raises(RuntimeError, match="cannot verify the snapshot"):
+            check_snapshot()
+
+
+class TestSerializeSnapshot:
+    def test_is_stable_regardless_of_key_order(self):
+        assert serialize_snapshot({"b": {"x": 1}, "a": {"y": 2}}) == serialize_snapshot({"a": {"y": 2}, "b": {"x": 1}})
+
+    def test_ends_with_a_trailing_newline(self):
+        assert serialize_snapshot({"a": {}}).endswith("}\n")