litellm/tests/claude_code/resolver.py
mateo-berri e87e5c1199 RALPH: compat matrix slice 4 - daily cron VM publishes matrix to docs (#26480, PRD #26476)
Slice 4 of the Claude Code Compatibility Matrix: stand up the daily-cron
pipeline that publishes `compatibility-matrix.json` to the docs repo. After
this slice lands, the hand-authored matrix in the docs repo is replaced by
auto-generated output, and the docs page begins reflecting real test runs
against the latest stable LiteLLM release.

What landed:

- tests/claude_code/resolver.py
  Latest Stable LiteLLM Resolver. Calls the GitHub Releases API and
  returns the newest tag matching `v*-stable`. Sort is numeric on
  (major, minor, patch) so v1.10.0-stable correctly outranks
  v1.9.5-stable. Injectable `http_get` so tests run offline.

- tests/claude_code/publisher.py
  Daily-cron orchestrator. Resolves the latest stable tag, pulls
  `ghcr.io/berriai/litellm:<tag>`, starts it as the proxy, installs
  `@anthropic-ai/claude-code@latest`, runs `pytest tests/claude_code/`,
  invokes the Matrix JSON Builder, and direct-pushes
  `compatibility-matrix.json` to the docs repo's main branch using a
  GitHub App installation token (`DOCS_REPO_TOKEN`). Idempotent: a no-op
  if the JSON is byte-identical to what's already on main.

- tests/claude_code/_publisher_unit_tests/test_resolver.py
  test_publisher.py
  14 unit tests covering the small pure helpers — version sort,
  non-stable filtering, http-get injection, commit message determinism,
  Docker image-name builder, and the file allowlist that enforces the
  "only `compatibility-matrix.json` ever ships" guarantee. Per the PRD's
  "Testing Decisions" section, the publisher's full subprocess
  orchestration intentionally ships without a unit-test harness; the
  daily-cron failure surface is itself the test.

- .github/workflows/claude_code_compat_matrix.yml
  GitHub Actions workflow with three triggers (daily cron at 06:00 UTC,
  `release: published` filtered to `*-stable` tags, and
  `workflow_dispatch`). Mints a docs-repo installation token from a
  GitHub App scoped to `BerriAI/litellm-docs` only with `contents:
  write`, then runs the publisher.

- .gitignore
  Add `compatibility-matrix.json` (cron VM output).

Key decisions:

- "Isolated VM" is realized as a GitHub-hosted ubuntu-latest runner —
  every run gets a fresh ephemeral VM, and the always-latest Claude
  Code CLI is only ever installed inside that ephemeral environment,
  so a malicious or broken Claude Code release cannot affect the
  trusted PR-gate CI in CircleCI.
- File-level restriction on the GitHub App's broad `contents: write`
  scope is enforced by `select_files_to_commit` (script correctness),
  per the PRD's explicit acknowledgement that GitHub does not support
  file-path-scoped tokens.
- `release` runs are filtered to tags ending in `-stable` at the
  workflow level, so a `v1.84.0-rc1` release does not republish the
  matrix.
- Resolver and publisher live under `tests/claude_code/` alongside
  `matrix_builder.py` and `cli_driver.py` — production code that
  supports the test suite, kept colocated with it to match the slice
  1+2 layout.

Out of scope / blockers for next iteration:

- Provisioning the GitHub App itself (creating it under BerriAI's
  org, installing it on litellm-docs only, generating the private key
  and registering `COMPAT_MATRIX_APP_ID` / `COMPAT_MATRIX_APP_PRIVATE_KEY`
  as repo secrets) is an operator/infra step that cannot land via a
  code change in this repo.
- The first successful cron run is what removes the hand-authored
  `compatibility-matrix.json` from the docs repo and replaces it with
  generated output — that happens after this PR merges and the App is
  installed; not a code change here.

Tests: 34 -> 45 passing (added 7 resolver tests + 7 publisher helper
tests, all unit-only and offline). The 12 per-cell failures under
`tests/claude_code/basic_messaging_non_streaming/` remain by design —
they require a running proxy which the cron VM provides.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-25 05:03:16 +00:00

91 lines
2.9 KiB
Python

"""Latest Stable LiteLLM Resolver.
Queries the GitHub Releases API for `BerriAI/litellm` and returns the
newest tag matching `v*-stable`. Used by the daily-cron publisher to
decide which Docker image to pull for the matrix run.
The resolver is intentionally tiny — its only state is the GitHub API
URL constant — and exposes a single public function so unit tests can
inject a fake HTTP getter and run offline.
"""
from __future__ import annotations
import json
import re
import urllib.request
from typing import Callable, List, Optional
GITHUB_RELEASES_URL = "https://api.github.com/repos/BerriAI/litellm/releases"
USER_AGENT = "litellm-compat-matrix-resolver"
REQUEST_TIMEOUT_SECONDS = 30
STABLE_TAG_RE = re.compile(r"^v(\d+)\.(\d+)\.(\d+)-stable$")
class ResolverError(RuntimeError):
"""Raised when the resolver cannot determine a latest-stable tag."""
def _default_http_get(url: str, *, token: Optional[str] = None) -> str:
"""Minimal urllib-based GET that the cron VM can call without extra deps.
Forwards `token` as a Bearer header when set so the cron job can lift
the unauthenticated GitHub rate limit by passing the GitHub App
installation token.
"""
request = urllib.request.Request(url)
request.add_header("User-Agent", USER_AGENT)
request.add_header("Accept", "application/vnd.github+json")
if token:
request.add_header("Authorization", f"Bearer {token}")
with urllib.request.urlopen( # noqa: S310 - URL is hardcoded above
request, timeout=REQUEST_TIMEOUT_SECONDS
) as response:
return response.read().decode("utf-8")
def latest_stable_litellm_tag(
*,
http_get: Optional[Callable[..., str]] = None,
token: Optional[str] = None,
) -> str:
"""Return the newest `v*-stable` tag published on BerriAI/litellm.
Sort is numeric on the (major, minor, patch) triple so v1.10.0-stable
correctly outranks v1.9.5-stable. Releases with no `tag_name` (drafts)
or that don't match the `v*-stable` shape are skipped.
"""
fetch = http_get or _default_http_get
payload = fetch(GITHUB_RELEASES_URL, token=token)
releases = json.loads(payload)
if not isinstance(releases, list):
raise ResolverError(
"github releases response is not a list; got " f"{type(releases).__name__}"
)
matched: List[tuple] = []
for release in releases:
if not isinstance(release, dict):
continue
tag = release.get("tag_name")
if not isinstance(tag, str):
continue
m = STABLE_TAG_RE.match(tag)
if not m:
continue
version_key = tuple(int(x) for x in m.groups())
matched.append((version_key, tag))
if not matched:
raise ResolverError("no v*-stable tags found in github releases response")
matched.sort(key=lambda pair: pair[0], reverse=True)
return matched[0][1]
__all__ = [
"GITHUB_RELEASES_URL",
"ResolverError",
"latest_stable_litellm_tag",
]