litellm/tests/claude_code/pr_gate_version_resolver.py
mateo-berri 069dba6e1d RALPH: compat matrix slice 3 - wire PR gate in CircleCI (#26479, PRD #26476)
Slice 3 of the Claude Code Compatibility Matrix: wire the
`tests/claude_code/` suite into CircleCI as a pre-merge gate. A red
status on the new `claude_code_compat_pr_gate` job blocks merge into
the staging branch.

What landed:

- tests/claude_code/pr_gate_version_resolver.py
  The Claude Code PR-Gate Version Resolver described in the PRD's
  "Version resolvers" section. Queries the npm registry for
  `@anthropic-ai/claude-code` and returns the newest version whose
  publish timestamp is at least 3 days old. The 3-day window is a
  security review buffer: a malicious or broken Claude Code release
  has at least 72 hours to be detected before it can land in our PR
  gate. Importable function (with `metadata=` / `fetcher=` / `as_of=`
  injection seams for tests) and a `python -m ...` CLI for the CI step.

- tests/claude_code/test_config.yaml
  Proxy routing config that maps the per-cell aliases the tests use
  (`claude-haiku-4-5`, `claude-haiku-4-5-bedrock-invoke`, ...,
  `claude-opus-4-7-vertex`) to real upstream model ids on Anthropic /
  Bedrock (Invoke + Converse) / Vertex AI. Azure intentionally has no
  entries here because every Azure × claude-code cell is
  `not_applicable` (Azure OpenAI doesn't host Claude).

- .circleci/config.yml
  New `claude_code_compat_pr_gate` job. Pattern modeled on
  `proxy_e2e_anthropic_messages_tests` (load PR-built docker image,
  start postgres, mount config.yaml). New step in the middle:
  resolve the Claude Code version from the resolver, install Node 20
  via the machine image's preinstalled nvm, and `npm install -g
  @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}` (pinned, never
  `latest`). Wired into `workflows.build_and_test` with a
  `requires: [build_docker_database_image]` gate and the same
  `*main_branches` filter the other proxy e2e job uses.

- tests/claude_code/_pr_gate_unit_tests/
  16 new unit tests:
  * 8 against the version resolver: boundary (>= 3d inclusive),
    empty / all-too-new metadata, semver-vs-publish-time tiebreak,
    custom min_age, fetcher injection, npm `time.created` /
    `time.modified` skipping.
  * 8 structural tests against `.circleci/config.yml`: job exists,
    is in the workflow, requires the docker image, invokes the
    resolver, install command is pinned (rejects unpinned `latest`),
    runs `tests/claude_code/`, mounts `test_config.yaml`, exports
    `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY`. Plus one
    regression test: the existing `proxy_e2e_anthropic_messages_tests`
    job is unchanged in shape (acceptance criterion).

Key decisions:

- "Newest version" in the resolver is by **publish time**, not by
  semver string ordering — if a patch lands on an older major after a
  newer release, the patched line is the eligible one. (Tested.)
- The resolver's CLI prints the announcement to stderr and the bare
  version to stdout, so the CI step can do
  `CLAUDE_CODE_VERSION=$(uv run python -m ...)` cleanly while still
  surfacing the selected version in the job log (acceptance criterion:
  "the selected Claude Code version is logged").
- The structural CircleCI tests live under `_pr_gate_unit_tests/` so
  the conftest path-inference hook skips them (the leading underscore
  is the existing convention from `_driver_unit_tests/` /
  `_builder_unit_tests/`); they don't pollute the matrix artifact.
- No `--no-verify` style supply-chain safety relaxation. Per the PRD,
  Claude Code's pinning is the 3-day publish-age window, not a fixed
  hash — by design, since the daily cron also pulls newer versions.

Tests: 47 -> 47 passing for the unit suite (16 new + 31 from slices
1 and 2). The end-to-end cells under `basic_messaging_non_streaming/`
require `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` and a
running proxy + `claude` CLI; they only run inside the new CircleCI
job.

Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- No docs PR is needed for this slice — the gate produces a status
  check, not a published artifact. The compat matrix JSON the docs
  page consumes is published by the daily-cron job (a future slice),
  not by the PR gate.

Notes for next iteration:
- The daily cron / matrix publisher is the next slice. Several
  pieces this slice introduces (the `tests/claude_code/test_config.yaml`
  proxy config, the structure of the compat-results.json artifact)
  will be reused by it.
- The bedrock-converse / vertex_ai aliases in `test_config.yaml` use
  best-guess upstream model ids (`us.anthropic.claude-{tier}` and
  `vertex_ai/claude-{tier}`); the real ids may need to be tightened
  once the gate runs against live AWS / GCP credentials and we see
  what resolves.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-25 05:05:18 +00:00

148 lines
5.3 KiB
Python

"""Claude Code PR-Gate Version Resolver.
Resolves the `@anthropic-ai/claude-code` npm version that the PR-gate CI
job installs. Selects the newest version (by publish timestamp) whose
publish timestamp is at least 3 days old. The 3-day window is a security
review buffer — see PRD #26476, "Version resolvers".
Two surfaces:
- ``resolve_pr_gate_version(...)`` — the importable function. Accepts
pre-fetched npm metadata (for unit tests) or a custom ``fetcher``
callable. The default fetcher hits the public npm registry.
- ``python -m tests.claude_code.pr_gate_version_resolver`` — prints the
resolved version string to stdout, suitable for piping into a shell
``$(...)`` substitution inside the CircleCI job.
The CLI form is what CircleCI runs at job start; engineers reading the
job log can see the selected version on a single line above the
``npm install -g`` step (acceptance criterion: "the selected Claude
Code version is logged in the CI output").
"""
from __future__ import annotations
import json
import sys
import urllib.request
from datetime import datetime, timedelta, timezone
from typing import Callable, Mapping, Optional
PACKAGE_NAME = "@anthropic-ai/claude-code"
NPM_REGISTRY_URL = "https://registry.npmjs.org/{package}"
DEFAULT_MIN_AGE = timedelta(days=3)
DEFAULT_FETCH_TIMEOUT_SECONDS = 30
# npm's `time` map mixes per-version timestamps with these meta keys.
_TIME_META_KEYS = frozenset({"created", "modified"})
class NoEligibleVersionError(RuntimeError):
"""Raised when no version in the npm metadata satisfies the min-age cutoff."""
def _parse_npm_timestamp(value: str) -> datetime:
"""Parse the ISO-8601 timestamps npm emits (always UTC, may use ``Z``)."""
if value.endswith("Z"):
value = value[:-1] + "+00:00"
parsed = datetime.fromisoformat(value)
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=timezone.utc)
return parsed
def _default_fetcher(package_name: str) -> dict:
"""Fetch the npm packument for ``package_name`` over HTTPS.
Uses urllib (stdlib) so this module has no extra dependencies in the
CI environment. Returns the raw JSON dict.
"""
# urllib.parse.quote would encode the leading '@' / '/' which the
# npm registry expects literally; do a minimal hand-roll instead.
url = NPM_REGISTRY_URL.format(package=package_name.replace("/", "%2F"))
req = urllib.request.Request(url, headers={"Accept": "application/json"})
with urllib.request.urlopen( # noqa: S310 — registry URL is constant
req, timeout=DEFAULT_FETCH_TIMEOUT_SECONDS
) as response:
body = response.read().decode("utf-8")
return json.loads(body)
def resolve_pr_gate_version(
*,
metadata: Optional[Mapping] = None,
fetcher: Optional[Callable[[str], Mapping]] = None,
as_of: Optional[datetime] = None,
min_age: timedelta = DEFAULT_MIN_AGE,
package_name: str = PACKAGE_NAME,
) -> str:
"""Return the newest npm version of ``package_name`` published >= ``min_age`` ago.
"Newest" means newest by **publish time**, not semver string order —
if a patch lands on an older major after a newer release, the
patched line is the eligible one.
Args:
metadata: Pre-fetched npm packument (skips the HTTP call). Useful
for unit tests.
fetcher: Callable taking a package name and returning the
packument. Defaults to a stdlib HTTPS fetcher.
as_of: The clock used to decide whether a version is "old
enough". Defaults to ``datetime.now(timezone.utc)``.
min_age: Minimum publish age. Defaults to 3 days.
package_name: Defaults to ``@anthropic-ai/claude-code``.
Raises:
NoEligibleVersionError: when no version in the registry meets
the age cutoff.
"""
if metadata is None:
fetch = fetcher or _default_fetcher
metadata = fetch(package_name)
times = metadata.get("time") or {}
if as_of is None:
as_of = datetime.now(timezone.utc)
cutoff = as_of - min_age
eligible: list[tuple[datetime, str]] = []
for version, raw_ts in times.items():
if version in _TIME_META_KEYS:
continue
if not isinstance(raw_ts, str):
continue
published = _parse_npm_timestamp(raw_ts)
if published <= cutoff:
eligible.append((published, version))
if not eligible:
raise NoEligibleVersionError(
f"no version of {package_name} is at least {min_age} old "
f"as of {as_of.isoformat()}"
)
eligible.sort(key=lambda pair: pair[0], reverse=True)
return eligible[0][1]
def _main(argv: list[str]) -> int:
"""Print the resolved version to stdout. Exit code 0 on success.
Stderr carries the human-readable announcement so the version can be
captured cleanly with ``$(python -m ...)`` in shell.
"""
try:
version = resolve_pr_gate_version()
except Exception as exc: # noqa: BLE001 — CLI surface, want everything
print(f"pr_gate_version_resolver: {exc}", file=sys.stderr) # noqa: T201
return 1
print( # noqa: T201
f"pr_gate_version_resolver: selected {PACKAGE_NAME}@{version}",
file=sys.stderr,
)
print(version) # noqa: T201
return 0
if __name__ == "__main__":
raise SystemExit(_main(sys.argv[1:]))