mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-17 23:51:30 +00:00
Slice 3 of the Claude Code Compatibility Matrix: wire the
`tests/claude_code/` suite into CircleCI as a pre-merge gate. A red
status on the new `claude_code_compat_pr_gate` job blocks merge into
the staging branch.
What landed:
- tests/claude_code/pr_gate_version_resolver.py
The Claude Code PR-Gate Version Resolver described in the PRD's
"Version resolvers" section. Queries the npm registry for
`@anthropic-ai/claude-code` and returns the newest version whose
publish timestamp is at least 3 days old. The 3-day window is a
security review buffer: a malicious or broken Claude Code release
has at least 72 hours to be detected before it can land in our PR
gate. Importable function (with `metadata=` / `fetcher=` / `as_of=`
injection seams for tests) and a `python -m ...` CLI for the CI step.
- tests/claude_code/test_config.yaml
Proxy routing config that maps the per-cell aliases the tests use
(`claude-haiku-4-5`, `claude-haiku-4-5-bedrock-invoke`, ...,
`claude-opus-4-7-vertex`) to real upstream model ids on Anthropic /
Bedrock (Invoke + Converse) / Vertex AI. Azure intentionally has no
entries here because every Azure × claude-code cell is
`not_applicable` (Azure OpenAI doesn't host Claude).
- .circleci/config.yml
New `claude_code_compat_pr_gate` job. Pattern modeled on
`proxy_e2e_anthropic_messages_tests` (load PR-built docker image,
start postgres, mount config.yaml). New step in the middle:
resolve the Claude Code version from the resolver, install Node 20
via the machine image's preinstalled nvm, and `npm install -g
@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}` (pinned, never
`latest`). Wired into `workflows.build_and_test` with a
`requires: [build_docker_database_image]` gate and the same
`*main_branches` filter the other proxy e2e job uses.
- tests/claude_code/_pr_gate_unit_tests/
16 new unit tests:
* 8 against the version resolver: boundary (>= 3d inclusive),
empty / all-too-new metadata, semver-vs-publish-time tiebreak,
custom min_age, fetcher injection, npm `time.created` /
`time.modified` skipping.
* 8 structural tests against `.circleci/config.yml`: job exists,
is in the workflow, requires the docker image, invokes the
resolver, install command is pinned (rejects unpinned `latest`),
runs `tests/claude_code/`, mounts `test_config.yaml`, exports
`LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY`. Plus one
regression test: the existing `proxy_e2e_anthropic_messages_tests`
job is unchanged in shape (acceptance criterion).
Key decisions:
- "Newest version" in the resolver is by **publish time**, not by
semver string ordering — if a patch lands on an older major after a
newer release, the patched line is the eligible one. (Tested.)
- The resolver's CLI prints the announcement to stderr and the bare
version to stdout, so the CI step can do
`CLAUDE_CODE_VERSION=$(uv run python -m ...)` cleanly while still
surfacing the selected version in the job log (acceptance criterion:
"the selected Claude Code version is logged").
- The structural CircleCI tests live under `_pr_gate_unit_tests/` so
the conftest path-inference hook skips them (the leading underscore
is the existing convention from `_driver_unit_tests/` /
`_builder_unit_tests/`); they don't pollute the matrix artifact.
- No `--no-verify` style supply-chain safety relaxation. Per the PRD,
Claude Code's pinning is the 3-day publish-age window, not a fixed
hash — by design, since the daily cron also pulls newer versions.
Tests: 47 -> 47 passing for the unit suite (16 new + 31 from slices
1 and 2). The end-to-end cells under `basic_messaging_non_streaming/`
require `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` and a
running proxy + `claude` CLI; they only run inside the new CircleCI
job.
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- No docs PR is needed for this slice — the gate produces a status
check, not a published artifact. The compat matrix JSON the docs
page consumes is published by the daily-cron job (a future slice),
not by the PR gate.
Notes for next iteration:
- The daily cron / matrix publisher is the next slice. Several
pieces this slice introduces (the `tests/claude_code/test_config.yaml`
proxy config, the structure of the compat-results.json artifact)
will be reused by it.
- The bedrock-converse / vertex_ai aliases in `test_config.yaml` use
best-guess upstream model ids (`us.anthropic.claude-{tier}` and
`vertex_ai/claude-{tier}`); the real ids may need to be tightened
once the gate runs against live AWS / GCP credentials and we see
what resolves.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
148 lines
5.3 KiB
Python
148 lines
5.3 KiB
Python
"""Claude Code PR-Gate Version Resolver.
|
|
|
|
Resolves the `@anthropic-ai/claude-code` npm version that the PR-gate CI
|
|
job installs. Selects the newest version (by publish timestamp) whose
|
|
publish timestamp is at least 3 days old. The 3-day window is a security
|
|
review buffer — see PRD #26476, "Version resolvers".
|
|
|
|
Two surfaces:
|
|
|
|
- ``resolve_pr_gate_version(...)`` — the importable function. Accepts
|
|
pre-fetched npm metadata (for unit tests) or a custom ``fetcher``
|
|
callable. The default fetcher hits the public npm registry.
|
|
- ``python -m tests.claude_code.pr_gate_version_resolver`` — prints the
|
|
resolved version string to stdout, suitable for piping into a shell
|
|
``$(...)`` substitution inside the CircleCI job.
|
|
|
|
The CLI form is what CircleCI runs at job start; engineers reading the
|
|
job log can see the selected version on a single line above the
|
|
``npm install -g`` step (acceptance criterion: "the selected Claude
|
|
Code version is logged in the CI output").
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import sys
|
|
import urllib.request
|
|
from datetime import datetime, timedelta, timezone
|
|
from typing import Callable, Mapping, Optional
|
|
|
|
PACKAGE_NAME = "@anthropic-ai/claude-code"
|
|
NPM_REGISTRY_URL = "https://registry.npmjs.org/{package}"
|
|
DEFAULT_MIN_AGE = timedelta(days=3)
|
|
DEFAULT_FETCH_TIMEOUT_SECONDS = 30
|
|
|
|
# npm's `time` map mixes per-version timestamps with these meta keys.
|
|
_TIME_META_KEYS = frozenset({"created", "modified"})
|
|
|
|
|
|
class NoEligibleVersionError(RuntimeError):
|
|
"""Raised when no version in the npm metadata satisfies the min-age cutoff."""
|
|
|
|
|
|
def _parse_npm_timestamp(value: str) -> datetime:
|
|
"""Parse the ISO-8601 timestamps npm emits (always UTC, may use ``Z``)."""
|
|
if value.endswith("Z"):
|
|
value = value[:-1] + "+00:00"
|
|
parsed = datetime.fromisoformat(value)
|
|
if parsed.tzinfo is None:
|
|
parsed = parsed.replace(tzinfo=timezone.utc)
|
|
return parsed
|
|
|
|
|
|
def _default_fetcher(package_name: str) -> dict:
|
|
"""Fetch the npm packument for ``package_name`` over HTTPS.
|
|
|
|
Uses urllib (stdlib) so this module has no extra dependencies in the
|
|
CI environment. Returns the raw JSON dict.
|
|
"""
|
|
# urllib.parse.quote would encode the leading '@' / '/' which the
|
|
# npm registry expects literally; do a minimal hand-roll instead.
|
|
url = NPM_REGISTRY_URL.format(package=package_name.replace("/", "%2F"))
|
|
req = urllib.request.Request(url, headers={"Accept": "application/json"})
|
|
with urllib.request.urlopen( # noqa: S310 — registry URL is constant
|
|
req, timeout=DEFAULT_FETCH_TIMEOUT_SECONDS
|
|
) as response:
|
|
body = response.read().decode("utf-8")
|
|
return json.loads(body)
|
|
|
|
|
|
def resolve_pr_gate_version(
|
|
*,
|
|
metadata: Optional[Mapping] = None,
|
|
fetcher: Optional[Callable[[str], Mapping]] = None,
|
|
as_of: Optional[datetime] = None,
|
|
min_age: timedelta = DEFAULT_MIN_AGE,
|
|
package_name: str = PACKAGE_NAME,
|
|
) -> str:
|
|
"""Return the newest npm version of ``package_name`` published >= ``min_age`` ago.
|
|
|
|
"Newest" means newest by **publish time**, not semver string order —
|
|
if a patch lands on an older major after a newer release, the
|
|
patched line is the eligible one.
|
|
|
|
Args:
|
|
metadata: Pre-fetched npm packument (skips the HTTP call). Useful
|
|
for unit tests.
|
|
fetcher: Callable taking a package name and returning the
|
|
packument. Defaults to a stdlib HTTPS fetcher.
|
|
as_of: The clock used to decide whether a version is "old
|
|
enough". Defaults to ``datetime.now(timezone.utc)``.
|
|
min_age: Minimum publish age. Defaults to 3 days.
|
|
package_name: Defaults to ``@anthropic-ai/claude-code``.
|
|
|
|
Raises:
|
|
NoEligibleVersionError: when no version in the registry meets
|
|
the age cutoff.
|
|
"""
|
|
if metadata is None:
|
|
fetch = fetcher or _default_fetcher
|
|
metadata = fetch(package_name)
|
|
|
|
times = metadata.get("time") or {}
|
|
if as_of is None:
|
|
as_of = datetime.now(timezone.utc)
|
|
cutoff = as_of - min_age
|
|
|
|
eligible: list[tuple[datetime, str]] = []
|
|
for version, raw_ts in times.items():
|
|
if version in _TIME_META_KEYS:
|
|
continue
|
|
if not isinstance(raw_ts, str):
|
|
continue
|
|
published = _parse_npm_timestamp(raw_ts)
|
|
if published <= cutoff:
|
|
eligible.append((published, version))
|
|
|
|
if not eligible:
|
|
raise NoEligibleVersionError(
|
|
f"no version of {package_name} is at least {min_age} old "
|
|
f"as of {as_of.isoformat()}"
|
|
)
|
|
|
|
eligible.sort(key=lambda pair: pair[0], reverse=True)
|
|
return eligible[0][1]
|
|
|
|
|
|
def _main(argv: list[str]) -> int:
|
|
"""Print the resolved version to stdout. Exit code 0 on success.
|
|
|
|
Stderr carries the human-readable announcement so the version can be
|
|
captured cleanly with ``$(python -m ...)`` in shell.
|
|
"""
|
|
try:
|
|
version = resolve_pr_gate_version()
|
|
except Exception as exc: # noqa: BLE001 — CLI surface, want everything
|
|
print(f"pr_gate_version_resolver: {exc}", file=sys.stderr) # noqa: T201
|
|
return 1
|
|
print( # noqa: T201
|
|
f"pr_gate_version_resolver: selected {PACKAGE_NAME}@{version}",
|
|
file=sys.stderr,
|
|
)
|
|
print(version) # noqa: T201
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(_main(sys.argv[1:]))
|