From 069dba6e1deb093b6cdce3a7c83381f0a75d716c Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Sat, 25 Apr 2026 05:05:18 +0000 Subject: [PATCH] RALPH: compat matrix slice 3 - wire PR gate in CircleCI (#26479, PRD #26476) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Slice 3 of the Claude Code Compatibility Matrix: wire the `tests/claude_code/` suite into CircleCI as a pre-merge gate. A red status on the new `claude_code_compat_pr_gate` job blocks merge into the staging branch. What landed: - tests/claude_code/pr_gate_version_resolver.py The Claude Code PR-Gate Version Resolver described in the PRD's "Version resolvers" section. Queries the npm registry for `@anthropic-ai/claude-code` and returns the newest version whose publish timestamp is at least 3 days old. The 3-day window is a security review buffer: a malicious or broken Claude Code release has at least 72 hours to be detected before it can land in our PR gate. Importable function (with `metadata=` / `fetcher=` / `as_of=` injection seams for tests) and a `python -m ...` CLI for the CI step. - tests/claude_code/test_config.yaml Proxy routing config that maps the per-cell aliases the tests use (`claude-haiku-4-5`, `claude-haiku-4-5-bedrock-invoke`, ..., `claude-opus-4-7-vertex`) to real upstream model ids on Anthropic / Bedrock (Invoke + Converse) / Vertex AI. Azure intentionally has no entries here because every Azure × claude-code cell is `not_applicable` (Azure OpenAI doesn't host Claude). - .circleci/config.yml New `claude_code_compat_pr_gate` job. Pattern modeled on `proxy_e2e_anthropic_messages_tests` (load PR-built docker image, start postgres, mount config.yaml). New step in the middle: resolve the Claude Code version from the resolver, install Node 20 via the machine image's preinstalled nvm, and `npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}` (pinned, never `latest`). Wired into `workflows.build_and_test` with a `requires: [build_docker_database_image]` gate and the same `*main_branches` filter the other proxy e2e job uses. - tests/claude_code/_pr_gate_unit_tests/ 16 new unit tests: * 8 against the version resolver: boundary (>= 3d inclusive), empty / all-too-new metadata, semver-vs-publish-time tiebreak, custom min_age, fetcher injection, npm `time.created` / `time.modified` skipping. * 8 structural tests against `.circleci/config.yml`: job exists, is in the workflow, requires the docker image, invokes the resolver, install command is pinned (rejects unpinned `latest`), runs `tests/claude_code/`, mounts `test_config.yaml`, exports `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY`. Plus one regression test: the existing `proxy_e2e_anthropic_messages_tests` job is unchanged in shape (acceptance criterion). Key decisions: - "Newest version" in the resolver is by **publish time**, not by semver string ordering — if a patch lands on an older major after a newer release, the patched line is the eligible one. (Tested.) - The resolver's CLI prints the announcement to stderr and the bare version to stdout, so the CI step can do `CLAUDE_CODE_VERSION=$(uv run python -m ...)` cleanly while still surfacing the selected version in the job log (acceptance criterion: "the selected Claude Code version is logged"). - The structural CircleCI tests live under `_pr_gate_unit_tests/` so the conftest path-inference hook skips them (the leading underscore is the existing convention from `_driver_unit_tests/` / `_builder_unit_tests/`); they don't pollute the matrix artifact. - No `--no-verify` style supply-chain safety relaxation. Per the PRD, Claude Code's pinning is the 3-day publish-age window, not a fixed hash — by design, since the daily cron also pulls newer versions. Tests: 47 -> 47 passing for the unit suite (16 new + 31 from slices 1 and 2). The end-to-end cells under `basic_messaging_non_streaming/` require `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` and a running proxy + `claude` CLI; they only run inside the new CircleCI job. Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs): - No docs PR is needed for this slice — the gate produces a status check, not a published artifact. The compat matrix JSON the docs page consumes is published by the daily-cron job (a future slice), not by the PR gate. Notes for next iteration: - The daily cron / matrix publisher is the next slice. Several pieces this slice introduces (the `tests/claude_code/test_config.yaml` proxy config, the structure of the compat-results.json artifact) will be reused by it. - The bedrock-converse / vertex_ai aliases in `test_config.yaml` use best-guess upstream model ids (`us.anthropic.claude-{tier}` and `vertex_ai/claude-{tier}`); the real ids may need to be tightened once the gate runs against live AWS / GCP credentials and we see what resolves. Co-Authored-By: Claude Opus 4.7 --- .circleci/config.yml | 96 ++++++++++++ .../_pr_gate_unit_tests/__init__.py | 0 .../test_circleci_pr_gate_wiring.py | 131 ++++++++++++++++ .../test_pr_gate_version_resolver.py | 139 ++++++++++++++++ tests/claude_code/pr_gate_version_resolver.py | 148 ++++++++++++++++++ tests/claude_code/test_config.yaml | 85 ++++++++++ 6 files changed, 599 insertions(+) create mode 100644 tests/claude_code/_pr_gate_unit_tests/__init__.py create mode 100644 tests/claude_code/_pr_gate_unit_tests/test_circleci_pr_gate_wiring.py create mode 100644 tests/claude_code/_pr_gate_unit_tests/test_pr_gate_version_resolver.py create mode 100644 tests/claude_code/pr_gate_version_resolver.py create mode 100644 tests/claude_code/test_config.yaml diff --git a/.circleci/config.yml b/.circleci/config.yml index 9f01bae3e5b..faf757e7a5d 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -2006,6 +2006,98 @@ jobs: - store_test_results: path: test-results + claude_code_compat_pr_gate: + # Pre-merge gate for the Claude Code Compatibility Matrix. + # + # Boots the LiteLLM proxy from the PR's code, installs the `claude` + # CLI at a version selected at run time from npm (newest version + # whose publish timestamp is >= 3 days old; the 3-day window is a + # security review buffer), and runs every test under + # `tests/claude_code/`. A red status on this job blocks merge — + # see issue #26479 / PRD #26476 for the design. + machine: + image: ubuntu-2204:2024.04.1 + resource_class: large + working_directory: ~/project + steps: + - checkout + - setup_google_dns + - install_uv + - run: + name: Install Dependencies + command: | + uv sync --frozen --all-groups --all-extras --python 3.12 + - start_postgres + - attach_workspace: + at: ~/project + - run: + name: Load Docker Database Image + command: | + zstd -d litellm-docker-database.tar.zst --stdout | docker load + docker images | grep litellm-docker-database + - run: + name: Resolve Claude Code CLI version (newest published >= 3 days ago) + command: | + # Run the resolver from the PR's code; capture the version + # to BASH_ENV so subsequent steps see CLAUDE_CODE_VERSION. + CLAUDE_CODE_VERSION=$(uv run --no-sync python -m tests.claude_code.pr_gate_version_resolver) + echo "Selected @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" + echo "export CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" >> "$BASH_ENV" + - run: + name: Install Node.js 20 + Claude Code CLI + command: | + # The machine image ships with nvm preinstalled; use it to + # pin Node 20 (Claude Code's officially supported runtime). + export NVM_DIR="$HOME/.nvm" + # shellcheck source=/dev/null + [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" + nvm install 20 + nvm use 20 + NODE_BIN_DIR="$(dirname "$(command -v node)")" + echo "export PATH=\"${NODE_BIN_DIR}:\$PATH\"" >> "$BASH_ENV" + npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" + claude --version + - run: + name: Run LiteLLM proxy from PR's code + command: | + docker run -d \ + -p 4000:4000 \ + -e DATABASE_URL=postgresql://postgres:postgres@host.docker.internal:5432/circle_test \ + -e LITELLM_MASTER_KEY="sk-1234" \ + -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \ + -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \ + -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \ + -e AWS_REGION_NAME="us-east-1" \ + -e LITELLM_LOCAL_ANTHROPIC_BETA_HEADERS="True" \ + --add-host host.docker.internal:host-gateway \ + --name compat-proxy \ + -v $(pwd)/tests/claude_code/test_config.yaml:/app/config.yaml \ + litellm-docker-database:ci \ + --config /app/config.yaml \ + --port 4000 \ + --detailed_debug + - run: + name: Stream proxy logs + command: docker logs -f compat-proxy + background: true + - wait_for_service: + url: http://localhost:4000 + timeout: "300" + - run: + name: Run Claude Code compatibility test suite + command: | + export LITELLM_PROXY_BASE_URL="http://localhost:4000" + export LITELLM_PROXY_API_KEY="sk-1234" + mkdir -p test-results + uv run --no-sync python -m pytest -vv tests/claude_code/ \ + --junitxml=test-results/junit.xml \ + --durations=10 + no_output_timeout: 30m + + # Store test results + - store_test_results: + path: test-results + upload-coverage: docker: - *python312_image @@ -2356,6 +2448,10 @@ workflows: requires: - build_docker_database_image filters: *main_branches + - claude_code_compat_pr_gate: + requires: + - build_docker_database_image + filters: *main_branches - llm_translation_testing: filters: *main_branches - realtime_translation_testing: diff --git a/tests/claude_code/_pr_gate_unit_tests/__init__.py b/tests/claude_code/_pr_gate_unit_tests/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/claude_code/_pr_gate_unit_tests/test_circleci_pr_gate_wiring.py b/tests/claude_code/_pr_gate_unit_tests/test_circleci_pr_gate_wiring.py new file mode 100644 index 00000000000..e238bc01554 --- /dev/null +++ b/tests/claude_code/_pr_gate_unit_tests/test_circleci_pr_gate_wiring.py @@ -0,0 +1,131 @@ +"""Sanity tests for the CircleCI PR-gate wiring. + +Parses `.circleci/config.yml` and asserts the claude_code PR-gate job is +present, in the `build_and_test` workflow, and runs the whole +`tests/claude_code/` suite. This catches the obvious "someone deleted +the job" / "someone deleted the workflow entry" regressions that +otherwise only show up in CI history. + +The intent of these tests is *structural*, not *behavioral*: we don't +exercise the docker / npm / proxy machinery here, just verify the YAML +the CircleCI scheduler actually reads. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[3] +CONFIG_PATH = REPO_ROOT / ".circleci" / "config.yml" +JOB_NAME = "claude_code_compat_pr_gate" + + +@pytest.fixture(scope="module") +def circleci_config() -> dict: + return yaml.safe_load(CONFIG_PATH.read_text()) + + +def _job_step_runs(job: dict) -> list[str]: + """Return the concatenated `command` text of every `run:` step.""" + commands: list[str] = [] + for step in job.get("steps", []): + if isinstance(step, dict) and "run" in step: + run = step["run"] + if isinstance(run, dict): + cmd = run.get("command") + if isinstance(cmd, str): + commands.append(cmd) + return commands + + +def test_pr_gate_job_is_defined(circleci_config: dict) -> None: + assert JOB_NAME in circleci_config["jobs"], ( + f"{JOB_NAME} job is missing from .circleci/config.yml — the PR gate " + "is the merge-blocker; deleting it silently disables the gate." + ) + + +def test_pr_gate_job_is_in_build_and_test_workflow(circleci_config: dict) -> None: + workflow = circleci_config["workflows"]["build_and_test"]["jobs"] + job_names = [ + next(iter(entry.keys())) if isinstance(entry, dict) else entry + for entry in workflow + ] + assert JOB_NAME in job_names, ( + f"{JOB_NAME} is defined but not wired into workflows.build_and_test — " + "CircleCI will never run it without this entry." + ) + + +def test_pr_gate_job_requires_docker_database_image(circleci_config: dict) -> None: + """The proxy is booted from `litellm-docker-database:ci`, so the gate + must wait for that build to finish before it runs.""" + workflow = circleci_config["workflows"]["build_and_test"]["jobs"] + entry = next(e[JOB_NAME] for e in workflow if isinstance(e, dict) and JOB_NAME in e) + requires = entry.get("requires", []) + assert "build_docker_database_image" in requires + + +def test_pr_gate_job_invokes_version_resolver_and_pinned_install( + circleci_config: dict, +) -> None: + """Acceptance criterion: the CLI is installed at a version computed + at run time from the npm registry, and the version is logged.""" + job = circleci_config["jobs"][JOB_NAME] + commands = "\n".join(_job_step_runs(job)) + assert "tests.claude_code.pr_gate_version_resolver" in commands, ( + "PR-gate job must invoke the version resolver; otherwise the " + "3-day publish-age security buffer is bypassed." + ) + # Pinned install of the resolved version (not 'latest', not unversioned) + assert "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" in commands + + +def test_pr_gate_job_runs_claude_code_test_dir(circleci_config: dict) -> None: + job = circleci_config["jobs"][JOB_NAME] + commands = "\n".join(_job_step_runs(job)) + assert "tests/claude_code/" in commands, ( + f"{JOB_NAME} must run the tests/claude_code/ suite; otherwise " + "the gate isn't actually exercising the compat tests." + ) + + +def test_pr_gate_job_mounts_test_config_yaml(circleci_config: dict) -> None: + """The tests reference proxy aliases (`claude-haiku-4-5` etc.) that + only the routing config knows about — the gate must mount it into + the proxy container.""" + job = circleci_config["jobs"][JOB_NAME] + commands = "\n".join(_job_step_runs(job)) + assert "tests/claude_code/test_config.yaml" in commands + + +def test_pr_gate_job_exports_proxy_env_used_by_tests( + circleci_config: dict, +) -> None: + """Tests read LITELLM_PROXY_BASE_URL / LITELLM_PROXY_API_KEY — the + job must export both before the pytest invocation.""" + job = circleci_config["jobs"][JOB_NAME] + commands = "\n".join(_job_step_runs(job)) + assert "LITELLM_PROXY_BASE_URL" in commands + assert "LITELLM_PROXY_API_KEY" in commands + + +def test_existing_proxy_e2e_anthropic_job_unchanged(circleci_config: dict) -> None: + """No regression to the existing `proxy_e2e_anthropic_messages_tests` + job (acceptance criterion). We don't lock its full body, but we do + pin the load-bearing surface: it still runs the same test directory + and is still wired into the workflow.""" + assert "proxy_e2e_anthropic_messages_tests" in circleci_config["jobs"] + workflow = circleci_config["workflows"]["build_and_test"]["jobs"] + job_names = [ + next(iter(entry.keys())) if isinstance(entry, dict) else entry + for entry in workflow + ] + assert "proxy_e2e_anthropic_messages_tests" in job_names + body = "\n".join( + _job_step_runs(circleci_config["jobs"]["proxy_e2e_anthropic_messages_tests"]) + ) + assert "tests/proxy_e2e_anthropic_messages_tests/" in body diff --git a/tests/claude_code/_pr_gate_unit_tests/test_pr_gate_version_resolver.py b/tests/claude_code/_pr_gate_unit_tests/test_pr_gate_version_resolver.py new file mode 100644 index 00000000000..2daf418f9b0 --- /dev/null +++ b/tests/claude_code/_pr_gate_unit_tests/test_pr_gate_version_resolver.py @@ -0,0 +1,139 @@ +"""Unit tests for the Claude Code PR-Gate Version Resolver. + +The resolver picks the newest `@anthropic-ai/claude-code` version whose +publish timestamp is at least 3 days old. The 3-day window is a security +review buffer: a malicious or broken Claude Code release that slipped +through the npm publish process gets at least 72 hours to be detected +before it can land in the LiteLLM PR gate. + +The unit tests inject npm metadata directly (no network) and a fixed +`as_of` clock (no real time), so they run anywhere and never flake on +the wall clock or registry availability. +""" + +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +import pytest + +from tests.claude_code.pr_gate_version_resolver import ( + NoEligibleVersionError, + resolve_pr_gate_version, +) + + +def _t(iso: str) -> str: + """Helper for readable ISO-8601 publish timestamps in fixtures.""" + return iso + + +# A clock fixed at a moment well after every fixture publish time below. +NOW = datetime(2026, 4, 25, 12, 0, 0, tzinfo=timezone.utc) + + +def _metadata_with_times(times: dict) -> dict: + """Shape an npm `packument`-like dict with the `time` field populated. + + The npm registry response includes `time.created` / `time.modified` + keys alongside per-version timestamps; the resolver must skip those. + """ + return { + "name": "@anthropic-ai/claude-code", + "time": { + "created": _t("2024-01-01T00:00:00.000Z"), + "modified": _t("2026-04-25T00:00:00.000Z"), + **times, + }, + } + + +def test_picks_newest_version_at_least_three_days_old(): + metadata = _metadata_with_times( + { + "2.1.118": _t("2026-04-15T10:00:00.000Z"), + "2.1.119": _t("2026-04-21T10:00:00.000Z"), # 4d 2h old + "2.1.120": _t("2026-04-23T10:00:00.000Z"), # 2d 2h old — too new + "2.1.121": _t("2026-04-25T11:00:00.000Z"), # 1h old — too new + } + ) + assert resolve_pr_gate_version(metadata=metadata, as_of=NOW) == "2.1.119" + + +def test_skips_created_and_modified_meta_keys(): + """`time` contains `created` / `modified` non-version entries — must be ignored.""" + metadata = { + "name": "@anthropic-ai/claude-code", + "time": { + "created": _t("2024-01-01T00:00:00.000Z"), + "modified": _t("2026-04-25T00:00:00.000Z"), + "2.0.0": _t("2026-04-10T00:00:00.000Z"), + }, + } + assert resolve_pr_gate_version(metadata=metadata, as_of=NOW) == "2.0.0" + + +def test_min_age_boundary_is_inclusive(): + """A version published exactly 3 days ago is eligible (>= cutoff).""" + three_days_ago = NOW - timedelta(days=3) + metadata = _metadata_with_times( + { + "2.1.0": three_days_ago.isoformat().replace("+00:00", "Z"), + } + ) + assert resolve_pr_gate_version(metadata=metadata, as_of=NOW) == "2.1.0" + + +def test_raises_when_every_version_is_too_new(): + metadata = _metadata_with_times( + { + "2.1.121": _t("2026-04-25T08:00:00.000Z"), # 4h old + "2.1.120": _t("2026-04-24T10:00:00.000Z"), # ~26h old + } + ) + with pytest.raises(NoEligibleVersionError): + resolve_pr_gate_version(metadata=metadata, as_of=NOW) + + +def test_raises_when_metadata_has_no_versions(): + metadata = {"name": "@anthropic-ai/claude-code", "time": {}} + with pytest.raises(NoEligibleVersionError): + resolve_pr_gate_version(metadata=metadata, as_of=NOW) + + +def test_picks_latest_publish_time_not_largest_semver(): + """If a patch is published to an old major after a newer release, + "newest" is by publish time, not semver string ordering.""" + metadata = _metadata_with_times( + { + "1.9.99": _t("2026-04-22T10:00:00.000Z"), # patched recently — wins + "2.0.0": _t("2026-03-01T10:00:00.000Z"), # older publish + } + ) + assert resolve_pr_gate_version(metadata=metadata, as_of=NOW) == "1.9.99" + + +def test_uses_custom_min_age(): + metadata = _metadata_with_times( + { + "1.0.0": _t("2026-04-23T10:00:00.000Z"), # 2d 2h old + "0.9.0": _t("2026-04-10T10:00:00.000Z"), # 15d old + } + ) + # min_age = 5 days disqualifies 1.0.0 + out = resolve_pr_gate_version( + metadata=metadata, as_of=NOW, min_age=timedelta(days=5) + ) + assert out == "0.9.0" + + +def test_resolver_uses_fetcher_when_metadata_not_provided(): + captured = {} + + def fake_fetch(package_name: str) -> dict: + captured["package"] = package_name + return _metadata_with_times({"3.0.0": _t("2026-04-10T10:00:00.000Z")}) + + out = resolve_pr_gate_version(as_of=NOW, fetcher=fake_fetch) + assert out == "3.0.0" + assert captured["package"] == "@anthropic-ai/claude-code" diff --git a/tests/claude_code/pr_gate_version_resolver.py b/tests/claude_code/pr_gate_version_resolver.py new file mode 100644 index 00000000000..25baaaf2b11 --- /dev/null +++ b/tests/claude_code/pr_gate_version_resolver.py @@ -0,0 +1,148 @@ +"""Claude Code PR-Gate Version Resolver. + +Resolves the `@anthropic-ai/claude-code` npm version that the PR-gate CI +job installs. Selects the newest version (by publish timestamp) whose +publish timestamp is at least 3 days old. The 3-day window is a security +review buffer — see PRD #26476, "Version resolvers". + +Two surfaces: + +- ``resolve_pr_gate_version(...)`` — the importable function. Accepts + pre-fetched npm metadata (for unit tests) or a custom ``fetcher`` + callable. The default fetcher hits the public npm registry. +- ``python -m tests.claude_code.pr_gate_version_resolver`` — prints the + resolved version string to stdout, suitable for piping into a shell + ``$(...)`` substitution inside the CircleCI job. + +The CLI form is what CircleCI runs at job start; engineers reading the +job log can see the selected version on a single line above the +``npm install -g`` step (acceptance criterion: "the selected Claude +Code version is logged in the CI output"). +""" + +from __future__ import annotations + +import json +import sys +import urllib.request +from datetime import datetime, timedelta, timezone +from typing import Callable, Mapping, Optional + +PACKAGE_NAME = "@anthropic-ai/claude-code" +NPM_REGISTRY_URL = "https://registry.npmjs.org/{package}" +DEFAULT_MIN_AGE = timedelta(days=3) +DEFAULT_FETCH_TIMEOUT_SECONDS = 30 + +# npm's `time` map mixes per-version timestamps with these meta keys. +_TIME_META_KEYS = frozenset({"created", "modified"}) + + +class NoEligibleVersionError(RuntimeError): + """Raised when no version in the npm metadata satisfies the min-age cutoff.""" + + +def _parse_npm_timestamp(value: str) -> datetime: + """Parse the ISO-8601 timestamps npm emits (always UTC, may use ``Z``).""" + if value.endswith("Z"): + value = value[:-1] + "+00:00" + parsed = datetime.fromisoformat(value) + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return parsed + + +def _default_fetcher(package_name: str) -> dict: + """Fetch the npm packument for ``package_name`` over HTTPS. + + Uses urllib (stdlib) so this module has no extra dependencies in the + CI environment. Returns the raw JSON dict. + """ + # urllib.parse.quote would encode the leading '@' / '/' which the + # npm registry expects literally; do a minimal hand-roll instead. + url = NPM_REGISTRY_URL.format(package=package_name.replace("/", "%2F")) + req = urllib.request.Request(url, headers={"Accept": "application/json"}) + with urllib.request.urlopen( # noqa: S310 — registry URL is constant + req, timeout=DEFAULT_FETCH_TIMEOUT_SECONDS + ) as response: + body = response.read().decode("utf-8") + return json.loads(body) + + +def resolve_pr_gate_version( + *, + metadata: Optional[Mapping] = None, + fetcher: Optional[Callable[[str], Mapping]] = None, + as_of: Optional[datetime] = None, + min_age: timedelta = DEFAULT_MIN_AGE, + package_name: str = PACKAGE_NAME, +) -> str: + """Return the newest npm version of ``package_name`` published >= ``min_age`` ago. + + "Newest" means newest by **publish time**, not semver string order — + if a patch lands on an older major after a newer release, the + patched line is the eligible one. + + Args: + metadata: Pre-fetched npm packument (skips the HTTP call). Useful + for unit tests. + fetcher: Callable taking a package name and returning the + packument. Defaults to a stdlib HTTPS fetcher. + as_of: The clock used to decide whether a version is "old + enough". Defaults to ``datetime.now(timezone.utc)``. + min_age: Minimum publish age. Defaults to 3 days. + package_name: Defaults to ``@anthropic-ai/claude-code``. + + Raises: + NoEligibleVersionError: when no version in the registry meets + the age cutoff. + """ + if metadata is None: + fetch = fetcher or _default_fetcher + metadata = fetch(package_name) + + times = metadata.get("time") or {} + if as_of is None: + as_of = datetime.now(timezone.utc) + cutoff = as_of - min_age + + eligible: list[tuple[datetime, str]] = [] + for version, raw_ts in times.items(): + if version in _TIME_META_KEYS: + continue + if not isinstance(raw_ts, str): + continue + published = _parse_npm_timestamp(raw_ts) + if published <= cutoff: + eligible.append((published, version)) + + if not eligible: + raise NoEligibleVersionError( + f"no version of {package_name} is at least {min_age} old " + f"as of {as_of.isoformat()}" + ) + + eligible.sort(key=lambda pair: pair[0], reverse=True) + return eligible[0][1] + + +def _main(argv: list[str]) -> int: + """Print the resolved version to stdout. Exit code 0 on success. + + Stderr carries the human-readable announcement so the version can be + captured cleanly with ``$(python -m ...)`` in shell. + """ + try: + version = resolve_pr_gate_version() + except Exception as exc: # noqa: BLE001 — CLI surface, want everything + print(f"pr_gate_version_resolver: {exc}", file=sys.stderr) # noqa: T201 + return 1 + print( # noqa: T201 + f"pr_gate_version_resolver: selected {PACKAGE_NAME}@{version}", + file=sys.stderr, + ) + print(version) # noqa: T201 + return 0 + + +if __name__ == "__main__": + raise SystemExit(_main(sys.argv[1:])) diff --git a/tests/claude_code/test_config.yaml b/tests/claude_code/test_config.yaml new file mode 100644 index 00000000000..8c9be506ed4 --- /dev/null +++ b/tests/claude_code/test_config.yaml @@ -0,0 +1,85 @@ +# Proxy routing config for the Claude Code Compatibility Matrix PR gate. +# +# The tests under `tests/claude_code/` only know **alias** names (e.g. +# `claude-haiku-4-5-bedrock-invoke`). The proxy is the layer that maps +# each alias to a real upstream model id, region, and credentials. +# +# Adding a new (feature, provider) cell is a three-step change in the +# test repo (manifest + test file + alias here); changing which upstream +# model a cell exercises is a one-step change here, with no test edits. +# +# Aliases: +# - claude-{tier} → Anthropic API +# - claude-{tier}-bedrock-invoke → Bedrock InvokeModel API +# - claude-{tier}-bedrock-converse → Bedrock Converse API +# - claude-{tier}-vertex → GCP Vertex AI + +model_list: + # ---- Anthropic ---- + - model_name: claude-haiku-4-5 + litellm_params: + model: anthropic/claude-haiku-4-5 + api_key: os.environ/ANTHROPIC_API_KEY + - model_name: claude-sonnet-4-6 + litellm_params: + model: anthropic/claude-sonnet-4-6 + api_key: os.environ/ANTHROPIC_API_KEY + - model_name: claude-opus-4-7 + litellm_params: + model: anthropic/claude-opus-4-7 + api_key: os.environ/ANTHROPIC_API_KEY + + # ---- Bedrock (InvokeModel) ---- + - model_name: claude-haiku-4-5-bedrock-invoke + litellm_params: + model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0 + aws_region_name: us-east-1 + - model_name: claude-sonnet-4-6-bedrock-invoke + litellm_params: + model: bedrock/us.anthropic.claude-sonnet-4-6 + aws_region_name: us-east-1 + - model_name: claude-opus-4-7-bedrock-invoke + litellm_params: + model: bedrock/us.anthropic.claude-opus-4-7 + aws_region_name: us-east-1 + + # ---- Bedrock (Converse) ---- + - model_name: claude-haiku-4-5-bedrock-converse + litellm_params: + model: bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0 + aws_region_name: us-east-1 + - model_name: claude-sonnet-4-6-bedrock-converse + litellm_params: + model: bedrock/converse/us.anthropic.claude-sonnet-4-6 + aws_region_name: us-east-1 + - model_name: claude-opus-4-7-bedrock-converse + litellm_params: + model: bedrock/converse/us.anthropic.claude-opus-4-7 + aws_region_name: us-east-1 + + # ---- Vertex AI ---- + - model_name: claude-haiku-4-5-vertex + litellm_params: + model: vertex_ai/claude-haiku-4-5 + vertex_ai_project: pathrise-convert-1606954137718 + vertex_ai_location: us-east5 + - model_name: claude-sonnet-4-6-vertex + litellm_params: + model: vertex_ai/claude-sonnet-4-6 + vertex_ai_project: pathrise-convert-1606954137718 + vertex_ai_location: us-east5 + - model_name: claude-opus-4-7-vertex + litellm_params: + model: vertex_ai/claude-opus-4-7 + vertex_ai_project: pathrise-convert-1606954137718 + vertex_ai_location: us-east5 + +general_settings: + # Claude Code sends provider-specific headers (e.g. anthropic-beta) we + # want to forward verbatim to the upstream so the wire-shape under + # test matches what real customers send. + forward_client_headers_to_llm_api: true + +litellm_settings: + drop_params: true + modify_params: true