scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-04 02:31:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 71

RALPH: compat matrix slice 3 - wire PR gate in CircleCI (#26479, PRD #26476)

Browse source 
Slice 3 of the Claude Code Compatibility Matrix: wire the
`tests/claude_code/` suite into CircleCI as a pre-merge gate. A red
status on the new `claude_code_compat_pr_gate` job blocks merge into
the staging branch.

What landed:

- tests/claude_code/pr_gate_version_resolver.py
  The Claude Code PR-Gate Version Resolver described in the PRD's
  "Version resolvers" section. Queries the npm registry for
  `@anthropic-ai/claude-code` and returns the newest version whose
  publish timestamp is at least 3 days old. The 3-day window is a
  security review buffer: a malicious or broken Claude Code release
  has at least 72 hours to be detected before it can land in our PR
  gate. Importable function (with `metadata=` / `fetcher=` / `as_of=`
  injection seams for tests) and a `python -m ...` CLI for the CI step.

- tests/claude_code/test_config.yaml
  Proxy routing config that maps the per-cell aliases the tests use
  (`claude-haiku-4-5`, `claude-haiku-4-5-bedrock-invoke`, ...,
  `claude-opus-4-7-vertex`) to real upstream model ids on Anthropic /
  Bedrock (Invoke + Converse) / Vertex AI. Azure intentionally has no
  entries here because every Azure × claude-code cell is
  `not_applicable` (Azure OpenAI doesn't host Claude).

- .circleci/config.yml
  New `claude_code_compat_pr_gate` job. Pattern modeled on
  `proxy_e2e_anthropic_messages_tests` (load PR-built docker image,
  start postgres, mount config.yaml). New step in the middle:
  resolve the Claude Code version from the resolver, install Node 20
  via the machine image's preinstalled nvm, and `npm install -g
  @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}` (pinned, never
  `latest`). Wired into `workflows.build_and_test` with a
  `requires: [build_docker_database_image]` gate and the same
  `*main_branches` filter the other proxy e2e job uses.

- tests/claude_code/_pr_gate_unit_tests/
  16 new unit tests:
  * 8 against the version resolver: boundary (>= 3d inclusive),
    empty / all-too-new metadata, semver-vs-publish-time tiebreak,
    custom min_age, fetcher injection, npm `time.created` /
    `time.modified` skipping.
  * 8 structural tests against `.circleci/config.yml`: job exists,
    is in the workflow, requires the docker image, invokes the
    resolver, install command is pinned (rejects unpinned `latest`),
    runs `tests/claude_code/`, mounts `test_config.yaml`, exports
    `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY`. Plus one
    regression test: the existing `proxy_e2e_anthropic_messages_tests`
    job is unchanged in shape (acceptance criterion).

Key decisions:

- "Newest version" in the resolver is by **publish time**, not by
  semver string ordering — if a patch lands on an older major after a
  newer release, the patched line is the eligible one. (Tested.)
- The resolver's CLI prints the announcement to stderr and the bare
  version to stdout, so the CI step can do
  `CLAUDE_CODE_VERSION=$(uv run python -m ...)` cleanly while still
  surfacing the selected version in the job log (acceptance criterion:
  "the selected Claude Code version is logged").
- The structural CircleCI tests live under `_pr_gate_unit_tests/` so
  the conftest path-inference hook skips them (the leading underscore
  is the existing convention from `_driver_unit_tests/` /
  `_builder_unit_tests/`); they don't pollute the matrix artifact.
- No `--no-verify` style supply-chain safety relaxation. Per the PRD,
  Claude Code's pinning is the 3-day publish-age window, not a fixed
  hash — by design, since the daily cron also pulls newer versions.

Tests: 47 -> 47 passing for the unit suite (16 new + 31 from slices
1 and 2). The end-to-end cells under `basic_messaging_non_streaming/`
require `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` and a
running proxy + `claude` CLI; they only run inside the new CircleCI
job.

Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- No docs PR is needed for this slice — the gate produces a status
  check, not a published artifact. The compat matrix JSON the docs
  page consumes is published by the daily-cron job (a future slice),
  not by the PR gate.

Notes for next iteration:
- The daily cron / matrix publisher is the next slice. Several
  pieces this slice introduces (the `tests/claude_code/test_config.yaml`
  proxy config, the structure of the compat-results.json artifact)
  will be reused by it.
- The bedrock-converse / vertex_ai aliases in `test_config.yaml` use
  best-guess upstream model ids (`us.anthropic.claude-{tier}` and
  `vertex_ai/claude-{tier}`); the real ids may need to be tightened
  once the gate runs against live AWS / GCP credentials and we see
  what resolves.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
mateo-berri 2026-04-25 05:05:18 +00:00
parent 658c41244c
commit 069dba6e1d
6 changed files with 599 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

96
.circleci/config.yml
View file

@ -2006,6 +2006,98 @@ jobs:
- store_test_results:
path: test-results
claude_code_compat_pr_gate:
# Pre-merge gate for the Claude Code Compatibility Matrix.
#
# Boots the LiteLLM proxy from the PR's code, installs the `claude`
# CLI at a version selected at run time from npm (newest version
# whose publish timestamp is >= 3 days old; the 3-day window is a
# security review buffer), and runs every test under
# `tests/claude_code/`. A red status on this job blocks merge —
# see issue #26479 / PRD #26476 for the design.
machine:
image: ubuntu-2204:2024.04.1
resource_class: large
working_directory: ~/project
steps:
- checkout
- setup_google_dns
- install_uv
- run:
name: Install Dependencies
command: |
uv sync --frozen --all-groups --all-extras --python 3.12
- start_postgres
- attach_workspace:
at: ~/project
- run:
name: Load Docker Database Image
command: |
zstd -d litellm-docker-database.tar.zst --stdout | docker load
docker images | grep litellm-docker-database
- run:
name: Resolve Claude Code CLI version (newest published >= 3 days ago)
command: |
# Run the resolver from the PR's code; capture the version
# to BASH_ENV so subsequent steps see CLAUDE_CODE_VERSION.
CLAUDE_CODE_VERSION=$(uv run --no-sync python -m tests.claude_code.pr_gate_version_resolver)
echo "Selected @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}"
echo "export CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" >> "$BASH_ENV"
- run:
name: Install Node.js 20 + Claude Code CLI
command: |
# The machine image ships with nvm preinstalled; use it to
# pin Node 20 (Claude Code's officially supported runtime).
export NVM_DIR="$HOME/.nvm"
# shellcheck source=/dev/null
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
nvm install 20
nvm use 20
NODE_BIN_DIR="$(dirname "$(command -v node)")"
echo "export PATH=\"${NODE_BIN_DIR}:\$PATH\"" >> "$BASH_ENV"
npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}"
claude --version
- run:
name: Run LiteLLM proxy from PR's code
command: |
docker run -d \
-p 4000:4000 \
-e DATABASE_URL=postgresql://postgres:postgres@host.docker.internal:5432/circle_test \
-e LITELLM_MASTER_KEY="sk-1234" \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-e AWS_REGION_NAME="us-east-1" \
-e LITELLM_LOCAL_ANTHROPIC_BETA_HEADERS="True" \
--add-host host.docker.internal:host-gateway \
--name compat-proxy \
-v $(pwd)/tests/claude_code/test_config.yaml:/app/config.yaml \
litellm-docker-database:ci \
--config /app/config.yaml \
--port 4000 \
--detailed_debug
- run:
name: Stream proxy logs
command: docker logs -f compat-proxy
background: true
- wait_for_service:
url: http://localhost:4000
timeout: "300"
- run:
name: Run Claude Code compatibility test suite
command: |
export LITELLM_PROXY_BASE_URL="http://localhost:4000"
export LITELLM_PROXY_API_KEY="sk-1234"
mkdir -p test-results
uv run --no-sync python -m pytest -vv tests/claude_code/ \
--junitxml=test-results/junit.xml \
--durations=10
no_output_timeout: 30m
# Store test results
- store_test_results:
path: test-results
upload-coverage:
docker:
- *python312_image
@ -2356,6 +2448,10 @@ workflows:
requires:
- build_docker_database_image
filters: *main_branches
- claude_code_compat_pr_gate:
requires:
- build_docker_database_image
filters: *main_branches
- llm_translation_testing:
filters: *main_branches
- realtime_translation_testing:

0
tests/claude_code/_pr_gate_unit_tests/__init__.py Normal file
View file

131
tests/claude_code/_pr_gate_unit_tests/test_circleci_pr_gate_wiring.py Normal file
View file

@ -0,0 +1,131 @@
"""Sanity tests for the CircleCI PR-gate wiring.
Parses `.circleci/config.yml` and asserts the claude_code PR-gate job is
present, in the `build_and_test` workflow, and runs the whole
`tests/claude_code/` suite. This catches the obvious "someone deleted
the job" / "someone deleted the workflow entry" regressions that
otherwise only show up in CI history.
The intent of these tests is *structural*, not *behavioral*: we don't
exercise the docker / npm / proxy machinery here, just verify the YAML
the CircleCI scheduler actually reads.
"""
from __future__ import annotations
from pathlib import Path
import pytest
import yaml
REPO_ROOT = Path(__file__).resolve().parents[3]
CONFIG_PATH = REPO_ROOT / ".circleci" / "config.yml"
JOB_NAME = "claude_code_compat_pr_gate"
@pytest.fixture(scope="module")
def circleci_config() -> dict:
return yaml.safe_load(CONFIG_PATH.read_text())
def _job_step_runs(job: dict) -> list[str]:
"""Return the concatenated `command` text of every `run:` step."""
commands: list[str] = []
for step in job.get("steps", []):
if isinstance(step, dict) and "run" in step:
run = step["run"]
if isinstance(run, dict):
cmd = run.get("command")
if isinstance(cmd, str):
commands.append(cmd)
return commands
def test_pr_gate_job_is_defined(circleci_config: dict) -> None:
assert JOB_NAME in circleci_config["jobs"], (
f"{JOB_NAME} job is missing from .circleci/config.yml — the PR gate "
"is the merge-blocker; deleting it silently disables the gate."
)
def test_pr_gate_job_is_in_build_and_test_workflow(circleci_config: dict) -> None:
workflow = circleci_config["workflows"]["build_and_test"]["jobs"]
job_names = [
next(iter(entry.keys())) if isinstance(entry, dict) else entry
for entry in workflow
]
assert JOB_NAME in job_names, (
f"{JOB_NAME} is defined but not wired into workflows.build_and_test — "
"CircleCI will never run it without this entry."
)
def test_pr_gate_job_requires_docker_database_image(circleci_config: dict) -> None:
"""The proxy is booted from `litellm-docker-database:ci`, so the gate
must wait for that build to finish before it runs."""
workflow = circleci_config["workflows"]["build_and_test"]["jobs"]
entry = next(e[JOB_NAME] for e in workflow if isinstance(e, dict) and JOB_NAME in e)
requires = entry.get("requires", [])
assert "build_docker_database_image" in requires
def test_pr_gate_job_invokes_version_resolver_and_pinned_install(
circleci_config: dict,
) -> None:
"""Acceptance criterion: the CLI is installed at a version computed
at run time from the npm registry, and the version is logged."""
job = circleci_config["jobs"][JOB_NAME]
commands = "\n".join(_job_step_runs(job))
assert "tests.claude_code.pr_gate_version_resolver" in commands, (
"PR-gate job must invoke the version resolver; otherwise the "
"3-day publish-age security buffer is bypassed."
)
# Pinned install of the resolved version (not 'latest', not unversioned)
assert "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" in commands
def test_pr_gate_job_runs_claude_code_test_dir(circleci_config: dict) -> None:
job = circleci_config["jobs"][JOB_NAME]
commands = "\n".join(_job_step_runs(job))
assert "tests/claude_code/" in commands, (
f"{JOB_NAME} must run the tests/claude_code/ suite; otherwise "
"the gate isn't actually exercising the compat tests."
)
def test_pr_gate_job_mounts_test_config_yaml(circleci_config: dict) -> None:
"""The tests reference proxy aliases (`claude-haiku-4-5` etc.) that
only the routing config knows about — the gate must mount it into
the proxy container."""
job = circleci_config["jobs"][JOB_NAME]
commands = "\n".join(_job_step_runs(job))
assert "tests/claude_code/test_config.yaml" in commands
def test_pr_gate_job_exports_proxy_env_used_by_tests(
circleci_config: dict,
) -> None:
"""Tests read LITELLM_PROXY_BASE_URL / LITELLM_PROXY_API_KEY — the
job must export both before the pytest invocation."""
job = circleci_config["jobs"][JOB_NAME]
commands = "\n".join(_job_step_runs(job))
assert "LITELLM_PROXY_BASE_URL" in commands
assert "LITELLM_PROXY_API_KEY" in commands
def test_existing_proxy_e2e_anthropic_job_unchanged(circleci_config: dict) -> None:
"""No regression to the existing `proxy_e2e_anthropic_messages_tests`
job (acceptance criterion). We don't lock its full body, but we do
pin the load-bearing surface: it still runs the same test directory
and is still wired into the workflow."""
assert "proxy_e2e_anthropic_messages_tests" in circleci_config["jobs"]
workflow = circleci_config["workflows"]["build_and_test"]["jobs"]
job_names = [
next(iter(entry.keys())) if isinstance(entry, dict) else entry
for entry in workflow
]
assert "proxy_e2e_anthropic_messages_tests" in job_names
body = "\n".join(
_job_step_runs(circleci_config["jobs"]["proxy_e2e_anthropic_messages_tests"])
)
assert "tests/proxy_e2e_anthropic_messages_tests/" in body

139
tests/claude_code/_pr_gate_unit_tests/test_pr_gate_version_resolver.py Normal file
View file

@ -0,0 +1,139 @@
"""Unit tests for the Claude Code PR-Gate Version Resolver.
The resolver picks the newest `@anthropic-ai/claude-code` version whose
publish timestamp is at least 3 days old. The 3-day window is a security
review buffer: a malicious or broken Claude Code release that slipped
through the npm publish process gets at least 72 hours to be detected
before it can land in the LiteLLM PR gate.
The unit tests inject npm metadata directly (no network) and a fixed
`as_of` clock (no real time), so they run anywhere and never flake on
the wall clock or registry availability.
"""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
import pytest
from tests.claude_code.pr_gate_version_resolver import (
NoEligibleVersionError,
resolve_pr_gate_version,
)
def _t(iso: str) -> str:
"""Helper for readable ISO-8601 publish timestamps in fixtures."""
return iso
# A clock fixed at a moment well after every fixture publish time below.
NOW = datetime(2026, 4, 25, 12, 0, 0, tzinfo=timezone.utc)
def _metadata_with_times(times: dict) -> dict:
"""Shape an npm `packument`-like dict with the `time` field populated.
The npm registry response includes `time.created` / `time.modified`
keys alongside per-version timestamps; the resolver must skip those.
"""
return {
"name": "@anthropic-ai/claude-code",
"time": {
"created": _t("2024-01-01T00:00:00.000Z"),
"modified": _t("2026-04-25T00:00:00.000Z"),
**times,
},
}
def test_picks_newest_version_at_least_three_days_old():
metadata = _metadata_with_times(
{
"2.1.118": _t("2026-04-15T10:00:00.000Z"),
"2.1.119": _t("2026-04-21T10:00:00.000Z"), # 4d 2h old
"2.1.120": _t("2026-04-23T10:00:00.000Z"), # 2d 2h old — too new
"2.1.121": _t("2026-04-25T11:00:00.000Z"), # 1h old — too new
}
)
assert resolve_pr_gate_version(metadata=metadata, as_of=NOW) == "2.1.119"
def test_skips_created_and_modified_meta_keys():
"""`time` contains `created` / `modified` non-version entries — must be ignored."""
metadata = {
"name": "@anthropic-ai/claude-code",
"time": {
"created": _t("2024-01-01T00:00:00.000Z"),
"modified": _t("2026-04-25T00:00:00.000Z"),
"2.0.0": _t("2026-04-10T00:00:00.000Z"),
},
}
assert resolve_pr_gate_version(metadata=metadata, as_of=NOW) == "2.0.0"
def test_min_age_boundary_is_inclusive():
"""A version published exactly 3 days ago is eligible (>= cutoff)."""
three_days_ago = NOW - timedelta(days=3)
metadata = _metadata_with_times(
{
"2.1.0": three_days_ago.isoformat().replace("+00:00", "Z"),
}
)
assert resolve_pr_gate_version(metadata=metadata, as_of=NOW) == "2.1.0"
def test_raises_when_every_version_is_too_new():
metadata = _metadata_with_times(
{
"2.1.121": _t("2026-04-25T08:00:00.000Z"), # 4h old
"2.1.120": _t("2026-04-24T10:00:00.000Z"), # ~26h old
}
)
with pytest.raises(NoEligibleVersionError):
resolve_pr_gate_version(metadata=metadata, as_of=NOW)
def test_raises_when_metadata_has_no_versions():
metadata = {"name": "@anthropic-ai/claude-code", "time": {}}
with pytest.raises(NoEligibleVersionError):
resolve_pr_gate_version(metadata=metadata, as_of=NOW)
def test_picks_latest_publish_time_not_largest_semver():
"""If a patch is published to an old major after a newer release,
"newest" is by publish time, not semver string ordering."""
metadata = _metadata_with_times(
{
"1.9.99": _t("2026-04-22T10:00:00.000Z"), # patched recently — wins
"2.0.0": _t("2026-03-01T10:00:00.000Z"), # older publish
}
)
assert resolve_pr_gate_version(metadata=metadata, as_of=NOW) == "1.9.99"
def test_uses_custom_min_age():
metadata = _metadata_with_times(
{
"1.0.0": _t("2026-04-23T10:00:00.000Z"), # 2d 2h old
"0.9.0": _t("2026-04-10T10:00:00.000Z"), # 15d old
}
)
# min_age = 5 days disqualifies 1.0.0
out = resolve_pr_gate_version(
metadata=metadata, as_of=NOW, min_age=timedelta(days=5)
)
assert out == "0.9.0"
def test_resolver_uses_fetcher_when_metadata_not_provided():
captured = {}
def fake_fetch(package_name: str) -> dict:
captured["package"] = package_name
return _metadata_with_times({"3.0.0": _t("2026-04-10T10:00:00.000Z")})
out = resolve_pr_gate_version(as_of=NOW, fetcher=fake_fetch)
assert out == "3.0.0"
assert captured["package"] == "@anthropic-ai/claude-code"

148
tests/claude_code/pr_gate_version_resolver.py Normal file
View file

@ -0,0 +1,148 @@
"""Claude Code PR-Gate Version Resolver.
Resolves the `@anthropic-ai/claude-code` npm version that the PR-gate CI
job installs. Selects the newest version (by publish timestamp) whose
publish timestamp is at least 3 days old. The 3-day window is a security
review buffer — see PRD #26476, "Version resolvers".
Two surfaces:
- ``resolve_pr_gate_version(...)`` — the importable function. Accepts
pre-fetched npm metadata (for unit tests) or a custom ``fetcher``
callable. The default fetcher hits the public npm registry.
- ``python -m tests.claude_code.pr_gate_version_resolver`` — prints the
resolved version string to stdout, suitable for piping into a shell
``$(...)`` substitution inside the CircleCI job.
The CLI form is what CircleCI runs at job start; engineers reading the
job log can see the selected version on a single line above the
``npm install -g`` step (acceptance criterion: "the selected Claude
Code version is logged in the CI output").
"""
from __future__ import annotations
import json
import sys
import urllib.request
from datetime import datetime, timedelta, timezone
from typing import Callable, Mapping, Optional
PACKAGE_NAME = "@anthropic-ai/claude-code"
NPM_REGISTRY_URL = "https://registry.npmjs.org/{package}"
DEFAULT_MIN_AGE = timedelta(days=3)
DEFAULT_FETCH_TIMEOUT_SECONDS = 30
# npm's `time` map mixes per-version timestamps with these meta keys.
_TIME_META_KEYS = frozenset({"created", "modified"})
class NoEligibleVersionError(RuntimeError):
"""Raised when no version in the npm metadata satisfies the min-age cutoff."""
def _parse_npm_timestamp(value: str) -> datetime:
"""Parse the ISO-8601 timestamps npm emits (always UTC, may use ``Z``)."""
if value.endswith("Z"):
value = value[:-1] + "+00:00"
parsed = datetime.fromisoformat(value)
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=timezone.utc)
return parsed
def _default_fetcher(package_name: str) -> dict:
"""Fetch the npm packument for ``package_name`` over HTTPS.
Uses urllib (stdlib) so this module has no extra dependencies in the
CI environment. Returns the raw JSON dict.
"""
# urllib.parse.quote would encode the leading '@' / '/' which the
# npm registry expects literally; do a minimal hand-roll instead.
url = NPM_REGISTRY_URL.format(package=package_name.replace("/", "%2F"))
req = urllib.request.Request(url, headers={"Accept": "application/json"})
with urllib.request.urlopen( # noqa: S310 — registry URL is constant
req, timeout=DEFAULT_FETCH_TIMEOUT_SECONDS
) as response:
body = response.read().decode("utf-8")
return json.loads(body)
def resolve_pr_gate_version(
*,
metadata: Optional[Mapping] = None,
fetcher: Optional[Callable[[str], Mapping]] = None,
as_of: Optional[datetime] = None,
min_age: timedelta = DEFAULT_MIN_AGE,
package_name: str = PACKAGE_NAME,
) -> str:
"""Return the newest npm version of ``package_name`` published >= ``min_age`` ago.
"Newest" means newest by **publish time**, not semver string order —
if a patch lands on an older major after a newer release, the
patched line is the eligible one.
Args:
metadata: Pre-fetched npm packument (skips the HTTP call). Useful
for unit tests.
fetcher: Callable taking a package name and returning the
packument. Defaults to a stdlib HTTPS fetcher.
as_of: The clock used to decide whether a version is "old
enough". Defaults to ``datetime.now(timezone.utc)``.
min_age: Minimum publish age. Defaults to 3 days.
package_name: Defaults to ``@anthropic-ai/claude-code``.
Raises:
NoEligibleVersionError: when no version in the registry meets
the age cutoff.
"""
if metadata is None:
fetch = fetcher or _default_fetcher
metadata = fetch(package_name)
times = metadata.get("time") or {}
if as_of is None:
as_of = datetime.now(timezone.utc)
cutoff = as_of - min_age
eligible: list[tuple[datetime, str]] = []
for version, raw_ts in times.items():
if version in _TIME_META_KEYS:
continue
if not isinstance(raw_ts, str):
continue
published = _parse_npm_timestamp(raw_ts)
if published <= cutoff:
eligible.append((published, version))
if not eligible:
raise NoEligibleVersionError(
f"no version of {package_name} is at least {min_age} old "
f"as of {as_of.isoformat()}"
)
eligible.sort(key=lambda pair: pair[0], reverse=True)
return eligible[0][1]
def _main(argv: list[str]) -> int:
"""Print the resolved version to stdout. Exit code 0 on success.
Stderr carries the human-readable announcement so the version can be
captured cleanly with ``$(python -m ...)`` in shell.
"""
try:
version = resolve_pr_gate_version()
except Exception as exc: # noqa: BLE001 — CLI surface, want everything
print(f"pr_gate_version_resolver: {exc}", file=sys.stderr) # noqa: T201
return 1
print( # noqa: T201
f"pr_gate_version_resolver: selected {PACKAGE_NAME}@{version}",
file=sys.stderr,
)
print(version) # noqa: T201
return 0
if __name__ == "__main__":
raise SystemExit(_main(sys.argv[1:]))

85
tests/claude_code/test_config.yaml Normal file
View file

@ -0,0 +1,85 @@
# Proxy routing config for the Claude Code Compatibility Matrix PR gate.
#
# The tests under `tests/claude_code/` only know **alias** names (e.g.
# `claude-haiku-4-5-bedrock-invoke`). The proxy is the layer that maps
# each alias to a real upstream model id, region, and credentials.
#
# Adding a new (feature, provider) cell is a three-step change in the
# test repo (manifest + test file + alias here); changing which upstream
# model a cell exercises is a one-step change here, with no test edits.
#
# Aliases:
# - claude-{tier} → Anthropic API
# - claude-{tier}-bedrock-invoke → Bedrock InvokeModel API
# - claude-{tier}-bedrock-converse → Bedrock Converse API
# - claude-{tier}-vertex → GCP Vertex AI
model_list:
# ---- Anthropic ----
- model_name: claude-haiku-4-5
litellm_params:
model: anthropic/claude-haiku-4-5
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-sonnet-4-6
litellm_params:
model: anthropic/claude-sonnet-4-6
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-opus-4-7
litellm_params:
model: anthropic/claude-opus-4-7
api_key: os.environ/ANTHROPIC_API_KEY
# ---- Bedrock (InvokeModel) ----
- model_name: claude-haiku-4-5-bedrock-invoke
litellm_params:
model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
aws_region_name: us-east-1
- model_name: claude-sonnet-4-6-bedrock-invoke
litellm_params:
model: bedrock/us.anthropic.claude-sonnet-4-6
aws_region_name: us-east-1
- model_name: claude-opus-4-7-bedrock-invoke
litellm_params:
model: bedrock/us.anthropic.claude-opus-4-7
aws_region_name: us-east-1
# ---- Bedrock (Converse) ----
- model_name: claude-haiku-4-5-bedrock-converse
litellm_params:
model: bedrock/converse/us.anthropic.claude-haiku-4-5-20251001-v1:0
aws_region_name: us-east-1
- model_name: claude-sonnet-4-6-bedrock-converse
litellm_params:
model: bedrock/converse/us.anthropic.claude-sonnet-4-6
aws_region_name: us-east-1
- model_name: claude-opus-4-7-bedrock-converse
litellm_params:
model: bedrock/converse/us.anthropic.claude-opus-4-7
aws_region_name: us-east-1
# ---- Vertex AI ----
- model_name: claude-haiku-4-5-vertex
litellm_params:
model: vertex_ai/claude-haiku-4-5
vertex_ai_project: pathrise-convert-1606954137718
vertex_ai_location: us-east5
- model_name: claude-sonnet-4-6-vertex
litellm_params:
model: vertex_ai/claude-sonnet-4-6
vertex_ai_project: pathrise-convert-1606954137718
vertex_ai_location: us-east5
- model_name: claude-opus-4-7-vertex
litellm_params:
model: vertex_ai/claude-opus-4-7
vertex_ai_project: pathrise-convert-1606954137718
vertex_ai_location: us-east5
general_settings:
# Claude Code sends provider-specific headers (e.g. anthropic-beta) we
# want to forward verbatim to the upstream so the wire-shape under
# test matches what real customers send.
forward_client_headers_to_llm_api: true
litellm_settings:
drop_params: true
modify_params: true