GitNexus/eval/workflow_bench/runner_artifacts.py
Abhigyan Patwari 136003377a
feat(eval): add opt-in baseline_cli arm to measure MCP catalog cost (#3762)
* feat(eval): add opt-in baseline_cli arm to measure MCP catalog cost

The MCP arm costs ~22-43% more than baseline_nomcp in every release
evaluation, with no solve-rate gain. Almost all of the extra is re-read
input context, not output. baseline_cli separates the tool catalog's
share of that from tool-output size and extra turns.

- cli_shim: a standard-library `gitnexus` command, mounted at the CLI
  path, forwards one call per invocation to the same per-cell isolated
  MCP endpoint the baseline arm uses, restricted to the same read-only
  tools. Help is on demand (`gitnexus help [command]`); no tool schema
  enters the model context, and the runtime stays in the sibling sandbox.
- runner/proposer_sandbox: wire the arm (Bubblewrap only) with a short
  prompt note and an empty MCP config.
- cli_arm_report: descriptive comparison with the paired arms, counting
  `gitnexus` shell calls from transcripts.
- release-evaluation: `cli_arm` dispatch input runs the arm on the
  candidate into its own output and uploads release-cli-arm-<sha>. The
  paired evidence, release report and gate are unchanged.
- tests: shim argument mapping, a fake-MCP round trip, the sandbox
  mount, the report, and a real-Bubblewrap canary (added to the ubuntu
  containment job and the pre-paid canary step).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(eval): forward explicit empty shim options; drop unused reuse entry

- cli_shim: `--flag=` now forwards an empty string instead of treating
  the option as valueless (test added).
- comparator_reuse: remove baseline_cli from REUSABLE_COMPARATOR_ARMS.
  The skill-digest gate rejects every non-CE arm without a digest, so
  listing it implied reuse that cannot happen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(eval): keep boolean shim switches from consuming positionals

`gitnexus impact --include-tests foo` took `foo` as the switch value and
left the required target unset. Boolean-only options now consume the
next token only when it is an explicit true/false (tests added).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(eval): let the baseline_cli shim open its MCP socket from Bash

The first paid baseline_cli cell showed every `gitnexus` call failing
with EPERM on socket(AF_UNIX): Claude Code's Bash sandbox blocks Unix
sockets (allowAllUnixSockets: false). The MCP arm never hit this because
Claude's MCP client runs socat outside the Bash sandbox. Shim sessions now
set allowAllUnixSockets; the bwrap mounts still expose only the cell's MCP
endpoint, and every other arm keeps the block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(eval): keep PR previews valid alongside opt-in descriptive arms

measurement_protocol required exactly one `-m workflow_bench.runner` step,
so the cli_arm step made it raise on the checked-in workflow; any later
workflow change would then break PR cost previews. It now still requires
exactly one ungated paid step, and allows extra runner steps only with an
exact opt-in gate (env.CLI_ARM / env.CODE_ARM) placed after the paid step,
outside the fingerprint. Tests cover the allowed case, misplaced and
unlisted extra arms, and the checked-in workflow itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(eval): let the shim's first value name the tool's subject

The first paid baseline_cli cell's opening call, `gitnexus impact
statusCommand --direction upstream`, failed with "too many values":
impact requires only `direction` (target is optional beside target_uid),
so no positional slot was left for the symbol. Positional values now fill
the first string property (target, name, search_query, from) before the
required ones (test added with the real impact requirement).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: expect the opt-in CLI_ARM key in the release evaluate env

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(eval): explicit report text join; count only command-position CLI calls

- cli_arm_report: join the summary sentence with `+` (CodeQL implicit
  string concatenation in a list).
- Count `gitnexus <command>` only in command position. In this repo
  `npm --prefix gitnexus test` names the package directory and was being
  counted as a CLI call (test added).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(eval): run the shim canary against the real impact contract

The real-Bubblewrap canary used a fixture where impact requires `target`,
and its fake server answered any call. It now serves REAL_IMPACT
(direction required, as in gitnexus/src/mcp/tools.ts), rejects calls
missing a required argument, and checks both the flag and positional
forms plus a rejected call without a direction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(eval): pin test_cli_shim.py in the containment job expectations

The ubuntu containment job now runs tests/test_cli_shim.py (its real-
Bubblewrap shim canary is skipped everywhere else), so the pinned
pytest selection and the bwrap-canary file list include it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(eval): compare CLI-arm deltas on matched task/run pairs

The CLI arm report averaged each arm separately, so an interrupted sweep
(or a cell with no billing) skewed the deltas with no warning. Deltas now
use only task/run pairs present once in both arms with the metric
measured in both, and are labelled with the pair count ("partial" when
fewer than expected). A metric missing on any cell leaves that arm's mean
unknown and is listed as a problem; duplicate pairs are flagged and
excluded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(eval): read CLI-arm transcripts from the persisted artifact records

Result rows store transcript_artifacts as the {"path", "sha256", "bytes",
"source"} records runner_sessions persists, not bare paths, so joining
results_dir with an entry raised TypeError on real output. Read the
record's path, and skip anything that resolves outside the results dir.
The report test now uses the real record shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
2026-10-29 11:46:52 +00:00

688 lines
26 KiB
Python

"""Workspace, patch, and verifier evidence for workflow benchmark runs."""
from __future__ import annotations
import hashlib
import os
import re
import shlex
import shutil
import stat
import sys
import tempfile
from dataclasses import dataclass
from pathlib import Path, PurePosixPath
from typing import Any, Iterator, Sequence
from .evolution import skill_fingerprint
from .process_control import ManagedProcessError, ManagedProcessResult, run_checked, run_managed
from .proposer_sandbox import SANDBOX_WORKSPACE, SandboxSession, build_sandbox_environment
MAX_PATCH_BYTES = 300_000
MAX_WORKSPACE_SNAPSHOT_ENTRIES = 100_000
MAX_WORKSPACE_SNAPSHOT_PATH_BYTES = 16 * 1024 * 1024
MAX_WORKSPACE_SNAPSHOT_FILE_BYTES = 1024 * 1024 * 1024
# Claude Code's own enableWeakerNestedSandbox bootstrap creates these paths on
# EVERY session regardless of task or model output -- reproduced empirically
# with a trivial "say OK" prompt: a synthetic package.json/lockfiles/
# node_modules, a full set of .env variants, and .claude/agents,
# .claude/commands, .claude/.cc-writes. None of this is something the model
# decided to write, so it must not count as an "unauthorized" workspace
# change during the planning-phase boundary check (the one thing this
# snapshot is used for -- see workspace_snapshot's callers). Mirrors the
# pre-existing .git exclusion below, which is the same kind of harness/tool
# noise rather than substantive diff.
WORKSPACE_SNAPSHOT_BOOTSTRAP_NOISE = frozenset(
{
".claude",
".bash_profile",
".bashrc",
".env",
".env.development",
".env.development.local",
".env.local",
".env.production",
".env.production.local",
".env.test",
".env.test.local",
".gitmodules",
".gitconfig",
".idea",
".npmrc",
".profile",
".ripgreprc",
".vscode",
".yarnrc",
".yarnrc.yml",
".zprofile",
".zshrc",
"bunfig.toml",
"node_modules",
"package-lock.json",
"package.json",
"pnpm-lock.yaml",
"yarn.lock",
}
)
# Claude Code may drop a workspace-root `scripts` *file* during bootstrap.
# Only that exact entry is noise — a `scripts/` directory is real workspace.
WORKSPACE_SNAPSHOT_ROOT_FILE_NOISE = frozenset({"scripts"})
# The set above is matched at the workspace ROOT only, because most of its
# entries (package.json, node_modules, the .env family) are also legitimate
# repository content further down the tree -- gitnexus/package.json and
# gitnexus/.claude/settings.local.json are both tracked files whose edits must
# still be caught. But Claude Code bootstraps into whatever directory it is
# running in, so a task whose prompt cd's into a subdirectory gets the same
# noise one level down. Observed in skill-evolution run 29861768554: 13 of 18
# sessions failed with "phase changed unauthorized workspace path(s):
# gitnexus/.claude/.cc-writes". That entry is matched at ANY depth -- never
# ".claude" itself, which holds real configuration.
#
# Deliberately only .cc-writes. Every excluded name is a blind spot: once a
# .claude directory already exists (gitnexus/.claude/settings.local.json is
# tracked), anything a phase writes underneath an excluded entry becomes
# invisible to this check, and Claude Code loads .claude/agents relative to
# its cwd -- which these tasks point at gitnexus/. Adding "agents" and
# "commands" here on the theory that they might also appear nested would let a
# planning phase plant a definition that the later work phase reads, with no
# evidence in the boundary check. Only .cc-writes was ever observed nested, so
# only .cc-writes is excluded; extend this set from an observed failure, never
# pre-emptively.
CLAUDE_BOOTSTRAP_DIR = ".claude"
CLAUDE_BOOTSTRAP_ENTRIES = frozenset({".cc-writes"})
IMPLEMENTATION_ARMS = frozenset(
{
"workflow",
"workflow_direct",
"ce_workflow",
"ce_workflow_direct",
"baseline",
"baseline_cli",
"baseline_nomcp",
}
)
@dataclass(frozen=True)
class VerificationResult:
"""Verifier output that preserves infrastructure terminal state."""
command: Sequence[str] | str
process: ManagedProcessResult
output: str
@property
def passed(self) -> bool:
return self.process.ok
def __iter__(self) -> Iterator[bool | str]:
# Preserve the historical two-value unpacking API for standalone
# callers while runner.py inspects ``process.state`` explicitly.
yield self.passed
yield self.output
def _is_bootstrap_noise(
relative: PurePosixPath,
*,
is_dir: bool = False,
is_file: bool = False,
) -> bool:
"""Report whether a walked entry is harness noise rather than workspace change."""
parts = relative.parts
if parts[0] == ".git" or parts[0] in WORKSPACE_SNAPSHOT_BOOTSTRAP_NOISE:
return True
if (
is_file
and not is_dir
and len(parts) == 1
and parts[0] in WORKSPACE_SNAPSHOT_ROOT_FILE_NOISE
):
return True
return len(parts) >= 2 and parts[-2] == CLAUDE_BOOTSTRAP_DIR and parts[-1] in CLAUDE_BOOTSTRAP_ENTRIES
def workspace_snapshot(worktree: Path) -> dict[str, str]:
"""Hash the workspace without following links, excluding Git internals
and Claude Code's own sandbox-bootstrap noise (see
WORKSPACE_SNAPSHOT_BOOTSTRAP_NOISE)."""
root = worktree.expanduser().absolute()
mode = root.lstat().st_mode
if stat.S_ISLNK(mode) or not stat.S_ISDIR(mode) or root.resolve(strict=True) != root:
raise ValueError(f"workspace snapshot root must be a real directory: {root}")
snapshot: dict[str, str] = {}
pending: list[tuple[Path, PurePosixPath]] = [(root, PurePosixPath())]
entry_count = 0
path_bytes = 0
file_bytes = 0
nofollow = getattr(os, "O_NOFOLLOW", 0)
while pending:
directory, relative_dir = pending.pop()
try:
children = sorted(os.scandir(directory), key=lambda entry: entry.name, reverse=True)
except OSError as exc:
raise ValueError(f"workspace snapshot directory is unreadable: {directory}: {exc}") from exc
for entry in children:
relative = relative_dir / entry.name
if _is_bootstrap_noise(
relative,
is_dir=entry.is_dir(follow_symlinks=False),
is_file=entry.is_file(follow_symlinks=False),
):
continue
entry_count += 1
path_bytes += len(relative.as_posix().encode())
if entry_count > MAX_WORKSPACE_SNAPSHOT_ENTRIES or path_bytes > MAX_WORKSPACE_SNAPSHOT_PATH_BYTES:
raise ValueError("workspace snapshot exceeds its bounded entry or path limit")
metadata = entry.stat(follow_symlinks=False)
permissions = stat.S_IMODE(metadata.st_mode)
if stat.S_ISDIR(metadata.st_mode):
snapshot[relative.as_posix()] = f"d:{permissions:o}"
pending.append((Path(entry.path), relative))
continue
if stat.S_ISLNK(metadata.st_mode):
snapshot[relative.as_posix()] = f"l:{permissions:o}:{os.readlink(entry.path)}"
continue
if not stat.S_ISREG(metadata.st_mode):
snapshot[relative.as_posix()] = f"s:{metadata.st_mode}"
continue
file_bytes += metadata.st_size
if file_bytes > MAX_WORKSPACE_SNAPSHOT_FILE_BYTES:
raise ValueError("workspace snapshot exceeds its bounded file-byte limit")
descriptor = os.open(entry.path, os.O_RDONLY | nofollow)
try:
opened = os.fstat(descriptor)
if (
not stat.S_ISREG(opened.st_mode)
or opened.st_dev != metadata.st_dev
or opened.st_ino != metadata.st_ino
):
raise ValueError(f"workspace file changed while opening: {entry.path}")
digest = hashlib.sha256()
while chunk := os.read(descriptor, 64 * 1024):
digest.update(chunk)
after = os.fstat(descriptor)
if (opened.st_size, opened.st_mtime_ns) != (after.st_size, after.st_mtime_ns):
raise ValueError(f"workspace file changed while hashing: {entry.path}")
finally:
os.close(descriptor)
snapshot[relative.as_posix()] = f"f:{permissions:o}:{metadata.st_size}:{digest.hexdigest()}"
return snapshot
def enforce_phase_workspace(
worktree: Path,
before: dict[str, str],
*,
allowed_artifact: Path | None,
) -> None:
"""Require a phase to change only its one explicit workspace artifact.
``allowed_artifact=None`` is the stricter contract: the phase must leave
the workspace byte-identical. That is what a review phase whose artifact
lives outside the workspace has to satisfy — there is nothing in there it
is entitled to touch.
"""
root = worktree.expanduser().absolute()
if allowed_artifact is None:
after = workspace_snapshot(root)
changed = sorted(
path for path in before.keys() | after.keys() if before.get(path) != after.get(path)
)
if changed:
raise ValueError(f"phase changed the read-only workspace: {', '.join(changed[:5])}")
return
artifact = allowed_artifact.expanduser().absolute()
try:
relative = PurePosixPath(artifact.relative_to(root).as_posix())
except ValueError as exc:
raise ValueError(f"phase artifact escapes the workspace: {allowed_artifact}") from exc
after = workspace_snapshot(root)
changed = {path for path in before.keys() | after.keys() if before.get(path) != after.get(path)}
artifact_key = relative.as_posix()
artifact_state = after.get(artifact_key)
if before.get(artifact_key) == artifact_state:
raise ValueError(f"phase did not create or change its required artifact: {relative}")
if artifact_state is None or not artifact_state.startswith("f:"):
raise ValueError(f"phase artifact must be a regular non-symlink file: {relative}")
try:
metadata = artifact.lstat()
descriptor = os.open(artifact, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0))
except OSError as exc:
raise ValueError(f"phase artifact must be a readable regular non-symlink file: {relative}") from exc
try:
opened = os.fstat(descriptor)
if (
stat.S_ISLNK(metadata.st_mode)
or not stat.S_ISREG(metadata.st_mode)
or not stat.S_ISREG(opened.st_mode)
or metadata.st_dev != opened.st_dev
or metadata.st_ino != opened.st_ino
):
raise ValueError(f"phase artifact must be a regular non-symlink file: {relative}")
finally:
os.close(descriptor)
allowed = {artifact_key}
parent = relative.parent
while parent.parts:
parent_key = parent.as_posix()
if parent_key not in before and after.get(parent_key, "").startswith("d:"):
allowed.add(parent_key)
parent = parent.parent
unauthorized = sorted(changed - allowed)
if unauthorized:
preview = ", ".join(unauthorized[:8])
suffix = " …" if len(unauthorized) > 8 else ""
raise ValueError(f"phase changed unauthorized workspace path(s): {preview}{suffix}")
def require_skill_fingerprint(worktree: Path, arm: str, expected: str | None, *, phase: str) -> None:
"""Fail closed when a bounded phase changes the evaluated prompt roots."""
try:
observed = skill_fingerprint(worktree, arm)
except (OSError, ValueError) as exc:
raise ValueError(f"{phase} changed the evaluated skill fingerprint") from exc
if observed != expected:
raise ValueError(f"{phase} changed the evaluated skill fingerprint")
def snapshot_plan_docs(worktree: Path) -> dict[Path, str]:
"""Hash direct, regular plan artifacts without following links."""
plans = worktree / "docs" / "plans"
if not plans.exists():
return {}
if plans.is_symlink() or not plans.is_dir():
raise ValueError(f"plan directory must be a real directory: {plans}")
snapshot: dict[Path, str] = {}
for path in sorted(plans.iterdir()):
if path.suffix.lower() not in {".md", ".html"}:
continue
metadata = path.lstat()
if stat.S_ISLNK(metadata.st_mode):
raise ValueError(f"plan artifact cannot be a symlink: {path}")
if not stat.S_ISREG(metadata.st_mode):
raise ValueError(f"plan artifact must be a regular file: {path}")
descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0))
try:
opened = os.fstat(descriptor)
if not stat.S_ISREG(opened.st_mode) or opened.st_dev != metadata.st_dev or opened.st_ino != metadata.st_ino:
raise ValueError(f"plan artifact changed while opening: {path}")
with os.fdopen(descriptor, "rb", closefd=False) as handle:
snapshot[path] = hashlib.file_digest(handle, "sha256").hexdigest()
after = os.fstat(descriptor)
if (opened.st_size, opened.st_mtime_ns) != (after.st_size, after.st_mtime_ns):
raise ValueError(f"plan artifact changed while hashing: {path}")
finally:
os.close(descriptor)
return snapshot
def new_plan_doc(worktree: Path, before: dict[Path, str]) -> Path:
"""Return the sole new or modified plan, rejecting ambiguous evidence."""
after = snapshot_plan_docs(worktree)
deleted = sorted(path for path in before if path not in after)
if deleted:
raise ValueError("planning deleted existing plan artifact(s): " + ", ".join(str(path) for path in deleted))
changed = sorted(path for path, digest in after.items() if before.get(path) != digest)
if len(changed) != 1:
raise ValueError(f"planning must create or modify exactly one plan artifact; observed {len(changed)}")
return changed[0]
def _assert_self_contained_git_objects(clone: Path) -> None:
"""Refuse clones that share pack/object bytes with another repository."""
alternates = clone / ".git" / "objects" / "info" / "alternates"
if alternates.exists():
raise RuntimeError(f"clone unexpectedly has an external object alternate: {alternates}")
objects = clone / ".git" / "objects"
if not objects.is_dir():
raise RuntimeError(f"clone is missing a git object store: {clone}")
for obj in objects.rglob("*"):
if obj.is_file() and obj.stat().st_nlink > 1:
raise RuntimeError(f"clone object is hardlinked to host storage: {obj}")
def copy_isolated_tree(source: Path, parent: Path) -> Path:
"""Copy a sanitized clone without sharing git objects or a ref namespace.
``git clone --no-local`` of GitNexus plus ``sanitize_clone_for_hidden_oracles``
(repack/prune/fsck) is minutes per cell. After sanitization the snapshot is
one parentless commit; copying that tree is the isolation boundary the
contamination bug actually required (a private ``.git``), not a second
fetch of full history. Prefer ``cp --reflink=auto`` so XFS/btrfs pay COW;
fall back to a full copy on filesystems that cannot reflink.
"""
try:
source_meta = source.expanduser().lstat()
except OSError as exc:
raise RuntimeError(f"clone template is unavailable: {source}: {exc}") from exc
if stat.S_ISLNK(source_meta.st_mode) or not stat.S_ISDIR(source_meta.st_mode):
raise RuntimeError(f"clone template must be a real directory: {source}")
source = source.expanduser().resolve()
target = Path(tempfile.mkdtemp(prefix="wfbench-", dir=parent))
target.rmdir()
try:
copied = run_managed(
["cp", "-a", "--reflink=auto", str(source), str(target)],
timeout=600,
)
if not copied.ok:
# The fallback is for a filesystem that cannot reflink, which shows
# up as a normal nonzero exit. A cancellation or timeout is reported
# the same way (run_managed returns it rather than raising), and
# copytree cannot be cancelled — so falling back there makes the
# outage breaker wait out the full copy it set the event to avoid.
if copied.state != "exited":
raise ManagedProcessError(["cp", "-a", "--reflink=auto", str(source), str(target)], copied)
shutil.copytree(source, target, symlinks=True, copy_function=shutil.copy2)
_assert_self_contained_git_objects(target)
return target
except BaseException as primary:
if target.exists():
try:
shutil.rmtree(target)
except OSError as cleanup:
primary.add_note(f"clone copy cleanup also failed: {type(cleanup).__name__}: {cleanup}")
raise
def make_worktree(repo: Path, ref: str, parent: Path) -> Path:
"""Create a self-contained clone per benchmark arm."""
target = Path(tempfile.mkdtemp(prefix="wfbench-", dir=parent))
target.rmdir()
try:
run_checked(
[
"git",
"clone",
"--no-local",
"--no-hardlinks",
"--no-tags",
"--quiet",
str(repo),
str(target),
],
timeout=600,
)
_assert_self_contained_git_objects(target)
for candidate in (ref, f"origin/{ref}"):
proc = run_managed(
["git", "-C", str(target), "checkout", "--detach", "--quiet", candidate],
timeout=60,
)
if proc.ok:
return target
raise RuntimeError(f"ref {ref!r} not found in clone of {repo}")
except BaseException as primary:
if target.exists():
try:
shutil.rmtree(target)
except OSError as cleanup:
primary.add_note(f"clone cleanup also failed: {type(cleanup).__name__}: {cleanup}")
raise
def remove_clone(clone: Path) -> None:
"""Delete one throwaway arm clone (created by make_worktree)."""
shutil.rmtree(clone)
def parse_shortstat(text: str) -> dict[str, int]:
"""Parse `git diff --shortstat` output into churn counters."""
keys = {
"file": "diff_files",
"insertion": "diff_insertions",
"deletion": "diff_deletions",
}
out = dict.fromkeys(keys.values(), 0)
for count, word in re.findall(r"(\d+) (file|insertion|deletion)", text):
out[keys[word]] = int(count)
return out
def _sandbox_git(sandbox: SandboxSession, args: list[str], *, timeout: int = 60) -> str:
command = ["/usr/bin/git", "-c", "core.fsmonitor=false", *args]
result = sandbox.run(command, timeout=timeout, env=build_sandbox_environment())
if not result.ok:
raise ManagedProcessError(command, result)
return result.stdout_tail
def _prepare_untracked_for_diff(sandbox: SandboxSession) -> None:
_sandbox_git(sandbox, ["add", "--intent-to-add", "-A"])
def _synthetic_guidance_exclusions(sandbox: SandboxSession) -> list[str]:
return [f":(exclude,literal){path}" for path in sandbox.synthetic_guidance_paths]
def implementation_diff_digest(
sandbox: SandboxSession,
orig_sha: str,
*,
prepare_untracked: bool = True,
) -> str:
"""Digest non-plan final work entirely inside the containment boundary."""
if not re.fullmatch(r"[0-9a-fA-F]{40,64}", orig_sha):
raise ValueError(f"unsafe git object id: {orig_sha!r}")
if prepare_untracked:
_prepare_untracked_for_diff(sandbox)
exclusions = " ".join(shlex.quote(path) for path in _synthetic_guidance_exclusions(sandbox))
command = (
"/usr/bin/git -c core.fsmonitor=false diff --no-ext-diff --no-textconv --binary "
f"{orig_sha} -- . ':(exclude)docs/plans' ':(exclude).claude/skills' "
f"{exclusions} | /usr/bin/sha256sum"
)
result = sandbox.run(
["/bin/sh", "-c", command],
timeout=60,
env=build_sandbox_environment(),
)
if not result.ok:
raise ManagedProcessError(command, result)
digest = result.stdout_tail.strip().split()[0] if result.stdout_tail.strip() else ""
if not re.fullmatch(r"[0-9a-f]{64}", digest):
raise RuntimeError("sandboxed git diff did not produce a SHA-256 digest")
return digest
def diff_churn(
sandbox: SandboxSession,
orig_sha: str,
*,
prepare_untracked: bool = True,
) -> dict[str, int]:
"""Return code churn versus the arm's starting SHA."""
if prepare_untracked:
_prepare_untracked_for_diff(sandbox)
output = _sandbox_git(
sandbox,
[
"diff",
"--no-ext-diff",
"--no-textconv",
"--shortstat",
orig_sha,
"--",
".",
":(exclude)docs/plans",
":(exclude).claude/skills",
*_synthetic_guidance_exclusions(sandbox),
],
)
return parse_shortstat(output)
def _bounded_regular_bytes(path: Path, *, limit: int) -> bytes:
"""Read at most ``limit`` bytes without following a generated link."""
metadata = path.lstat()
if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode):
raise RuntimeError(f"generated artifact is not a regular non-symlink file: {path}")
nofollow = getattr(os, "O_NOFOLLOW", 0)
descriptor = os.open(path, os.O_RDONLY | nofollow)
try:
opened = os.fstat(descriptor)
if not stat.S_ISREG(opened.st_mode):
raise RuntimeError(f"generated artifact changed type while opening: {path}")
chunks: list[bytes] = []
remaining = limit
while remaining > 0:
chunk = os.read(descriptor, min(64 * 1024, remaining))
if not chunk:
break
chunks.append(chunk)
remaining -= len(chunk)
return b"".join(chunks)
finally:
os.close(descriptor)
def capture_patch(sandbox: SandboxSession, worktree: Path, orig_sha: str) -> bytes:
"""Stream a final patch inside the sandbox while retaining a bounded prefix."""
artifact_dir = Path(tempfile.mkdtemp(prefix=".wfbench-artifact-", dir=worktree))
artifact_dir.chmod(0o700)
patch = artifact_dir / "final.patch"
sandbox_path = f"{SANDBOX_WORKSPACE}/{artifact_dir.relative_to(worktree).as_posix()}/final.patch"
sink = """\
import subprocess
import sys
limit = int(sys.argv[1])
output = sys.argv[2]
command = sys.argv[3:]
with open(output, "xb") as handle:
process = subprocess.Popen(command, stdout=subprocess.PIPE)
assert process.stdout is not None
remaining = limit
for chunk in iter(lambda: process.stdout.read(65536), b""):
if remaining:
retained = chunk[:remaining]
handle.write(retained)
remaining -= len(retained)
process.stdout.close()
returncode = process.wait()
if returncode:
raise SystemExit(returncode)
"""
command = [
"/usr/bin/python3",
"-I",
"-c",
sink,
str(MAX_PATCH_BYTES),
sandbox_path,
"/usr/bin/git",
"-c",
"core.fsmonitor=false",
"diff",
"--no-ext-diff",
"--no-textconv",
"--binary",
orig_sha,
"--",
".",
":(exclude).wfbench-artifact-*",
*_synthetic_guidance_exclusions(sandbox),
]
primary: BaseException | None = None
try:
result = sandbox.run(command, timeout=60, env=build_sandbox_environment())
if not result.ok:
raise ManagedProcessError(command, result)
return _bounded_regular_bytes(patch, limit=MAX_PATCH_BYTES)
except BaseException as exc:
primary = exc
raise
finally:
# The returned bytes are the evidence; this temporary sink must not
# become an agent edit in later Git diffs or repeated captures.
try:
shutil.rmtree(artifact_dir)
except OSError as cleanup:
# A failed removal must neither discard the patch already read nor
# replace the real failure. The leftover directory is excluded from
# the patch pathspec above and dies with the disposable clone, so
# it cannot corrupt the evidence; still report it rather than hide it.
detail = f"patch artifact cleanup failed: {artifact_dir}: {type(cleanup).__name__}: {cleanup}"
if primary is not None:
primary.add_note(detail)
else:
print(f"warning: {detail}", file=sys.stderr, flush=True)
def enforce_work_evidence(
record: dict[str, Any],
*,
arm: str,
before_digest: str,
after_digest: str,
) -> None:
if arm not in IMPLEMENTATION_ARMS or not record.get("resolved"):
return
if before_digest != after_digest:
return
record["resolved"] = False
record["error_kind"] = "no-work-produced"
record["error_detail"] = "verifier passed but the implementation arm produced no non-plan repository change"
def run_verify(
command: str,
cwd: Path,
timeout: int,
*,
command_prefix: list[str] | None = None,
env: dict[str, str] | None = None,
require_pid_namespace: bool = False,
) -> VerificationResult:
"""Run the task's verify command; keep its output tail for diagnosis."""
if command_prefix:
# HOME is writable during model execution. A non-login shell prevents
# candidate-created profile files from running inside trusted evidence
# collection or either verifier.
managed_command: list[str] | str = [*command_prefix, "/bin/sh", "-c", command]
shell = False
managed_cwd: Path | None = None
else:
managed_command = command
shell = True
managed_cwd = cwd
proc = run_managed(
managed_command,
shell=shell,
cwd=managed_cwd,
env=env,
timeout=timeout,
require_pid_namespace=require_pid_namespace,
)
output = proc.stdout_tail + "\n" + proc.stderr_tail
if proc.detail:
output += f"\n[{proc.state}] {proc.detail}"
return VerificationResult(
command=managed_command,
process=proc,
output=output[-4000:],
)