diff --git a/eval/README.md b/eval/README.md index 92e45e180..addf2db31 100644 --- a/eval/README.md +++ b/eval/README.md @@ -24,6 +24,8 @@ Evaluate whether GitNexus code intelligence improves AI agent performance on rea - DeepSeek - Any model supported by litellm (add a YAML config) +For long-horizon tasks, [`deepswe/`](deepswe/README.md) runs a paired baseline vs GitNexus CLI comparison on [DeepSWE](https://deepswe.datacurve.ai/) with its own harness (pier) on a dedicated EC2 instance. + ## Prerequisites - Python 3.11+ diff --git a/eval/deepswe/README.md b/eval/deepswe/README.md new file mode 100644 index 000000000..934aa4265 --- /dev/null +++ b/eval/deepswe/README.md @@ -0,0 +1,90 @@ +# DeepSWE: baseline vs GitNexus CLI + +[DeepSWE](https://deepswe.datacurve.ai/) has 113 long-horizon tasks across 91 repositories in five languages. The tasks were written from scratch, and the solutions need about 5.5x more code than SWE-bench Pro. This directory runs a paired comparison on a sampled subset with the benchmark's own harness, [pier](https://github.com/datacurve-ai/pier): + +| Arm | What the agent gets | +| --- | --- | +| `baseline` | The DeepSWE task, unchanged | +| `gitnexus` | The same task, plus the `gitnexus` CLI and an index of `/app` built into the image, and a short note on the CLI appended to the instruction | + +The default agent is Claude Code (pier's `claude-code`, pinned to the release evaluation's CLI). GitNexus is reached through the real CLI (`gitnexus query`, `context`, `impact`, `detect-changes`) rather than MCP. The arms differ only in that layer and that note. + +- **Toolset.** Claude Code gets only `Read`, `Edit`, `Write` and `Bash`, matching the release evaluation's single-agent toolset. Subagents, background tasks, skills and web tools are disallowed, and pier's forced background-task env is switched off. Without this, one smoke trial ran 237 background agents past its budget cap. +- **Guidance.** The note (`--guidance product`, the default) is the CLI form of the rules GitNexus ships in AGENTS.md: graph first, `impact` before editing a symbol, `detect-changes` before committing. In the release evaluation, arms load those directives the same way. `--guidance optional` only lists the commands; in a smoke trial the agent then never used them. + +The workflow_bench `baseline_cli` arm points the same way: on the small release tasks, a shell command cost far less than the MCP tool catalog. `--agent mini-swe-agent`, the leaderboard's agent, also works. + +### Model access + +- **Non-Claude models** (default `gpt-5.6-sol`, `OPENAI_API_KEY`): `run.sh` starts a LiteLLM gateway container. It is routed like `workflow_bench.model_gateway`, with Claude Code's Anthropic Messages served from OpenAI's Responses API, a random master key, and the LiteLLM version from `eval/uv.lock`. + - It listens on host port 80, because pier's egress proxy only allows ports 80 and 443. Keep that port closed to the outside; `launch.sh`'s security group only opens SSH. + - Trials reach it as `host.docker.internal` on Docker Desktop, or as the EC2 private DNS name. +- **Claude models** (`--model anthropic/`): use `ANTHROPIC_API_KEY` directly. + +Costs come from Claude Code's own per-session accounting (`total_cost_usd`). Through the gateway that is Claude Code's price table applied to a non-Claude model, so it is consistent across arms but is not the provider's invoice. + +## How the GitNexus arm is built + +Task agents run with no network (`allow_internet = false`; pier allowlists only the model API). Nothing can be installed at run time, so `prepare.py` gives each `gitnexus` task a derived Dockerfile: + +1. `FROM` the task's prebuilt image. +2. Node from the official tarball, checked against `SHASUMS256.txt`. +3. `npm install -g` of a `gitnexus` tarball built from this checkout. +4. `gitnexus analyze --index-only`. This writes no AGENTS.md, CLAUDE.md or skills, and `.gitnexus/` is git-excluded. The build fails if `git status` changed, because that would leak into the graded patch. + +The task is renamed `-gitnexus`, so pier's per-task image tag never reuses a baseline image. The verifier, its separate image, the patch collector and the timeouts are untouched; only the build timeout is raised to fit indexing. `report.py` strips the suffix and pairs trials by task id. + +## Run it on EC2 + +Use a dedicated instance, not the self-hosted release-evaluation runner. It needs an authenticated AWS CLI (`aws login`). + +```bash +eval/deepswe/ec2/launch.sh --ref main # add --dry-run to print the plan only +ssh -i ~/.ssh/gitnexus-deepswe.pem ubuntu@ # wait for ~/BOOTSTRAP_DONE +export OPENAI_API_KEY=... # in the session, never in user-data +cd ~/GitNexus && eval/deepswe/run.sh --n-tasks 10 --concurrency 4 +``` + +`launch.sh` does the following: + +- Starts Ubuntu 24.04 on an `m7i.4xlarge` (16 vCPU / 64 GiB) with a 400 GB gp3 volume. +- Tags the instance `purpose=gitnexus-deepswe`. +- Opens SSH to your current IP only. +- Stops the instance after `--max-hours` (default 12). + +It prints the `scp` and `terminate-instances` commands for when you're done. + +Task images are about 4 GB each, plus pier's agent layer and the GitNexus layer, so 400 GB fits roughly 40–50 tasks across both arms. For all 113 tasks, pass `--volume-gb 1000` or run `docker image prune -a` between batches. + +`run.sh` performs these steps: + +1. Pins `deep-swe` to a commit. +2. Builds the gitnexus package (the same steps as the CI package job). +3. Prepares both arms on one seeded subset. +4. Runs `pier run` per arm. +5. Writes `deepswe-comparison.{md,json}`. + +The main options are listed below; `run.sh --help` shows them all. + +| Option | Default | Notes | +| --- | --- | --- | +| `--n-tasks`, `--seed`, `--task` | 10, 0 | `--task` (repeatable) overrides the sample; `--n-tasks 0` runs all 113 | +| `--agent`, `--model`, `--effort` | `claude-code`, `gpt-5.6-sol`, `low` | `--model anthropic/` skips the gateway. `--agent mini-swe-agent` takes litellm names such as `openai/gpt-5.6-sol`. | +| `--cost-limit` | `5` | Per-trial agent cost cap in USD, the same for both arms. A capped trial ends unsolved. | +| `--concurrency` | `2` | Trials per arm at once. Each needs 2 CPUs and 8 GB. | +| `--package` | build from checkout | Pass a prebuilt `npm pack` tarball to skip the build | + +## Reading the report + +`report.py` pairs the two pier jobs by task id and attempt: + +- Deltas use only pairs graded in both arms with the metric measured in both. +- Ungraded trials, interrupted sweeps and missing cost are listed under **Problems**. A grader crash writes reward `-1`, which counts as ungraded. +- A trial whose agent hit `--cost-limit` or exited nonzero still had its patch graded. It counts as an unresolved attempt, listed as an agent error, not dropped. +- `GitNexus usage` counts `gitnexus ` calls at command position in the ATIF trajectories. + +```bash +python3 -m deepswe.report --baseline /baseline- --gitnexus /gitnexus- --out +``` + +A sampled subset gives descriptive evidence, not statistical significance. DeepSWE trials can run for up to 3 hours each, and leaderboard costs run from about $2 to $12 per task at high effort, so start with a small `--n-tasks` and a cost cap. diff --git a/eval/deepswe/__init__.py b/eval/deepswe/__init__.py new file mode 100644 index 000000000..20572e902 --- /dev/null +++ b/eval/deepswe/__init__.py @@ -0,0 +1 @@ +"""DeepSWE (github.com/datacurve-ai/deep-swe) paired runs: baseline vs GitNexus CLI.""" diff --git a/eval/deepswe/ec2/bootstrap.sh b/eval/deepswe/ec2/bootstrap.sh new file mode 100755 index 000000000..7c1d5f301 --- /dev/null +++ b/eval/deepswe/ec2/bootstrap.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# EC2 user-data for a DeepSWE runner (Ubuntu 24.04). launch.sh fills in the +# @...@ values. No model key is passed here: user-data is readable from the +# instance metadata service, so export the key in your SSH session instead. +set -euxo pipefail + +GITNEXUS_REPO='@GITNEXUS_REPO@' +GITNEXUS_REF='@GITNEXUS_REF@' +MAX_HOURS='@MAX_HOURS@' +NODE_VERSION=22.18.0 +PIER_VERSION=0.3.1 +USER_NAME=ubuntu +HOME_DIR=/home/$USER_NAME + +# Cost guard: stop (not terminate, so results survive) after MAX_HOURS. +shutdown -h "+$((MAX_HOURS * 60))" "DeepSWE runner max lifetime reached" + +export DEBIAN_FRONTEND=noninteractive +apt-get update +apt-get install -y --no-install-recommends docker.io docker-compose-v2 git python3 curl ca-certificates jq tmux +systemctl enable --now docker +usermod -aG docker "$USER_NAME" + +case "$(uname -m)" in x86_64) arch=x64 ;; aarch64) arch=arm64 ;; *) echo "unsupported arch" >&2; exit 1 ;; esac +tarball="node-v${NODE_VERSION}-linux-${arch}.tar.gz" +cd /tmp +curl -fsSLO "https://nodejs.org/dist/v${NODE_VERSION}/${tarball}" +curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/SHASUMS256.txt" | grep " ${tarball}$" | sha256sum -c - +tar -xzf "$tarball" -C /usr/local --strip-components=1 +rm "$tarball" + +sudo -iu "$USER_NAME" bash -lc "curl -LsSf https://astral.sh/uv/install.sh | sh" +sudo -iu "$USER_NAME" bash -lc "\$HOME/.local/bin/uv tool install datacurve-pier==${PIER_VERSION}" +sudo -iu "$USER_NAME" git clone --quiet "$GITNEXUS_REPO" "$HOME_DIR/GitNexus" +sudo -iu "$USER_NAME" git -C "$HOME_DIR/GitNexus" fetch --quiet origin "$GITNEXUS_REF" +sudo -iu "$USER_NAME" git -C "$HOME_DIR/GitNexus" checkout --quiet --detach FETCH_HEAD + +echo "ready $(date -u +%FT%TZ) gitnexus=$(sudo -iu "$USER_NAME" git -C "$HOME_DIR/GitNexus" rev-parse HEAD)" \ + > "$HOME_DIR/BOOTSTRAP_DONE" +chown "$USER_NAME:$USER_NAME" "$HOME_DIR/BOOTSTRAP_DONE" diff --git a/eval/deepswe/ec2/launch.sh b/eval/deepswe/ec2/launch.sh new file mode 100755 index 000000000..9b330dd43 --- /dev/null +++ b/eval/deepswe/ec2/launch.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +# Launch a dedicated EC2 instance for DeepSWE runs (separate from the +# self-hosted release-evaluation runner). Prints the SSH and cleanup commands. +# +# eval/deepswe/ec2/launch.sh --ref main --instance-type m7i.4xlarge +# +# Needs an authenticated AWS CLI. Creates, if missing, an SSH key pair +# (private key saved under ~/.ssh) and a security group open on port 22 to +# your current public IP only. The instance stops itself after --max-hours. +set -euo pipefail + +REGION="${AWS_REGION:-$(aws configure get region 2>/dev/null || true)}" +REGION="${REGION:-us-east-1}" +INSTANCE_TYPE=m7i.4xlarge # 16 vCPU / 64 GiB: room for 4 concurrent 2-CPU / 8 GB trials plus builds +VOLUME_GB=400 # task images are several GB each; derived images add the index +MAX_HOURS=12 +KEY_NAME=gitnexus-deepswe +SG_NAME=gitnexus-deepswe-ssh +NAME="gitnexus-deepswe-$(date -u +%Y%m%d-%H%M)" +GITNEXUS_REPO=https://github.com/abhigyanpatwari/GitNexus.git +GITNEXUS_REF=main +DRY_RUN=0 + +while [ $# -gt 0 ]; do + case "$1" in + --region) REGION="$2"; shift 2 ;; + --instance-type) INSTANCE_TYPE="$2"; shift 2 ;; + --volume-gb) VOLUME_GB="$2"; shift 2 ;; + --max-hours) MAX_HOURS="$2"; shift 2 ;; + --key-name) KEY_NAME="$2"; shift 2 ;; + --name) NAME="$2"; shift 2 ;; + --repo) GITNEXUS_REPO="$2"; shift 2 ;; + --ref) GITNEXUS_REF="$2"; shift 2 ;; + --dry-run) DRY_RUN=1; shift ;; + -h|--help) sed -n '2,10p' "$0"; exit 0 ;; + *) echo "unknown option: $1" >&2; exit 2 ;; + esac +done + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +user_data="$(mktemp)" +trap 'rm -f "$user_data"' EXIT +sed -e "s|@GITNEXUS_REPO@|$GITNEXUS_REPO|" -e "s|@GITNEXUS_REF@|$GITNEXUS_REF|" -e "s|@MAX_HOURS@|$MAX_HOURS|" \ + "$HERE/bootstrap.sh" > "$user_data" +if grep -q '@[A-Z_]*@' "$user_data"; then echo "unfilled bootstrap placeholder" >&2; exit 1; fi + +aws_() { aws --region "$REGION" "$@"; } +identity="$(aws_ sts get-caller-identity --query Arn --output text)" +ami="$(aws_ ssm get-parameter --name /aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id \ + --query Parameter.Value --output text)" +echo "account identity: $identity" +echo "region=$REGION ami=$ami type=$INSTANCE_TYPE volume=${VOLUME_GB}GB max-hours=$MAX_HOURS ref=$GITNEXUS_REF" +if [ "$DRY_RUN" = 1 ]; then + echo "--dry-run: nothing created"; exit 0 +fi + +if ! aws_ ec2 describe-key-pairs --key-names "$KEY_NAME" >/dev/null 2>&1; then + mkdir -p "$HOME/.ssh" + aws_ ec2 create-key-pair --key-name "$KEY_NAME" --query KeyMaterial --output text > "$HOME/.ssh/$KEY_NAME.pem" + chmod 600 "$HOME/.ssh/$KEY_NAME.pem" + echo "created key pair $KEY_NAME -> ~/.ssh/$KEY_NAME.pem" +fi + +vpc="$(aws_ ec2 describe-vpcs --filters Name=is-default,Values=true --query 'Vpcs[0].VpcId' --output text)" +[ "$vpc" != None ] || { echo "no default VPC in $REGION" >&2; exit 1; } +sg="$(aws_ ec2 describe-security-groups --filters Name=group-name,Values="$SG_NAME" Name=vpc-id,Values="$vpc" \ + --query 'SecurityGroups[0].GroupId' --output text)" +if [ "$sg" = None ]; then + sg="$(aws_ ec2 create-security-group --group-name "$SG_NAME" --vpc-id "$vpc" \ + --description "SSH to GitNexus DeepSWE runners" --query GroupId --output text)" +fi +my_ip="$(curl -fsS https://checkip.amazonaws.com | tr -d '[:space:]')" +aws_ ec2 authorize-security-group-ingress --group-id "$sg" --protocol tcp --port 22 --cidr "$my_ip/32" \ + >/dev/null 2>&1 || true # already allowed + +instance="$(aws_ ec2 run-instances --image-id "$ami" --instance-type "$INSTANCE_TYPE" --key-name "$KEY_NAME" \ + --security-group-ids "$sg" --user-data "file://$user_data" \ + --instance-initiated-shutdown-behavior stop \ + --metadata-options HttpTokens=required,HttpEndpoint=enabled \ + --block-device-mappings "DeviceName=/dev/sda1,Ebs={VolumeSize=$VOLUME_GB,VolumeType=gp3,DeleteOnTermination=true}" \ + --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=$NAME},{Key=purpose,Value=gitnexus-deepswe}]" \ + --query 'Instances[0].InstanceId' --output text)" +aws_ ec2 wait instance-running --instance-ids "$instance" +ip="$(aws_ ec2 describe-instances --instance-ids "$instance" --query 'Reservations[0].Instances[0].PublicIpAddress' \ + --output text)" + +cat <&1 | tee ~/deepswe-run.log' + # results: + scp -i ~/.ssh/$KEY_NAME.pem -r ubuntu@$ip:deepswe-work/report-\* . + # when finished (deletes the instance and its volume): + aws --region $REGION ec2 terminate-instances --instance-ids $instance +EOF diff --git a/eval/deepswe/prepare.py b/eval/deepswe/prepare.py new file mode 100644 index 000000000..71d6a0a5a --- /dev/null +++ b/eval/deepswe/prepare.py @@ -0,0 +1,213 @@ +"""Prepare DeepSWE task copies for a paired baseline vs GitNexus comparison. + +DeepSWE tasks are Harbor tasks with a prebuilt image and no agent network. The +`baseline` arm copies them unchanged. The `gitnexus` arm gives each task a +derived image: the prebuilt one plus Node, the GitNexus package and an index of +`/app` built at image-build time, so the no-network agent sandbox has a ready +graph and no install step. It also appends a short CLI note to the instruction. +Both arms get the same seeded subset, so `report.py` can pair trials by task. +""" + +from __future__ import annotations + +import argparse +import json +import random +import re +import shutil +import tomllib +from pathlib import Path +from typing import Any + +ARMS = ("baseline", "gitnexus") +# Renamed so pier's per-task image tag (`hb__`) never reuses a baseline image. +GITNEXUS_TASK_SUFFIX = "-gitnexus" +NODE_VERSION = "22.18.0" # gitnexus/package.json engines: ^22.18.0 +PACKAGE_FILE = "gitnexus.tgz" +MIN_BUILD_TIMEOUT_SEC = 3600.0 # the index is built inside the image build + +# Appended to the gitnexus arm's instruction. "product" is the CLI form of the +# rules GitNexus ships in AGENTS.md (graph first, impact before edits, +# detect-changes before committing), as the release evaluation's arms load. +# "optional" only lists the commands, for an ablation. +GUIDANCE = { + "product": """ + +## GitNexus + +A GitNexus code graph of `/app` (indexed at the starting commit) is installed; the `gitnexus` command works offline. Follow GitNexus's agent rules: + +- Explore with `gitnexus query ""` for concepts and execution flows, and `gitnexus context ` for a named symbol's callers, callees and flows. Graph first; use text search for literals or when the graph has no answer. +- Before editing a function, class or method, run `gitnexus impact --direction upstream` and check its callers and risk. An empty result is not proof that nothing depends on it. +- Before committing, run `gitnexus detect-changes` to see which symbols and flows your edits affect. + +Run `gitnexus --help` for options. +""", + "optional": """ + +## GitNexus + +A GitNexus code graph of `/app` (indexed at the starting commit) is installed. The `gitnexus` command works offline: + +- `gitnexus query ""`: code related to a concept, grouped by execution flow +- `gitnexus context `: callers, callees and flows of a symbol +- `gitnexus impact --direction upstream`: what depends on a symbol +- `gitnexus detect-changes`: symbols and flows touched by your uncommitted edits + +Run `gitnexus --help` for options. Using it is optional. +""", +} + +DOCKERFILE = """FROM {image} + +# GitNexus layer, written by eval/deepswe/prepare.py. The agent sandbox has no +# network, so Node, the package and the index of /app are all built in here. +ARG NODE_VERSION={node} +RUN set -eu; \\ + command -v curl >/dev/null || (apt-get update && apt-get install -y --no-install-recommends curl ca-certificates); \\ + case "$(uname -m)" in x86_64) arch=x64 ;; aarch64) arch=arm64 ;; *) echo "unsupported arch" >&2; exit 1 ;; esac; \\ + tarball="node-v${{NODE_VERSION}}-linux-${{arch}}.tar.gz"; \\ + cd /tmp; \\ + curl -fsSLO "https://nodejs.org/dist/v${{NODE_VERSION}}/${{tarball}}"; \\ + curl -fsSL "https://nodejs.org/dist/v${{NODE_VERSION}}/SHASUMS256.txt" | grep " ${{tarball}}$" | sha256sum -c -; \\ + mkdir -p /opt/gitnexus-node; \\ + tar -xzf "${{tarball}}" -C /opt/gitnexus-node --strip-components=1; \\ + rm "${{tarball}}" +COPY {package} /tmp/{package} +# PATH is left alone so a task's own node/npm are not shadowed. The only new +# command is a /usr/local/bin wrapper pinned to this Node, which login shells keep too. +# An explicit --prefix, because task images may carry their own npm config. +RUN export PATH=/opt/gitnexus-node/bin:$PATH \\ + && npm install --global --prefix /opt/gitnexus --no-fund --no-audit /tmp/{package} \\ + && rm /tmp/{package} \\ + && pkg=/opt/gitnexus/lib/node_modules/gitnexus \\ + && cli="$pkg/$(node -p "require('$pkg/package.json').bin.gitnexus")" \\ + && printf '#!/bin/sh\\nPATH=/opt/gitnexus-node/bin:$PATH exec node %s "$@"\\n' "$cli" > /usr/local/bin/gitnexus \\ + && chmod 755 /usr/local/bin/gitnexus \\ + && gitnexus --version +# --index-only writes no AGENTS.md/CLAUDE.md/skills; .gitnexus/ is git-excluded. +# Fail the build if indexing changed what git sees, since that would leak into the graded patch. +RUN cd /app \\ + && before="$(git status --porcelain=v1 --untracked-files=all | sha256sum)" \\ + && gitnexus analyze --index-only \\ + && after="$(git status --porcelain=v1 --untracked-files=all | sha256sum)" \\ + && test "$before" = "$after" +""" + + +def task_ids(tasks_root: Path) -> list[str]: + return sorted(path.parent.name for path in tasks_root.glob("*/task.toml")) + + +def select_tasks(available: list[str], *, include: list[str] | None, n_tasks: int | None, seed: int) -> list[str]: + """The same subset for every arm: explicit ids, or a seeded sample of the sorted ids.""" + + if include: + missing = sorted(set(include) - set(available)) + if missing: + raise SystemExit(f"unknown task id(s): {', '.join(missing)}") + return sorted(dict.fromkeys(include)) + if n_tasks is None or n_tasks >= len(available): + return sorted(available) + return sorted(random.Random(seed).sample(sorted(available), n_tasks)) + + +def _docker_image(task_toml: str) -> str: + image = (tomllib.loads(task_toml).get("environment") or {}).get("docker_image") + if not isinstance(image, str) or not image: + raise ValueError("task has no [environment] docker_image to build the GitNexus layer on") + return image + + +def gitnexus_task_toml(task_toml: str) -> str: + """Build from the derived Dockerfile, under a distinct name, with room for indexing.""" + + data = tomllib.loads(task_toml) + name = data["task"]["name"] + lines, section = [], "" + for line in task_toml.splitlines(): + header = re.match(r"^\s*\[([^\[\]]+)\]\s*$", line) + if header: + section = header.group(1).strip() + key = line.split("=", 1)[0].strip() if "=" in line and not line.lstrip().startswith("#") else "" + if section == "environment" and key == "docker_image": + continue + if section == "task" and key == "name": + line = f"name = {json.dumps(name + GITNEXUS_TASK_SUFFIX)}" + if section == "environment" and key == "build_timeout_sec": + current = float(data["environment"]["build_timeout_sec"]) + line = f"build_timeout_sec = {max(current, MIN_BUILD_TIMEOUT_SEC)}" + lines.append(line) + rewritten = "\n".join(lines) + "\n" + tomllib.loads(rewritten) # still valid TOML + return rewritten + + +def prepare_task(source: Path, dest: Path, *, arm: str, package: Path | None, guidance: str = "product") -> None: + if dest.exists(): + shutil.rmtree(dest) + shutil.copytree(source, dest) + if arm == "baseline": + return + if package is None: + raise ValueError("the gitnexus arm needs --gitnexus-package") + task_toml = (source / "task.toml").read_text(encoding="utf-8") + image = _docker_image(task_toml) + environment = dest / "environment" + environment.mkdir(exist_ok=True) + (environment / "Dockerfile").write_text( + DOCKERFILE.format(image=image, node=NODE_VERSION, package=PACKAGE_FILE), encoding="utf-8" + ) + shutil.copyfile(package, environment / PACKAGE_FILE) + (dest / "task.toml").write_text(gitnexus_task_toml(task_toml), encoding="utf-8") + instruction = dest / "instruction.md" + instruction.write_text(instruction.read_text(encoding="utf-8").rstrip() + "\n" + GUIDANCE[guidance], encoding="utf-8") + + +def prepare( + *, tasks_root: Path, out: Path, arms: list[str], selected: list[str], package: Path | None, + guidance: str = "product", +) -> dict[str, Any]: + for arm in arms: + for task in selected: + prepare_task(tasks_root / task, out / arm / task, arm=arm, package=package, guidance=guidance) + manifest = { + "schema": "gitnexus.deepswe-prepare/v1", + "arms": arms, + "tasks": selected, + "gitnexus_task_suffix": GITNEXUS_TASK_SUFFIX, + "guidance": guidance, + "node_version": NODE_VERSION, + "gitnexus_package": package.name if package else None, + } + (out / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + return manifest + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--deep-swe", required=True, type=Path, help="checkout of github.com/datacurve-ai/deep-swe") + parser.add_argument("--out", required=True, type=Path, help="writes /// and manifest.json") + parser.add_argument("--arms", nargs="+", choices=ARMS, default=list(ARMS)) + parser.add_argument("--gitnexus-package", type=Path, help="`npm pack` tarball of gitnexus/ (gitnexus arm)") + parser.add_argument("--task", action="append", dest="include", help="task id; repeatable") + parser.add_argument("--n-tasks", type=int, help="seeded random subset size (default: all tasks)") + parser.add_argument("--sample-seed", type=int, default=0) + parser.add_argument("--guidance", choices=sorted(GUIDANCE), default="product", + help="note appended to the gitnexus arm's instruction (default: product)") + args = parser.parse_args(argv) + tasks_root = args.deep_swe / "tasks" + available = task_ids(tasks_root) + if not available: + parser.error(f"no tasks under {tasks_root}") + if "gitnexus" in args.arms and (args.gitnexus_package is None or not args.gitnexus_package.is_file()): + parser.error("--gitnexus-package must name an existing `npm pack` tarball for the gitnexus arm") + selected = select_tasks(available, include=args.include, n_tasks=args.n_tasks, seed=args.sample_seed) + manifest = prepare(tasks_root=tasks_root, out=args.out, arms=args.arms, selected=selected, + package=args.gitnexus_package, guidance=args.guidance) + print(f"prepared {len(selected)} task(s) x {len(manifest['arms'])} arm(s) under {args.out}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eval/deepswe/report.py b/eval/deepswe/report.py new file mode 100644 index 000000000..13d58b4f4 --- /dev/null +++ b/eval/deepswe/report.py @@ -0,0 +1,235 @@ +"""Paired report for DeepSWE pier jobs: baseline vs GitNexus. + +Reads the `result.json` pier writes under `//`. Trials pair on the +task id (the gitnexus arm's `-gitnexus` suffix removed) and attempt order. +Deltas use only pairs graded in both arms with the metric measured in both; +missing, duplicated, ungraded or unbilled trials are listed as problems. +""" + +from __future__ import annotations + +import argparse +import json +import re +import statistics +from collections import Counter, defaultdict +from datetime import datetime +from pathlib import Path +from typing import Any + +from .prepare import GITNEXUS_TASK_SUFFIX + +ARMS = ("baseline", "gitnexus") +METRICS = ("cost_usd", "n_input_tokens", "n_cache_tokens", "n_output_tokens", "n_agent_steps", "agent_seconds") +# Command position only, so `ls gitnexus/` or a path argument does not count. +_CLI_COMMAND = re.compile(r"(?:^|[;&|(`\n]\s*|\$\(\s*)gitnexus\s+([a-z][a-z_-]*)") + + +def _measured(value: Any) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) + + +def task_id(task_name: str) -> str: + name = task_name.rsplit("/", 1)[-1] + return name.removesuffix(GITNEXUS_TASK_SUFFIX) + + +def _seconds(timing: dict[str, Any] | None) -> float | None: + if not timing or not timing.get("started_at") or not timing.get("finished_at"): + return None + start, end = (datetime.fromisoformat(str(timing[key]).replace("Z", "+00:00")) for key in ("started_at", "finished_at")) + return (end - start).total_seconds() + + +def gitnexus_commands(trial_dir: Path) -> Counter: + """`gitnexus ` calls in the trial's ATIF trajectory tool calls.""" + + counts: Counter = Counter() + path = trial_dir / "agent" / "trajectory.json" + if not path.is_file(): + return counts + try: + steps = json.loads(path.read_text(encoding="utf-8", errors="replace")).get("steps") or [] + except ValueError: + return counts + for step in steps: + for call in step.get("tool_calls") or []: + for value in (call.get("arguments") or {}).values(): + if isinstance(value, str): + counts.update(match.group(1) for match in _CLI_COMMAND.finditer(value)) + return counts + + +def load_trial(trial_dir: Path) -> dict[str, Any] | None: + path = trial_dir / "result.json" + if not path.is_file(): + return None + result = json.loads(path.read_text(encoding="utf-8")) + agent = result.get("agent_result") or {} + rewards = (result.get("verifier_result") or {}).get("rewards") or {} + reward = rewards.get("reward") + # Graded means the verifier produced a 0/1 reward. A trial whose agent hit the + # cost cap or exited nonzero is still graded on its patch; dropping it would + # hide expensive failures. -1 (the grader's crash sentinel) or no reward is not. + graded = reward in (0, 1) and not isinstance(reward, bool) + row = { + "task": task_id(str(result.get("task_name", trial_dir.name))), + "trial": result.get("trial_name") or trial_dir.name, + "started_at": result.get("started_at") or "", + "graded": graded, + "resolved": graded and reward == 1, + "error": (result.get("exception_info") or {}).get("exception_type"), + "agent_seconds": _seconds(result.get("agent_execution")), + "gitnexus_commands": dict(gitnexus_commands(trial_dir)), + } + for metric in METRICS[:-1]: + row[metric] = agent.get(metric) + return row + + +def load_job(job_dir: Path) -> list[dict[str, Any]]: + if not job_dir.is_dir(): + return [] + trials = [load_trial(path) for path in sorted(job_dir.iterdir()) if path.is_dir()] + return [trial for trial in trials if trial is not None] + + +def _keyed(trials: list[dict[str, Any]]) -> dict[tuple[str, int], dict[str, Any]]: + by_task: dict[str, list[dict[str, Any]]] = defaultdict(list) + for trial in trials: + by_task[trial["task"]].append(trial) + keyed = {} + for task, rows in by_task.items(): + for attempt, row in enumerate(sorted(rows, key=lambda r: (r["started_at"], r["trial"]))): + keyed[(task, attempt)] = row + return keyed + + +def build_report(jobs: dict[str, Path]) -> dict[str, Any]: + report: dict[str, Any] = {"schema": "gitnexus.deepswe-comparison/v1", "arms": {}, "changes": {}, "problems": []} + keyed: dict[str, dict[tuple[str, int], dict[str, Any]]] = {} + for arm in ARMS: + trials = load_job(jobs[arm]) + if not trials: + report["problems"].append(f"{arm}: no trial results under {jobs[arm]}") + keyed[arm] = _keyed(trials) + graded = [row for row in trials if row["graded"]] + ungraded = [row for row in trials if not row["graded"]] + if ungraded: + kinds = Counter(row["error"] or "no reward" for row in ungraded) + report["problems"].append(f"{arm}: {len(ungraded)} of {len(trials)} trial(s) ungraded ({dict(kinds)})") + agent_errors = Counter(row["error"] for row in graded if row["error"]) + if agent_errors: + report["problems"].append( + f"{arm}: {sum(agent_errors.values())} graded trial(s) ended with an agent error " + + f"({dict(agent_errors)}), e.g. the cost cap; their patches were still graded" + ) + summary: dict[str, Any] = {"trials": len(trials), "graded": len(graded), + "resolved": sum(row["resolved"] for row in graded), "incomplete_metrics": []} + for metric in METRICS: + values = [row[metric] for row in graded if _measured(row.get(metric))] + if graded and len(values) == len(graded): + summary[f"mean_{metric}"] = statistics.mean(values) + else: + summary[f"mean_{metric}"] = None + if graded: + summary["incomplete_metrics"].append(metric) + report["problems"].append( + f"{arm}: {metric} missing on {len(graded) - len(values)} of {len(graded)} graded trial(s)" + ) + if arm == "gitnexus": + per_trial = [Counter(row["gitnexus_commands"]) for row in graded] + summary["trials_using_gitnexus"] = sum(1 for counts in per_trial if counts) + summary["mean_gitnexus_calls"] = statistics.mean(sum(c.values()) for c in per_trial) if per_trial else None + summary["gitnexus_commands"] = dict(sum(per_trial, Counter()).most_common()) + report["arms"][arm] = summary + + expected = set(keyed["baseline"]) | set(keyed["gitnexus"]) + shared = sorted(key for key in expected if all(key in keyed[arm] and keyed[arm][key]["graded"] for arm in ARMS)) + if len(shared) < len(expected): + report["problems"].append( + f"{len(shared)} of {len(expected)} task/attempt pair(s) graded in both arms; deltas use those only" + ) + report["pairs"] = {"graded_in_both": len(shared), "expected": len(expected)} + if shared: + report["paired_resolved"] = {arm: sum(keyed[arm][key]["resolved"] for key in shared) for arm in ARMS} + for metric in METRICS: + matched = [key for key in shared if all(_measured(keyed[arm][key].get(metric)) for arm in ARMS)] + if not matched: + continue + base = statistics.mean(keyed["baseline"][key][metric] for key in matched) + value = statistics.mean(keyed["gitnexus"][key][metric] for key in matched) + if base: + report["changes"][metric] = {"percent": 100 * (value - base) / base, "pairs": len(matched), + "expected_pairs": len(expected)} + report["per_task"] = { + f"{task}#{attempt}": {arm: _cell(keyed[arm].get((task, attempt))) for arm in ARMS} + for task, attempt in sorted(expected) + } + return report + + +def _cell(row: dict[str, Any] | None) -> dict[str, Any] | None: + if row is None: + return None + return {key: row.get(key) for key in ("graded", "resolved", "cost_usd", "n_agent_steps", "error")} + + +def render_markdown(report: dict[str, Any]) -> str: + lines = [ + "## DeepSWE: baseline vs GitNexus (descriptive)", + "", + "| Arm | Resolved / graded | Mean cost | Mean input tokens | Mean steps | Mean agent time | GitNexus calls/trial |", + "| --- | --- | --- | --- | --- | --- | --- |", + ] + for arm, values in report["arms"].items(): + + def show(metric: str, pattern: str) -> str: + if metric.removeprefix("mean_") in values.get("incomplete_metrics", ()): + return "incomplete" + value = values.get(metric) + return "-" if value is None else pattern.format(value) + + lines.append( + f"| {arm} | {values['resolved']}/{values['graded']} | {show('mean_cost_usd', '${:.3f}')} " + f"| {show('mean_n_input_tokens', '{:,.0f}')} | {show('mean_n_agent_steps', '{:.1f}')} " + f"| {show('mean_agent_seconds', '{:.0f}s')} | {show('mean_gitnexus_calls', '{:.1f}')} |" + ) + pairs = report["pairs"] + if "paired_resolved" in report: + resolved = report["paired_resolved"] + lines += ["", f"Paired ({pairs['graded_in_both']}/{pairs['expected']} task/attempt pairs): " + f"baseline resolved {resolved['baseline']}, gitnexus resolved {resolved['gitnexus']}."] + if report["changes"]: + lines.append("") + for metric, change in report["changes"].items(): + sample = f"{change['pairs']}/{change['expected_pairs']} pairs" + if change["pairs"] < change["expected_pairs"]: + sample += ", partial" + lines.append(f"- gitnexus vs baseline: {metric}: {change['percent']:+.1f}% ({sample})") + gitnexus = report["arms"].get("gitnexus") + if gitnexus and gitnexus.get("graded"): + lines += ["", f"GitNexus usage: {gitnexus['trials_using_gitnexus']}/{gitnexus['graded']} graded trials; " + f"commands {gitnexus['gitnexus_commands']}."] + if report["problems"]: + lines += ["", "**Problems:**", *[f"- {problem}" for problem in report["problems"]]] + lines += ["", "Descriptive evidence on a sampled subset, not statistical significance.", ""] + return "\n".join(lines) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--baseline", required=True, type=Path, help="pier job directory of the baseline arm") + parser.add_argument("--gitnexus", required=True, type=Path, help="pier job directory of the gitnexus arm") + parser.add_argument("--out", required=True, type=Path) + args = parser.parse_args(argv) + report = build_report({"baseline": args.baseline, "gitnexus": args.gitnexus}) + args.out.mkdir(parents=True, exist_ok=True) + (args.out / "deepswe-comparison.json").write_text(json.dumps(report, indent=2, sort_keys=True) + "\n") + (args.out / "deepswe-comparison.md").write_text(render_markdown(report)) + print(render_markdown(report)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eval/deepswe/run.sh b/eval/deepswe/run.sh new file mode 100755 index 000000000..1e0ac7faf --- /dev/null +++ b/eval/deepswe/run.sh @@ -0,0 +1,213 @@ +#!/usr/bin/env bash +# Paired DeepSWE run: prepare both arms, run pier on each, write the report. +# +# OPENAI_API_KEY=... eval/deepswe/run.sh --n-tasks 10 --concurrency 4 +# +# The default agent is Claude Code. A non-Claude model (default gpt-5.6-sol) +# goes through a local LiteLLM gateway, as in workflow_bench; an `anthropic/` +# or `claude-` model uses ANTHROPIC_API_KEY directly. Needs docker, git, +# python3.11+, pier (`uv tool install datacurve-pier==0.3.1`) and, to build the +# package, node/npm 22.18+. Everything lands under --work (default ~/deepswe-work). +set -euo pipefail + +DEEP_SWE_REPO=https://github.com/datacurve-ai/deep-swe +DEEP_SWE_COMMIT=0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea +PIER_VERSION=0.3.1 +CLAUDE_CODE_VERSION=2.1.214 # the release evaluation's pinned CLI +LITELLM_VERSION=1.100.4 # eval/uv.lock +MINI_SWE_AGENT_VERSION=2.2.7 # eval/uv.lock +GATEWAY_CONTAINER=deepswe-litellm-gateway +GATEWAY_PORT=80 # pier's egress proxy only allows ports 80 and 443 +# Everything CLI 2.1.214 offers beyond Read/Edit/Write/Bash, matching the release +# evaluation's single-agent toolset. Subagents and background tasks otherwise fan +# out unbounded: one smoke trial ran 237 background agents past its budget cap. +CLAUDE_DISALLOWED_TOOLS="EnterPlanMode Task Agent TaskCreate TaskGet TaskList TaskOutput TaskStop TaskUpdate SendMessage Skill Workflow CronCreate CronDelete CronList ScheduleWakeup EnterWorktree ExitWorktree NotebookEdit ReportFindings WebFetch WebSearch" + +AGENT=claude-code +MODEL=gpt-5.6-sol +EFFORT=low +N_TASKS=10 +SEED=0 +CONCURRENCY=2 +COST_LIMIT=5 +WORK="$HOME/deepswe-work" +PACKAGE="" +GATEWAY_HOST="" +GUIDANCE=product +TASKS=() +ARMS=(baseline gitnexus) + +usage() { + cat <; + mini-swe-agent: a litellm name such as openai/gpt-5.6-sol + --effort E reasoning effort (default $EFFORT) + --cost-limit USD per-trial agent cost cap, same for both arms (default $COST_LIMIT; 0 = none) + --package TGZ prebuilt \`npm pack\` tarball of gitnexus/ (default: build from this checkout) + --gateway-host H hostname the trial egress proxy uses to reach this host's gateway + (default: host.docker.internal if it resolves, else the EC2 private DNS name) + --guidance G note for the gitnexus arm: product (GitNexus's agent rules, default) or optional + --arms "A B" arms to run (default: "${ARMS[*]}") + --work DIR working directory (default $WORK) +EOF +} + +while [ $# -gt 0 ]; do + case "$1" in + --n-tasks) N_TASKS="$2"; shift 2 ;; + --seed) SEED="$2"; shift 2 ;; + --task) TASKS+=("$2"); shift 2 ;; + --concurrency) CONCURRENCY="$2"; shift 2 ;; + --agent) AGENT="$2"; shift 2 ;; + --model) MODEL="$2"; shift 2 ;; + --effort) EFFORT="$2"; shift 2 ;; + --cost-limit) COST_LIMIT="$2"; shift 2 ;; + --package) PACKAGE="$2"; shift 2 ;; + --gateway-host) GATEWAY_HOST="$2"; shift 2 ;; + --guidance) GUIDANCE="$2"; shift 2 ;; + --arms) read -r -a ARMS <<<"$2"; shift 2 ;; + --work) WORK="$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown option: $1" >&2; usage >&2; exit 2 ;; + esac +done + +EVAL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +REPO_ROOT="$(cd "$EVAL_DIR/.." && pwd)" +needed=(docker git python3 pier curl) +[[ " ${ARMS[*]} " == *" gitnexus "* ]] && [ -z "$PACKAGE" ] && needed+=(node npm) # only to build the package +for tool in "${needed[@]}"; do + command -v "$tool" >/dev/null || { echo "missing $tool on PATH" >&2; exit 1; } +done +pier_version="$(pier --version 2>/dev/null | grep -Eo '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true)" +[ "$pier_version" = "$PIER_VERSION" ] || echo "warning: pier $pier_version, pinned $PIER_VERSION" >&2 + +agent_args=(--agent "$AGENT" --model "$MODEL") +use_gateway=0 +case "$AGENT" in + claude-code) + agent_args+=(--ak "version=$CLAUDE_CODE_VERSION" --ak "reasoning_effort=$EFFORT" + --ak "disallowed_tools=$CLAUDE_DISALLOWED_TOOLS" + # pier forces both on; --ae is merged last, so these win. + --ae ENABLE_BACKGROUND_TASKS=0 --ae FORCE_AUTO_BACKGROUND_TASKS=0) + [ "$COST_LIMIT" != 0 ] && agent_args+=(--ak "max_budget_usd=$COST_LIMIT") + case "$MODEL" in + anthropic/*|claude-*) : "${ANTHROPIC_API_KEY:?ANTHROPIC_API_KEY is required for $MODEL}" ;; + */*) echo "claude-code through the gateway takes a bare model name, not $MODEL" >&2; exit 2 ;; + *) : "${OPENAI_API_KEY:?OPENAI_API_KEY is required for the $MODEL gateway}"; use_gateway=1 ;; + esac + ;; + mini-swe-agent) + # litellm's Responses path imports `mcp`, which a bare mini-swe-agent install lacks. + agent_args+=(--ak "version=$MINI_SWE_AGENT_VERSION" --ak "reasoning_effort=$EFFORT" + --ak "cost_limit=$COST_LIMIT" --ak 'extra_python_packages=["mcp"]') + case "$MODEL" in + openai/*) : "${OPENAI_API_KEY:?OPENAI_API_KEY is required for $MODEL}" ;; + anthropic/*) : "${ANTHROPIC_API_KEY:?ANTHROPIC_API_KEY is required for $MODEL}" ;; + esac + ;; + *) echo "unsupported agent: $AGENT" >&2; exit 2 ;; +esac +mkdir -p "$WORK" + +start_gateway() { + if [ -z "$GATEWAY_HOST" ]; then + if getent hosts host.docker.internal >/dev/null; then + GATEWAY_HOST=host.docker.internal + else + imds_token="$(curl -fsS -X PUT -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' \ + http://169.254.169.254/latest/api/token 2>/dev/null || true)" + GATEWAY_HOST="$(curl -fsS -H "X-aws-ec2-metadata-token: $imds_token" \ + http://169.254.169.254/latest/meta-data/local-hostname 2>/dev/null || hostname -f)" + fi + fi + local dir="$WORK/gateway" + mkdir -p "$dir" + # Same routing as workflow_bench.model_gateway: Claude Code speaks Anthropic + # Messages; LiteLLM serves them from OpenAI's Responses API. + cat > "$dir/litellm.yaml" </dev/null 2>&1 || true + docker run -d --name "$GATEWAY_CONTAINER" -p "$GATEWAY_PORT:4000" \ + -e OPENAI_API_KEY -e LITELLM_MASTER_KEY="$GATEWAY_KEY" -e LITELLM_LOCAL_MODEL_COST_MAP=True \ + -v "$dir:/cfg:ro" python:3.12-slim \ + sh -c "pip install --quiet --no-cache-dir 'litellm[proxy]==$LITELLM_VERSION' && exec litellm --config /cfg/litellm.yaml --host 0.0.0.0 --port 4000" \ + >/dev/null + trap 'docker rm -f "$GATEWAY_CONTAINER" >/dev/null 2>&1 || true' EXIT + echo "waiting for the LiteLLM gateway on http://$GATEWAY_HOST:$GATEWAY_PORT" + for _ in $(seq 1 180); do + if curl -fsS -o /dev/null -H "Authorization: Bearer $GATEWAY_KEY" \ + "http://$GATEWAY_HOST:$GATEWAY_PORT/health/liveliness" 2>/dev/null; then + agent_args+=(--ae "ANTHROPIC_BASE_URL=http://$GATEWAY_HOST:$GATEWAY_PORT" --ae "ANTHROPIC_AUTH_TOKEN=$GATEWAY_KEY") + return 0 + fi + docker inspect -f '{{.State.Running}}' "$GATEWAY_CONTAINER" 2>/dev/null | grep -q true || break + sleep 2 + done + echo "gateway did not become ready; last log lines:" >&2 + docker logs --tail 20 "$GATEWAY_CONTAINER" >&2 || true + exit 1 +} + +if [ ! -d "$WORK/deep-swe/.git" ]; then + git clone --quiet "$DEEP_SWE_REPO" "$WORK/deep-swe" +fi +git -C "$WORK/deep-swe" fetch --quiet origin "$DEEP_SWE_COMMIT" 2>/dev/null || true +git -C "$WORK/deep-swe" checkout --quiet --detach "$DEEP_SWE_COMMIT" + +if [[ " ${ARMS[*]} " == *" gitnexus "* ]] && [ -z "$PACKAGE" ]; then + echo "building the gitnexus package from $REPO_ROOT" + # Same steps as the CI package job: prepack compiles the CLI and web UI into the tarball. + (cd "$REPO_ROOT/gitnexus" && npm ci --ignore-scripts --no-fund --no-audit) + (cd "$REPO_ROOT/gitnexus-shared" && node ../gitnexus/node_modules/typescript/lib/tsc.js) + (cd "$REPO_ROOT/gitnexus-web" && npm ci --no-fund --no-audit) + (cd "$REPO_ROOT/gitnexus" && npm pack --pack-destination "$WORK" >/dev/null) + PACKAGE="$(ls -t "$WORK"/gitnexus-*.tgz | head -1)" +fi + +select_args=() +if [ ${#TASKS[@]} -gt 0 ]; then + for task in "${TASKS[@]}"; do select_args+=(--task "$task"); done +elif [ "$N_TASKS" != 0 ]; then + select_args+=(--n-tasks "$N_TASKS" --sample-seed "$SEED") +fi +package_args=() +[ -n "$PACKAGE" ] && package_args+=(--gitnexus-package "$PACKAGE") +(cd "$EVAL_DIR" && python3 -m deepswe.prepare --deep-swe "$WORK/deep-swe" --out "$WORK/tasks" \ + --arms "${ARMS[@]}" --guidance "$GUIDANCE" "${package_args[@]}" "${select_args[@]}") + +[ "$use_gateway" = 1 ] && start_gateway + +stamp="$(date -u +%Y%m%dT%H%M%SZ)" +for arm in "${ARMS[@]}"; do + echo "== pier: $arm" + # A failed trial is recorded in the job and reported; keep going to the other arm. + pier run -p "$WORK/tasks/$arm" --env docker "${agent_args[@]}" \ + --n-concurrent "$CONCURRENCY" --jobs-dir "$WORK/jobs" --job-name "$arm-$stamp" --yes || \ + echo "pier exited nonzero for $arm; see $WORK/jobs/$arm-$stamp" >&2 +done + +if [ ${#ARMS[@]} -eq 2 ]; then + (cd "$EVAL_DIR" && python3 -m deepswe.report --baseline "$WORK/jobs/baseline-$stamp" \ + --gitnexus "$WORK/jobs/gitnexus-$stamp" --out "$WORK/report-$stamp") + echo "report: $WORK/report-$stamp/deepswe-comparison.md" +fi diff --git a/eval/tests/test_deepswe.py b/eval/tests/test_deepswe.py new file mode 100644 index 000000000..82bfb56c7 --- /dev/null +++ b/eval/tests/test_deepswe.py @@ -0,0 +1,239 @@ +"""eval/deepswe: task preparation for the paired DeepSWE arms, and the paired report.""" + +from __future__ import annotations + +import json +import re +import tomllib +from pathlib import Path + +import pytest + +from deepswe import prepare, report + +DEEPSWE_DIR = Path(__file__).resolve().parents[1] / "deepswe" + +# Trimmed from deep-swe@0b9fabbb tasks/abs-module-cache-flags/task.toml. +TASK_TOML = """schema_version = "1.3" +artifacts = ["/logs/artifacts/model.patch"] +[task] +name = "datacurve/abs-module-cache-flags" +description = "" +[metadata] +task_id = "abs-module-cache-flags" +language = "go" +base_commit_hash = "cb1b3b671d0ee9fa9da9f7b02f86967953ffd10a" +[verifier] +network_mode = "no-network" +environment_mode = "separate" +timeout_sec = 1800.0 +[verifier.environment] +build_timeout_sec = 1800.0 +docker_image = "verifier-image-stays" +[[verifier.collect]] +command = "cd /app && git diff --binary cb1b3b671d0ee9fa9da9f7b02f86967953ffd10a HEAD > /logs/artifacts/model.patch" +timeout_sec = 300.0 +[agent] +network_mode = "no-network" +timeout_sec = 10800.0 +[environment] +build_timeout_sec = 1800.0 +docker_image = "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh75679ajj3b8dtd7se3h7z0a1833y6r-v1.1" +os = "linux" +cpus = 2 +memory_mb = 8192 +mcp_servers = [] +""" + + +def _task(root: Path, task_id: str, toml: str = TASK_TOML) -> Path: + task = root / task_id + (task / "environment").mkdir(parents=True) + (task / "environment" / "Dockerfile").write_text("FROM upstream-build\n") + (task / "tests").mkdir() + (task / "tests" / "test.sh").write_text("#!/bin/bash\n") + (task / "instruction.md").write_text("Implement the thing.\n") + (task / "task.toml").write_text(toml.replace("abs-module-cache-flags", task_id)) + return task + + +def test_select_tasks_is_seeded_and_validates_ids(): + ids = [f"t{i:02d}" for i in range(20)] + first = prepare.select_tasks(ids, include=None, n_tasks=5, seed=3) + assert first == prepare.select_tasks(list(reversed(ids)), include=None, n_tasks=5, seed=3) + assert len(first) == 5 and first == sorted(first) + assert prepare.select_tasks(ids, include=None, n_tasks=None, seed=0) == ids + assert prepare.select_tasks(ids, include=["t03", "t01", "t03"], n_tasks=5, seed=0) == ["t01", "t03"] + with pytest.raises(SystemExit, match="nope"): + prepare.select_tasks(ids, include=["nope"], n_tasks=None, seed=0) + + +def test_gitnexus_task_toml_builds_from_the_derived_dockerfile_under_a_new_name(): + data = tomllib.loads(prepare.gitnexus_task_toml(TASK_TOML)) + assert data["task"]["name"] == "datacurve/abs-module-cache-flags-gitnexus" + assert "docker_image" not in data["environment"] + assert data["environment"]["build_timeout_sec"] == prepare.MIN_BUILD_TIMEOUT_SEC + # Everything else, including the separate verifier image and the patch collector, is untouched. + original = tomllib.loads(TASK_TOML) + assert data["verifier"] == original["verifier"] + assert data["agent"] == original["agent"] and data["metadata"] == original["metadata"] + assert {k: v for k, v in data["environment"].items() if k != "build_timeout_sec"} == { + k: v for k, v in original["environment"].items() if k not in ("docker_image", "build_timeout_sec") + } + + +def test_prepare_copies_baseline_unchanged_and_layers_gitnexus(tmp_path): + tasks = tmp_path / "deep-swe" / "tasks" + for task_id in ("alpha-task", "beta-task"): + _task(tasks, task_id) + package = tmp_path / "gitnexus-9.9.9.tgz" + package.write_bytes(b"tarball") + out = tmp_path / "out" + + assert prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(out), + "--gitnexus-package", str(package), "--task", "alpha-task"]) == 0 + + manifest = json.loads((out / "manifest.json").read_text()) + assert manifest["tasks"] == ["alpha-task"] and manifest["arms"] == ["baseline", "gitnexus"] + assert not (out / "baseline" / "beta-task").exists() + for name in ("task.toml", "instruction.md", "environment/Dockerfile", "tests/test.sh"): + assert (out / "baseline" / "alpha-task" / name).read_bytes() == (tasks / "alpha-task" / name).read_bytes() + + derived = out / "gitnexus" / "alpha-task" + dockerfile = (derived / "environment" / "Dockerfile").read_text() + assert dockerfile.startswith( + "FROM public.ecr.aws/d3j8x8q7/swe-bench-202605:kh75679ajj3b8dtd7se3h7z0a1833y6r-v1.1\n" + ) + assert f"ARG NODE_VERSION={prepare.NODE_VERSION}" in dockerfile and "sha256sum -c -" in dockerfile + assert "gitnexus analyze --index-only" in dockerfile and 'test "$before" = "$after"' in dockerfile + # A task's own node/npm stay first on PATH; only a pinned wrapper is added. + assert "ENV PATH" not in dockerfile and "/usr/local/bin/gitnexus" in dockerfile + assert "{" not in dockerfile.replace("${", "") # every template field was filled + assert (derived / "environment" / prepare.PACKAGE_FILE).read_bytes() == b"tarball" + instruction = (derived / "instruction.md").read_text() + assert instruction.startswith("Implement the thing.\n") and instruction.endswith(prepare.GUIDANCE["product"]) + assert (derived / "tests" / "test.sh").read_bytes() == (tasks / "alpha-task" / "tests" / "test.sh").read_bytes() + + +def test_gitnexus_arm_requires_a_package_and_a_prebuilt_image(tmp_path): + tasks = tmp_path / "deep-swe" / "tasks" + _task(tasks, "gamma-task", TASK_TOML.replace('docker_image = "public.ecr', 'base_image = "public.ecr')) + with pytest.raises(SystemExit): + prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(tmp_path / "o")]) + package = tmp_path / "p.tgz" + package.write_bytes(b"x") + with pytest.raises(ValueError, match="docker_image"): + prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(tmp_path / "o"), + "--gitnexus-package", str(package)]) + + +@pytest.mark.parametrize("guidance", sorted(prepare.GUIDANCE)) +def test_gitnexus_guidance_names_only_real_cli_commands(guidance): + note = prepare.GUIDANCE[guidance] + commands = set(re.findall(r"`gitnexus ([a-z-]+)", note)) - {""} + assert commands == {"query", "context", "impact", "detect-changes"} + assert len(note.encode()) < 1000 + + +def test_prepare_records_the_guidance_variant(tmp_path): + _task(tmp_path / "deep-swe" / "tasks", "delta-task") + package = tmp_path / "p.tgz" + package.write_bytes(b"x") + prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(tmp_path / "o"), "--arms", "gitnexus", + "--gitnexus-package", str(package), "--guidance", "optional"]) + assert json.loads((tmp_path / "o" / "manifest.json").read_text())["guidance"] == "optional" + assert (tmp_path / "o" / "gitnexus" / "delta-task" / "instruction.md").read_text().endswith( + prepare.GUIDANCE["optional"] + ) + + +def _trial(job: Path, name: str, task: str, *, reward=1, cost=1.0, steps=10, exception=None, commands=()): + trial = job / name + (trial / "agent").mkdir(parents=True) + result = { + "task_name": f"datacurve/{task}", "trial_name": name, "started_at": f"2026-10-30T00:00:{len(name):02d}Z", + "agent_result": {"cost_usd": cost, "n_input_tokens": 1000, "n_cache_tokens": 800, "n_output_tokens": 50, + "n_agent_steps": steps}, + "verifier_result": None if reward is None else {"rewards": {"reward": reward}}, + "exception_info": exception, + "agent_execution": {"started_at": "2026-10-30T00:00:00Z", "finished_at": "2026-10-30T00:01:40Z"}, + } + (trial / "result.json").write_text(json.dumps(result)) + calls = [{"function_name": "bash", "arguments": {"command": command}} for command in commands] + (trial / "agent" / "trajectory.json").write_text(json.dumps({"steps": [{"tool_calls": calls}]})) + + +def test_report_pairs_on_task_id_and_compares_matched_pairs_only(tmp_path): + base, gitnexus = tmp_path / "baseline-job", tmp_path / "gitnexus-job" + _trial(base, "a__1", "alpha", cost=1.0) + _trial(base, "b__1", "beta", cost=9.0, reward=0) + _trial(base, "c__1", "gamma", cost=2.0) + _trial(gitnexus, "a__1", "alpha-gitnexus", cost=2.0, + commands=["gitnexus query parse && gitnexus impact Foo", "ls gitnexus/ && cat gitnexus.json"]) + _trial(gitnexus, "b__1", "beta-gitnexus", cost=9.0, reward=-1) # grader crash sentinel + # gamma never ran in the gitnexus arm (interrupted sweep) + _trial(base, "d__1", "delta", cost=3.0, reward=0, exception={"exception_type": "NonZeroAgentExitCodeError"}) + _trial(gitnexus, "d__1", "delta-gitnexus", cost=4.0, reward=1) + (base / "config.json").write_text("{}") # job-level files are not trials + + result = report.build_report({"baseline": base, "gitnexus": gitnexus}) + + # The capped baseline trial (agent error, reward 0) is graded and paired, not dropped. + assert result["pairs"] == {"graded_in_both": 2, "expected": 4} + assert result["changes"]["cost_usd"] == {"percent": 50.0, "pairs": 2, "expected_pairs": 4} + assert result["paired_resolved"] == {"baseline": 1, "gitnexus": 2} + assert result["arms"]["baseline"]["resolved"] == 2 and result["arms"]["baseline"]["graded"] == 4 + assert result["arms"]["gitnexus"]["gitnexus_commands"] == {"query": 1, "impact": 1} + assert result["arms"]["gitnexus"]["trials_using_gitnexus"] == 1 + assert any("gitnexus: 1 of 3 trial(s) ungraded" in problem for problem in result["problems"]) + assert any("2 of 4 task/attempt pair(s) graded in both arms" in problem for problem in result["problems"]) + assert any("baseline: 1 graded trial(s) ended with an agent error" in problem for problem in result["problems"]) + markdown = report.render_markdown(result) + assert "gitnexus vs baseline: cost_usd: +50.0% (2/4 pairs, partial)" in markdown + assert result["per_task"]["gamma#0"]["gitnexus"] is None + + +def test_report_marks_missing_billing_incomplete(tmp_path): + base, gitnexus = tmp_path / "b", tmp_path / "g" + for name, task in (("x__1", "x"), ("y__1", "y")): + _trial(base, name, task, cost=1.0) + _trial(gitnexus, "x__1", "x-gitnexus", cost=None) + _trial(gitnexus, "y__1", "y-gitnexus", cost=3.0) + + result = report.build_report({"baseline": base, "gitnexus": gitnexus}) + + assert result["arms"]["gitnexus"]["mean_cost_usd"] is None + assert result["arms"]["gitnexus"]["incomplete_metrics"] == ["cost_usd"] + assert result["changes"]["cost_usd"]["pairs"] == 1 and result["changes"]["n_agent_steps"]["pairs"] == 2 + assert "| gitnexus | 2/2 | incomplete |" in report.render_markdown(result) + + +def test_report_survives_an_arm_that_never_produced_a_job(tmp_path): + _trial(tmp_path / "b", "x__1", "x") + result = report.build_report({"baseline": tmp_path / "b", "gitnexus": tmp_path / "never-created"}) + assert any(problem.startswith("gitnexus: no trial results") for problem in result["problems"]) + assert result["pairs"] == {"graded_in_both": 0, "expected": 1} and result["changes"] == {} + assert "| gitnexus | 0/0 |" in report.render_markdown(result) + + +def test_scripts_pin_the_same_versions_and_fill_every_bootstrap_placeholder(): + run_sh = (DEEPSWE_DIR / "run.sh").read_text() + bootstrap = (DEEPSWE_DIR / "ec2" / "bootstrap.sh").read_text() + launch = (DEEPSWE_DIR / "ec2" / "launch.sh").read_text() + + def pin(text: str, name: str) -> str: + match = re.search(rf"^{name}=(\S+)", text, re.MULTILINE) + assert match, name + return match.group(1) + + assert pin(run_sh, "PIER_VERSION") == pin(bootstrap, "PIER_VERSION") + # The gateway and the optional mini-swe-agent use the versions eval/ is locked to. + lock = (DEEPSWE_DIR.parent / "uv.lock").read_text() + for package, name in (("litellm", "LITELLM_VERSION"), ("mini-swe-agent", "MINI_SWE_AGENT_VERSION")): + assert f'name = "{package}"\nversion = "{pin(run_sh, name)}"' in lock, package + assert re.fullmatch(r"\d+\.\d+\.\d+", pin(run_sh, "CLAUDE_CODE_VERSION")) + assert pin(bootstrap, "NODE_VERSION") == prepare.NODE_VERSION + assert re.fullmatch(r"[0-9a-f]{40}", pin(run_sh, "DEEP_SWE_COMMIT")) + placeholders = set(re.findall(r"@([A-Z_]+)@", bootstrap)) + assert placeholders == set(re.findall(r"s\|@([A-Z_]+)@\|", launch)) == {"GITNEXUS_REPO", "GITNEXUS_REF", "MAX_HOURS"} + assert "API_KEY" not in bootstrap.replace("# No model key", "")