mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-11-14 09:01:33 +00:00
Merge c85c638129 into 5b968a098c
This commit is contained in:
Abhigyan Patwari
GitHub
commit
405d1a828f
9 changed files with 1132 additions and 0 deletions
|
|
@ -24,6 +24,8 @@ Evaluate whether GitNexus code intelligence improves AI agent performance on rea
|
|||
- DeepSeek
|
||||
- Any model supported by litellm (add a YAML config)
|
||||
|
||||
For long-horizon tasks, [`deepswe/`](deepswe/README.md) runs a paired baseline vs GitNexus CLI comparison on [DeepSWE](https://deepswe.datacurve.ai/) with its own harness (pier) on a dedicated EC2 instance.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.11+
|
||||
|
|
|
|||
90
eval/deepswe/README.md
Normal file
90
eval/deepswe/README.md
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
# DeepSWE: baseline vs GitNexus CLI
|
||||
|
||||
[DeepSWE](https://deepswe.datacurve.ai/) has 113 long-horizon tasks across 91 repositories in five languages. The tasks were written from scratch, and the solutions need about 5.5x more code than SWE-bench Pro. This directory runs a paired comparison on a sampled subset with the benchmark's own harness, [pier](https://github.com/datacurve-ai/pier):
|
||||
|
||||
| Arm | What the agent gets |
|
||||
| --- | --- |
|
||||
| `baseline` | The DeepSWE task, unchanged |
|
||||
| `gitnexus` | The same task, plus the `gitnexus` CLI and an index of `/app` built into the image, and a short note on the CLI appended to the instruction |
|
||||
|
||||
The default agent is Claude Code (pier's `claude-code`, pinned to the release evaluation's CLI). GitNexus is reached through the real CLI (`gitnexus query`, `context`, `impact`, `detect-changes`) rather than MCP. The arms differ only in that layer and that note.
|
||||
|
||||
- **Toolset.** Claude Code gets only `Read`, `Edit`, `Write` and `Bash`, matching the release evaluation's single-agent toolset. Subagents, background tasks, skills and web tools are disallowed, and pier's forced background-task env is switched off. Without this, one smoke trial ran 237 background agents past its budget cap.
|
||||
- **Guidance.** The note (`--guidance product`, the default) is the CLI form of the rules GitNexus ships in AGENTS.md: graph first, `impact` before editing a symbol, `detect-changes` before committing. In the release evaluation, arms load those directives the same way. `--guidance optional` only lists the commands; in a smoke trial the agent then never used them.
|
||||
|
||||
The workflow_bench `baseline_cli` arm points the same way: on the small release tasks, a shell command cost far less than the MCP tool catalog. `--agent mini-swe-agent`, the leaderboard's agent, also works.
|
||||
|
||||
### Model access
|
||||
|
||||
- **Non-Claude models** (default `gpt-5.6-sol`, `OPENAI_API_KEY`): `run.sh` starts a LiteLLM gateway container. It is routed like `workflow_bench.model_gateway`, with Claude Code's Anthropic Messages served from OpenAI's Responses API, a random master key, and the LiteLLM version from `eval/uv.lock`.
|
||||
- It listens on host port 80, because pier's egress proxy only allows ports 80 and 443. Keep that port closed to the outside; `launch.sh`'s security group only opens SSH.
|
||||
- Trials reach it as `host.docker.internal` on Docker Desktop, or as the EC2 private DNS name.
|
||||
- **Claude models** (`--model anthropic/<model>`): use `ANTHROPIC_API_KEY` directly.
|
||||
|
||||
Costs come from Claude Code's own per-session accounting (`total_cost_usd`). Through the gateway that is Claude Code's price table applied to a non-Claude model, so it is consistent across arms but is not the provider's invoice.
|
||||
|
||||
## How the GitNexus arm is built
|
||||
|
||||
Task agents run with no network (`allow_internet = false`; pier allowlists only the model API). Nothing can be installed at run time, so `prepare.py` gives each `gitnexus` task a derived Dockerfile:
|
||||
|
||||
1. `FROM` the task's prebuilt image.
|
||||
2. Node from the official tarball, checked against `SHASUMS256.txt`.
|
||||
3. `npm install -g` of a `gitnexus` tarball built from this checkout.
|
||||
4. `gitnexus analyze --index-only`. This writes no AGENTS.md, CLAUDE.md or skills, and `.gitnexus/` is git-excluded. The build fails if `git status` changed, because that would leak into the graded patch.
|
||||
|
||||
The task is renamed `<name>-gitnexus`, so pier's per-task image tag never reuses a baseline image. The verifier, its separate image, the patch collector and the timeouts are untouched; only the build timeout is raised to fit indexing. `report.py` strips the suffix and pairs trials by task id.
|
||||
|
||||
## Run it on EC2
|
||||
|
||||
Use a dedicated instance, not the self-hosted release-evaluation runner. It needs an authenticated AWS CLI (`aws login`).
|
||||
|
||||
```bash
|
||||
eval/deepswe/ec2/launch.sh --ref main # add --dry-run to print the plan only
|
||||
ssh -i ~/.ssh/gitnexus-deepswe.pem ubuntu@<ip> # wait for ~/BOOTSTRAP_DONE
|
||||
export OPENAI_API_KEY=... # in the session, never in user-data
|
||||
cd ~/GitNexus && eval/deepswe/run.sh --n-tasks 10 --concurrency 4
|
||||
```
|
||||
|
||||
`launch.sh` does the following:
|
||||
|
||||
- Starts Ubuntu 24.04 on an `m7i.4xlarge` (16 vCPU / 64 GiB) with a 400 GB gp3 volume.
|
||||
- Tags the instance `purpose=gitnexus-deepswe`.
|
||||
- Opens SSH to your current IP only.
|
||||
- Stops the instance after `--max-hours` (default 12).
|
||||
|
||||
It prints the `scp` and `terminate-instances` commands for when you're done.
|
||||
|
||||
Task images are about 4 GB each, plus pier's agent layer and the GitNexus layer, so 400 GB fits roughly 40–50 tasks across both arms. For all 113 tasks, pass `--volume-gb 1000` or run `docker image prune -a` between batches.
|
||||
|
||||
`run.sh` performs these steps:
|
||||
|
||||
1. Pins `deep-swe` to a commit.
|
||||
2. Builds the gitnexus package (the same steps as the CI package job).
|
||||
3. Prepares both arms on one seeded subset.
|
||||
4. Runs `pier run` per arm.
|
||||
5. Writes `deepswe-comparison.{md,json}`.
|
||||
|
||||
The main options are listed below; `run.sh --help` shows them all.
|
||||
|
||||
| Option | Default | Notes |
|
||||
| --- | --- | --- |
|
||||
| `--n-tasks`, `--seed`, `--task` | 10, 0 | `--task` (repeatable) overrides the sample; `--n-tasks 0` runs all 113 |
|
||||
| `--agent`, `--model`, `--effort` | `claude-code`, `gpt-5.6-sol`, `low` | `--model anthropic/<model>` skips the gateway. `--agent mini-swe-agent` takes litellm names such as `openai/gpt-5.6-sol`. |
|
||||
| `--cost-limit` | `5` | Per-trial agent cost cap in USD, the same for both arms. A capped trial ends unsolved. |
|
||||
| `--concurrency` | `2` | Trials per arm at once. Each needs 2 CPUs and 8 GB. |
|
||||
| `--package` | build from checkout | Pass a prebuilt `npm pack` tarball to skip the build |
|
||||
|
||||
## Reading the report
|
||||
|
||||
`report.py` pairs the two pier jobs by task id and attempt:
|
||||
|
||||
- Deltas use only pairs graded in both arms with the metric measured in both.
|
||||
- Ungraded trials, interrupted sweeps and missing cost are listed under **Problems**. A grader crash writes reward `-1`, which counts as ungraded.
|
||||
- A trial whose agent hit `--cost-limit` or exited nonzero still had its patch graded. It counts as an unresolved attempt, listed as an agent error, not dropped.
|
||||
- `GitNexus usage` counts `gitnexus <command>` calls at command position in the ATIF trajectories.
|
||||
|
||||
```bash
|
||||
python3 -m deepswe.report --baseline <jobs>/baseline-<stamp> --gitnexus <jobs>/gitnexus-<stamp> --out <dir>
|
||||
```
|
||||
|
||||
A sampled subset gives descriptive evidence, not statistical significance. DeepSWE trials can run for up to 3 hours each, and leaderboard costs run from about $2 to $12 per task at high effort, so start with a small `--n-tasks` and a cost cap.
|
||||
1
eval/deepswe/__init__.py
Normal file
1
eval/deepswe/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""DeepSWE (github.com/datacurve-ai/deep-swe) paired runs: baseline vs GitNexus CLI."""
|
||||
40
eval/deepswe/ec2/bootstrap.sh
Executable file
40
eval/deepswe/ec2/bootstrap.sh
Executable file
|
|
@ -0,0 +1,40 @@
|
|||
#!/usr/bin/env bash
|
||||
# EC2 user-data for a DeepSWE runner (Ubuntu 24.04). launch.sh fills in the
|
||||
# @...@ values. No model key is passed here: user-data is readable from the
|
||||
# instance metadata service, so export the key in your SSH session instead.
|
||||
set -euxo pipefail
|
||||
|
||||
GITNEXUS_REPO='@GITNEXUS_REPO@'
|
||||
GITNEXUS_REF='@GITNEXUS_REF@'
|
||||
MAX_HOURS='@MAX_HOURS@'
|
||||
NODE_VERSION=22.18.0
|
||||
PIER_VERSION=0.3.1
|
||||
USER_NAME=ubuntu
|
||||
HOME_DIR=/home/$USER_NAME
|
||||
|
||||
# Cost guard: stop (not terminate, so results survive) after MAX_HOURS.
|
||||
shutdown -h "+$((MAX_HOURS * 60))" "DeepSWE runner max lifetime reached"
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends docker.io docker-compose-v2 git python3 curl ca-certificates jq tmux
|
||||
systemctl enable --now docker
|
||||
usermod -aG docker "$USER_NAME"
|
||||
|
||||
case "$(uname -m)" in x86_64) arch=x64 ;; aarch64) arch=arm64 ;; *) echo "unsupported arch" >&2; exit 1 ;; esac
|
||||
tarball="node-v${NODE_VERSION}-linux-${arch}.tar.gz"
|
||||
cd /tmp
|
||||
curl -fsSLO "https://nodejs.org/dist/v${NODE_VERSION}/${tarball}"
|
||||
curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/SHASUMS256.txt" | grep " ${tarball}$" | sha256sum -c -
|
||||
tar -xzf "$tarball" -C /usr/local --strip-components=1
|
||||
rm "$tarball"
|
||||
|
||||
sudo -iu "$USER_NAME" bash -lc "curl -LsSf https://astral.sh/uv/install.sh | sh"
|
||||
sudo -iu "$USER_NAME" bash -lc "\$HOME/.local/bin/uv tool install datacurve-pier==${PIER_VERSION}"
|
||||
sudo -iu "$USER_NAME" git clone --quiet "$GITNEXUS_REPO" "$HOME_DIR/GitNexus"
|
||||
sudo -iu "$USER_NAME" git -C "$HOME_DIR/GitNexus" fetch --quiet origin "$GITNEXUS_REF"
|
||||
sudo -iu "$USER_NAME" git -C "$HOME_DIR/GitNexus" checkout --quiet --detach FETCH_HEAD
|
||||
|
||||
echo "ready $(date -u +%FT%TZ) gitnexus=$(sudo -iu "$USER_NAME" git -C "$HOME_DIR/GitNexus" rev-parse HEAD)" \
|
||||
> "$HOME_DIR/BOOTSTRAP_DONE"
|
||||
chown "$USER_NAME:$USER_NAME" "$HOME_DIR/BOOTSTRAP_DONE"
|
||||
99
eval/deepswe/ec2/launch.sh
Executable file
99
eval/deepswe/ec2/launch.sh
Executable file
|
|
@ -0,0 +1,99 @@
|
|||
#!/usr/bin/env bash
|
||||
# Launch a dedicated EC2 instance for DeepSWE runs (separate from the
|
||||
# self-hosted release-evaluation runner). Prints the SSH and cleanup commands.
|
||||
#
|
||||
# eval/deepswe/ec2/launch.sh --ref main --instance-type m7i.4xlarge
|
||||
#
|
||||
# Needs an authenticated AWS CLI. Creates, if missing, an SSH key pair
|
||||
# (private key saved under ~/.ssh) and a security group open on port 22 to
|
||||
# your current public IP only. The instance stops itself after --max-hours.
|
||||
set -euo pipefail
|
||||
|
||||
REGION="${AWS_REGION:-$(aws configure get region 2>/dev/null || true)}"
|
||||
REGION="${REGION:-us-east-1}"
|
||||
INSTANCE_TYPE=m7i.4xlarge # 16 vCPU / 64 GiB: room for 4 concurrent 2-CPU / 8 GB trials plus builds
|
||||
VOLUME_GB=400 # task images are several GB each; derived images add the index
|
||||
MAX_HOURS=12
|
||||
KEY_NAME=gitnexus-deepswe
|
||||
SG_NAME=gitnexus-deepswe-ssh
|
||||
NAME="gitnexus-deepswe-$(date -u +%Y%m%d-%H%M)"
|
||||
GITNEXUS_REPO=https://github.com/abhigyanpatwari/GitNexus.git
|
||||
GITNEXUS_REF=main
|
||||
DRY_RUN=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--region) REGION="$2"; shift 2 ;;
|
||||
--instance-type) INSTANCE_TYPE="$2"; shift 2 ;;
|
||||
--volume-gb) VOLUME_GB="$2"; shift 2 ;;
|
||||
--max-hours) MAX_HOURS="$2"; shift 2 ;;
|
||||
--key-name) KEY_NAME="$2"; shift 2 ;;
|
||||
--name) NAME="$2"; shift 2 ;;
|
||||
--repo) GITNEXUS_REPO="$2"; shift 2 ;;
|
||||
--ref) GITNEXUS_REF="$2"; shift 2 ;;
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
-h|--help) sed -n '2,10p' "$0"; exit 0 ;;
|
||||
*) echo "unknown option: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
user_data="$(mktemp)"
|
||||
trap 'rm -f "$user_data"' EXIT
|
||||
sed -e "s|@GITNEXUS_REPO@|$GITNEXUS_REPO|" -e "s|@GITNEXUS_REF@|$GITNEXUS_REF|" -e "s|@MAX_HOURS@|$MAX_HOURS|" \
|
||||
"$HERE/bootstrap.sh" > "$user_data"
|
||||
if grep -q '@[A-Z_]*@' "$user_data"; then echo "unfilled bootstrap placeholder" >&2; exit 1; fi
|
||||
|
||||
aws_() { aws --region "$REGION" "$@"; }
|
||||
identity="$(aws_ sts get-caller-identity --query Arn --output text)"
|
||||
ami="$(aws_ ssm get-parameter --name /aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id \
|
||||
--query Parameter.Value --output text)"
|
||||
echo "account identity: $identity"
|
||||
echo "region=$REGION ami=$ami type=$INSTANCE_TYPE volume=${VOLUME_GB}GB max-hours=$MAX_HOURS ref=$GITNEXUS_REF"
|
||||
if [ "$DRY_RUN" = 1 ]; then
|
||||
echo "--dry-run: nothing created"; exit 0
|
||||
fi
|
||||
|
||||
if ! aws_ ec2 describe-key-pairs --key-names "$KEY_NAME" >/dev/null 2>&1; then
|
||||
mkdir -p "$HOME/.ssh"
|
||||
aws_ ec2 create-key-pair --key-name "$KEY_NAME" --query KeyMaterial --output text > "$HOME/.ssh/$KEY_NAME.pem"
|
||||
chmod 600 "$HOME/.ssh/$KEY_NAME.pem"
|
||||
echo "created key pair $KEY_NAME -> ~/.ssh/$KEY_NAME.pem"
|
||||
fi
|
||||
|
||||
vpc="$(aws_ ec2 describe-vpcs --filters Name=is-default,Values=true --query 'Vpcs[0].VpcId' --output text)"
|
||||
[ "$vpc" != None ] || { echo "no default VPC in $REGION" >&2; exit 1; }
|
||||
sg="$(aws_ ec2 describe-security-groups --filters Name=group-name,Values="$SG_NAME" Name=vpc-id,Values="$vpc" \
|
||||
--query 'SecurityGroups[0].GroupId' --output text)"
|
||||
if [ "$sg" = None ]; then
|
||||
sg="$(aws_ ec2 create-security-group --group-name "$SG_NAME" --vpc-id "$vpc" \
|
||||
--description "SSH to GitNexus DeepSWE runners" --query GroupId --output text)"
|
||||
fi
|
||||
my_ip="$(curl -fsS https://checkip.amazonaws.com | tr -d '[:space:]')"
|
||||
aws_ ec2 authorize-security-group-ingress --group-id "$sg" --protocol tcp --port 22 --cidr "$my_ip/32" \
|
||||
>/dev/null 2>&1 || true # already allowed
|
||||
|
||||
instance="$(aws_ ec2 run-instances --image-id "$ami" --instance-type "$INSTANCE_TYPE" --key-name "$KEY_NAME" \
|
||||
--security-group-ids "$sg" --user-data "file://$user_data" \
|
||||
--instance-initiated-shutdown-behavior stop \
|
||||
--metadata-options HttpTokens=required,HttpEndpoint=enabled \
|
||||
--block-device-mappings "DeviceName=/dev/sda1,Ebs={VolumeSize=$VOLUME_GB,VolumeType=gp3,DeleteOnTermination=true}" \
|
||||
--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=$NAME},{Key=purpose,Value=gitnexus-deepswe}]" \
|
||||
--query 'Instances[0].InstanceId' --output text)"
|
||||
aws_ ec2 wait instance-running --instance-ids "$instance"
|
||||
ip="$(aws_ ec2 describe-instances --instance-ids "$instance" --query 'Reservations[0].Instances[0].PublicIpAddress' \
|
||||
--output text)"
|
||||
|
||||
cat <<EOF
|
||||
instance: $instance ($NAME) in $REGION, public IP $ip
|
||||
bootstrap takes a few minutes; it is done when ~/BOOTSTRAP_DONE exists.
|
||||
|
||||
ssh -i ~/.ssh/$KEY_NAME.pem ubuntu@$ip
|
||||
# on the instance:
|
||||
export OPENAI_API_KEY=... # never put it in user-data
|
||||
tmux new -s deepswe 'cd ~/GitNexus && eval/deepswe/run.sh --n-tasks 10 --concurrency 4 2>&1 | tee ~/deepswe-run.log'
|
||||
# results:
|
||||
scp -i ~/.ssh/$KEY_NAME.pem -r ubuntu@$ip:deepswe-work/report-\* .
|
||||
# when finished (deletes the instance and its volume):
|
||||
aws --region $REGION ec2 terminate-instances --instance-ids $instance
|
||||
EOF
|
||||
213
eval/deepswe/prepare.py
Normal file
213
eval/deepswe/prepare.py
Normal file
|
|
@ -0,0 +1,213 @@
|
|||
"""Prepare DeepSWE task copies for a paired baseline vs GitNexus comparison.
|
||||
|
||||
DeepSWE tasks are Harbor tasks with a prebuilt image and no agent network. The
|
||||
`baseline` arm copies them unchanged. The `gitnexus` arm gives each task a
|
||||
derived image: the prebuilt one plus Node, the GitNexus package and an index of
|
||||
`/app` built at image-build time, so the no-network agent sandbox has a ready
|
||||
graph and no install step. It also appends a short CLI note to the instruction.
|
||||
Both arms get the same seeded subset, so `report.py` can pair trials by task.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
import shutil
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
ARMS = ("baseline", "gitnexus")
|
||||
# Renamed so pier's per-task image tag (`hb__<task name>`) never reuses a baseline image.
|
||||
GITNEXUS_TASK_SUFFIX = "-gitnexus"
|
||||
NODE_VERSION = "22.18.0" # gitnexus/package.json engines: ^22.18.0
|
||||
PACKAGE_FILE = "gitnexus.tgz"
|
||||
MIN_BUILD_TIMEOUT_SEC = 3600.0 # the index is built inside the image build
|
||||
|
||||
# Appended to the gitnexus arm's instruction. "product" is the CLI form of the
|
||||
# rules GitNexus ships in AGENTS.md (graph first, impact before edits,
|
||||
# detect-changes before committing), as the release evaluation's arms load.
|
||||
# "optional" only lists the commands, for an ablation.
|
||||
GUIDANCE = {
|
||||
"product": """
|
||||
|
||||
## GitNexus
|
||||
|
||||
A GitNexus code graph of `/app` (indexed at the starting commit) is installed; the `gitnexus` command works offline. Follow GitNexus's agent rules:
|
||||
|
||||
- Explore with `gitnexus query "<concept>"` for concepts and execution flows, and `gitnexus context <symbol>` for a named symbol's callers, callees and flows. Graph first; use text search for literals or when the graph has no answer.
|
||||
- Before editing a function, class or method, run `gitnexus impact <symbol> --direction upstream` and check its callers and risk. An empty result is not proof that nothing depends on it.
|
||||
- Before committing, run `gitnexus detect-changes` to see which symbols and flows your edits affect.
|
||||
|
||||
Run `gitnexus <command> --help` for options.
|
||||
""",
|
||||
"optional": """
|
||||
|
||||
## GitNexus
|
||||
|
||||
A GitNexus code graph of `/app` (indexed at the starting commit) is installed. The `gitnexus` command works offline:
|
||||
|
||||
- `gitnexus query "<concept>"`: code related to a concept, grouped by execution flow
|
||||
- `gitnexus context <symbol>`: callers, callees and flows of a symbol
|
||||
- `gitnexus impact <symbol> --direction upstream`: what depends on a symbol
|
||||
- `gitnexus detect-changes`: symbols and flows touched by your uncommitted edits
|
||||
|
||||
Run `gitnexus <command> --help` for options. Using it is optional.
|
||||
""",
|
||||
}
|
||||
|
||||
DOCKERFILE = """FROM {image}
|
||||
|
||||
# GitNexus layer, written by eval/deepswe/prepare.py. The agent sandbox has no
|
||||
# network, so Node, the package and the index of /app are all built in here.
|
||||
ARG NODE_VERSION={node}
|
||||
RUN set -eu; \\
|
||||
command -v curl >/dev/null || (apt-get update && apt-get install -y --no-install-recommends curl ca-certificates); \\
|
||||
case "$(uname -m)" in x86_64) arch=x64 ;; aarch64) arch=arm64 ;; *) echo "unsupported arch" >&2; exit 1 ;; esac; \\
|
||||
tarball="node-v${{NODE_VERSION}}-linux-${{arch}}.tar.gz"; \\
|
||||
cd /tmp; \\
|
||||
curl -fsSLO "https://nodejs.org/dist/v${{NODE_VERSION}}/${{tarball}}"; \\
|
||||
curl -fsSL "https://nodejs.org/dist/v${{NODE_VERSION}}/SHASUMS256.txt" | grep " ${{tarball}}$" | sha256sum -c -; \\
|
||||
mkdir -p /opt/gitnexus-node; \\
|
||||
tar -xzf "${{tarball}}" -C /opt/gitnexus-node --strip-components=1; \\
|
||||
rm "${{tarball}}"
|
||||
COPY {package} /tmp/{package}
|
||||
# PATH is left alone so a task's own node/npm are not shadowed. The only new
|
||||
# command is a /usr/local/bin wrapper pinned to this Node, which login shells keep too.
|
||||
# An explicit --prefix, because task images may carry their own npm config.
|
||||
RUN export PATH=/opt/gitnexus-node/bin:$PATH \\
|
||||
&& npm install --global --prefix /opt/gitnexus --no-fund --no-audit /tmp/{package} \\
|
||||
&& rm /tmp/{package} \\
|
||||
&& pkg=/opt/gitnexus/lib/node_modules/gitnexus \\
|
||||
&& cli="$pkg/$(node -p "require('$pkg/package.json').bin.gitnexus")" \\
|
||||
&& printf '#!/bin/sh\\nPATH=/opt/gitnexus-node/bin:$PATH exec node %s "$@"\\n' "$cli" > /usr/local/bin/gitnexus \\
|
||||
&& chmod 755 /usr/local/bin/gitnexus \\
|
||||
&& gitnexus --version
|
||||
# --index-only writes no AGENTS.md/CLAUDE.md/skills; .gitnexus/ is git-excluded.
|
||||
# Fail the build if indexing changed what git sees, since that would leak into the graded patch.
|
||||
RUN cd /app \\
|
||||
&& before="$(git status --porcelain=v1 --untracked-files=all | sha256sum)" \\
|
||||
&& gitnexus analyze --index-only \\
|
||||
&& after="$(git status --porcelain=v1 --untracked-files=all | sha256sum)" \\
|
||||
&& test "$before" = "$after"
|
||||
"""
|
||||
|
||||
|
||||
def task_ids(tasks_root: Path) -> list[str]:
|
||||
return sorted(path.parent.name for path in tasks_root.glob("*/task.toml"))
|
||||
|
||||
|
||||
def select_tasks(available: list[str], *, include: list[str] | None, n_tasks: int | None, seed: int) -> list[str]:
|
||||
"""The same subset for every arm: explicit ids, or a seeded sample of the sorted ids."""
|
||||
|
||||
if include:
|
||||
missing = sorted(set(include) - set(available))
|
||||
if missing:
|
||||
raise SystemExit(f"unknown task id(s): {', '.join(missing)}")
|
||||
return sorted(dict.fromkeys(include))
|
||||
if n_tasks is None or n_tasks >= len(available):
|
||||
return sorted(available)
|
||||
return sorted(random.Random(seed).sample(sorted(available), n_tasks))
|
||||
|
||||
|
||||
def _docker_image(task_toml: str) -> str:
|
||||
image = (tomllib.loads(task_toml).get("environment") or {}).get("docker_image")
|
||||
if not isinstance(image, str) or not image:
|
||||
raise ValueError("task has no [environment] docker_image to build the GitNexus layer on")
|
||||
return image
|
||||
|
||||
|
||||
def gitnexus_task_toml(task_toml: str) -> str:
|
||||
"""Build from the derived Dockerfile, under a distinct name, with room for indexing."""
|
||||
|
||||
data = tomllib.loads(task_toml)
|
||||
name = data["task"]["name"]
|
||||
lines, section = [], ""
|
||||
for line in task_toml.splitlines():
|
||||
header = re.match(r"^\s*\[([^\[\]]+)\]\s*$", line)
|
||||
if header:
|
||||
section = header.group(1).strip()
|
||||
key = line.split("=", 1)[0].strip() if "=" in line and not line.lstrip().startswith("#") else ""
|
||||
if section == "environment" and key == "docker_image":
|
||||
continue
|
||||
if section == "task" and key == "name":
|
||||
line = f"name = {json.dumps(name + GITNEXUS_TASK_SUFFIX)}"
|
||||
if section == "environment" and key == "build_timeout_sec":
|
||||
current = float(data["environment"]["build_timeout_sec"])
|
||||
line = f"build_timeout_sec = {max(current, MIN_BUILD_TIMEOUT_SEC)}"
|
||||
lines.append(line)
|
||||
rewritten = "\n".join(lines) + "\n"
|
||||
tomllib.loads(rewritten) # still valid TOML
|
||||
return rewritten
|
||||
|
||||
|
||||
def prepare_task(source: Path, dest: Path, *, arm: str, package: Path | None, guidance: str = "product") -> None:
|
||||
if dest.exists():
|
||||
shutil.rmtree(dest)
|
||||
shutil.copytree(source, dest)
|
||||
if arm == "baseline":
|
||||
return
|
||||
if package is None:
|
||||
raise ValueError("the gitnexus arm needs --gitnexus-package")
|
||||
task_toml = (source / "task.toml").read_text(encoding="utf-8")
|
||||
image = _docker_image(task_toml)
|
||||
environment = dest / "environment"
|
||||
environment.mkdir(exist_ok=True)
|
||||
(environment / "Dockerfile").write_text(
|
||||
DOCKERFILE.format(image=image, node=NODE_VERSION, package=PACKAGE_FILE), encoding="utf-8"
|
||||
)
|
||||
shutil.copyfile(package, environment / PACKAGE_FILE)
|
||||
(dest / "task.toml").write_text(gitnexus_task_toml(task_toml), encoding="utf-8")
|
||||
instruction = dest / "instruction.md"
|
||||
instruction.write_text(instruction.read_text(encoding="utf-8").rstrip() + "\n" + GUIDANCE[guidance], encoding="utf-8")
|
||||
|
||||
|
||||
def prepare(
|
||||
*, tasks_root: Path, out: Path, arms: list[str], selected: list[str], package: Path | None,
|
||||
guidance: str = "product",
|
||||
) -> dict[str, Any]:
|
||||
for arm in arms:
|
||||
for task in selected:
|
||||
prepare_task(tasks_root / task, out / arm / task, arm=arm, package=package, guidance=guidance)
|
||||
manifest = {
|
||||
"schema": "gitnexus.deepswe-prepare/v1",
|
||||
"arms": arms,
|
||||
"tasks": selected,
|
||||
"gitnexus_task_suffix": GITNEXUS_TASK_SUFFIX,
|
||||
"guidance": guidance,
|
||||
"node_version": NODE_VERSION,
|
||||
"gitnexus_package": package.name if package else None,
|
||||
}
|
||||
(out / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
|
||||
return manifest
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--deep-swe", required=True, type=Path, help="checkout of github.com/datacurve-ai/deep-swe")
|
||||
parser.add_argument("--out", required=True, type=Path, help="writes <out>/<arm>/<task-id>/ and manifest.json")
|
||||
parser.add_argument("--arms", nargs="+", choices=ARMS, default=list(ARMS))
|
||||
parser.add_argument("--gitnexus-package", type=Path, help="`npm pack` tarball of gitnexus/ (gitnexus arm)")
|
||||
parser.add_argument("--task", action="append", dest="include", help="task id; repeatable")
|
||||
parser.add_argument("--n-tasks", type=int, help="seeded random subset size (default: all tasks)")
|
||||
parser.add_argument("--sample-seed", type=int, default=0)
|
||||
parser.add_argument("--guidance", choices=sorted(GUIDANCE), default="product",
|
||||
help="note appended to the gitnexus arm's instruction (default: product)")
|
||||
args = parser.parse_args(argv)
|
||||
tasks_root = args.deep_swe / "tasks"
|
||||
available = task_ids(tasks_root)
|
||||
if not available:
|
||||
parser.error(f"no tasks under {tasks_root}")
|
||||
if "gitnexus" in args.arms and (args.gitnexus_package is None or not args.gitnexus_package.is_file()):
|
||||
parser.error("--gitnexus-package must name an existing `npm pack` tarball for the gitnexus arm")
|
||||
selected = select_tasks(available, include=args.include, n_tasks=args.n_tasks, seed=args.sample_seed)
|
||||
manifest = prepare(tasks_root=tasks_root, out=args.out, arms=args.arms, selected=selected,
|
||||
package=args.gitnexus_package, guidance=args.guidance)
|
||||
print(f"prepared {len(selected)} task(s) x {len(manifest['arms'])} arm(s) under {args.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
235
eval/deepswe/report.py
Normal file
235
eval/deepswe/report.py
Normal file
|
|
@ -0,0 +1,235 @@
|
|||
"""Paired report for DeepSWE pier jobs: baseline vs GitNexus.
|
||||
|
||||
Reads the `result.json` pier writes under `<job>/<trial>/`. Trials pair on the
|
||||
task id (the gitnexus arm's `-gitnexus` suffix removed) and attempt order.
|
||||
Deltas use only pairs graded in both arms with the metric measured in both;
|
||||
missing, duplicated, ungraded or unbilled trials are listed as problems.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import statistics
|
||||
from collections import Counter, defaultdict
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .prepare import GITNEXUS_TASK_SUFFIX
|
||||
|
||||
ARMS = ("baseline", "gitnexus")
|
||||
METRICS = ("cost_usd", "n_input_tokens", "n_cache_tokens", "n_output_tokens", "n_agent_steps", "agent_seconds")
|
||||
# Command position only, so `ls gitnexus/` or a path argument does not count.
|
||||
_CLI_COMMAND = re.compile(r"(?:^|[;&|(`\n]\s*|\$\(\s*)gitnexus\s+([a-z][a-z_-]*)")
|
||||
|
||||
|
||||
def _measured(value: Any) -> bool:
|
||||
return isinstance(value, (int, float)) and not isinstance(value, bool)
|
||||
|
||||
|
||||
def task_id(task_name: str) -> str:
|
||||
name = task_name.rsplit("/", 1)[-1]
|
||||
return name.removesuffix(GITNEXUS_TASK_SUFFIX)
|
||||
|
||||
|
||||
def _seconds(timing: dict[str, Any] | None) -> float | None:
|
||||
if not timing or not timing.get("started_at") or not timing.get("finished_at"):
|
||||
return None
|
||||
start, end = (datetime.fromisoformat(str(timing[key]).replace("Z", "+00:00")) for key in ("started_at", "finished_at"))
|
||||
return (end - start).total_seconds()
|
||||
|
||||
|
||||
def gitnexus_commands(trial_dir: Path) -> Counter:
|
||||
"""`gitnexus <command>` calls in the trial's ATIF trajectory tool calls."""
|
||||
|
||||
counts: Counter = Counter()
|
||||
path = trial_dir / "agent" / "trajectory.json"
|
||||
if not path.is_file():
|
||||
return counts
|
||||
try:
|
||||
steps = json.loads(path.read_text(encoding="utf-8", errors="replace")).get("steps") or []
|
||||
except ValueError:
|
||||
return counts
|
||||
for step in steps:
|
||||
for call in step.get("tool_calls") or []:
|
||||
for value in (call.get("arguments") or {}).values():
|
||||
if isinstance(value, str):
|
||||
counts.update(match.group(1) for match in _CLI_COMMAND.finditer(value))
|
||||
return counts
|
||||
|
||||
|
||||
def load_trial(trial_dir: Path) -> dict[str, Any] | None:
|
||||
path = trial_dir / "result.json"
|
||||
if not path.is_file():
|
||||
return None
|
||||
result = json.loads(path.read_text(encoding="utf-8"))
|
||||
agent = result.get("agent_result") or {}
|
||||
rewards = (result.get("verifier_result") or {}).get("rewards") or {}
|
||||
reward = rewards.get("reward")
|
||||
# Graded means the verifier produced a 0/1 reward. A trial whose agent hit the
|
||||
# cost cap or exited nonzero is still graded on its patch; dropping it would
|
||||
# hide expensive failures. -1 (the grader's crash sentinel) or no reward is not.
|
||||
graded = reward in (0, 1) and not isinstance(reward, bool)
|
||||
row = {
|
||||
"task": task_id(str(result.get("task_name", trial_dir.name))),
|
||||
"trial": result.get("trial_name") or trial_dir.name,
|
||||
"started_at": result.get("started_at") or "",
|
||||
"graded": graded,
|
||||
"resolved": graded and reward == 1,
|
||||
"error": (result.get("exception_info") or {}).get("exception_type"),
|
||||
"agent_seconds": _seconds(result.get("agent_execution")),
|
||||
"gitnexus_commands": dict(gitnexus_commands(trial_dir)),
|
||||
}
|
||||
for metric in METRICS[:-1]:
|
||||
row[metric] = agent.get(metric)
|
||||
return row
|
||||
|
||||
|
||||
def load_job(job_dir: Path) -> list[dict[str, Any]]:
|
||||
if not job_dir.is_dir():
|
||||
return []
|
||||
trials = [load_trial(path) for path in sorted(job_dir.iterdir()) if path.is_dir()]
|
||||
return [trial for trial in trials if trial is not None]
|
||||
|
||||
|
||||
def _keyed(trials: list[dict[str, Any]]) -> dict[tuple[str, int], dict[str, Any]]:
|
||||
by_task: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for trial in trials:
|
||||
by_task[trial["task"]].append(trial)
|
||||
keyed = {}
|
||||
for task, rows in by_task.items():
|
||||
for attempt, row in enumerate(sorted(rows, key=lambda r: (r["started_at"], r["trial"]))):
|
||||
keyed[(task, attempt)] = row
|
||||
return keyed
|
||||
|
||||
|
||||
def build_report(jobs: dict[str, Path]) -> dict[str, Any]:
|
||||
report: dict[str, Any] = {"schema": "gitnexus.deepswe-comparison/v1", "arms": {}, "changes": {}, "problems": []}
|
||||
keyed: dict[str, dict[tuple[str, int], dict[str, Any]]] = {}
|
||||
for arm in ARMS:
|
||||
trials = load_job(jobs[arm])
|
||||
if not trials:
|
||||
report["problems"].append(f"{arm}: no trial results under {jobs[arm]}")
|
||||
keyed[arm] = _keyed(trials)
|
||||
graded = [row for row in trials if row["graded"]]
|
||||
ungraded = [row for row in trials if not row["graded"]]
|
||||
if ungraded:
|
||||
kinds = Counter(row["error"] or "no reward" for row in ungraded)
|
||||
report["problems"].append(f"{arm}: {len(ungraded)} of {len(trials)} trial(s) ungraded ({dict(kinds)})")
|
||||
agent_errors = Counter(row["error"] for row in graded if row["error"])
|
||||
if agent_errors:
|
||||
report["problems"].append(
|
||||
f"{arm}: {sum(agent_errors.values())} graded trial(s) ended with an agent error "
|
||||
+ f"({dict(agent_errors)}), e.g. the cost cap; their patches were still graded"
|
||||
)
|
||||
summary: dict[str, Any] = {"trials": len(trials), "graded": len(graded),
|
||||
"resolved": sum(row["resolved"] for row in graded), "incomplete_metrics": []}
|
||||
for metric in METRICS:
|
||||
values = [row[metric] for row in graded if _measured(row.get(metric))]
|
||||
if graded and len(values) == len(graded):
|
||||
summary[f"mean_{metric}"] = statistics.mean(values)
|
||||
else:
|
||||
summary[f"mean_{metric}"] = None
|
||||
if graded:
|
||||
summary["incomplete_metrics"].append(metric)
|
||||
report["problems"].append(
|
||||
f"{arm}: {metric} missing on {len(graded) - len(values)} of {len(graded)} graded trial(s)"
|
||||
)
|
||||
if arm == "gitnexus":
|
||||
per_trial = [Counter(row["gitnexus_commands"]) for row in graded]
|
||||
summary["trials_using_gitnexus"] = sum(1 for counts in per_trial if counts)
|
||||
summary["mean_gitnexus_calls"] = statistics.mean(sum(c.values()) for c in per_trial) if per_trial else None
|
||||
summary["gitnexus_commands"] = dict(sum(per_trial, Counter()).most_common())
|
||||
report["arms"][arm] = summary
|
||||
|
||||
expected = set(keyed["baseline"]) | set(keyed["gitnexus"])
|
||||
shared = sorted(key for key in expected if all(key in keyed[arm] and keyed[arm][key]["graded"] for arm in ARMS))
|
||||
if len(shared) < len(expected):
|
||||
report["problems"].append(
|
||||
f"{len(shared)} of {len(expected)} task/attempt pair(s) graded in both arms; deltas use those only"
|
||||
)
|
||||
report["pairs"] = {"graded_in_both": len(shared), "expected": len(expected)}
|
||||
if shared:
|
||||
report["paired_resolved"] = {arm: sum(keyed[arm][key]["resolved"] for key in shared) for arm in ARMS}
|
||||
for metric in METRICS:
|
||||
matched = [key for key in shared if all(_measured(keyed[arm][key].get(metric)) for arm in ARMS)]
|
||||
if not matched:
|
||||
continue
|
||||
base = statistics.mean(keyed["baseline"][key][metric] for key in matched)
|
||||
value = statistics.mean(keyed["gitnexus"][key][metric] for key in matched)
|
||||
if base:
|
||||
report["changes"][metric] = {"percent": 100 * (value - base) / base, "pairs": len(matched),
|
||||
"expected_pairs": len(expected)}
|
||||
report["per_task"] = {
|
||||
f"{task}#{attempt}": {arm: _cell(keyed[arm].get((task, attempt))) for arm in ARMS}
|
||||
for task, attempt in sorted(expected)
|
||||
}
|
||||
return report
|
||||
|
||||
|
||||
def _cell(row: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
if row is None:
|
||||
return None
|
||||
return {key: row.get(key) for key in ("graded", "resolved", "cost_usd", "n_agent_steps", "error")}
|
||||
|
||||
|
||||
def render_markdown(report: dict[str, Any]) -> str:
|
||||
lines = [
|
||||
"## DeepSWE: baseline vs GitNexus (descriptive)",
|
||||
"",
|
||||
"| Arm | Resolved / graded | Mean cost | Mean input tokens | Mean steps | Mean agent time | GitNexus calls/trial |",
|
||||
"| --- | --- | --- | --- | --- | --- | --- |",
|
||||
]
|
||||
for arm, values in report["arms"].items():
|
||||
|
||||
def show(metric: str, pattern: str) -> str:
|
||||
if metric.removeprefix("mean_") in values.get("incomplete_metrics", ()):
|
||||
return "incomplete"
|
||||
value = values.get(metric)
|
||||
return "-" if value is None else pattern.format(value)
|
||||
|
||||
lines.append(
|
||||
f"| {arm} | {values['resolved']}/{values['graded']} | {show('mean_cost_usd', '${:.3f}')} "
|
||||
f"| {show('mean_n_input_tokens', '{:,.0f}')} | {show('mean_n_agent_steps', '{:.1f}')} "
|
||||
f"| {show('mean_agent_seconds', '{:.0f}s')} | {show('mean_gitnexus_calls', '{:.1f}')} |"
|
||||
)
|
||||
pairs = report["pairs"]
|
||||
if "paired_resolved" in report:
|
||||
resolved = report["paired_resolved"]
|
||||
lines += ["", f"Paired ({pairs['graded_in_both']}/{pairs['expected']} task/attempt pairs): "
|
||||
f"baseline resolved {resolved['baseline']}, gitnexus resolved {resolved['gitnexus']}."]
|
||||
if report["changes"]:
|
||||
lines.append("")
|
||||
for metric, change in report["changes"].items():
|
||||
sample = f"{change['pairs']}/{change['expected_pairs']} pairs"
|
||||
if change["pairs"] < change["expected_pairs"]:
|
||||
sample += ", partial"
|
||||
lines.append(f"- gitnexus vs baseline: {metric}: {change['percent']:+.1f}% ({sample})")
|
||||
gitnexus = report["arms"].get("gitnexus")
|
||||
if gitnexus and gitnexus.get("graded"):
|
||||
lines += ["", f"GitNexus usage: {gitnexus['trials_using_gitnexus']}/{gitnexus['graded']} graded trials; "
|
||||
f"commands {gitnexus['gitnexus_commands']}."]
|
||||
if report["problems"]:
|
||||
lines += ["", "**Problems:**", *[f"- {problem}" for problem in report["problems"]]]
|
||||
lines += ["", "Descriptive evidence on a sampled subset, not statistical significance.", ""]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--baseline", required=True, type=Path, help="pier job directory of the baseline arm")
|
||||
parser.add_argument("--gitnexus", required=True, type=Path, help="pier job directory of the gitnexus arm")
|
||||
parser.add_argument("--out", required=True, type=Path)
|
||||
args = parser.parse_args(argv)
|
||||
report = build_report({"baseline": args.baseline, "gitnexus": args.gitnexus})
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
(args.out / "deepswe-comparison.json").write_text(json.dumps(report, indent=2, sort_keys=True) + "\n")
|
||||
(args.out / "deepswe-comparison.md").write_text(render_markdown(report))
|
||||
print(render_markdown(report))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
213
eval/deepswe/run.sh
Executable file
213
eval/deepswe/run.sh
Executable file
|
|
@ -0,0 +1,213 @@
|
|||
#!/usr/bin/env bash
|
||||
# Paired DeepSWE run: prepare both arms, run pier on each, write the report.
|
||||
#
|
||||
# OPENAI_API_KEY=... eval/deepswe/run.sh --n-tasks 10 --concurrency 4
|
||||
#
|
||||
# The default agent is Claude Code. A non-Claude model (default gpt-5.6-sol)
|
||||
# goes through a local LiteLLM gateway, as in workflow_bench; an `anthropic/`
|
||||
# or `claude-` model uses ANTHROPIC_API_KEY directly. Needs docker, git,
|
||||
# python3.11+, pier (`uv tool install datacurve-pier==0.3.1`) and, to build the
|
||||
# package, node/npm 22.18+. Everything lands under --work (default ~/deepswe-work).
|
||||
set -euo pipefail
|
||||
|
||||
DEEP_SWE_REPO=https://github.com/datacurve-ai/deep-swe
|
||||
DEEP_SWE_COMMIT=0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea
|
||||
PIER_VERSION=0.3.1
|
||||
CLAUDE_CODE_VERSION=2.1.214 # the release evaluation's pinned CLI
|
||||
LITELLM_VERSION=1.100.4 # eval/uv.lock
|
||||
MINI_SWE_AGENT_VERSION=2.2.7 # eval/uv.lock
|
||||
GATEWAY_CONTAINER=deepswe-litellm-gateway
|
||||
GATEWAY_PORT=80 # pier's egress proxy only allows ports 80 and 443
|
||||
# Everything CLI 2.1.214 offers beyond Read/Edit/Write/Bash, matching the release
|
||||
# evaluation's single-agent toolset. Subagents and background tasks otherwise fan
|
||||
# out unbounded: one smoke trial ran 237 background agents past its budget cap.
|
||||
CLAUDE_DISALLOWED_TOOLS="EnterPlanMode Task Agent TaskCreate TaskGet TaskList TaskOutput TaskStop TaskUpdate SendMessage Skill Workflow CronCreate CronDelete CronList ScheduleWakeup EnterWorktree ExitWorktree NotebookEdit ReportFindings WebFetch WebSearch"
|
||||
|
||||
AGENT=claude-code
|
||||
MODEL=gpt-5.6-sol
|
||||
EFFORT=low
|
||||
N_TASKS=10
|
||||
SEED=0
|
||||
CONCURRENCY=2
|
||||
COST_LIMIT=5
|
||||
WORK="$HOME/deepswe-work"
|
||||
PACKAGE=""
|
||||
GATEWAY_HOST=""
|
||||
GUIDANCE=product
|
||||
TASKS=()
|
||||
ARMS=(baseline gitnexus)
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $0 [options]
|
||||
--n-tasks N seeded random subset size (default $N_TASKS; 0 = all 113)
|
||||
--seed S sample seed (default $SEED)
|
||||
--task ID run this task id; repeatable, overrides --n-tasks
|
||||
--concurrency C pier --n-concurrent per arm (default $CONCURRENCY; each trial uses 2 CPUs / 8 GB)
|
||||
--agent A claude-code (default) or mini-swe-agent
|
||||
--model M default $MODEL; claude-code: a gateway model name or anthropic/<claude model>;
|
||||
mini-swe-agent: a litellm name such as openai/gpt-5.6-sol
|
||||
--effort E reasoning effort (default $EFFORT)
|
||||
--cost-limit USD per-trial agent cost cap, same for both arms (default $COST_LIMIT; 0 = none)
|
||||
--package TGZ prebuilt \`npm pack\` tarball of gitnexus/ (default: build from this checkout)
|
||||
--gateway-host H hostname the trial egress proxy uses to reach this host's gateway
|
||||
(default: host.docker.internal if it resolves, else the EC2 private DNS name)
|
||||
--guidance G note for the gitnexus arm: product (GitNexus's agent rules, default) or optional
|
||||
--arms "A B" arms to run (default: "${ARMS[*]}")
|
||||
--work DIR working directory (default $WORK)
|
||||
EOF
|
||||
}
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--n-tasks) N_TASKS="$2"; shift 2 ;;
|
||||
--seed) SEED="$2"; shift 2 ;;
|
||||
--task) TASKS+=("$2"); shift 2 ;;
|
||||
--concurrency) CONCURRENCY="$2"; shift 2 ;;
|
||||
--agent) AGENT="$2"; shift 2 ;;
|
||||
--model) MODEL="$2"; shift 2 ;;
|
||||
--effort) EFFORT="$2"; shift 2 ;;
|
||||
--cost-limit) COST_LIMIT="$2"; shift 2 ;;
|
||||
--package) PACKAGE="$2"; shift 2 ;;
|
||||
--gateway-host) GATEWAY_HOST="$2"; shift 2 ;;
|
||||
--guidance) GUIDANCE="$2"; shift 2 ;;
|
||||
--arms) read -r -a ARMS <<<"$2"; shift 2 ;;
|
||||
--work) WORK="$2"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "unknown option: $1" >&2; usage >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
EVAL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
REPO_ROOT="$(cd "$EVAL_DIR/.." && pwd)"
|
||||
needed=(docker git python3 pier curl)
|
||||
[[ " ${ARMS[*]} " == *" gitnexus "* ]] && [ -z "$PACKAGE" ] && needed+=(node npm) # only to build the package
|
||||
for tool in "${needed[@]}"; do
|
||||
command -v "$tool" >/dev/null || { echo "missing $tool on PATH" >&2; exit 1; }
|
||||
done
|
||||
pier_version="$(pier --version 2>/dev/null | grep -Eo '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true)"
|
||||
[ "$pier_version" = "$PIER_VERSION" ] || echo "warning: pier $pier_version, pinned $PIER_VERSION" >&2
|
||||
|
||||
agent_args=(--agent "$AGENT" --model "$MODEL")
|
||||
use_gateway=0
|
||||
case "$AGENT" in
|
||||
claude-code)
|
||||
agent_args+=(--ak "version=$CLAUDE_CODE_VERSION" --ak "reasoning_effort=$EFFORT"
|
||||
--ak "disallowed_tools=$CLAUDE_DISALLOWED_TOOLS"
|
||||
# pier forces both on; --ae is merged last, so these win.
|
||||
--ae ENABLE_BACKGROUND_TASKS=0 --ae FORCE_AUTO_BACKGROUND_TASKS=0)
|
||||
[ "$COST_LIMIT" != 0 ] && agent_args+=(--ak "max_budget_usd=$COST_LIMIT")
|
||||
case "$MODEL" in
|
||||
anthropic/*|claude-*) : "${ANTHROPIC_API_KEY:?ANTHROPIC_API_KEY is required for $MODEL}" ;;
|
||||
*/*) echo "claude-code through the gateway takes a bare model name, not $MODEL" >&2; exit 2 ;;
|
||||
*) : "${OPENAI_API_KEY:?OPENAI_API_KEY is required for the $MODEL gateway}"; use_gateway=1 ;;
|
||||
esac
|
||||
;;
|
||||
mini-swe-agent)
|
||||
# litellm's Responses path imports `mcp`, which a bare mini-swe-agent install lacks.
|
||||
agent_args+=(--ak "version=$MINI_SWE_AGENT_VERSION" --ak "reasoning_effort=$EFFORT"
|
||||
--ak "cost_limit=$COST_LIMIT" --ak 'extra_python_packages=["mcp"]')
|
||||
case "$MODEL" in
|
||||
openai/*) : "${OPENAI_API_KEY:?OPENAI_API_KEY is required for $MODEL}" ;;
|
||||
anthropic/*) : "${ANTHROPIC_API_KEY:?ANTHROPIC_API_KEY is required for $MODEL}" ;;
|
||||
esac
|
||||
;;
|
||||
*) echo "unsupported agent: $AGENT" >&2; exit 2 ;;
|
||||
esac
|
||||
mkdir -p "$WORK"
|
||||
|
||||
start_gateway() {
|
||||
if [ -z "$GATEWAY_HOST" ]; then
|
||||
if getent hosts host.docker.internal >/dev/null; then
|
||||
GATEWAY_HOST=host.docker.internal
|
||||
else
|
||||
imds_token="$(curl -fsS -X PUT -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' \
|
||||
http://169.254.169.254/latest/api/token 2>/dev/null || true)"
|
||||
GATEWAY_HOST="$(curl -fsS -H "X-aws-ec2-metadata-token: $imds_token" \
|
||||
http://169.254.169.254/latest/meta-data/local-hostname 2>/dev/null || hostname -f)"
|
||||
fi
|
||||
fi
|
||||
local dir="$WORK/gateway"
|
||||
mkdir -p "$dir"
|
||||
# Same routing as workflow_bench.model_gateway: Claude Code speaks Anthropic
|
||||
# Messages; LiteLLM serves them from OpenAI's Responses API.
|
||||
cat > "$dir/litellm.yaml" <<EOF
|
||||
model_list:
|
||||
- model_name: $MODEL
|
||||
litellm_params:
|
||||
model: openai/$MODEL
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
timeout: 900
|
||||
model_info:
|
||||
mode: responses
|
||||
litellm_settings:
|
||||
request_timeout: 900
|
||||
general_settings:
|
||||
master_key: os.environ/LITELLM_MASTER_KEY
|
||||
EOF
|
||||
GATEWAY_KEY="sk-deepswe-$(python3 -c 'import secrets; print(secrets.token_hex(16))')"
|
||||
docker rm -f "$GATEWAY_CONTAINER" >/dev/null 2>&1 || true
|
||||
docker run -d --name "$GATEWAY_CONTAINER" -p "$GATEWAY_PORT:4000" \
|
||||
-e OPENAI_API_KEY -e LITELLM_MASTER_KEY="$GATEWAY_KEY" -e LITELLM_LOCAL_MODEL_COST_MAP=True \
|
||||
-v "$dir:/cfg:ro" python:3.12-slim \
|
||||
sh -c "pip install --quiet --no-cache-dir 'litellm[proxy]==$LITELLM_VERSION' && exec litellm --config /cfg/litellm.yaml --host 0.0.0.0 --port 4000" \
|
||||
>/dev/null
|
||||
trap 'docker rm -f "$GATEWAY_CONTAINER" >/dev/null 2>&1 || true' EXIT
|
||||
echo "waiting for the LiteLLM gateway on http://$GATEWAY_HOST:$GATEWAY_PORT"
|
||||
for _ in $(seq 1 180); do
|
||||
if curl -fsS -o /dev/null -H "Authorization: Bearer $GATEWAY_KEY" \
|
||||
"http://$GATEWAY_HOST:$GATEWAY_PORT/health/liveliness" 2>/dev/null; then
|
||||
agent_args+=(--ae "ANTHROPIC_BASE_URL=http://$GATEWAY_HOST:$GATEWAY_PORT" --ae "ANTHROPIC_AUTH_TOKEN=$GATEWAY_KEY")
|
||||
return 0
|
||||
fi
|
||||
docker inspect -f '{{.State.Running}}' "$GATEWAY_CONTAINER" 2>/dev/null | grep -q true || break
|
||||
sleep 2
|
||||
done
|
||||
echo "gateway did not become ready; last log lines:" >&2
|
||||
docker logs --tail 20 "$GATEWAY_CONTAINER" >&2 || true
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [ ! -d "$WORK/deep-swe/.git" ]; then
|
||||
git clone --quiet "$DEEP_SWE_REPO" "$WORK/deep-swe"
|
||||
fi
|
||||
git -C "$WORK/deep-swe" fetch --quiet origin "$DEEP_SWE_COMMIT" 2>/dev/null || true
|
||||
git -C "$WORK/deep-swe" checkout --quiet --detach "$DEEP_SWE_COMMIT"
|
||||
|
||||
if [[ " ${ARMS[*]} " == *" gitnexus "* ]] && [ -z "$PACKAGE" ]; then
|
||||
echo "building the gitnexus package from $REPO_ROOT"
|
||||
# Same steps as the CI package job: prepack compiles the CLI and web UI into the tarball.
|
||||
(cd "$REPO_ROOT/gitnexus" && npm ci --ignore-scripts --no-fund --no-audit)
|
||||
(cd "$REPO_ROOT/gitnexus-shared" && node ../gitnexus/node_modules/typescript/lib/tsc.js)
|
||||
(cd "$REPO_ROOT/gitnexus-web" && npm ci --no-fund --no-audit)
|
||||
(cd "$REPO_ROOT/gitnexus" && npm pack --pack-destination "$WORK" >/dev/null)
|
||||
PACKAGE="$(ls -t "$WORK"/gitnexus-*.tgz | head -1)"
|
||||
fi
|
||||
|
||||
select_args=()
|
||||
if [ ${#TASKS[@]} -gt 0 ]; then
|
||||
for task in "${TASKS[@]}"; do select_args+=(--task "$task"); done
|
||||
elif [ "$N_TASKS" != 0 ]; then
|
||||
select_args+=(--n-tasks "$N_TASKS" --sample-seed "$SEED")
|
||||
fi
|
||||
package_args=()
|
||||
[ -n "$PACKAGE" ] && package_args+=(--gitnexus-package "$PACKAGE")
|
||||
(cd "$EVAL_DIR" && python3 -m deepswe.prepare --deep-swe "$WORK/deep-swe" --out "$WORK/tasks" \
|
||||
--arms "${ARMS[@]}" --guidance "$GUIDANCE" "${package_args[@]}" "${select_args[@]}")
|
||||
|
||||
[ "$use_gateway" = 1 ] && start_gateway
|
||||
|
||||
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
for arm in "${ARMS[@]}"; do
|
||||
echo "== pier: $arm"
|
||||
# A failed trial is recorded in the job and reported; keep going to the other arm.
|
||||
pier run -p "$WORK/tasks/$arm" --env docker "${agent_args[@]}" \
|
||||
--n-concurrent "$CONCURRENCY" --jobs-dir "$WORK/jobs" --job-name "$arm-$stamp" --yes || \
|
||||
echo "pier exited nonzero for $arm; see $WORK/jobs/$arm-$stamp" >&2
|
||||
done
|
||||
|
||||
if [ ${#ARMS[@]} -eq 2 ]; then
|
||||
(cd "$EVAL_DIR" && python3 -m deepswe.report --baseline "$WORK/jobs/baseline-$stamp" \
|
||||
--gitnexus "$WORK/jobs/gitnexus-$stamp" --out "$WORK/report-$stamp")
|
||||
echo "report: $WORK/report-$stamp/deepswe-comparison.md"
|
||||
fi
|
||||
239
eval/tests/test_deepswe.py
Normal file
239
eval/tests/test_deepswe.py
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
"""eval/deepswe: task preparation for the paired DeepSWE arms, and the paired report."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from deepswe import prepare, report
|
||||
|
||||
DEEPSWE_DIR = Path(__file__).resolve().parents[1] / "deepswe"
|
||||
|
||||
# Trimmed from deep-swe@0b9fabbb tasks/abs-module-cache-flags/task.toml.
|
||||
TASK_TOML = """schema_version = "1.3"
|
||||
artifacts = ["/logs/artifacts/model.patch"]
|
||||
[task]
|
||||
name = "datacurve/abs-module-cache-flags"
|
||||
description = ""
|
||||
[metadata]
|
||||
task_id = "abs-module-cache-flags"
|
||||
language = "go"
|
||||
base_commit_hash = "cb1b3b671d0ee9fa9da9f7b02f86967953ffd10a"
|
||||
[verifier]
|
||||
network_mode = "no-network"
|
||||
environment_mode = "separate"
|
||||
timeout_sec = 1800.0
|
||||
[verifier.environment]
|
||||
build_timeout_sec = 1800.0
|
||||
docker_image = "verifier-image-stays"
|
||||
[[verifier.collect]]
|
||||
command = "cd /app && git diff --binary cb1b3b671d0ee9fa9da9f7b02f86967953ffd10a HEAD > /logs/artifacts/model.patch"
|
||||
timeout_sec = 300.0
|
||||
[agent]
|
||||
network_mode = "no-network"
|
||||
timeout_sec = 10800.0
|
||||
[environment]
|
||||
build_timeout_sec = 1800.0
|
||||
docker_image = "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh75679ajj3b8dtd7se3h7z0a1833y6r-v1.1"
|
||||
os = "linux"
|
||||
cpus = 2
|
||||
memory_mb = 8192
|
||||
mcp_servers = []
|
||||
"""
|
||||
|
||||
|
||||
def _task(root: Path, task_id: str, toml: str = TASK_TOML) -> Path:
|
||||
task = root / task_id
|
||||
(task / "environment").mkdir(parents=True)
|
||||
(task / "environment" / "Dockerfile").write_text("FROM upstream-build\n")
|
||||
(task / "tests").mkdir()
|
||||
(task / "tests" / "test.sh").write_text("#!/bin/bash\n")
|
||||
(task / "instruction.md").write_text("Implement the thing.\n")
|
||||
(task / "task.toml").write_text(toml.replace("abs-module-cache-flags", task_id))
|
||||
return task
|
||||
|
||||
|
||||
def test_select_tasks_is_seeded_and_validates_ids():
|
||||
ids = [f"t{i:02d}" for i in range(20)]
|
||||
first = prepare.select_tasks(ids, include=None, n_tasks=5, seed=3)
|
||||
assert first == prepare.select_tasks(list(reversed(ids)), include=None, n_tasks=5, seed=3)
|
||||
assert len(first) == 5 and first == sorted(first)
|
||||
assert prepare.select_tasks(ids, include=None, n_tasks=None, seed=0) == ids
|
||||
assert prepare.select_tasks(ids, include=["t03", "t01", "t03"], n_tasks=5, seed=0) == ["t01", "t03"]
|
||||
with pytest.raises(SystemExit, match="nope"):
|
||||
prepare.select_tasks(ids, include=["nope"], n_tasks=None, seed=0)
|
||||
|
||||
|
||||
def test_gitnexus_task_toml_builds_from_the_derived_dockerfile_under_a_new_name():
|
||||
data = tomllib.loads(prepare.gitnexus_task_toml(TASK_TOML))
|
||||
assert data["task"]["name"] == "datacurve/abs-module-cache-flags-gitnexus"
|
||||
assert "docker_image" not in data["environment"]
|
||||
assert data["environment"]["build_timeout_sec"] == prepare.MIN_BUILD_TIMEOUT_SEC
|
||||
# Everything else, including the separate verifier image and the patch collector, is untouched.
|
||||
original = tomllib.loads(TASK_TOML)
|
||||
assert data["verifier"] == original["verifier"]
|
||||
assert data["agent"] == original["agent"] and data["metadata"] == original["metadata"]
|
||||
assert {k: v for k, v in data["environment"].items() if k != "build_timeout_sec"} == {
|
||||
k: v for k, v in original["environment"].items() if k not in ("docker_image", "build_timeout_sec")
|
||||
}
|
||||
|
||||
|
||||
def test_prepare_copies_baseline_unchanged_and_layers_gitnexus(tmp_path):
|
||||
tasks = tmp_path / "deep-swe" / "tasks"
|
||||
for task_id in ("alpha-task", "beta-task"):
|
||||
_task(tasks, task_id)
|
||||
package = tmp_path / "gitnexus-9.9.9.tgz"
|
||||
package.write_bytes(b"tarball")
|
||||
out = tmp_path / "out"
|
||||
|
||||
assert prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(out),
|
||||
"--gitnexus-package", str(package), "--task", "alpha-task"]) == 0
|
||||
|
||||
manifest = json.loads((out / "manifest.json").read_text())
|
||||
assert manifest["tasks"] == ["alpha-task"] and manifest["arms"] == ["baseline", "gitnexus"]
|
||||
assert not (out / "baseline" / "beta-task").exists()
|
||||
for name in ("task.toml", "instruction.md", "environment/Dockerfile", "tests/test.sh"):
|
||||
assert (out / "baseline" / "alpha-task" / name).read_bytes() == (tasks / "alpha-task" / name).read_bytes()
|
||||
|
||||
derived = out / "gitnexus" / "alpha-task"
|
||||
dockerfile = (derived / "environment" / "Dockerfile").read_text()
|
||||
assert dockerfile.startswith(
|
||||
"FROM public.ecr.aws/d3j8x8q7/swe-bench-202605:kh75679ajj3b8dtd7se3h7z0a1833y6r-v1.1\n"
|
||||
)
|
||||
assert f"ARG NODE_VERSION={prepare.NODE_VERSION}" in dockerfile and "sha256sum -c -" in dockerfile
|
||||
assert "gitnexus analyze --index-only" in dockerfile and 'test "$before" = "$after"' in dockerfile
|
||||
# A task's own node/npm stay first on PATH; only a pinned wrapper is added.
|
||||
assert "ENV PATH" not in dockerfile and "/usr/local/bin/gitnexus" in dockerfile
|
||||
assert "{" not in dockerfile.replace("${", "") # every template field was filled
|
||||
assert (derived / "environment" / prepare.PACKAGE_FILE).read_bytes() == b"tarball"
|
||||
instruction = (derived / "instruction.md").read_text()
|
||||
assert instruction.startswith("Implement the thing.\n") and instruction.endswith(prepare.GUIDANCE["product"])
|
||||
assert (derived / "tests" / "test.sh").read_bytes() == (tasks / "alpha-task" / "tests" / "test.sh").read_bytes()
|
||||
|
||||
|
||||
def test_gitnexus_arm_requires_a_package_and_a_prebuilt_image(tmp_path):
|
||||
tasks = tmp_path / "deep-swe" / "tasks"
|
||||
_task(tasks, "gamma-task", TASK_TOML.replace('docker_image = "public.ecr', 'base_image = "public.ecr'))
|
||||
with pytest.raises(SystemExit):
|
||||
prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(tmp_path / "o")])
|
||||
package = tmp_path / "p.tgz"
|
||||
package.write_bytes(b"x")
|
||||
with pytest.raises(ValueError, match="docker_image"):
|
||||
prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(tmp_path / "o"),
|
||||
"--gitnexus-package", str(package)])
|
||||
|
||||
|
||||
@pytest.mark.parametrize("guidance", sorted(prepare.GUIDANCE))
|
||||
def test_gitnexus_guidance_names_only_real_cli_commands(guidance):
|
||||
note = prepare.GUIDANCE[guidance]
|
||||
commands = set(re.findall(r"`gitnexus ([a-z-]+)", note)) - {"<command>"}
|
||||
assert commands == {"query", "context", "impact", "detect-changes"}
|
||||
assert len(note.encode()) < 1000
|
||||
|
||||
|
||||
def test_prepare_records_the_guidance_variant(tmp_path):
|
||||
_task(tmp_path / "deep-swe" / "tasks", "delta-task")
|
||||
package = tmp_path / "p.tgz"
|
||||
package.write_bytes(b"x")
|
||||
prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(tmp_path / "o"), "--arms", "gitnexus",
|
||||
"--gitnexus-package", str(package), "--guidance", "optional"])
|
||||
assert json.loads((tmp_path / "o" / "manifest.json").read_text())["guidance"] == "optional"
|
||||
assert (tmp_path / "o" / "gitnexus" / "delta-task" / "instruction.md").read_text().endswith(
|
||||
prepare.GUIDANCE["optional"]
|
||||
)
|
||||
|
||||
|
||||
def _trial(job: Path, name: str, task: str, *, reward=1, cost=1.0, steps=10, exception=None, commands=()):
|
||||
trial = job / name
|
||||
(trial / "agent").mkdir(parents=True)
|
||||
result = {
|
||||
"task_name": f"datacurve/{task}", "trial_name": name, "started_at": f"2026-10-30T00:00:{len(name):02d}Z",
|
||||
"agent_result": {"cost_usd": cost, "n_input_tokens": 1000, "n_cache_tokens": 800, "n_output_tokens": 50,
|
||||
"n_agent_steps": steps},
|
||||
"verifier_result": None if reward is None else {"rewards": {"reward": reward}},
|
||||
"exception_info": exception,
|
||||
"agent_execution": {"started_at": "2026-10-30T00:00:00Z", "finished_at": "2026-10-30T00:01:40Z"},
|
||||
}
|
||||
(trial / "result.json").write_text(json.dumps(result))
|
||||
calls = [{"function_name": "bash", "arguments": {"command": command}} for command in commands]
|
||||
(trial / "agent" / "trajectory.json").write_text(json.dumps({"steps": [{"tool_calls": calls}]}))
|
||||
|
||||
|
||||
def test_report_pairs_on_task_id_and_compares_matched_pairs_only(tmp_path):
|
||||
base, gitnexus = tmp_path / "baseline-job", tmp_path / "gitnexus-job"
|
||||
_trial(base, "a__1", "alpha", cost=1.0)
|
||||
_trial(base, "b__1", "beta", cost=9.0, reward=0)
|
||||
_trial(base, "c__1", "gamma", cost=2.0)
|
||||
_trial(gitnexus, "a__1", "alpha-gitnexus", cost=2.0,
|
||||
commands=["gitnexus query parse && gitnexus impact Foo", "ls gitnexus/ && cat gitnexus.json"])
|
||||
_trial(gitnexus, "b__1", "beta-gitnexus", cost=9.0, reward=-1) # grader crash sentinel
|
||||
# gamma never ran in the gitnexus arm (interrupted sweep)
|
||||
_trial(base, "d__1", "delta", cost=3.0, reward=0, exception={"exception_type": "NonZeroAgentExitCodeError"})
|
||||
_trial(gitnexus, "d__1", "delta-gitnexus", cost=4.0, reward=1)
|
||||
(base / "config.json").write_text("{}") # job-level files are not trials
|
||||
|
||||
result = report.build_report({"baseline": base, "gitnexus": gitnexus})
|
||||
|
||||
# The capped baseline trial (agent error, reward 0) is graded and paired, not dropped.
|
||||
assert result["pairs"] == {"graded_in_both": 2, "expected": 4}
|
||||
assert result["changes"]["cost_usd"] == {"percent": 50.0, "pairs": 2, "expected_pairs": 4}
|
||||
assert result["paired_resolved"] == {"baseline": 1, "gitnexus": 2}
|
||||
assert result["arms"]["baseline"]["resolved"] == 2 and result["arms"]["baseline"]["graded"] == 4
|
||||
assert result["arms"]["gitnexus"]["gitnexus_commands"] == {"query": 1, "impact": 1}
|
||||
assert result["arms"]["gitnexus"]["trials_using_gitnexus"] == 1
|
||||
assert any("gitnexus: 1 of 3 trial(s) ungraded" in problem for problem in result["problems"])
|
||||
assert any("2 of 4 task/attempt pair(s) graded in both arms" in problem for problem in result["problems"])
|
||||
assert any("baseline: 1 graded trial(s) ended with an agent error" in problem for problem in result["problems"])
|
||||
markdown = report.render_markdown(result)
|
||||
assert "gitnexus vs baseline: cost_usd: +50.0% (2/4 pairs, partial)" in markdown
|
||||
assert result["per_task"]["gamma#0"]["gitnexus"] is None
|
||||
|
||||
|
||||
def test_report_marks_missing_billing_incomplete(tmp_path):
|
||||
base, gitnexus = tmp_path / "b", tmp_path / "g"
|
||||
for name, task in (("x__1", "x"), ("y__1", "y")):
|
||||
_trial(base, name, task, cost=1.0)
|
||||
_trial(gitnexus, "x__1", "x-gitnexus", cost=None)
|
||||
_trial(gitnexus, "y__1", "y-gitnexus", cost=3.0)
|
||||
|
||||
result = report.build_report({"baseline": base, "gitnexus": gitnexus})
|
||||
|
||||
assert result["arms"]["gitnexus"]["mean_cost_usd"] is None
|
||||
assert result["arms"]["gitnexus"]["incomplete_metrics"] == ["cost_usd"]
|
||||
assert result["changes"]["cost_usd"]["pairs"] == 1 and result["changes"]["n_agent_steps"]["pairs"] == 2
|
||||
assert "| gitnexus | 2/2 | incomplete |" in report.render_markdown(result)
|
||||
|
||||
|
||||
def test_report_survives_an_arm_that_never_produced_a_job(tmp_path):
|
||||
_trial(tmp_path / "b", "x__1", "x")
|
||||
result = report.build_report({"baseline": tmp_path / "b", "gitnexus": tmp_path / "never-created"})
|
||||
assert any(problem.startswith("gitnexus: no trial results") for problem in result["problems"])
|
||||
assert result["pairs"] == {"graded_in_both": 0, "expected": 1} and result["changes"] == {}
|
||||
assert "| gitnexus | 0/0 |" in report.render_markdown(result)
|
||||
|
||||
|
||||
def test_scripts_pin_the_same_versions_and_fill_every_bootstrap_placeholder():
|
||||
run_sh = (DEEPSWE_DIR / "run.sh").read_text()
|
||||
bootstrap = (DEEPSWE_DIR / "ec2" / "bootstrap.sh").read_text()
|
||||
launch = (DEEPSWE_DIR / "ec2" / "launch.sh").read_text()
|
||||
|
||||
def pin(text: str, name: str) -> str:
|
||||
match = re.search(rf"^{name}=(\S+)", text, re.MULTILINE)
|
||||
assert match, name
|
||||
return match.group(1)
|
||||
|
||||
assert pin(run_sh, "PIER_VERSION") == pin(bootstrap, "PIER_VERSION")
|
||||
# The gateway and the optional mini-swe-agent use the versions eval/ is locked to.
|
||||
lock = (DEEPSWE_DIR.parent / "uv.lock").read_text()
|
||||
for package, name in (("litellm", "LITELLM_VERSION"), ("mini-swe-agent", "MINI_SWE_AGENT_VERSION")):
|
||||
assert f'name = "{package}"\nversion = "{pin(run_sh, name)}"' in lock, package
|
||||
assert re.fullmatch(r"\d+\.\d+\.\d+", pin(run_sh, "CLAUDE_CODE_VERSION"))
|
||||
assert pin(bootstrap, "NODE_VERSION") == prepare.NODE_VERSION
|
||||
assert re.fullmatch(r"[0-9a-f]{40}", pin(run_sh, "DEEP_SWE_COMMIT"))
|
||||
placeholders = set(re.findall(r"@([A-Z_]+)@", bootstrap))
|
||||
assert placeholders == set(re.findall(r"s\|@([A-Z_]+)@\|", launch)) == {"GITNEXUS_REPO", "GITNEXUS_REF", "MAX_HOURS"}
|
||||
assert "API_KEY" not in bootstrap.replace("# No model key", "")
|
||||
Loading…
Add table
Reference in a new issue