diff --git a/eval/deepswe/README.md b/eval/deepswe/README.md index 7c7ac34a1..934aa4265 100644 --- a/eval/deepswe/README.md +++ b/eval/deepswe/README.md @@ -7,9 +7,21 @@ | `baseline` | The DeepSWE task, unchanged | | `gitnexus` | The same task, plus the `gitnexus` CLI and an index of `/app` built into the image, and a short note on the CLI appended to the instruction | -The default agent is `mini-swe-agent`, which the leaderboard uses. It is bash-only, so GitNexus is reached through the real CLI (`gitnexus query`, `context`, `impact`, `detect-changes`) rather than MCP. The arms differ only in that layer and that note. +The default agent is Claude Code (pier's `claude-code`, pinned to the release evaluation's CLI). GitNexus is reached through the real CLI (`gitnexus query`, `context`, `impact`, `detect-changes`) rather than MCP. The arms differ only in that layer and that note. -The workflow_bench `baseline_cli` arm points the same way: on the small release tasks, a shell command cost far less than the MCP tool catalog. +- **Toolset.** Claude Code gets only `Read`, `Edit`, `Write` and `Bash`, matching the release evaluation's single-agent toolset. Subagents, background tasks, skills and web tools are disallowed, and pier's forced background-task env is switched off. Without this, one smoke trial ran 237 background agents past its budget cap. +- **Guidance.** The note (`--guidance product`, the default) is the CLI form of the rules GitNexus ships in AGENTS.md: graph first, `impact` before editing a symbol, `detect-changes` before committing. In the release evaluation, arms load those directives the same way. `--guidance optional` only lists the commands; in a smoke trial the agent then never used them. + +The workflow_bench `baseline_cli` arm points the same way: on the small release tasks, a shell command cost far less than the MCP tool catalog. `--agent mini-swe-agent`, the leaderboard's agent, also works. + +### Model access + +- **Non-Claude models** (default `gpt-5.6-sol`, `OPENAI_API_KEY`): `run.sh` starts a LiteLLM gateway container. It is routed like `workflow_bench.model_gateway`, with Claude Code's Anthropic Messages served from OpenAI's Responses API, a random master key, and the LiteLLM version from `eval/uv.lock`. + - It listens on host port 80, because pier's egress proxy only allows ports 80 and 443. Keep that port closed to the outside; `launch.sh`'s security group only opens SSH. + - Trials reach it as `host.docker.internal` on Docker Desktop, or as the EC2 private DNS name. +- **Claude models** (`--model anthropic/`): use `ANTHROPIC_API_KEY` directly. + +Costs come from Claude Code's own per-session accounting (`total_cost_usd`). Through the gateway that is Claude Code's price table applied to a non-Claude model, so it is consistent across arms but is not the provider's invoice. ## How the GitNexus arm is built @@ -57,7 +69,7 @@ The main options are listed below; `run.sh --help` shows them all. | Option | Default | Notes | | --- | --- | --- | | `--n-tasks`, `--seed`, `--task` | 10, 0 | `--task` (repeatable) overrides the sample; `--n-tasks 0` runs all 113 | -| `--agent`, `--model`, `--effort` | `mini-swe-agent`, `openai/gpt-5.6-sol`, `low` | `--agent claude-code` also works with `ANTHROPIC_API_KEY` | +| `--agent`, `--model`, `--effort` | `claude-code`, `gpt-5.6-sol`, `low` | `--model anthropic/` skips the gateway. `--agent mini-swe-agent` takes litellm names such as `openai/gpt-5.6-sol`. | | `--cost-limit` | `5` | Per-trial agent cost cap in USD, the same for both arms. A capped trial ends unsolved. | | `--concurrency` | `2` | Trials per arm at once. Each needs 2 CPUs and 8 GB. | | `--package` | build from checkout | Pass a prebuilt `npm pack` tarball to skip the build | @@ -68,6 +80,7 @@ The main options are listed below; `run.sh --help` shows them all. - Deltas use only pairs graded in both arms with the metric measured in both. - Ungraded trials, interrupted sweeps and missing cost are listed under **Problems**. A grader crash writes reward `-1`, which counts as ungraded. +- A trial whose agent hit `--cost-limit` or exited nonzero still had its patch graded. It counts as an unresolved attempt, listed as an agent error, not dropped. - `GitNexus usage` counts `gitnexus ` calls at command position in the ATIF trajectories. ```bash diff --git a/eval/deepswe/prepare.py b/eval/deepswe/prepare.py index 15c6e3e5a..71d6a0a5a 100644 --- a/eval/deepswe/prepare.py +++ b/eval/deepswe/prepare.py @@ -26,7 +26,24 @@ NODE_VERSION = "22.18.0" # gitnexus/package.json engines: ^22.18.0 PACKAGE_FILE = "gitnexus.tgz" MIN_BUILD_TIMEOUT_SEC = 3600.0 # the index is built inside the image build -GITNEXUS_NOTE = """ +# Appended to the gitnexus arm's instruction. "product" is the CLI form of the +# rules GitNexus ships in AGENTS.md (graph first, impact before edits, +# detect-changes before committing), as the release evaluation's arms load. +# "optional" only lists the commands, for an ablation. +GUIDANCE = { + "product": """ + +## GitNexus + +A GitNexus code graph of `/app` (indexed at the starting commit) is installed; the `gitnexus` command works offline. Follow GitNexus's agent rules: + +- Explore with `gitnexus query ""` for concepts and execution flows, and `gitnexus context ` for a named symbol's callers, callees and flows. Graph first; use text search for literals or when the graph has no answer. +- Before editing a function, class or method, run `gitnexus impact --direction upstream` and check its callers and risk. An empty result is not proof that nothing depends on it. +- Before committing, run `gitnexus detect-changes` to see which symbols and flows your edits affect. + +Run `gitnexus --help` for options. +""", + "optional": """ ## GitNexus @@ -38,7 +55,8 @@ A GitNexus code graph of `/app` (indexed at the starting commit) is installed. T - `gitnexus detect-changes`: symbols and flows touched by your uncommitted edits Run `gitnexus --help` for options. Using it is optional. -""" +""", +} DOCKERFILE = """FROM {image} @@ -125,7 +143,7 @@ def gitnexus_task_toml(task_toml: str) -> str: return rewritten -def prepare_task(source: Path, dest: Path, *, arm: str, package: Path | None) -> None: +def prepare_task(source: Path, dest: Path, *, arm: str, package: Path | None, guidance: str = "product") -> None: if dest.exists(): shutil.rmtree(dest) shutil.copytree(source, dest) @@ -143,20 +161,22 @@ def prepare_task(source: Path, dest: Path, *, arm: str, package: Path | None) -> shutil.copyfile(package, environment / PACKAGE_FILE) (dest / "task.toml").write_text(gitnexus_task_toml(task_toml), encoding="utf-8") instruction = dest / "instruction.md" - instruction.write_text(instruction.read_text(encoding="utf-8").rstrip() + "\n" + GITNEXUS_NOTE, encoding="utf-8") + instruction.write_text(instruction.read_text(encoding="utf-8").rstrip() + "\n" + GUIDANCE[guidance], encoding="utf-8") def prepare( - *, tasks_root: Path, out: Path, arms: list[str], selected: list[str], package: Path | None + *, tasks_root: Path, out: Path, arms: list[str], selected: list[str], package: Path | None, + guidance: str = "product", ) -> dict[str, Any]: for arm in arms: for task in selected: - prepare_task(tasks_root / task, out / arm / task, arm=arm, package=package) + prepare_task(tasks_root / task, out / arm / task, arm=arm, package=package, guidance=guidance) manifest = { "schema": "gitnexus.deepswe-prepare/v1", "arms": arms, "tasks": selected, "gitnexus_task_suffix": GITNEXUS_TASK_SUFFIX, + "guidance": guidance, "node_version": NODE_VERSION, "gitnexus_package": package.name if package else None, } @@ -173,6 +193,8 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument("--task", action="append", dest="include", help="task id; repeatable") parser.add_argument("--n-tasks", type=int, help="seeded random subset size (default: all tasks)") parser.add_argument("--sample-seed", type=int, default=0) + parser.add_argument("--guidance", choices=sorted(GUIDANCE), default="product", + help="note appended to the gitnexus arm's instruction (default: product)") args = parser.parse_args(argv) tasks_root = args.deep_swe / "tasks" available = task_ids(tasks_root) @@ -182,7 +204,7 @@ def main(argv: list[str] | None = None) -> int: parser.error("--gitnexus-package must name an existing `npm pack` tarball for the gitnexus arm") selected = select_tasks(available, include=args.include, n_tasks=args.n_tasks, seed=args.sample_seed) manifest = prepare(tasks_root=tasks_root, out=args.out, arms=args.arms, selected=selected, - package=args.gitnexus_package) + package=args.gitnexus_package, guidance=args.guidance) print(f"prepared {len(selected)} task(s) x {len(manifest['arms'])} arm(s) under {args.out}") return 0 diff --git a/eval/deepswe/report.py b/eval/deepswe/report.py index 65d569956..13d58b4f4 100644 --- a/eval/deepswe/report.py +++ b/eval/deepswe/report.py @@ -68,7 +68,10 @@ def load_trial(trial_dir: Path) -> dict[str, Any] | None: agent = result.get("agent_result") or {} rewards = (result.get("verifier_result") or {}).get("rewards") or {} reward = rewards.get("reward") - graded = result.get("exception_info") is None and reward in (0, 1) + # Graded means the verifier produced a 0/1 reward. A trial whose agent hit the + # cost cap or exited nonzero is still graded on its patch; dropping it would + # hide expensive failures. -1 (the grader's crash sentinel) or no reward is not. + graded = reward in (0, 1) and not isinstance(reward, bool) row = { "task": task_id(str(result.get("task_name", trial_dir.name))), "trial": result.get("trial_name") or trial_dir.name, @@ -115,6 +118,12 @@ def build_report(jobs: dict[str, Path]) -> dict[str, Any]: if ungraded: kinds = Counter(row["error"] or "no reward" for row in ungraded) report["problems"].append(f"{arm}: {len(ungraded)} of {len(trials)} trial(s) ungraded ({dict(kinds)})") + agent_errors = Counter(row["error"] for row in graded if row["error"]) + if agent_errors: + report["problems"].append( + f"{arm}: {sum(agent_errors.values())} graded trial(s) ended with an agent error " + + f"({dict(agent_errors)}), e.g. the cost cap; their patches were still graded" + ) summary: dict[str, Any] = {"trials": len(trials), "graded": len(graded), "resolved": sum(row["resolved"] for row in graded), "incomplete_metrics": []} for metric in METRICS: diff --git a/eval/deepswe/run.sh b/eval/deepswe/run.sh index facd7c3de..1e0ac7faf 100755 --- a/eval/deepswe/run.sh +++ b/eval/deepswe/run.sh @@ -18,6 +18,10 @@ LITELLM_VERSION=1.100.4 # eval/uv.lock MINI_SWE_AGENT_VERSION=2.2.7 # eval/uv.lock GATEWAY_CONTAINER=deepswe-litellm-gateway GATEWAY_PORT=80 # pier's egress proxy only allows ports 80 and 443 +# Everything CLI 2.1.214 offers beyond Read/Edit/Write/Bash, matching the release +# evaluation's single-agent toolset. Subagents and background tasks otherwise fan +# out unbounded: one smoke trial ran 237 background agents past its budget cap. +CLAUDE_DISALLOWED_TOOLS="EnterPlanMode Task Agent TaskCreate TaskGet TaskList TaskOutput TaskStop TaskUpdate SendMessage Skill Workflow CronCreate CronDelete CronList ScheduleWakeup EnterWorktree ExitWorktree NotebookEdit ReportFindings WebFetch WebSearch" AGENT=claude-code MODEL=gpt-5.6-sol @@ -29,6 +33,7 @@ COST_LIMIT=5 WORK="$HOME/deepswe-work" PACKAGE="" GATEWAY_HOST="" +GUIDANCE=product TASKS=() ARMS=(baseline gitnexus) @@ -47,6 +52,7 @@ Usage: $0 [options] --package TGZ prebuilt \`npm pack\` tarball of gitnexus/ (default: build from this checkout) --gateway-host H hostname the trial egress proxy uses to reach this host's gateway (default: host.docker.internal if it resolves, else the EC2 private DNS name) + --guidance G note for the gitnexus arm: product (GitNexus's agent rules, default) or optional --arms "A B" arms to run (default: "${ARMS[*]}") --work DIR working directory (default $WORK) EOF @@ -64,6 +70,7 @@ while [ $# -gt 0 ]; do --cost-limit) COST_LIMIT="$2"; shift 2 ;; --package) PACKAGE="$2"; shift 2 ;; --gateway-host) GATEWAY_HOST="$2"; shift 2 ;; + --guidance) GUIDANCE="$2"; shift 2 ;; --arms) read -r -a ARMS <<<"$2"; shift 2 ;; --work) WORK="$2"; shift 2 ;; -h|--help) usage; exit 0 ;; @@ -85,7 +92,10 @@ agent_args=(--agent "$AGENT" --model "$MODEL") use_gateway=0 case "$AGENT" in claude-code) - agent_args+=(--ak "version=$CLAUDE_CODE_VERSION" --ak "reasoning_effort=$EFFORT") + agent_args+=(--ak "version=$CLAUDE_CODE_VERSION" --ak "reasoning_effort=$EFFORT" + --ak "disallowed_tools=$CLAUDE_DISALLOWED_TOOLS" + # pier forces both on; --ae is merged last, so these win. + --ae ENABLE_BACKGROUND_TASKS=0 --ae FORCE_AUTO_BACKGROUND_TASKS=0) [ "$COST_LIMIT" != 0 ] && agent_args+=(--ak "max_budget_usd=$COST_LIMIT") case "$MODEL" in anthropic/*|claude-*) : "${ANTHROPIC_API_KEY:?ANTHROPIC_API_KEY is required for $MODEL}" ;; @@ -183,7 +193,7 @@ fi package_args=() [ -n "$PACKAGE" ] && package_args+=(--gitnexus-package "$PACKAGE") (cd "$EVAL_DIR" && python3 -m deepswe.prepare --deep-swe "$WORK/deep-swe" --out "$WORK/tasks" \ - --arms "${ARMS[@]}" "${package_args[@]}" "${select_args[@]}") + --arms "${ARMS[@]}" --guidance "$GUIDANCE" "${package_args[@]}" "${select_args[@]}") [ "$use_gateway" = 1 ] && start_gateway diff --git a/eval/tests/test_deepswe.py b/eval/tests/test_deepswe.py index f9d10ad8f..82bfb56c7 100644 --- a/eval/tests/test_deepswe.py +++ b/eval/tests/test_deepswe.py @@ -111,7 +111,7 @@ def test_prepare_copies_baseline_unchanged_and_layers_gitnexus(tmp_path): assert "{" not in dockerfile.replace("${", "") # every template field was filled assert (derived / "environment" / prepare.PACKAGE_FILE).read_bytes() == b"tarball" instruction = (derived / "instruction.md").read_text() - assert instruction.startswith("Implement the thing.\n") and instruction.endswith(prepare.GITNEXUS_NOTE) + assert instruction.startswith("Implement the thing.\n") and instruction.endswith(prepare.GUIDANCE["product"]) assert (derived / "tests" / "test.sh").read_bytes() == (tasks / "alpha-task" / "tests" / "test.sh").read_bytes() @@ -127,10 +127,24 @@ def test_gitnexus_arm_requires_a_package_and_a_prebuilt_image(tmp_path): "--gitnexus-package", str(package)]) -def test_gitnexus_note_names_only_real_cli_commands(): - commands = set(re.findall(r"`gitnexus ([a-z-]+)", prepare.GITNEXUS_NOTE)) - {""} +@pytest.mark.parametrize("guidance", sorted(prepare.GUIDANCE)) +def test_gitnexus_guidance_names_only_real_cli_commands(guidance): + note = prepare.GUIDANCE[guidance] + commands = set(re.findall(r"`gitnexus ([a-z-]+)", note)) - {""} assert commands == {"query", "context", "impact", "detect-changes"} - assert len(prepare.GITNEXUS_NOTE.encode()) < 700 + assert len(note.encode()) < 1000 + + +def test_prepare_records_the_guidance_variant(tmp_path): + _task(tmp_path / "deep-swe" / "tasks", "delta-task") + package = tmp_path / "p.tgz" + package.write_bytes(b"x") + prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(tmp_path / "o"), "--arms", "gitnexus", + "--gitnexus-package", str(package), "--guidance", "optional"]) + assert json.loads((tmp_path / "o" / "manifest.json").read_text())["guidance"] == "optional" + assert (tmp_path / "o" / "gitnexus" / "delta-task" / "instruction.md").read_text().endswith( + prepare.GUIDANCE["optional"] + ) def _trial(job: Path, name: str, task: str, *, reward=1, cost=1.0, steps=10, exception=None, commands=()): @@ -158,20 +172,24 @@ def test_report_pairs_on_task_id_and_compares_matched_pairs_only(tmp_path): commands=["gitnexus query parse && gitnexus impact Foo", "ls gitnexus/ && cat gitnexus.json"]) _trial(gitnexus, "b__1", "beta-gitnexus", cost=9.0, reward=-1) # grader crash sentinel # gamma never ran in the gitnexus arm (interrupted sweep) + _trial(base, "d__1", "delta", cost=3.0, reward=0, exception={"exception_type": "NonZeroAgentExitCodeError"}) + _trial(gitnexus, "d__1", "delta-gitnexus", cost=4.0, reward=1) (base / "config.json").write_text("{}") # job-level files are not trials result = report.build_report({"baseline": base, "gitnexus": gitnexus}) - assert result["pairs"] == {"graded_in_both": 1, "expected": 3} - assert result["changes"]["cost_usd"] == {"percent": 100.0, "pairs": 1, "expected_pairs": 3} - assert result["paired_resolved"] == {"baseline": 1, "gitnexus": 1} - assert result["arms"]["baseline"]["resolved"] == 2 and result["arms"]["baseline"]["graded"] == 3 + # The capped baseline trial (agent error, reward 0) is graded and paired, not dropped. + assert result["pairs"] == {"graded_in_both": 2, "expected": 4} + assert result["changes"]["cost_usd"] == {"percent": 50.0, "pairs": 2, "expected_pairs": 4} + assert result["paired_resolved"] == {"baseline": 1, "gitnexus": 2} + assert result["arms"]["baseline"]["resolved"] == 2 and result["arms"]["baseline"]["graded"] == 4 assert result["arms"]["gitnexus"]["gitnexus_commands"] == {"query": 1, "impact": 1} assert result["arms"]["gitnexus"]["trials_using_gitnexus"] == 1 - assert any("gitnexus: 1 of 2 trial(s) ungraded" in problem for problem in result["problems"]) - assert any("1 of 3 task/attempt pair(s) graded in both arms" in problem for problem in result["problems"]) + assert any("gitnexus: 1 of 3 trial(s) ungraded" in problem for problem in result["problems"]) + assert any("2 of 4 task/attempt pair(s) graded in both arms" in problem for problem in result["problems"]) + assert any("baseline: 1 graded trial(s) ended with an agent error" in problem for problem in result["problems"]) markdown = report.render_markdown(result) - assert "gitnexus vs baseline: cost_usd: +100.0% (1/3 pairs, partial)" in markdown + assert "gitnexus vs baseline: cost_usd: +50.0% (2/4 pairs, partial)" in markdown assert result["per_task"]["gamma#0"]["gitnexus"] is None