mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-11-16 09:22:00 +00:00
fix(eval): bound DeepSWE Claude Code trials and grade capped attempts
A local smoke trial showed pier's claude-code forcing background tasks: the baseline agent fanned out into 237 background agents. Its main session hit the $3 cap while the background work kept going ($40.82 by Claude Code's own accounting). run.sh now gives Claude Code only Read/Edit/Write/Bash, as the release evaluation does, and overrides pier's background-task env. The gitnexus arm's agent ignored an "optional" note, so the default note is now the CLI form of GitNexus's shipped agent rules (graph first, impact before edits, detect-changes before commit). `--guidance optional` keeps the plain listing for an ablation. report.py grades a trial on its verifier reward even when the agent hit the cost cap or exited nonzero, and lists those as agent errors. Dropping them would hide expensive failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
7b1d9a3ee2
commit
5dffaccc2e
5 changed files with 96 additions and 24 deletions
|
|
@ -7,9 +7,21 @@
|
|||
| `baseline` | The DeepSWE task, unchanged |
|
||||
| `gitnexus` | The same task, plus the `gitnexus` CLI and an index of `/app` built into the image, and a short note on the CLI appended to the instruction |
|
||||
|
||||
The default agent is `mini-swe-agent`, which the leaderboard uses. It is bash-only, so GitNexus is reached through the real CLI (`gitnexus query`, `context`, `impact`, `detect-changes`) rather than MCP. The arms differ only in that layer and that note.
|
||||
The default agent is Claude Code (pier's `claude-code`, pinned to the release evaluation's CLI). GitNexus is reached through the real CLI (`gitnexus query`, `context`, `impact`, `detect-changes`) rather than MCP. The arms differ only in that layer and that note.
|
||||
|
||||
The workflow_bench `baseline_cli` arm points the same way: on the small release tasks, a shell command cost far less than the MCP tool catalog.
|
||||
- **Toolset.** Claude Code gets only `Read`, `Edit`, `Write` and `Bash`, matching the release evaluation's single-agent toolset. Subagents, background tasks, skills and web tools are disallowed, and pier's forced background-task env is switched off. Without this, one smoke trial ran 237 background agents past its budget cap.
|
||||
- **Guidance.** The note (`--guidance product`, the default) is the CLI form of the rules GitNexus ships in AGENTS.md: graph first, `impact` before editing a symbol, `detect-changes` before committing. In the release evaluation, arms load those directives the same way. `--guidance optional` only lists the commands; in a smoke trial the agent then never used them.
|
||||
|
||||
The workflow_bench `baseline_cli` arm points the same way: on the small release tasks, a shell command cost far less than the MCP tool catalog. `--agent mini-swe-agent`, the leaderboard's agent, also works.
|
||||
|
||||
### Model access
|
||||
|
||||
- **Non-Claude models** (default `gpt-5.6-sol`, `OPENAI_API_KEY`): `run.sh` starts a LiteLLM gateway container. It is routed like `workflow_bench.model_gateway`, with Claude Code's Anthropic Messages served from OpenAI's Responses API, a random master key, and the LiteLLM version from `eval/uv.lock`.
|
||||
- It listens on host port 80, because pier's egress proxy only allows ports 80 and 443. Keep that port closed to the outside; `launch.sh`'s security group only opens SSH.
|
||||
- Trials reach it as `host.docker.internal` on Docker Desktop, or as the EC2 private DNS name.
|
||||
- **Claude models** (`--model anthropic/<model>`): use `ANTHROPIC_API_KEY` directly.
|
||||
|
||||
Costs come from Claude Code's own per-session accounting (`total_cost_usd`). Through the gateway that is Claude Code's price table applied to a non-Claude model, so it is consistent across arms but is not the provider's invoice.
|
||||
|
||||
## How the GitNexus arm is built
|
||||
|
||||
|
|
@ -57,7 +69,7 @@ The main options are listed below; `run.sh --help` shows them all.
|
|||
| Option | Default | Notes |
|
||||
| --- | --- | --- |
|
||||
| `--n-tasks`, `--seed`, `--task` | 10, 0 | `--task` (repeatable) overrides the sample; `--n-tasks 0` runs all 113 |
|
||||
| `--agent`, `--model`, `--effort` | `mini-swe-agent`, `openai/gpt-5.6-sol`, `low` | `--agent claude-code` also works with `ANTHROPIC_API_KEY` |
|
||||
| `--agent`, `--model`, `--effort` | `claude-code`, `gpt-5.6-sol`, `low` | `--model anthropic/<model>` skips the gateway. `--agent mini-swe-agent` takes litellm names such as `openai/gpt-5.6-sol`. |
|
||||
| `--cost-limit` | `5` | Per-trial agent cost cap in USD, the same for both arms. A capped trial ends unsolved. |
|
||||
| `--concurrency` | `2` | Trials per arm at once. Each needs 2 CPUs and 8 GB. |
|
||||
| `--package` | build from checkout | Pass a prebuilt `npm pack` tarball to skip the build |
|
||||
|
|
@ -68,6 +80,7 @@ The main options are listed below; `run.sh --help` shows them all.
|
|||
|
||||
- Deltas use only pairs graded in both arms with the metric measured in both.
|
||||
- Ungraded trials, interrupted sweeps and missing cost are listed under **Problems**. A grader crash writes reward `-1`, which counts as ungraded.
|
||||
- A trial whose agent hit `--cost-limit` or exited nonzero still had its patch graded. It counts as an unresolved attempt, listed as an agent error, not dropped.
|
||||
- `GitNexus usage` counts `gitnexus <command>` calls at command position in the ATIF trajectories.
|
||||
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -26,7 +26,24 @@ NODE_VERSION = "22.18.0" # gitnexus/package.json engines: ^22.18.0
|
|||
PACKAGE_FILE = "gitnexus.tgz"
|
||||
MIN_BUILD_TIMEOUT_SEC = 3600.0 # the index is built inside the image build
|
||||
|
||||
GITNEXUS_NOTE = """
|
||||
# Appended to the gitnexus arm's instruction. "product" is the CLI form of the
|
||||
# rules GitNexus ships in AGENTS.md (graph first, impact before edits,
|
||||
# detect-changes before committing), as the release evaluation's arms load.
|
||||
# "optional" only lists the commands, for an ablation.
|
||||
GUIDANCE = {
|
||||
"product": """
|
||||
|
||||
## GitNexus
|
||||
|
||||
A GitNexus code graph of `/app` (indexed at the starting commit) is installed; the `gitnexus` command works offline. Follow GitNexus's agent rules:
|
||||
|
||||
- Explore with `gitnexus query "<concept>"` for concepts and execution flows, and `gitnexus context <symbol>` for a named symbol's callers, callees and flows. Graph first; use text search for literals or when the graph has no answer.
|
||||
- Before editing a function, class or method, run `gitnexus impact <symbol> --direction upstream` and check its callers and risk. An empty result is not proof that nothing depends on it.
|
||||
- Before committing, run `gitnexus detect-changes` to see which symbols and flows your edits affect.
|
||||
|
||||
Run `gitnexus <command> --help` for options.
|
||||
""",
|
||||
"optional": """
|
||||
|
||||
## GitNexus
|
||||
|
||||
|
|
@ -38,7 +55,8 @@ A GitNexus code graph of `/app` (indexed at the starting commit) is installed. T
|
|||
- `gitnexus detect-changes`: symbols and flows touched by your uncommitted edits
|
||||
|
||||
Run `gitnexus <command> --help` for options. Using it is optional.
|
||||
"""
|
||||
""",
|
||||
}
|
||||
|
||||
DOCKERFILE = """FROM {image}
|
||||
|
||||
|
|
@ -125,7 +143,7 @@ def gitnexus_task_toml(task_toml: str) -> str:
|
|||
return rewritten
|
||||
|
||||
|
||||
def prepare_task(source: Path, dest: Path, *, arm: str, package: Path | None) -> None:
|
||||
def prepare_task(source: Path, dest: Path, *, arm: str, package: Path | None, guidance: str = "product") -> None:
|
||||
if dest.exists():
|
||||
shutil.rmtree(dest)
|
||||
shutil.copytree(source, dest)
|
||||
|
|
@ -143,20 +161,22 @@ def prepare_task(source: Path, dest: Path, *, arm: str, package: Path | None) ->
|
|||
shutil.copyfile(package, environment / PACKAGE_FILE)
|
||||
(dest / "task.toml").write_text(gitnexus_task_toml(task_toml), encoding="utf-8")
|
||||
instruction = dest / "instruction.md"
|
||||
instruction.write_text(instruction.read_text(encoding="utf-8").rstrip() + "\n" + GITNEXUS_NOTE, encoding="utf-8")
|
||||
instruction.write_text(instruction.read_text(encoding="utf-8").rstrip() + "\n" + GUIDANCE[guidance], encoding="utf-8")
|
||||
|
||||
|
||||
def prepare(
|
||||
*, tasks_root: Path, out: Path, arms: list[str], selected: list[str], package: Path | None
|
||||
*, tasks_root: Path, out: Path, arms: list[str], selected: list[str], package: Path | None,
|
||||
guidance: str = "product",
|
||||
) -> dict[str, Any]:
|
||||
for arm in arms:
|
||||
for task in selected:
|
||||
prepare_task(tasks_root / task, out / arm / task, arm=arm, package=package)
|
||||
prepare_task(tasks_root / task, out / arm / task, arm=arm, package=package, guidance=guidance)
|
||||
manifest = {
|
||||
"schema": "gitnexus.deepswe-prepare/v1",
|
||||
"arms": arms,
|
||||
"tasks": selected,
|
||||
"gitnexus_task_suffix": GITNEXUS_TASK_SUFFIX,
|
||||
"guidance": guidance,
|
||||
"node_version": NODE_VERSION,
|
||||
"gitnexus_package": package.name if package else None,
|
||||
}
|
||||
|
|
@ -173,6 +193,8 @@ def main(argv: list[str] | None = None) -> int:
|
|||
parser.add_argument("--task", action="append", dest="include", help="task id; repeatable")
|
||||
parser.add_argument("--n-tasks", type=int, help="seeded random subset size (default: all tasks)")
|
||||
parser.add_argument("--sample-seed", type=int, default=0)
|
||||
parser.add_argument("--guidance", choices=sorted(GUIDANCE), default="product",
|
||||
help="note appended to the gitnexus arm's instruction (default: product)")
|
||||
args = parser.parse_args(argv)
|
||||
tasks_root = args.deep_swe / "tasks"
|
||||
available = task_ids(tasks_root)
|
||||
|
|
@ -182,7 +204,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
parser.error("--gitnexus-package must name an existing `npm pack` tarball for the gitnexus arm")
|
||||
selected = select_tasks(available, include=args.include, n_tasks=args.n_tasks, seed=args.sample_seed)
|
||||
manifest = prepare(tasks_root=tasks_root, out=args.out, arms=args.arms, selected=selected,
|
||||
package=args.gitnexus_package)
|
||||
package=args.gitnexus_package, guidance=args.guidance)
|
||||
print(f"prepared {len(selected)} task(s) x {len(manifest['arms'])} arm(s) under {args.out}")
|
||||
return 0
|
||||
|
||||
|
|
|
|||
|
|
@ -68,7 +68,10 @@ def load_trial(trial_dir: Path) -> dict[str, Any] | None:
|
|||
agent = result.get("agent_result") or {}
|
||||
rewards = (result.get("verifier_result") or {}).get("rewards") or {}
|
||||
reward = rewards.get("reward")
|
||||
graded = result.get("exception_info") is None and reward in (0, 1)
|
||||
# Graded means the verifier produced a 0/1 reward. A trial whose agent hit the
|
||||
# cost cap or exited nonzero is still graded on its patch; dropping it would
|
||||
# hide expensive failures. -1 (the grader's crash sentinel) or no reward is not.
|
||||
graded = reward in (0, 1) and not isinstance(reward, bool)
|
||||
row = {
|
||||
"task": task_id(str(result.get("task_name", trial_dir.name))),
|
||||
"trial": result.get("trial_name") or trial_dir.name,
|
||||
|
|
@ -115,6 +118,12 @@ def build_report(jobs: dict[str, Path]) -> dict[str, Any]:
|
|||
if ungraded:
|
||||
kinds = Counter(row["error"] or "no reward" for row in ungraded)
|
||||
report["problems"].append(f"{arm}: {len(ungraded)} of {len(trials)} trial(s) ungraded ({dict(kinds)})")
|
||||
agent_errors = Counter(row["error"] for row in graded if row["error"])
|
||||
if agent_errors:
|
||||
report["problems"].append(
|
||||
f"{arm}: {sum(agent_errors.values())} graded trial(s) ended with an agent error "
|
||||
+ f"({dict(agent_errors)}), e.g. the cost cap; their patches were still graded"
|
||||
)
|
||||
summary: dict[str, Any] = {"trials": len(trials), "graded": len(graded),
|
||||
"resolved": sum(row["resolved"] for row in graded), "incomplete_metrics": []}
|
||||
for metric in METRICS:
|
||||
|
|
|
|||
|
|
@ -18,6 +18,10 @@ LITELLM_VERSION=1.100.4 # eval/uv.lock
|
|||
MINI_SWE_AGENT_VERSION=2.2.7 # eval/uv.lock
|
||||
GATEWAY_CONTAINER=deepswe-litellm-gateway
|
||||
GATEWAY_PORT=80 # pier's egress proxy only allows ports 80 and 443
|
||||
# Everything CLI 2.1.214 offers beyond Read/Edit/Write/Bash, matching the release
|
||||
# evaluation's single-agent toolset. Subagents and background tasks otherwise fan
|
||||
# out unbounded: one smoke trial ran 237 background agents past its budget cap.
|
||||
CLAUDE_DISALLOWED_TOOLS="EnterPlanMode Task Agent TaskCreate TaskGet TaskList TaskOutput TaskStop TaskUpdate SendMessage Skill Workflow CronCreate CronDelete CronList ScheduleWakeup EnterWorktree ExitWorktree NotebookEdit ReportFindings WebFetch WebSearch"
|
||||
|
||||
AGENT=claude-code
|
||||
MODEL=gpt-5.6-sol
|
||||
|
|
@ -29,6 +33,7 @@ COST_LIMIT=5
|
|||
WORK="$HOME/deepswe-work"
|
||||
PACKAGE=""
|
||||
GATEWAY_HOST=""
|
||||
GUIDANCE=product
|
||||
TASKS=()
|
||||
ARMS=(baseline gitnexus)
|
||||
|
||||
|
|
@ -47,6 +52,7 @@ Usage: $0 [options]
|
|||
--package TGZ prebuilt \`npm pack\` tarball of gitnexus/ (default: build from this checkout)
|
||||
--gateway-host H hostname the trial egress proxy uses to reach this host's gateway
|
||||
(default: host.docker.internal if it resolves, else the EC2 private DNS name)
|
||||
--guidance G note for the gitnexus arm: product (GitNexus's agent rules, default) or optional
|
||||
--arms "A B" arms to run (default: "${ARMS[*]}")
|
||||
--work DIR working directory (default $WORK)
|
||||
EOF
|
||||
|
|
@ -64,6 +70,7 @@ while [ $# -gt 0 ]; do
|
|||
--cost-limit) COST_LIMIT="$2"; shift 2 ;;
|
||||
--package) PACKAGE="$2"; shift 2 ;;
|
||||
--gateway-host) GATEWAY_HOST="$2"; shift 2 ;;
|
||||
--guidance) GUIDANCE="$2"; shift 2 ;;
|
||||
--arms) read -r -a ARMS <<<"$2"; shift 2 ;;
|
||||
--work) WORK="$2"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
|
|
@ -85,7 +92,10 @@ agent_args=(--agent "$AGENT" --model "$MODEL")
|
|||
use_gateway=0
|
||||
case "$AGENT" in
|
||||
claude-code)
|
||||
agent_args+=(--ak "version=$CLAUDE_CODE_VERSION" --ak "reasoning_effort=$EFFORT")
|
||||
agent_args+=(--ak "version=$CLAUDE_CODE_VERSION" --ak "reasoning_effort=$EFFORT"
|
||||
--ak "disallowed_tools=$CLAUDE_DISALLOWED_TOOLS"
|
||||
# pier forces both on; --ae is merged last, so these win.
|
||||
--ae ENABLE_BACKGROUND_TASKS=0 --ae FORCE_AUTO_BACKGROUND_TASKS=0)
|
||||
[ "$COST_LIMIT" != 0 ] && agent_args+=(--ak "max_budget_usd=$COST_LIMIT")
|
||||
case "$MODEL" in
|
||||
anthropic/*|claude-*) : "${ANTHROPIC_API_KEY:?ANTHROPIC_API_KEY is required for $MODEL}" ;;
|
||||
|
|
@ -183,7 +193,7 @@ fi
|
|||
package_args=()
|
||||
[ -n "$PACKAGE" ] && package_args+=(--gitnexus-package "$PACKAGE")
|
||||
(cd "$EVAL_DIR" && python3 -m deepswe.prepare --deep-swe "$WORK/deep-swe" --out "$WORK/tasks" \
|
||||
--arms "${ARMS[@]}" "${package_args[@]}" "${select_args[@]}")
|
||||
--arms "${ARMS[@]}" --guidance "$GUIDANCE" "${package_args[@]}" "${select_args[@]}")
|
||||
|
||||
[ "$use_gateway" = 1 ] && start_gateway
|
||||
|
||||
|
|
|
|||
|
|
@ -111,7 +111,7 @@ def test_prepare_copies_baseline_unchanged_and_layers_gitnexus(tmp_path):
|
|||
assert "{" not in dockerfile.replace("${", "") # every template field was filled
|
||||
assert (derived / "environment" / prepare.PACKAGE_FILE).read_bytes() == b"tarball"
|
||||
instruction = (derived / "instruction.md").read_text()
|
||||
assert instruction.startswith("Implement the thing.\n") and instruction.endswith(prepare.GITNEXUS_NOTE)
|
||||
assert instruction.startswith("Implement the thing.\n") and instruction.endswith(prepare.GUIDANCE["product"])
|
||||
assert (derived / "tests" / "test.sh").read_bytes() == (tasks / "alpha-task" / "tests" / "test.sh").read_bytes()
|
||||
|
||||
|
||||
|
|
@ -127,10 +127,24 @@ def test_gitnexus_arm_requires_a_package_and_a_prebuilt_image(tmp_path):
|
|||
"--gitnexus-package", str(package)])
|
||||
|
||||
|
||||
def test_gitnexus_note_names_only_real_cli_commands():
|
||||
commands = set(re.findall(r"`gitnexus ([a-z-]+)", prepare.GITNEXUS_NOTE)) - {"<command>"}
|
||||
@pytest.mark.parametrize("guidance", sorted(prepare.GUIDANCE))
|
||||
def test_gitnexus_guidance_names_only_real_cli_commands(guidance):
|
||||
note = prepare.GUIDANCE[guidance]
|
||||
commands = set(re.findall(r"`gitnexus ([a-z-]+)", note)) - {"<command>"}
|
||||
assert commands == {"query", "context", "impact", "detect-changes"}
|
||||
assert len(prepare.GITNEXUS_NOTE.encode()) < 700
|
||||
assert len(note.encode()) < 1000
|
||||
|
||||
|
||||
def test_prepare_records_the_guidance_variant(tmp_path):
|
||||
_task(tmp_path / "deep-swe" / "tasks", "delta-task")
|
||||
package = tmp_path / "p.tgz"
|
||||
package.write_bytes(b"x")
|
||||
prepare.main(["--deep-swe", str(tmp_path / "deep-swe"), "--out", str(tmp_path / "o"), "--arms", "gitnexus",
|
||||
"--gitnexus-package", str(package), "--guidance", "optional"])
|
||||
assert json.loads((tmp_path / "o" / "manifest.json").read_text())["guidance"] == "optional"
|
||||
assert (tmp_path / "o" / "gitnexus" / "delta-task" / "instruction.md").read_text().endswith(
|
||||
prepare.GUIDANCE["optional"]
|
||||
)
|
||||
|
||||
|
||||
def _trial(job: Path, name: str, task: str, *, reward=1, cost=1.0, steps=10, exception=None, commands=()):
|
||||
|
|
@ -158,20 +172,24 @@ def test_report_pairs_on_task_id_and_compares_matched_pairs_only(tmp_path):
|
|||
commands=["gitnexus query parse && gitnexus impact Foo", "ls gitnexus/ && cat gitnexus.json"])
|
||||
_trial(gitnexus, "b__1", "beta-gitnexus", cost=9.0, reward=-1) # grader crash sentinel
|
||||
# gamma never ran in the gitnexus arm (interrupted sweep)
|
||||
_trial(base, "d__1", "delta", cost=3.0, reward=0, exception={"exception_type": "NonZeroAgentExitCodeError"})
|
||||
_trial(gitnexus, "d__1", "delta-gitnexus", cost=4.0, reward=1)
|
||||
(base / "config.json").write_text("{}") # job-level files are not trials
|
||||
|
||||
result = report.build_report({"baseline": base, "gitnexus": gitnexus})
|
||||
|
||||
assert result["pairs"] == {"graded_in_both": 1, "expected": 3}
|
||||
assert result["changes"]["cost_usd"] == {"percent": 100.0, "pairs": 1, "expected_pairs": 3}
|
||||
assert result["paired_resolved"] == {"baseline": 1, "gitnexus": 1}
|
||||
assert result["arms"]["baseline"]["resolved"] == 2 and result["arms"]["baseline"]["graded"] == 3
|
||||
# The capped baseline trial (agent error, reward 0) is graded and paired, not dropped.
|
||||
assert result["pairs"] == {"graded_in_both": 2, "expected": 4}
|
||||
assert result["changes"]["cost_usd"] == {"percent": 50.0, "pairs": 2, "expected_pairs": 4}
|
||||
assert result["paired_resolved"] == {"baseline": 1, "gitnexus": 2}
|
||||
assert result["arms"]["baseline"]["resolved"] == 2 and result["arms"]["baseline"]["graded"] == 4
|
||||
assert result["arms"]["gitnexus"]["gitnexus_commands"] == {"query": 1, "impact": 1}
|
||||
assert result["arms"]["gitnexus"]["trials_using_gitnexus"] == 1
|
||||
assert any("gitnexus: 1 of 2 trial(s) ungraded" in problem for problem in result["problems"])
|
||||
assert any("1 of 3 task/attempt pair(s) graded in both arms" in problem for problem in result["problems"])
|
||||
assert any("gitnexus: 1 of 3 trial(s) ungraded" in problem for problem in result["problems"])
|
||||
assert any("2 of 4 task/attempt pair(s) graded in both arms" in problem for problem in result["problems"])
|
||||
assert any("baseline: 1 graded trial(s) ended with an agent error" in problem for problem in result["problems"])
|
||||
markdown = report.render_markdown(result)
|
||||
assert "gitnexus vs baseline: cost_usd: +100.0% (1/3 pairs, partial)" in markdown
|
||||
assert "gitnexus vs baseline: cost_usd: +50.0% (2/4 pairs, partial)" in markdown
|
||||
assert result["per_task"]["gamma#0"]["gitnexus"] is None
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue