mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-24 00:51:53 +00:00
13095bc4bc
* fix(eval): ignore Claude Code bootstrap noise nested below the workspace root The planning-phase boundary check excluded Claude Code's own sandbox-bootstrap paths only at the workspace root: workspace_snapshot tested relative.parts[0] against WORKSPACE_SNAPSHOT_BOOTSTRAP_NOISE. But Claude Code bootstraps into whatever directory it is running in, and the benchmark's task prompts cd into gitnexus/, so the same noise landed one level down as gitnexus/.claude/.cc-writes -- whose parts[0] is "gitnexus", so it was never excluded. In skill-evolution run 29861768554 that accounted for 13 of 18 sessions, each failing with error_kind plan-evidence-invalid and the identical error_detail "phase changed unauthorized workspace path(s): gitnexus/.claude/.cc-writes". The same code path also guards the review phase (runner.py:499), so review arms hit it as review-evidence-invalid. Widening the whole set to match at any depth would be wrong: it also contains package.json, package-lock.json, node_modules and the .env family, and both gitnexus/package.json and gitnexus/.claude/settings.local.json are real tracked files whose edits must still be caught. So the root-anchored rule is unchanged, and a second narrow rule matches only the entries Claude Code itself creates inside a .claude directory (.cc-writes, agents, commands) at any depth -- never .claude itself. The predicate moves into _is_bootstrap_noise so it is directly testable. It is still evaluated before pending.append, so an excluded directory is never descended into. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(eval): mount the node install prefix so npx and npm resolve in the sandbox _runtime_mount_args bound only the `node` binary itself to SANDBOX_NODE. npm and npx are not standalone binaries -- they are symlinks into ../lib/node_modules/npm/bin/*-cli.js -- so the install prefix carrying both bin/ and lib/node_modules has to be mounted for them to resolve at all. On GitHub-hosted images node lives in /usr/local/bin, whose prefix (/usr/local) is already inside the wholesale /usr read-only bind, so npm and npx came along for free and the gap stayed invisible. A self-hosted runner's actions/setup-node installs into its own tool cache, outside /usr, so only the single node file was bound. Every task's verify command is "cd gitnexus && npx tsc --noEmit && npx vitest run <test>", so in skill-evolution run 29861768554 all 18 of 18 result records carried the identical verify_output "/bin/sh: 1: npx: not found" -- no run could resolve regardless of model output. It reached the model too: the session transcripts show 12 "npm: not found" failures, with gitnexus/scripts/build.js dying on `npm ci` with status 127. Binds Path(node_bin).resolve().parent.parent read-only at /opt/claude/nodejs, a fresh target outside the already-read-only trees (same constraint that put SANDBOX_NODE under /opt/claude), and adds its bin/ to SANDBOX_PATH. The bind is skipped when the prefix already sits inside /usr, /bin, /lib or /lib64, so the already-covered case does not widen the mount surface redundantly. SANDBOX_NODE is deliberately unchanged -- sanitized_graph.py and runner_sessions.py invoke it directly. SANDBOX_PATH is now derived from SANDBOX_NODE_PREFIX so the two cannot drift, and the minimal-mounts probe asserts against the constant instead of a duplicated literal. The real-Bubblewrap npx canary lives in test_proposer_sandbox.py deliberately: test_workflow_bench.py pins the set of files carrying the canary marker, and it runs in the eval-containment-linux job, where actions/setup-node also installs into the tool cache -- so the canary exercises the real failure shape. Combines plan steps 3-5 into one commit: the mount, SANDBOX_PATH and the pinned probe assertion are one behavioural change, and splitting them would leave a commit whose asserted PATH disagrees with the mounted reality. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(eval): only bind a verified node prefix, and stop excluding .claude/agents Addresses two findings from the branch review of the two preceding commits. 1. The prefix was derived as Path(node_bin).resolve().parent.parent with no check that the layout is really <prefix>/bin/node. Probed: /opt/bin/node bound ALL of /opt (every tool cache on a hosted runner), /mnt/tools/node bound /mnt, and a bare <dir>/node bound <dir>'s parent. That last shape is not hypothetical -- the pre-existing real-Bubblewrap node canary builds exactly it (tmp_path/toolcache/node), so eval-containment-linux would have silently read-only mounted the whole pytest tmp_path inside a containment test, passing while doing it. This function exists to keep the sandbox surface minimal, so an unrecognized layout now binds nothing extra and simply leaves npx unavailable, exactly as before the mount was added. 2. CLAUDE_BOOTSTRAP_ENTRIES also excluded "agents" and "commands" on the theory that they might appear nested too; only .cc-writes ever was observed. Every excluded name is a blind spot: once a .claude directory exists (gitnexus/.claude/settings.local.json is tracked) anything written under an excluded entry is invisible to the phase-boundary check, and Claude Code loads .claude/agents relative to its cwd -- which these tasks point at gitnexus/. Probed: a planning phase could plant gitnexus/.claude/agents/planted.md with the check reporting nothing, then the work phase reads it. Narrowed to .cc-writes alone; extend the set from an observed failure, never pre-emptively. Re-probed after both fixes: the over-broad mounts are gone while a genuine tool-cache prefix carrying npm still binds; planted agents/commands content is caught again; gitnexus/.claude/.cc-writes (the real run-29861768554 failure) stays ignored; and edits to gitnexus/.claude/settings.local.json are still caught. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(eval): gate the node-prefix bind on a working npx, not on an npm directory The guard tested (prefix)/lib/node_modules/npm as a proxy for "this prefix supplies npx". Test the property actually required instead: a working npx sitting beside node in a real bin/ directory. .exists() follows the symlink, so a dangling npx correctly fails the check -- it would not survive the mount either. The "bin" name requirement stays, because it is what keeps the parent.parent derivation honest; an npx sitting directly beside node in a flat directory would make that derivation name the wrong prefix. This matters because the guard can silently disable the fix it guards: if a runner's layout failed the proxy check, the prefix would not be bound and npx would still be missing, reproducing the original failure with no signal. Testing npx directly means the guard can only pass when the bind will actually achieve its purpose. Validated against a real extracted Node distribution (the official nodejs.org tarball layout that actions/setup-node unpacks into the tool cache) staged at a tool-cache-shaped path: bin/node is a real file, bin/npx resolves to ../lib/node_modules/npm/bin/npx-cli.js, and the prefix binds while SANDBOX_NODE is preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar@icloud.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_ce_plugin_runtime.py | ||
| test_errors.py | ||
| test_evolve.py | ||
| test_mcp_bridge.py | ||
| test_oracle_assets.py | ||
| test_parse_run_id.py | ||
| test_process_control.py | ||
| test_promotion_apply.py | ||
| test_property_based.py | ||
| test_proposer_sandbox.py | ||
| test_runner_hardening.py | ||
| test_sanitized_graph.py | ||
| test_task_assets.py | ||
| test_tool_scripts.py | ||
| test_workflow_bench.py | ||
| test_workflow_bench_evolution.py | ||
| test_workflow_bench_sessions.py | ||