GitNexus/eval/tests
Gergő Magyar bba25b2103
fix(eval): bind resolved node to a fresh sandbox path (corrects #2607) (#2609)
* fix(eval): bind the resolved node to a fresh sandbox path, not one under /usr

#2607 bound the resolved `node` to /usr/local/bin/node, but that path lives
inside the /usr tree that _runtime_mount_args already read-only-binds
wholesale. The second real workflow_dispatch run on the self-hosted runner
(https://github.com/abhigyanpatwari/GitNexus/actions/runs/29840270554)
failed immediately in the bubblewrap preflight: "bwrap: Can't create file
at /usr/local/bin/node: Read-only file system" -- bwrap can't create a new
mount-point file inside a tree it already bound read-only when the real
path doesn't already exist there on the host, which is exactly the
self-hosted case this bind exists to fix.

Introduces SANDBOX_NODE (/opt/claude/node), a fresh path outside every
tree _runtime_mount_args binds, following the same pattern SANDBOX_CLAUDE
and SANDBOX_PYTHON3 already use. Updates the two real call sites
(sanitized_graph.py, runner_sessions.py) to use the constant instead of
the hardcoded literal, so the fix can't drift out of sync with itself
again, and re-exports it from runner.py alongside the other SANDBOX_*
names for the real-bwrap tests that reference it directly.

Adds a real-bwrap test (gated behind GITNEXUS_REQUIRE_BWRAP_CANARY, same
as the existing ones) that copies a real node binary to a path outside
every bound tree and actually launches bwrap against it -- an
argv-construction test alone can't catch a bwrap-level "Read-only file
system" error, only a real invocation can, and that's exactly the gap
that let #2607's version of this fix through review looking correct.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(eval): don't let the new real-bwrap test's node-mock break bwrap's own resolution

CI caught this immediately: the new test_real_bubblewrap_runs_node_from_outside_the_bound_trees
monkeypatched shutil.which to return None for anything but "node", but
prepare_sandbox's own bwrap/claude resolution (_resolve_executable) goes
through shutil.which too -- so the test broke bwrap discovery before the
sandbox it's supposed to exercise could even be built ("SandboxError:
required executable is unavailable: bwrap").

Delegate to the real shutil.which for every other name instead of
blanket-returning None.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-21 16:09:45 +01:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py feat(eval): run the skill-evolution loop online (#2571) 2026-07-20 05:37:07 +01:00
test_mcp_bridge.py fix: start MCP bridge correctly when using npx (#1114) 2026-04-27 18:19:02 +01:00
test_oracle_assets.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_promotion_apply.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(eval): bind resolved node to a fresh sandbox path (corrects #2607) (#2609) 2026-07-21 16:09:45 +01:00
test_runner_hardening.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_sanitized_graph.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_task_assets.py fix(eval): size the buffered-fallback budget for the real graph index 2026-07-20 14:42:50 +00:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py fix(eval): self-hosted skill-evolution runner + sandbox Python 3 trust fix (#2600) 2026-07-21 14:40:20 +01:00
test_workflow_bench_evolution.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_workflow_bench_sessions.py fix(eval): bind resolved node to a fresh sandbox path (corrects #2607) (#2609) 2026-07-21 16:09:45 +01:00