GitNexus/eval/tests
Abhigyan Patwari a84e029066
fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630)
* fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts

Every task verify command and every hidden oracle ends in `npx vitest run
<test>`, and both run through run_verify with read_only_workspace=True. Vite
transpiles a TypeScript config by writing
<node_modules>/.vite-temp/<config>.timestamp-*.mjs before it loads anything, so
against a read-only dependency mount vitest dies with EROFS before a single
test executes:

  EROFS ... /workspace/gitnexus/node_modules/.vite-temp/vitest.config.ts.timestamp-*.mjs

This is pre-existing and was masked: until #2627 the verify command died at
`npx: not found`, short-circuiting the `&&` chain before vitest ran. Confirmed
by reproducing it at that merge base with npx bypassed entirely
(`./node_modules/.bin/vitest`), so it is independent of the node-prefix mount.
Because it blocks the oracle as well as the authored-test verify, `resolved`
stays 0/N without this.

bwrap cannot create a mount point inside an already-read-only bind -- the same
constraint that put SANDBOX_NODE under /opt/claude -- so overlaying a tmpfs only
works if the directory already exists in the mounted bytes. It cannot be
mkdir'd into the dependency snapshot after capture either: the snapshot is
digest-bound and validate_dependency_binding fails closed on drift. So the empty
directory is captured during dependency capture, before the manifest and both
dependency digests are computed, making it part of the snapshot rather than an
untracked mutation of it. The sandbox then overlays a tmpfs on exactly that
path; everything else in the mount, and the whole workspace, stays read-only,
and the overlay never reaches the host clone the credited patch comes from.

Scoped to dependency mounts whose target basename is node_modules, so hidden
oracle and skill mounts stay wholly read-only with no writable island.

Note: this shifts sandbox_dependency_content_digest and
sandbox_dependency_manifest_digest, so promotion evidence recorded before this
change is no longer comparable. That is already true of any harness fix that
changes what the sandbox exposes.

Verified on the self-hosted runner through the real path -- TaskAssetCache
.prepare -> stage_task_assets -> prepare_sandbox -> run_verify with the actual
trivial-version-alias verify string: passed, 15/15 tests, no EROFS. Full eval
suite there with GITNEXUS_REQUIRE_BWRAP_CANARY=1: 337 passed, 4 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(eval): only overlay .vite-temp where the mount source actually carries it

The tmpfs overlay keyed purely on the mount target basename being
node_modules, which also matched the trusted GitNexus runtime mount at
/opt/gitnexus/node_modules. That mount's source is the built runtime and does
not carry a .vite-temp, and bwrap cannot create a mount point inside an
already-read-only bind, so the containment CI job failed:

  bwrap: Can't mkdir /opt/gitnexus/node_modules/.vite-temp: Read-only file system
  FAILED test_real_bubblewrap_runtime_mount_imports_cli_without_exposing_checkout

My runner probe only exercised the dependency-mount path, so it missed this.

Gate the overlay on the mount SOURCE actually containing the directory rather
than on the target name. task_assets.py captures .vite-temp only into
dependency-snapshot node_modules, so the overlay now fires exactly there and
never on the runtime mount -- and the gate is correct by construction, since a
tmpfs can only overlay a mount point that already exists in the bound bytes.

Adds a regression test for a node_modules mount whose source has no captured
.vite-temp (the runtime-mount shape) getting no overlay, and updates the
positive test to create the directory in its mount source.

Verified on the self-hosted runner: the exact failing test now passes, and the
full containment selection is 124 passed, 4 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:16:25 +01:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py feat(eval): run the skill-evolution loop online (#2571) 2026-07-20 05:37:07 +01:00
test_mcp_bridge.py fix: start MCP bridge correctly when using npx (#1114) 2026-04-27 18:19:02 +01:00
test_oracle_assets.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_promotion_apply.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_runner_hardening.py fix(eval): mount the node prefix for npx and catch nested Claude Code bootstrap noise (#2627) 2026-07-22 08:25:46 +01:00
test_sanitized_graph.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_task_assets.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py fix(eval): self-hosted skill-evolution runner + sandbox Python 3 trust fix (#2600) 2026-07-21 14:40:20 +01:00
test_workflow_bench_evolution.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_workflow_bench_sessions.py fix(eval): bind resolved node to a fresh sandbox path (corrects #2607) (#2609) 2026-07-21 16:09:45 +01:00