GitNexus/eval/tests
Gergo Magyar a522a4fbc2 fix(eval): stop failing the benchmark when main's version bumps
PINNED_GITNEXUS_VERSION did not select a runtime — the sandbox already
mounts gitnexus/dist built from the checkout by the workflow's own
`npm run build`, so the benchmark has always run main, not a release. The
constant only asserted that the checkout's package.json said "1.6.9" and
raised SandboxError otherwise.

That makes it a tripwire aimed at the wrong thing. main is 1.6.9 today,
so it passes; the next release bumps it and every skill-evolution run
hard-fails until someone edits this line — discovered on a Saturday, on a
lane that runs unattended once a week, after the box has already been
started and the proposer session paid for.

What the constant was guarding is still guarded, and better: the sandbox
canary now checks the version reported from inside the sandbox against
the checkout's own package.json, so it still proves the mounted runtime
is the one this checkout built, without a literal that has to be
maintained in lockstep with releases. The digest bindings in
promotion.json (sandbox_dependency_content_digest, graph and oracle
digests) remain what actually pins the substrate a candidate was measured
against.

The remaining check keeps package.json readable and versioned, so a
malformed runtime still fails closed.
2026-08-02 15:37:30 +00:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py fix(eval): stop letting a task neither arm can solve veto every promotion 2026-08-01 17:07:14 +00:00
test_mcp_bridge.py fix: start MCP bridge correctly when using npx (#1114) 2026-04-27 18:19:02 +01:00
test_oracle_assets.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py fix(eval): drain with read1 so progress surfaces before the child exits 2026-08-02 12:12:08 +00:00
test_promotion_apply.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_runner_hardening.py fix(eval): redact the token from the one failure line that now reaches CI 2026-08-01 20:11:00 +00:00
test_sanitized_graph.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_task_assets.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py fix(eval): self-hosted skill-evolution runner + sandbox Python 3 trust fix (#2600) 2026-07-21 14:40:20 +01:00
test_workflow_bench_evolution.py fix(ci): make the evolution lane survive its own deadlines and remember prior runs 2026-08-01 17:42:48 +00:00
test_workflow_bench_sessions.py fix(eval): stop failing the benchmark when main's version bumps 2026-08-02 15:37:30 +00:00