GitNexus/eval/tests
Gergo Magyar b30f530698 refactor(eval): make a benchmark cell a callable instead of loop-body scope
The sweep's innermost body — clone, sandbox, sessions, verify, oracle,
teardown — was ~260 lines of `main()`'s scope, reachable only by running
the whole sweep. Nothing tested it, and it could not run anywhere except
that loop.

`run_cell(ctx, run_idx, arm)` now owns one (run, arm) cell and returns
its row; `TaskCellContext` is a frozen dataclass holding the per-task
inputs a cell reads, so a cell depends on named fields rather than on
whatever `main()` happens to have in scope. The try/except/finally moves
verbatim, exception whitelist unchanged: a harness bug still escapes
rather than being recorded as an ordinary infra-error and averaged into
the evidence.

The task asset snapshot is now prepared once per task, next to the graph
snapshot, instead of lazily inside whichever cell reached it first.
`TaskAssetCache` is a plain dict (task_assets.py:222-226,340), so the
lazy build was a read-then-write race waiting for a caller that is not
strictly serial. One behavior delta: when that preparation fails, every
cell of the task now reports the same wrapped RuntimeError, where before
the first cell reported the original OSError/SandboxError/ValueError.

The loop keeps all the bookkeeping — progress counter, prints, the
results.jsonl append, per-arm accumulation, the outage streak. Behavior
is otherwise unchanged; this is the seam, not the concurrency.

Six new tests cover it, the first coverage this body has ever had: the
row's task/arm/run/digest bindings, each of the five expected failure
kinds still removing the clone, an unexpected KeyError escaping while
cleanup still runs, cleanup failure overriding the primary outcome, and a
failed per-task snapshot failing every cell closed.
2026-08-01 18:48:18 +00:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py fix(eval): stop letting a task neither arm can solve veto every promotion 2026-08-01 17:07:14 +00:00
test_mcp_bridge.py fix: start MCP bridge correctly when using npx (#1114) 2026-04-27 18:19:02 +01:00
test_oracle_assets.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py feat(eval): report evolution progress while the generation is still running 2026-08-01 17:01:03 +00:00
test_promotion_apply.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_runner_hardening.py refactor(eval): make a benchmark cell a callable instead of loop-body scope 2026-08-01 18:48:18 +00:00
test_sanitized_graph.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_task_assets.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py fix(eval): self-hosted skill-evolution runner + sandbox Python 3 trust fix (#2600) 2026-07-21 14:40:20 +01:00
test_workflow_bench_evolution.py fix(ci): make the evolution lane survive its own deadlines and remember prior runs 2026-08-01 17:42:48 +00:00
test_workflow_bench_sessions.py refactor(eval): fold the review cleanups into the evolution fixes 2026-08-01 17:18:43 +00:00