GitNexus/eval/tests
Gergo Magyar 63d29c384f feat(eval): run a task's benchmark cells in waves instead of one at a time
18 cells at ~48 min each, strictly serial, is 97.4% of a generation's
14.7h. The cells are independent — the wall clock was a scheduling
choice, not a measurement requirement.

`--workers` (default 1) runs the cells of one task concurrently; tasks
stay sequential, so the sanitized graph snapshot each task already builds
before its cells stays a single-writer affair. Threads, not processes:
cells are subprocess-bound and `run_managed` keeps every piece of
ownership state local to its own call, so nothing is shared to race on.

Waves, not one fan-out. The outage breaker counts CONSECUTIVE systemic
failures, and "consecutive" means nothing in completion order — folding
as futures landed would make the trip point flaky between identical runs.
Each wave is folded in submission order once complete, and the next wave
starts only if the breaker held, so the breaker overruns by at most
`workers - 1` cells (the ones already in flight) rather than by a whole
task.

`--workers 1` calls the cell directly rather than using a pool of one.
That is not an optimisation: an async KeyboardInterrupt is delivered only
to the main thread, so a cell on a worker thread is outside the reach of
the ownership cleanup that kills its sandboxed process tree. The default
therefore stays exactly what it is today, Ctrl-C included, and above 1
the flag's help says what is given up.

All bookkeeping stays on the main thread — the results.jsonl append, the
per-arm accumulation, the progress prints, the streak fold. No lock is
needed anywhere, the progress counter cannot race, prints do not
interleave, and results.jsonl keeps its canonical order.

Every future is read. An exception a cell did not expect stays parked
inside its Future until something asks for it; unread, a harness bug
would become a silently missing run instead of a crash.
2026-08-01 18:48:19 +00:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py fix(eval): stop letting a task neither arm can solve veto every promotion 2026-08-01 17:07:14 +00:00
test_mcp_bridge.py fix: start MCP bridge correctly when using npx (#1114) 2026-04-27 18:19:02 +01:00
test_oracle_assets.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py feat(eval): report evolution progress while the generation is still running 2026-08-01 17:01:03 +00:00
test_promotion_apply.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_runner_hardening.py feat(eval): run a task's benchmark cells in waves instead of one at a time 2026-08-01 18:48:19 +00:00
test_sanitized_graph.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_task_assets.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py fix(eval): self-hosted skill-evolution runner + sandbox Python 3 trust fix (#2600) 2026-07-21 14:40:20 +01:00
test_workflow_bench_evolution.py fix(ci): make the evolution lane survive its own deadlines and remember prior runs 2026-08-01 17:42:48 +00:00
test_workflow_bench_sessions.py refactor(eval): fold the review cleanups into the evolution fixes 2026-08-01 17:18:43 +00:00