GitNexus/eval/tests
Gergo Magyar bb09ce28e0 fix(eval): stop discarding completed benchmark sessions as unverifiable
The evolution loop has not been able to promote anything since it went
online. Run 29907431284 (the last green run) reached the gate and threw
away 5 of its 18 runs, and the gate requires zero excluded runs in both
paired arms — so the generation could never produce a verdict on merit.

Two causes, both in the session layer:

1. Claude Code drains background-task bookkeeping after the final result
   event (`background_tasks_changed`, `task_updated`, `task_notification`,
   all `type: "system"`). The parent-stream check required the result to
   be the literal last event, so three sessions that had exited 0 with a
   complete result and usage payload were recorded as session errors.
   Trailing `system` events carry no tool_use/tool_result/usage payload
   and cannot forge skill or cost evidence; anything else after the
   result still fails closed.

2. The 3600s per-session ceiling killed two `workflow` incumbent runs on
   inv-bug-pdg-note mid-verification. Successful `workflow` rows in the
   same run finished in ~1600-2600s across both sessions, so the ceiling
   moves to 5400s and now lives in one shared constant instead of two
   argparse defaults that could drift apart.

Also marks the activation checklist against reality: the secrets, the
Environment, the runner, and the validation dispatch are all in place;
the repository variable GITNEXUS_EVOLUTION_ENABLED is the one remaining
gap, and until it is set the Saturday cron skips the job in seconds while
the EventBridge schedule still starts the runner for the day.
2026-08-01 16:45:48 +00:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py feat(eval): run the skill-evolution loop online (#2571) 2026-07-20 05:37:07 +01:00
test_mcp_bridge.py fix: start MCP bridge correctly when using npx (#1114) 2026-04-27 18:19:02 +01:00
test_oracle_assets.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_promotion_apply.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_runner_hardening.py fix(eval): mount the node prefix for npx and catch nested Claude Code bootstrap noise (#2627) 2026-07-22 08:25:46 +01:00
test_sanitized_graph.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_task_assets.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py fix(eval): self-hosted skill-evolution runner + sandbox Python 3 trust fix (#2600) 2026-07-21 14:40:20 +01:00
test_workflow_bench_evolution.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_workflow_bench_sessions.py fix(eval): stop discarding completed benchmark sessions as unverifiable 2026-08-01 16:45:48 +00:00