GitNexus/eval/tests
Gergo Magyar 1506a250dc fix(eval): review artifact path, evidence classification, comparator reuse
Extracted from the combined skill-evolution branch. This is the runtime change
set: everything that alters how a sweep executes and what it records. The
packed scheduler and its measurement harness were separated onto
perf/skill-evolution-packed-scheduler, which is purely additive.

Correctness. The review artifact was mounted as a writable FILE inside a
read-only workspace while the agent's Write tool writes atomically - temp file
beside the target, then rename - so the temp create failed EROFS and the
artifact was never written. Four layers gate that path and three named the old
location: the CLI's own allowWrite/allowRead policy, the task corpus verify
command run in its own sandbox invocation, and host_text/host_path for the
host-unsafe backend. Fixing the translator exposed a second defect, since
"/review-output" appears twice in "/review-output/review-output.json"; matching
is now anchored to a path boundary. parse_review_output folded OSError,
UnicodeError and JSONDecodeError into one message, so an artifact that was
never written looked like an encoding fault; each cause now names itself.

Health classification. broken_incumbent_arms inferred a broken environment from
an arm resolving zero tasks, which a reviewer facing a hard corpus falsifies -
Actions run 33962002890 is exactly that shape. Arms are now classified from
fresh execution and evidence outcomes as UNKNOWN, OBSERVED_OK, DEGRADED or
UNUSABLE, and only UNUSABLE aborts. Resolution count is not consulted. The
guard names no cause: an empty artifact establishes unusable evidence, not that
a mount rejected the write.

Comparator reuse. Reuse accepted evidence it should have rejected: a row
without a runtime_digest passed the drift lock, recorded_at was overwritten with
the copy time so a row could outlive its own max_age, and the binding ignored
task-asset and dependency digests although this change alters
sandbox_dependencies in the review corpus. Closing the last one required
preparing asset snapshots before the reuse decision, which also removes the
concurrent TaskAssetCache.prepare the prefetch thread could race.

These three concerns share aggregate() and _run_sweep, which is why they ship
together: separating them further would mean hunk-level surgery on a function
all three modify, and the risk of a silent omission outweighs the reviewability
gain.

592 eval tests pass at this base. The two test_model_gateway.py failures,
test_locked_litellm_translates_messages_to_offline_responses and
test_openai_gateway_never_leaves_proxy_output_on_an_undrained_pipe, fail
identically on origin/main in this environment.

Known gap, and the reason this is not ready to merge: no test drives _run_sweep
end to end. enforce_measurement_health is unit-tested including the
below-breaker unusable case, and the caller wiring is pinned structurally by
reading _run_sweep's compiled code object, but interruption semantics, exit
precedence and persisted artifacts are not exercised through the real path.
2026-09-07 14:23:38 +00:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_comparator_reuse.py refactor(eval): name the reuse directory check for the promise it makes 2026-09-07 14:21:16 +00:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py fix(review): close the artifact path in every layer that gates it 2026-09-07 14:21:15 +00:00
test_mcp_bridge.py feat(eval): evolve review skills against historical PRs 2026-09-04 05:32:31 +00:00
test_model_gateway.py fix(eval): require finite gateway startup budgets 2026-09-05 11:03:23 +00:00
test_oracle_assets.py fix(eval): close CI and remaining review gaps 2026-09-05 10:29:58 +00:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py fix(eval): close CI and remaining review gaps 2026-09-05 10:29:58 +00:00
test_promotion_apply.py fix(eval): make evolution evidence valid and bounded 2026-09-05 10:08:35 +00:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(review): close the artifact path in every layer that gates it 2026-09-07 14:21:15 +00:00
test_review_corpus.py fix(eval): cut skill-evolution wall clock without shrinking the gate 2026-09-07 14:21:14 +00:00
test_review_scoring.py refactor(eval): consolidate duplicated harness logic after the review round 2026-09-07 14:21:16 +00:00
test_runner_hardening.py refactor(eval): consolidate duplicated harness logic after the review round 2026-09-07 14:21:16 +00:00
test_sanitized_graph.py fix(eval): cut skill-evolution wall clock without shrinking the gate 2026-09-07 14:21:14 +00:00
test_session_progress.py fix(eval): cut skill-evolution wall clock without shrinking the gate 2026-09-07 14:21:14 +00:00
test_task_assets.py Address PR review feedback (#2785) 2026-09-04 18:59:32 +00:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py fix(eval): review artifact path, evidence classification, comparator reuse 2026-09-07 14:23:38 +00:00
test_workflow_bench_evolution.py fix(eval): make evolution evidence valid and bounded 2026-09-05 10:08:35 +00:00
test_workflow_bench_sessions.py refactor(eval): consolidate duplicated harness logic after the review round 2026-09-07 14:21:16 +00:00