GitNexus/eval/tests
Gergo Magyar a2fdf9e93c fix(eval): hide the hidden harness from the proposer; drop inert hooks
The proposer authors the overlay the benchmark arms are scored with, but its
clone was never sanitized: it could read eval/workflow_bench, i.e. the task
prompts and the hidden oracles it was about to be graded against. The last
diagnostic run did exactly that, reading
inv-feature-list-repos-filter.oracle.test.ts directly, so a proposal could
win the gate by encoding expected behavior into a skill instead of being a
better skill. Sanitize the proposer clone exactly as run_cell already does.

Also remove the PreToolUse tool-input normalizer. It never ran: headless
`claude -p` (2.1.247) dispatches no hooks from inline --settings, a settings
file, project/user/local --setting-sources, or a trusted ~/.claude.json
project entry. Keeping it would read as a control in review while enforcing
nothing, and it was the sole reason the proposer stopped using --bare —
which stays off on its own merits, since bare ignores --tools and would cost
the proposer Grep and Glob.

Blank optional arguments from the OpenAI adapter remain handled where the
code is ours: MCP aliases in local-backend normalizeToolParams. Built-in
Read still rejects pages:"" and the model self-corrects on the next turn.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-03 20:03:09 +00:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py fix(eval): hide the hidden harness from the proposer; drop inert hooks 2026-09-03 20:03:09 +00:00
test_mcp_bridge.py fix: start MCP bridge correctly when using npx (#1114) 2026-04-27 18:19:02 +01:00
test_model_gateway.py fix(eval): stop the OpenAI gateway from hanging on its own log pipe 2026-09-03 12:36:01 +00:00
test_oracle_assets.py feat(eval): route skill evolution through OpenAI 2026-09-03 10:52:37 +00:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py fix(eval): drain with read1 so progress surfaces before the child exits 2026-08-02 12:12:08 +00:00
test_promotion_apply.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(eval): hide the hidden harness from the proposer; drop inert hooks 2026-09-03 20:03:09 +00:00
test_runner_hardening.py fix(workflow-bench): repair the harness defects the verbose proposer logs exposed 2026-09-03 19:01:45 +00:00
test_sanitized_graph.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_session_progress.py fix(workflow-bench): repair the harness defects the verbose proposer logs exposed 2026-09-03 19:01:45 +00:00
test_task_assets.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py fix(workflow-bench): repair the harness defects the verbose proposer logs exposed 2026-09-03 19:01:45 +00:00
test_workflow_bench_evolution.py fix(eval): close skill evolution review gaps (#2785) 2026-09-03 07:38:01 +00:00
test_workflow_bench_sessions.py feat(eval): route skill evolution through OpenAI 2026-09-03 10:52:37 +00:00