GitNexus/eval/tests
Gergo Magyar 3fa42e4d83 feat(eval): report live progress for long headless sessions
A proposer or benchmark session could run for an hour with nothing in the
log between "proposing…" and its final result, so a wedged run looked
exactly like a working one. The last CI failure spent 66 minutes silently
retrying a dead endpoint before saying so.

A session's stdout is evidence and is only written out after redaction,
so it can never be echoed. Add a stdout_observer hook to run_managed that
sees the stream without copying it anywhere, and a SessionProgress
reporter that prints only what can be derived safely: turn counts, tool
names, API retries, and a heartbeat while the session is quiet. API
retries are called out by name because that is the signature of the
gateway wedging.

Progress goes to stdout so the benchmark sweep's lines reach the log
live through the existing echo_stdout passthrough, rather than as a
bounded stderr tail after the fact.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-03 12:45:29 +00:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py fix(eval): trim oversized proposer evidence instead of aborting 2026-09-03 11:19:34 +00:00
test_mcp_bridge.py fix: start MCP bridge correctly when using npx (#1114) 2026-04-27 18:19:02 +01:00
test_model_gateway.py fix(eval): stop the OpenAI gateway from hanging on its own log pipe 2026-09-03 12:36:01 +00:00
test_oracle_assets.py feat(eval): route skill evolution through OpenAI 2026-09-03 10:52:37 +00:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py fix(eval): drain with read1 so progress surfaces before the child exits 2026-08-02 12:12:08 +00:00
test_promotion_apply.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_runner_hardening.py feat(eval): route skill evolution through OpenAI 2026-09-03 10:52:37 +00:00
test_sanitized_graph.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_session_progress.py feat(eval): report live progress for long headless sessions 2026-09-03 12:45:29 +00:00
test_task_assets.py fix(eval): give vitest a writable .vite-temp inside read-only dependency mounts (#2630) 2026-07-22 10:16:25 +01:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py feat(eval): route skill evolution through OpenAI 2026-09-03 10:52:37 +00:00
test_workflow_bench_evolution.py fix(eval): close skill evolution review gaps (#2785) 2026-09-03 07:38:01 +00:00
test_workflow_bench_sessions.py feat(eval): route skill evolution through OpenAI 2026-09-03 10:52:37 +00:00