Commit graph

4 commits

Author SHA1 Message Date
Gergo Magyar
d541105340 fix(workflow-bench): repair the harness defects the verbose proposer logs exposed
The first fully-logged skill-evolution run failed for five deterministic
reasons that had nothing to do with the candidate under test. Each is fixed
at the layer that actually owns the contract:

- Strict provider adapters materialize omitted optional string arguments as
  "". The MCP alias normalizer now treats a blank optional alias as absent
  (a blank REQUIRED target is still rejected), and a trusted PreToolUse hook
  strips blank strings before Read/GitNexus tool calls.
- MCP semantic errors rode home in a successful envelope and logged as
  result=ok. SessionProgress now inspects the payload and reports them as
  semantic-error.
- Claude Code's nested sandbox overlays absent root dotfiles with device
  nodes, which the provenance snapshot read as unauthorized workspace
  changes. Those names are excluded at the workspace root and hidden from
  git via an immutable excludes file.
- The proposer could not read /evidence from Bash (missing allowRead entry)
  and had no offline gitnexus runner, so it fell back to npx and hit the
  network. Both are now mounted; ripgrep is installed in CI.
- selected-rows.json advertised host artifact names that do not exist in the
  mount. Rows now name their staged patch_file/transcript_files, the prompt
  describes the real layout, and oversized bundles compact artifacts before
  dropping evidence rows so no row is silently lost.

Also replaces two benchmark scenarios that main already satisfies
(trivial-version-alias, inv-bug-pdg-note) with non-vacuous ones, verified to
fail against a pristine checkout.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-03 19:01:45 +00:00
Gergo Magyar
636d45364a feat(eval): log why a benchmark cell failed
The sweep printed error_kind=plan-evidence-invalid and nothing else, so
the reason a cell failed stayed in results.jsonl — an artifact uploaded
after the run, not something a watcher can read while it is still going.

Print the cell's error_detail next to its result line, redacted through
the same credential list as the artifact and bounded, since a
session-error detail carries stdout/stderr tails.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-03 17:33:59 +00:00
Gergo Magyar
227a3502b8 feat(eval): log bounded tool inputs and results
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-03 17:28:13 +00:00
Gergo Magyar
3fa42e4d83 feat(eval): report live progress for long headless sessions
A proposer or benchmark session could run for an hour with nothing in the
log between "proposing…" and its final result, so a wedged run looked
exactly like a working one. The last CI failure spent 66 minutes silently
retrying a dead endpoint before saying so.

A session's stdout is evidence and is only written out after redaction,
so it can never be echoed. Add a stdout_observer hook to run_managed that
sees the stream without copying it anywhere, and a SessionProgress
reporter that prints only what can be derived safely: turn counts, tool
names, API retries, and a heartbeat while the session is quiet. API
retries are called out by name because that is the signature of the
gateway wedging.

Progress goes to stdout so the benchmark sweep's lines reach the log
live through the existing echo_stdout passthrough, rather than as a
bounded stderr tail after the fact.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-03 12:45:29 +00:00