mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-09-15 23:32:49 +00:00

History

Gergő Magyar e2da8d90ce test(eval): run the benchmark offline against a scripted provider (#3235 ) * feat(eval): a scriptable stand-in for Anthropic and OpenAI Every defect this harness shipped last round was invisible to its own tests for one reason: the tests exercised a layer BELOW where the code runs. The usage log was never written because the proxy is a subprocess with a constructed environment. The callback could not be imported because LiteLLM loads it by path, not as a package. Failures went unrecorded because only the async hook was overridden. CI or review caught all three; no unit test could, because each called the function directly instead of driving the path that calls it. This closes that gap without spending money. It speaks the two wire protocols the harness actually depends on - Anthropic Messages, streaming and not, and OpenAI Responses - so a run can go through the real sandbox, the real CLI, the real gateway and the real usage callback with only the model faked. The runner already supports pointing at it: --base-url is the same path the free-model proxy documentation uses. Scripted rather than simulated. A test decides what the model says, which tools it asks for, and exactly what usage it reports. That last part is what makes provider-native accounting testable at all: real cache hits are not reproducible on demand, but a declared cache_read of 44,000 is. One Reply served down both protocols is also the cleanest demonstration that the same billed work is stated as a sum on one side and as a whole on the other. Tool blocks are the mechanism for artifact-producing cells. The CLI runs what it is asked to run, so a scripted Write block makes it write that file inside the sandbox for real - no model deciding anything. The end-to-end test drives the real proxy against the mock and asserts the usage log records the provider's own arithmetic through the Anthropic-shaped translation. It SKIPS here, because litellm's console script is absent in this environment, so it is unverified until CI runs it - the same footing the bubblewrap canary started on, and that one found a real bug on its first CI run. Not yet built: driving a whole sweep against this. That needs a scripted reply sequence that carries a cell to a scored artifact, which is the next step and the point of the exercise. 668 eval tests pass, 17 skipped; the two test_model_gateway.py failures are the pre-existing environmental ones. * test(eval): run a real session against the scripted provider The mock only proves something once the harness runs against it. This adds the stand-in CLI and the first integration tests that use it, so a session goes through the real code with only the model faked. tests/fixtures/fake_claude.py does what the CLI does at the two boundaries the harness depends on: it calls ANTHROPIC_BASE_URL for a turn, EXECUTES the tool blocks that come back, and prints the stream-json sequence the parent parses. Everything between - the session runner, the event-stream parse, the usage extraction, the artifact capture, the scorer - stays real. Four tests, chosen for the layers that have actually broken here: the usage a provider reported survives to the row, a scripted Write produces an artifact parse_review_output accepts, the prompt the harness meant to send is what arrived, and an upstream 529 lands as a failed session rather than a usable measurement. Writing the stand-in found two things worth keeping. The prompt arrives on STDIN under "-p --input-format text"; scanning argv for a non-flag token picks up a flag's value instead, and the prompt-fidelity test is what caught it. And three of these tests had been holding a sandbox they never applied, since no command_prefix is passed - that implied coverage which was not there, so the sandbox is gone from them and stays only in the artifact test, which needs its review directory. What these do NOT cover, checked rather than assumed: making the stand-in write in place instead of atomically still passes. On the host-unsafe backend there is no read-only mount to refuse it, so the atomic-write requirement remains a bubblewrap mount property that only the real-sandbox canary can prove. Dropping cache_read from the recorded usage does fail, so that half is genuinely pinned. 672 eval tests pass, 17 skipped; the two test_model_gateway.py failures are the environmental ones. * fix(eval): the usage adapter read a shape the callback never receives Running the gateway against the scripted provider proved the accounting merged in #3220 does not work, and the same run showed why nothing had caught it. LiteLLM does not hand a logger the upstream body. It normalises usage into its own Chat-Completions-shaped object first, so an OpenAI Responses reply reaches the callback as prompt_tokens / prompt_tokens_details.cached_tokens - never the input_tokens / input_tokens_details the shipped adapter reads. Every field came back unknown. The observed call_type is "anthropic_messages" as well, because Claude Code calls the Anthropic-shaped endpoint, so canonical_provider returned None and normalize_usage would have refused outright. Both were assumptions about a boundary I had only read about. The unit tests agreed with them because their fixture was written in the same wrong shape, so producer and consumer were consistent and both wrong - the exact failure the producer/consumer round trip exists to catch, one layer further out. Adds a LITELLM_NORMALIZED adapter for the object that actually arrives. The arithmetic is still OpenAI's - prompt_tokens is the whole, the details are subsets - so ordinary input is recovered by subtraction. The Responses adapter stays for a raw upstream body, which the mock still serves and tests directly. An unrecognised provider is still refused rather than guessed. The fixtures now carry the measured shape, and the end-to-end test asserts it through a real proxy: 48k prompt tokens with 44k cached is read back as 3k ordinary rather than as silence. 676 eval tests pass, 16 skipped, none failing. * test(eval): run a whole sweep offline, with negative controls The layers between a model turn and a promotion decision had never been exercised together. Unit tests covered each alone, and the paid runs that would have covered the composition kept dying, so the contracts BETWEEN them went unverified - which is where this harness has repeatedly shipped bugs. Drives runner.main() the way the workflow does. Real task selection, hidden oracle capture, sandbox, CLI subprocess, artifact capture, scoring against the oracle, aggregation, health guard and promotion gate. Only the model is scripted. Getting to green meant satisfying nine real contracts nothing had exercised end to end, and each failure was the harness correctly refusing bad evidence: --unsafe-no-bwrap is restricted to the paired review arms; ce_* needs a plugin carrying ce-plan, ce-work and ce-code-review; candidate_* needs an overlay; the clone needs .gitnexus/meta.json with indexedAt and lastCommit; the evidence gate needs a Skill request with a non-error result; review findings need exactly ten fields with severity in critical/high/medium/low; and the hidden labels use a DIFFERENT schema from the review output - line_start/line_end, six fields. That last one only a real run surfaces. Three negative controls, because a scorer that cannot be wrong measures nothing. A finding in the wrong place is tp=0 fp=1 fn=1 and oracle-failed, while its evidence stays VALID - being wrong is a quality result, not a broken measurement. Approving defective code is a miss with no false positive, and precision is None rather than 0, because it is undefined with no predictions. One run cannot promote: the gate says it needs three valid paired runs. A fourth control exists because a mutation demanded it. Forcing skill_was_invoked_events to return True left every other test here passing, so nothing pinned the gate that separates measuring a SKILL from measuring a model. Writing it turned up behaviour worth recording rather than assuming: a skill-not-invoked row still carries its score AND still counts toward the arm median, because aggregate() drops EXCLUDED_ERROR_KINDS and evidence_valid=False and skill-not-invoked is neither. The health guard stops the sweep, so a single-run sweep cannot promote on it, but a mixed run's median would include a cell whose skill never ran. Pinned as-is so it cannot change silently in either direction; changing it is a promotion-semantics decision, not a test fix. Two provisioning steps are stubbed and neither is harness logic: the pinned runtime mounts (no node_modules in a worktree) and the sanitized graph build (needs the gitnexus CLI at a mounted path). Containment is host-unsafe here; bubblewrap stays with the real-sandbox canary. 681 eval tests pass, 16 skipped, none failing. Runs in ~18s. * fix(eval): an uninvoked skill must not move the arm's quality median Found by the offline sweep: a skill-not-invoked row still carried its score into the arm's quality median. aggregate()'s filter dropped EXCLUDED_ERROR_KINDS and evidence_valid=False, and skill-not-invoked is neither, so an arm could be credited for a review it never performed with the skill under test - which is the one thing an arm exists to measure. Excluded from the QUALITY metrics only. Cost and duration still count that row, because the session really ran and really was billed, and the promotion gate still sees it, because it has its own vocabulary for a candidate that never loaded its skill. Two wider fixes were tried and abandoned, both because the tests said so rather than because I reasoned it out first. Reusing the health guard's evidence_failed predicate also excluded transcript-missing rows, but test_aggregate_excludes_session_error_rows_from_medians pins those as counting: that session ran, only its transcript is unverifiable. Excluding the row from `valid` outright turned a candidate whose skill never loaded from keep_incumbent into insufficient_evidence - the safety property held either way, but the decision vocabulary is promotion semantics and not mine to change on a measurement fix. Mutation-checked: putting the rows back into the quality median fails the new test. Both directions asserted, since a filter that excludes everything would also pass - a wrong-but-valid review still moves quality, because being wrong is exactly what a quality median should reflect. 682 eval tests pass, 16 skipped. * test(eval): run the offline sweep unstubbed in the job that can, and probe CLI identity Items 5 and 6 turned out to be one change. The containment (ubuntu) job already installs bubblewrap, the pinned Claude CLI, node_modules and a built GitNexus - everything the sweep's two provisioning stubs stand in for. So the stubs are not a property of the test, only of a machine that lacks those things. GITNEXUS_REQUIRE_FULL_SWEEP=1 makes the sweep run with nothing stubbed: real containment instead of --unsafe-no-bwrap, the real runtime mounts, the real sanitized graph. Set in that job, following the GITNEXUS_REQUIRE_BWRAP_CANARY pattern already there. The gate FAILS on a missing piece rather than degrading to the stubbed path, which is the point - a green tick that silently tested less is what the bubblewrap canary was written to prevent. Verified both states here: default green, and gate-on fails on this machine rather than skipping, since it cannot create user namespaces. Item 7 is an experiment, not an answer. Per-cell attribution needs an identifier that travels WITH the request, because one proxy serves the whole sweep and anything read from its environment is identical for every call. What the real CLI sends is not documented anywhere I can check, and guessing a wire format is exactly how the last three accounting bugs happened. So the probe drives the REAL pinned CLI against the mock and records the identity-bearing headers and body keys that arrive. It asserts only that a request was made; the recorded evidence is the deliverable, and the job log preserves it. Skips without CLAUDE_CANARY_BIN. Two guards caught this rather than review: the repo pins the containment job's env and its exact test list, so both had to be updated deliberately - which is the guard working, not friction. 682 eval tests pass, 17 skipped. * test(eval): make the offline sweep cross-task, so a scheduler change is checkable The sweep fixture had one task, and a single task cannot show the thing a cross-task scheduler changes: waves are per-task, so ordering, packing and a breaker spanning a task boundary are all invisible with one. A second task with its defect in a DIFFERENT file, and its own hidden labels, makes per-task routing observable. The scripted reply is now task-aware, which matters for the same reason: replying with the first task's finding scores the second task wrong. The load-bearing assertion is that each task scored against ITS OWN oracle. That is the dangerous failure mode of interleaving cells from different tasks - a mis-routed context or artifact scores one task against another's labels, and every row still looks green. Mutation-checked: pointing every cell at the first task's oracle snapshot fails it. This is the safety net the packed-scheduler wiring needs. Measured earlier against the real sweep_packed_cells, that change is worth -27% on a cold sweep and -37% weekly, with breaker fidelity holding at three injected failure positions - but it restructures a 125-line loop across ~92 names that also holds graph prefetch, reuse selection, oracle staging and the canary drop. Landing that on top of a one-task fixture would have been unverifiable, which is why this comes first and separately. 682 eval tests pass, 17 skipped. * fix(eval): commit the stand-in CLI's executable bit The file was created and chmod +x'd locally, but committed 100644 - so the mode existed only in my working tree. Any fresh checkout, CI included, gets a non-executable file and every cell dies with "required executable is not an executable regular file". Found by accident: checking out origin/main and back to compare a flaky test restored the file from the index and stripped the bit, which turned 5 green tests into 9 failures. Without that detour this would have failed on the first CI run instead. Same shape as the bugs this branch exists to catch - something that works only because of local state, breaking where the code actually runs. * fix(eval): apply code review findings Seven local reviewers and an independent cross-model pass. The headline is that a fix I added in this branch was worse than the gap it closed. Reverted the aggregate() quality-median filter. Excluding skill-not-invoked rows from the quality metrics left valid_runs and excluded_runs still counting them, so the promotion gate saw N clean runs while the median came from fewer. The dropped rows are systematically an arm's worst, so it biased toward PROMOTING - reproduced: one real run at 0.9 plus two uninvoked rows at 0.0 gave the gate 3 valid runs, zero exclusions and a 0.9 median, flipping keep_incumbent to promote. Three verdict fields compounded it: they are all() reducers still reading the wider set, so one uninvoked cell flipped a whole arm. Five reviewers found the two halves independently. Closing it honestly needs a scored-run count plus a paired-equality check in the gate, which is promotion semantics rather than an aggregation fix. The gap is now pinned by a test that states why the half-fix was reverted. Stopped forging the absence of CI. The runner refuses --unsafe-no-bwrap when CI is set because that mode runs sessions with bypassPermissions behind a boundary its own docstring calls "not a security boundary"; the sweep test deleted CI to get past it, so eval / locked pytest ran an uncontained agent sweep on the runner holding the checkout and credentials. It skips under CI instead - the containment job still runs it for real with GITNEXUS_REQUIRE_FULL_SWEEP=1. The stand-in CLI was lying in three ways. It never set is_error, so a refused write read as a completed one. It had no Skill branch at all, so honoring is_error revealed the evidence gate had been satisfied by a tool the fixture never ran - the gate was measuring the fixture, not a skill. And a reply with no usage became four zero-valued fields plus a fabricated cost, which is exactly the unknown-is-not-zero confusion the accounting it feeds exists to prevent. A provider failure also crashed the subprocess with no terminal result event. The identity probe never ran anywhere. test_mock_provider.py was in no job's file list, and the only job setting CLAUDE_CANARY_BIN runs a fixed list. My commit message claimed the next containment run would produce the answer; it would not have. Now wired in, with the CI-shape test updated to pin it. Also: the regex-miss fallback wrote a predictable name in shared /tmp through a symlink-following stage, now scoped to the test's own directory; and the canonical_provider docstring plus the callback comment still asserted a call_type branch the code no longer has. Deferred as design decisions rather than review fixes: the containment sweep uses the stand-in CLI rather than the pinned real one, the full-sweep path bypasses the gateway so native usage accounting is unexercised there, _normalize_litellm duplicates the Responses algorithm, and OPENAI_RESPONSES is now unreachable from canonical_provider. 682 eval tests pass, 17 skipped, ruff clean. * fix(eval): carry scripted tools over the Responses protocol Review round on #3235. Three real items; five more were already fixed in `a20f94e1c` and are answered on their threads rather than re-fixed. `_openai_response` emitted only an `output_text` item and never read `reply.tools`, so a reply scripted with a Write or Skill crossed the gateway with the tool silently dropped. Responses is the protocol the gateway is configured for BECAUSE it carries tool use, so the mock was wrong about the wire on the one path that matters most. Function-call items now accompany the message. Mutation-checked: reverting the emit fails the new test on "the scripted tool must cross the Responses path". The artifact session now takes `command_prefix` and `require_pid_namespace` from the sandbox the way `run_arm` does instead of calling `run_claude` bare. On host-unsafe `command_prefix_for` returns `[]` by construction, so this pins the wiring, not the isolation - the comment says so rather than implying more. CodeQL's three unused-variable reports on one line were one finding: a call whose result is entirely discarded. Unpack nothing there. 683 passed, 17 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(eval): forward the usage the provider reported instead of zero-filling it Review round on #3235; all three findings valid. The stand-in CLI defaulted absent cache fields to 0. That fabricated a complete measurement out of an incomplete reply, and the second-order effect was worse than the first: `runner_sessions` requires all four USAGE_FIELDS before it calls a session measured, so a stand-in that always emitted four fields made that guard unfirable from any offline test. It was always satisfied. It now forwards exactly what arrived. `Reply`'s cache fields accept None to script absence, since a consumer that cannot tell "omitted" from "zero" is the bug this harness exists to catch. Mutation-checked: restoring the zero-fill fails the new test. Also corrected a comment claiming aggregate() excludes skill-not-invoked rows from the quality median. It does not - that was the filter reverted in `a20f94e1c` for inverting a promotion, and the comment survived the revert describing the opposite of what the test pins. Dropped an unused monkeypatch fixture arg. 684 passed, 17 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(eval): keep an omitted cache field omitted on the Responses wire Review round on #3235. The main finding is a miss in my own previous commit: that one taught the Anthropic path to forward absence instead of zero-filling, but `_openai_response` still serialized both `input_tokens_details` keys unconditionally. Collapsing None to 0 is right for the arithmetic - an unreported field adds nothing to the total - and wrong on the wire, because `_int_or_none` reads an absent key as unknown and a present 0 as a measured zero. So a reply scripted with `cache_read_input_tokens=None` was indistinguishable from a provider-reported zero on exactly one of the two protocols. Half-applying the invariant was arguably worse than not applying it: the Anthropic test passing made the pair look covered. Mutation-checked: restoring the unconditional keys fails the new test. Also: the module docstring claimed the stand-in executes the tool blocks that come back, without noting Bash is stubbed; and dropped an unused tmp_path fixture arg. 685 passed, 17 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(eval): validate every usage field the stand-in forwards Review round on #3235. The guard checked input_tokens and output_tokens for type and sign, but the forwarding comprehension passed the cache fields through unchecked whenever present. The parent's well_formed test only asks whether the four keys are PRESENT, so a negative, boolean, or non-integer cache value rode into a `success` result and was recorded as a usable measurement. Same shape as the previous two rounds: the required half of a pair was handled and the optional half was not. A field good enough to report is good enough to check. Mutation-checked: dropping the added clause fails all three parametrized cases (negative, boolean, string). 688 passed, 17 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(eval): say which stand-in tools execute and which are modelled Review round on #3235. The previous commit's docstring fix said "Write and Skill really run" while correcting the Bash claim. Only Write really runs: Skill validates the request and returns a synthetic result. Fourth round of the same shape - the reported half of a pair gets fixed and the sibling keeps the overclaim. Both docstrings now name each of the three branches and what it actually does. 688 passed, 17 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>		2026-09-09 12:34:00 +01:00
..
oracles	fix(eval): make evolution evidence valid and bounded	2026-09-05 10:08:35 +00:00
review_cases	fix(eval): make evolution evidence valid and bounded	2026-09-05 10:08:35 +00:00
__init__.py	feat(skills): GitNexus Engineering Tool Kits (#2566 )	2026-07-19 15:07:24 +01:00
comparator_reuse.py	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
evolution.py	fix(eval): make evolution evidence valid and bounded	2026-09-05 10:08:35 +00:00
evolve.py	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
free-model.litellm.yaml	feat(eval): route skill evolution through OpenAI	2026-09-03 10:52:37 +00:00
gateway_supervisor.py	fix(eval): repair native containment checks	2026-09-05 10:48:39 +00:00
learnings.jsonl	fix(scope-resolution): a closure binding is a call SOURCE in every language, and function-local values carry their own identity (closes #2699 ) (#2718 )	2026-07-28 18:25:19 +01:00
litellm_usage_callback.py	test(eval): run the benchmark offline against a scripted provider (#3235 )	2026-09-09 12:34:00 +01:00
measure_evolution_cost.py	feat(eval): Add bounded packed-scheduler primitives and offline replay benchmarks (#3206 )	2026-09-08 08:22:12 +01:00
mock_provider.py	test(eval): run the benchmark offline against a scripted provider (#3235 )	2026-09-09 12:34:00 +01:00
model_gateway.py	feat(eval): record provider-native usage at the gateway instead of inferring it after translation (#3220 )	2026-09-08 18:22:04 +01:00
oracle_assets.py	fix(eval): stop hiding review patches from sandboxed git apply	2026-09-04 19:43:47 +00:00
process_control.py	fix(eval): make evolution evidence valid and bounded	2026-09-05 10:08:35 +00:00
promotion_apply.py	feat(eval): evolve review skills against historical PRs	2026-09-04 05:32:31 +00:00
proposer_sandbox.py	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
provider_usage.py	test(eval): run the benchmark offline against a scripted provider (#3235 )	2026-09-09 12:34:00 +01:00
README.md	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
review_scoring.py	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
run-evolution.sh	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
runner.py	test(eval): run the benchmark offline against a scripted provider (#3235 )	2026-09-09 12:34:00 +01:00
runner_artifacts.py	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
runner_sessions.py	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
runner_tasks.py	feat(skills): GitNexus Engineering Tool Kits (#2566 )	2026-07-19 15:07:24 +01:00
runtime_mounts.py	fix(eval): close CI and remaining review gaps	2026-09-05 10:29:58 +00:00
sanitized_graph.py	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
session_durations.json	feat(eval): Add bounded packed-scheduler primitives and offline replay benchmarks (#3206 )	2026-09-08 08:22:12 +01:00
simulate_sweep.py	feat(eval): Add bounded packed-scheduler primitives and offline replay benchmarks (#3206 )	2026-09-08 08:22:12 +01:00
task_assets.py	fix(eval): make historical review evolution score instead of aborting	2026-09-04 18:37:39 +00:00
tasks.review.scenarios.yaml	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00
tasks.scenarios.yaml	fix(eval): sweep evidence handling and measurement health, with guarded comparator reuse (#3207 )	2026-09-08 09:25:45 +00:00

README.md

Skill benchmark — evolve review quality, measure workflow cost

Measures whether the gitnexus-plan → gitnexus-work engineering workflow actually saves tokens versus a baseline agent on the same tasks, using real headless Claude Code sessions. Nothing is estimated: every number comes from the CLI's own final event in its parent-captured --output-format stream-json report.

What it compares

Arm	Sessions	Notes
`workflow`	`gitnexus-plan` on the task, then `gitnexus-work` on the produced plan	The skills must be installed (`gitnexus setup`, or repo-local `.claude/skills/`)
`candidate_workflow`	same sessions as `workflow`, with a candidate skill overlay	Paired with `workflow` on the same task/ref/model
`workflow_direct`	one `gitnexus-work` direct-mode session	The middle option — execution discipline without a planning pass
`candidate_workflow_direct`	same session as `workflow_direct`, with a candidate skill overlay	Paired with `workflow_direct` on the same task/ref/model
`ce_workflow`	`ce-plan` on the task, then `ce-work` on the produced plan	External comparator: the explicitly supplied, pinned compound-engineering plugin's plan→work family
`ce_workflow_direct`	one `ce-work` direct-mode session	External comparator paired with `workflow_direct`
`review`	one `gitnexus-review` session over an immutable historical PR snapshot	Emits strict `review-output.json`; hidden human labels score quality after the session
`candidate_review`	the same review with a `gitnexus-review` candidate overlay	Paired with `review` on the same case/ref/model/runtime
`ce_review`	one pinned `ce-code-review` session over the same changes	External comparator paired with both review arms
`baseline`	one session with the identical task text	`--disallowedTools Skill` so it cannot borrow the workflow; same repo, same MCP tools
`baseline_nomcp`	like baseline, graph tools also disallowed	Separates the workflow-discipline question from the GitNexus-tools question (off by default)

Every arm runs in a fresh detached git worktree of the task's ref, once per --runs. The model-visible verify command is recorded as authored_tests_passed, but cannot certify its own solution: resolved also requires the task's harness-owned hidden behavioral oracle to pass. Token savings on a failed task are flagged, not celebrated, and diff churn (files/+insertions/−deletions vs the starting commit) is recorded as a cheap over-engineering proxy. Task class labels (trivial → investigation → cross-module) make the report readable as a routing table: the boundary where workflow starts beating workflow_direct and baseline is the boundary lfg's gate and work's direct-mode triage should encode.

Quick start

cd eval
export GITNEXUS_BENCH_ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY"
uv run --locked --extra dev python -m workflow_bench.runner \
  --tasks workflow_bench/tasks.scenarios.yaml --runs 3 \
  --model claude-sonnet-4-20250514

Scenarios marked expensive: true are skipped unless --include-expensive is supplied. The report names both selected and skipped tasks so an omitted cell cannot be mistaken for evidence.

CE comparator arms never discover a user-level plugin. Supply an exact plugin release explicitly; both flags are mandatory whenever any ce_* arm is selected:

uv run --locked --extra dev python -m workflow_bench.runner \
  --tasks workflow_bench/tasks.scenarios.yaml --runs 3 \
  --model claude-sonnet-4-20250514 \
  --arms workflow ce_workflow \
  --ce-plugin-dir /opt/operator-input/compound-engineering-3.19.0 \
  --ce-plugin-version 3.19.0

The runner verifies the manifest version, copies only the plugin manifests, skills, scripts, and assets into a bounded no-symlink snapshot, and mounts that snapshot read-only only for CE arms. Every CE result records its exact plugin version and content-manifest digest.

Output: results/wfbench-<timestamp>/results.jsonl (every run, with session ids for transcript drill-down) and report.md (medians per task per arm, plus a savings row: input / cache / output tokens, cost, wall time).

Trust model — fail-closed Linux containment

Task files and candidate prose remain untrusted executable inputs. Every setup, verifier, incumbent, and candidate cell therefore runs in a preflighted Bubblewrap boundary with a private home/config/temp, a self-contained clone, a PID namespace, bounded process-tree ownership, and a deny-by-default environment. Task-declared dependency roots are mounted read-only, while graph assets are rebuilt by the harness as described below. Claude runs in bare, dontAsk mode with strict clone-local MCP configuration; Bash children do not inherit the model credential and their network sandbox denies all domains.

Prebuilt task .gitnexus assets are rejected. For each task commit, the harness creates the deterministic parentless snapshot first, removes every analyzer-visible path or stored source reference to the benchmark harness, neutralizes target-controlled GitNexus config/ignore files, and builds one fresh PDG index offline with --pdg --index-only --no-stats. It then proves that neither whole graph nodes nor relationships contain a harness marker and caches only the bound metadata/database assets for reuse by paired arms.

Each selected task also declares a bounded hidden oracle command and file set. The harness captures those regular, non-symlink files into an immutable in-memory snapshot before any arm runs and binds the command, paths, sizes, and raw bytes into the task digest. Before any task asset or model session, each disposable clone that contains the benchmark harness is rewritten to a clean, parentless snapshot without eval/workflow_bench; all original refs, reflogs, and unreachable Git objects are pruned so git show cannot recover the hidden bytes. Only after the model exits (and after the authored-test signal is collected) does the harness materialize the oracle beneath a private host root, mount it read-only at a random workspace sibling, and supply that mount through GITNEXUS_BENCH_ORACLE_ROOT. This layout preserves hidden tests' ../gitnexus imports as the credited candidate checkout. Authored and hidden verifiers run with the complete workspace read-only and networking unshared; hidden stdout/stderr is never persisted. The harness re-checks every oracle byte and erases the mountpoint before churn/patch capture. Shipped Vitest oracles use the staged, digest-bound vitest.config.mts; a candidate cannot replace repo test config or setup hooks to make the hidden test vacuously pass. The hidden command invokes the read-only dependency's Vitest binary directly, without an npx configuration/resolution layer.

Every evaluated repo-local skill root is over-mounted read-only for the full model session, and an immutable empty user-level skills directory prevents a writable $HOME skill from shadowing it. Skill-use evidence comes only from the bounded stream captured directly from Claude stdout by the parent. The runner parses every event through EOF, requires one final result, correlates an exact Skill request ID with one later successful result, structurally redacts the event objects, and stores the canonical redacted JSONL with a digest. Files written beneath the agent's $HOME are never trusted as evidence.

Bare mode is deliberately non-interactive: it does not consult a stored Claude login/keychain or ANTHROPIC_AUTH_TOKEN. Supply an Anthropic API key through GITNEXUS_BENCH_ANTHROPIC_API_KEY (preferred) or --anthropic-api-key; the harness maps it to ANTHROPIC_API_KEY only for the trusted Claude parent and scrubs it from agent-launched tools. GITNEXUS_BENCH_AUTH_TOKEN and --auth-token remain as aliases. OpenAI keys are not a drop-in replacement: pass --openai-api-key / GITNEXUS_BENCH_OPENAI_API_KEY with gpt-* / o* / openai/* model ids and the harness starts a loopback LiteLLM proxy. The OpenAI key stays on that host process; Claude still sees only a minted ANTHROPIC_API_KEY plus ANTHROPIC_BASE_URL.

The trusted Claude CLI still needs outbound access to the explicitly supplied model endpoint. This is not a network broker, so the CLI itself retains that egress; agent-launched tools do not. Missing Bubblewrap, unsupported hosts, invalid mounts, or namespace preflight failure stop before model invocation. Native benchmark execution is therefore Linux/WSL2-only. Evidence assembly and hand-authored overlay preparation can happen elsewhere, but --initial-overlay does not bypass containment.

For a local diagnostic inside a container that blocks user namespaces, an operator may explicitly choose the non-containment host backend:

UNSAFE_NO_BWRAP=1 RUNS=1 ./workflow_bench/run-evolution.sh

This mode runs review sessions directly in disposable host worktrees and is not a security boundary: it does not isolate the network or create a PID namespace, and a session that can chmod can undo the workspace lock. The harness drops write bits on the whole clone, with no carve-out, so accidental npm install / analyze writes cannot invalidate review evidence. The review artifact is not in the clone at all: it lives in a writable directory bound at /review-output, outside the workspace. Sandbox cleanup restores owner write bits before deleting the private TMPDIR, because a session that copytrees the locked clone would otherwise leave non-empty 0555 directories that rmtree cannot remove. Historical review SHAs that gitignore .claude/skills/* are force-added when the harness seeds or overlays the evaluated gitnexus-review skill. Treat model and verifier processes as able to access host files and credentials available to the invoking user. It is restricted to the review benchmark, forbidden with --apply and whenever CI is set; promotion-capable and CI runs must use Bubblewrap.

Prompt and skill evolution loop

Prompts age as models and tool harnesses change. Treat the current skills and router thresholds as an incumbent policy, not permanent truth. Candidate changes run offline in the same throwaway clones as the incumbent; production skills never rewrite themselves from a live task.

On the self-hosted evolution box, run-evolution.sh passes --max-runtime-from-instance-window and the CLI derives its own cap from /proc/uptime at startup (24h EventBridge window minus a 90-minute upload reserve), in the same breath as it starts the clock that cap is measured against — a budget computed anywhere earlier is spent by the seconds between. A workflow_dispatch that lands on an already-running instance therefore exits in-process instead of vanishing when the box stops — a cancelled GitHub job skips even if: always(), which is how run 33962002890 lost 51 finished sessions. Local runs are uncapped.

A review generation is 6 tasks × 3 arms × 3 runs. Serial workers=1 at ~19 minutes per session is a 16-hour job (run 33962002890). Two harness changes cut that without shrinking the gate:

Comparator reuse. evolve.py forwards the seed / prior generation as --reuse-results. Incumbent review and ce_review rows are copied into the new results.jsonl when model, effort, task SHA, prompt digest, oracle bytes, incumbent skill digest, CE plugin digest, and sandbox backend still match. Candidate arms always run. A weekly generation with an unchanged incumbent therefore pays 18 sessions, not 54. A promotion, model change, task-corpus change, or harness RUNTIME_DIGEST change invalidates the lock and re-runs the comparators.
Sanitized clone templates. Each unique task SHA is cloned and sanitized once. Cells copy that parentless snapshot (reflink when the filesystem allows) instead of git clone --no-local plus repack/prune/fsck 54 times. Isolation is a private .git, not a second copy of full history.

Dispatch defaults to --workers 3 so those 18 paid cells can overlap. Size workers to the host: a cell that loses CPU and hits the session ceiling is an excluded run the gate refuses.

Build an overlay that mirrors only the canonical repo-local skill paths:

/tmp/gn-skill-candidate/
└── .claude/skills/
    ├── gitnexus-plan/SKILL.md
    └── gitnexus-work/SKILL.md

The overlay may contain Markdown files from either of those two skill trees. The runner rejects every other path, including source, test, and MCP configuration files, so a candidate cannot improve its score by changing the task or verifier. Arm selection is derived from the touched skill and must be exact: a plan-only overlay runs the workflow pair; any work overlay runs both workflow and direct-work pairs. Subsets and unrelated extra pairs fail before paid work. For a work overlay:

cd eval
uv run --locked --extra dev python -m workflow_bench.runner \
  --tasks workflow_bench/tasks.scenarios.yaml \
  --runs 3 --workers 1 --model claude-sonnet-4-20250514 \
  --arms workflow candidate_workflow \
         workflow_direct candidate_workflow_direct \
  --candidate-overlay /tmp/gn-skill-candidate

Candidate runs start from the same task commit, then receive a clean ephemeral commit containing the overlay. results.jsonl records the named model, task commit, task-prompt digest, skill digest, overlay digest, hidden-oracle command/manifest/content digests, immutable dependency content/manifest digests, separate authored-test and oracle outcomes, timestamp, local session ids, and digest-bound parent-captured event-stream artifacts. Those artifacts are the trajectory evidence: cluster failures and expensive detours, propose one bounded prompt change, and feed it back as the next overlay.

When candidate arms are present the runner also writes schema-6 promotion.json. It binds the immutable overlay digest, benchmark model, truthful candidate origin (a named proposer model or manual-initial-overlay), selected task definitions, resolved commits, and exact hidden-oracle bytes/commands, immutable dependency bytes, committed base digest of every apply destination, exact required arms, thresholds, and evidence expiry. Its default deterministic gate is deliberately conservative:

Schema 6 binds a separate policy to each required candidate arm and records whether the sweep completed. Apply validates the paired metrics and recomputes each decision. Historical schema 5 reports remain readable; regenerate their benchmark evidence before applying an overlay. Editing a schema number does not supply the missing evidence.

Review candidates optimize weighted F1 with a minimum improvement of 0.01, complete paired evidence on every selected task, and no per-task quality regression. Complete misses score zero. Matching uses maximum cardinality throughout the 100-finding limit. Downgraded findings receive at most their reported severity's weight; only blocking-severity matches count toward blocker recall. Every valid candidate repeat must have the correct verdict, and the minimum blocker recall across repeats must not regress. Clean controls retain their false-positive and verdict safeguards. Implementation candidates retain the efficiency policy below:

at least 3 paired VALID runs per task, zero excluded runs in either arm (session/infra-error rows therefore block promotion), and a named model;
a fully measured task that neither arm ever resolves remains reported but is ungated from the quality comparison — only if its metric was measured in both arms and no run hit skill-not-invoked (a skill that never loaded is prompt evidence, not task health). An ungated task still ranks against a looser 100% failed-task regression cap on the promotion metric;
at least half the paired tasks must stay gated, and promotion.json discloses the gated/ungated split per decision; a set with no gated task at all is insufficient_evidence;
the candidate must pass the hidden oracle on every valid run of every gated task the incumbent resolves at least once — on a task the incumbent never resolves, partial candidate progress counts as improvement instead of failing the floor, so making some progress is never scored worse than making none;
no per-task resolution-rate regression (quality is lexicographically first);
promotion by resolution needs a margin of at least 2 resolved runs — a 1-run difference is noise at this run count and falls through to the efficiency comparison;
with equal quality, at least 5% median improvement on the promotion metric (default cost_usd — the only CLI-reported number that includes subagent spend; token metrics count only the main-loop session and flatter subagent-heavy candidates, so selecting one stamps a warning into promotion.json);
no individual task may regress the selected efficiency metric by more than 20%.

Tune the efficiency signal with --promotion-metric and the three --promotion-* thresholds. Applying requires one unique promote decision for every bound candidate arm. The driver then stages every canonical and shipped mirror, verifies that all destination bytes still match the bound bases, replaces them as one compare-and-swap set, verifies byte parity, and rolls every landed replacement back on failure or interruption. keep_incumbent and insufficient_evidence become the next learning queue; their raw results.jsonl rows carry the session_ids of the trajectories to inspect.

Re-run the paired suite whenever the named model or tool harness changes, and at least every 90 days otherwise. This is prompt-policy optimization using verified agent trajectories as reward evidence; it is intentionally not online model-weight RL. The same records can feed a later offline RL pipeline without weakening today's deterministic promotion boundary.

Closing the loop automatically (`evolve.py`)

The evolution workflow runs an offline containment preflight with the pinned Claude Code 2.1.214 binary before starting a paid proposer or benchmark. The review canary seals the workspace read-only and writes nothing into it: the artifact directory is bound at /review-output outside the workspace, and the file itself is deliberately absent until the session creates it, so its absence distinguishes "never written" from "written badly". Runtime mount placeholders are prepared in the disposable clone before sealing it; existing config bytes are preserved. Any pre-existing result entry, including a symlink, is rejected. Required canaries fail when their runtime or Bubblewrap is unavailable.

The default outage limit is five consecutive unusable results, across task boundaries. Invalid review JSON advances this limit even when a skill or session error was recorded first. A valid zero-quality review resets it. Concurrent waves can exceed the limit by at most workers - 1 completed cells; no further wave starts after a trip. Completed rows and redacted diagnostics remain in the partial report, the runner exits nonzero, and the evolution driver stops without applying or starting another generation.

SIGINT and SIGTERM propagate one cancellation event through managed commands, including clone, setup, Claude, and verification. Executor submissions copy the run context so indirect subprocess helpers receive the same event. Active process groups or Windows Job Objects are terminated and workers joined before shared assets or the gateway are released. Controlled cancellation tests require cleanup within 15 seconds. Cancellation remains distinct from timeout and quality failure in recorded evidence.

The gateway runs under a private supervisor watching a pipe owned only by the harness. Parent exit, including SIGKILL, closes that pipe and stops the proxy group; Windows also retains kill-on-close Job Object ownership. Keep completed JSONL rows, transcripts, the partial report, and gateway diagnostics when investigating an interrupted run. A subsequent paid comparison needs fresh evidence from all arms under the same dependency lock. LiteLLM pricing comes from that locked release's local cost map; compare no old/new-lock costs as quality evidence.

workflow_bench.evolve automates the three manual arrows — propose, benchmark, apply — without moving the trust boundary:

cd eval
./workflow_bench/run-evolution.sh                  # local; no working-tree apply
./workflow_bench/run-evolution.sh --apply          # CI; same argv the workflow uses
./workflow_bench/run-evolution.sh --dry-run        # print the evolve command

The GitHub skill-evolution job calls this script. Do not invoke python -m workflow_bench.evolve directly for a full loop. Environment knobs match the workflow: MODEL, PROPOSER_MODEL, GENERATIONS, RUNS, WORKERS, PROVIDER, EFFORT, SEED_RESULTS, INCLUDE_EXPENSIVE. The checked-in production defaults are PROVIDER=openai, MODEL=gpt-5.6-sol, PROPOSER_MODEL=gpt-5.6-sol, and EFFORT=xhigh.

The scheduled/default profile is read-only review evolution. Set EVOLUTION_PROFILE=implementation explicitly to run the legacy plan/work benchmark. Review mode requires CE_PLUGIN_DIR and CE_PLUGIN_VERSION.

Each review generation: a confined proposer session reads only the incumbent gitnexus-review skill, normalized CE/incumbent/candidate result rows, bounded review artifacts and session transcripts, and the rejected proposal.md when available (including a workflow seed from a prior run), and the learning queue, then writes ONE bounded candidate overlay plus a reviewer-facing proposal.md. The proposer's clone is sanitized exactly like an arm's before its session starts: it authors the artifact the arms are scored with, so letting it read eval/workflow_bench would hand it the task prompts and the hidden oracles it is about to be graded against, and a proposal could win the gate by encoding the expected behavior into a skill rather than by being a better skill. The overlay is re-validated by candidate_overlay_files (same boundary: Markdown under gitnexus-review, including exercised ci-personas/, nothing else), frozen, and exercised only by its exact required pairs. Task refs are resolved once before generation zero and the immutable task bindings are forwarded to every generated runner invocation, so a moving branch cannot change later evidence. The deterministic quality-first gate rejects blocker-recall regressions, new false positives on clean controls, and any weighted-score regression. Repeated evidence (RUNS>=3) is required for promotion; RUNS=1 is diagnostic-only. CE is the external comparator. Cost and latency are tiebreakers and never compensate for quality loss. promote stops the loop; with --apply the authorized frozen bytes are transactionally applied to the canonical .claude/skills/ trees and their shipped mirrors as an ordinary working-tree diff — committing, CI (shipped-skills-sync, skills-steering), and the PR merge stay human. keep_incumbent feeds that generation's trajectories to the next proposer. --initial-overlay skips the generation-0 proposer to benchmark a hand-written candidate; --proposer-model upgrades only the diagnosis session.

Learning queue. Live skill runs never self-edit (see each skill's "Skill feedback" section) — instead they may append one-line JSON notes to workflow_bench/learnings.jsonl (gitignored, machine-local like the transcripts they complement). The proposer reads the queue as hints, not ground truth: a learning only reaches a shipped skill by surviving the same paired benchmark as any other candidate.

For ad-hoc use, run the driver on the existing re-evaluation triggers (model/harness change or 90-day staleness). The repository workflow runs a deliberate weekly drift check: dispatch defaults to three concurrent cells of one task; scheduled concurrency still requires GITNEXUS_EVOLUTION_WORKERS=3 after a clean proof. --workers is bounded to 1–8 before paid work starts. --generations remains the only loop bound.

Free-model setup (no paid tokens)

Headless Claude Code honors ANTHROPIC_BASE_URL, and litellm (already an eval dependency) can proxy its Anthropic-compatible /v1/messages to a model that costs nothing — a hosted OpenRouter :free variant or a fully local Ollama model. Config template: free-model.litellm.yaml.

# 1. Choose a proxy master key and start the proxy
#    (pick/edit a model route in the yaml first; keep the proxy on loopback —
#    anyone who can reach the port with this key can spend the backend quota)
export LITELLM_MASTER_KEY="$(openssl rand -hex 16)"
uv run --locked --with 'litellm[proxy]' litellm --config workflow_bench/free-model.litellm.yaml --port 4000

# 2. Point the benchmark at it
uv run --locked --extra dev python -m workflow_bench.runner \
  --tasks workflow_bench/tasks.scenarios.yaml --runs 3 \
  --base-url http://localhost:4000 --anthropic-api-key "$LITELLM_MASTER_KEY" --model free-coder

OpenAI API keys

Claude Code still speaks Anthropic /v1/messages. For a paid OpenAI backend, do not point --anthropic-api-key at an sk-... OpenAI key. Export the OpenAI key and use OpenAI model ids; the driver starts the proxy itself:

export GITNEXUS_BENCH_OPENAI_API_KEY="$OPENAI_API_KEY"
PROVIDER=openai ./workflow_bench/run-evolution.sh

The GitHub skill-evolution workflow accepts GITNEXUS_BENCH_OPENAI_API_KEY on the gitnexus-evolution environment. Dispatch with provider=openai to force that backend even when an Anthropic token is also configured (otherwise auto keeps using Anthropic whenever that secret exists). Claude default model inputs are then rewritten to gpt-5.6-sol; every proposer and benchmark session receives --effort xhigh.

Caveats, honestly:

Both arms run on the same model, so the comparison stays fair at any quality level — but small free models follow skills less reliably, so expect lower resolve rates and noisier savings than on frontier models. Treat free-model runs as directional; confirm headline numbers with a small paid run.
Through a proxy cost_usd reads ~0, and the CLI's token counts are NOT a substitute "real metric": they cover only the main-loop session, so subagent spend is invisible to both. For efficiency ranking, prefer a paid run gated on cost_usd, or sum per-session usage from the transcripts (~/.claude/projects/<cwd-slug>/<session_id>.jsonl, deduplicating events that share one message.id).
OpenRouter :free variants are rate-limited (~50 req/day on a fresh account); local Ollama has no limits.
Codex users: codex exec --oss runs local models for free too, but this runner is Claude-Code-first; a codex engine is a straightforward extension (parse its --json usage events).

Historical ground base (2026-07-11, Claude Code 2.1.207, unnamed model, n=1/cell)

These figures predate mandatory model provenance and are retained only as historical calibration. They are not eligible promotion evidence and must not be combined with current named-model runs.

Three task classes × three arms, single-repo (GitNexus itself). Every arm resolved every task — at this difficulty, pass/fail quality is saturated and the comparison is pure cost:

task (class)	arm	resolved	cost $	wall	turns	vs baseline cost
trivial-version-alias	workflow	1/1	9.16	16m	63	−333%
trivial-version-alias	baseline	1/1	2.11	2.8m	16	—
inv-bug-pdg-note	workflow	1/1	14.56	21m	83	−331%
inv-bug-pdg-note	workflow_direct	1/1	5.23	7.5m	32	−55%
inv-bug-pdg-note	baseline	1/1	3.38	4.7m	22	—
inv-feature-list-repos-filter	workflow	1/1	13.22	19m	84	−211%
inv-feature-list-repos-filter	workflow_direct	1/1	4.87	4.8m	38	−15% (wall +14% faster)
inv-feature-list-repos-filter	baseline	1/1	4.25	5.5m	32	—

What the ground base says, honestly:

The full plan→work workflow never paid for itself at this task scale (tasks a baseline agent finishes in ≤35 turns). Its fixed cost — freshness gate incl. analyzer rebuild + re-index, a full 13-section plan, work-phase re-anchoring — is ~$9–11 per task and needs much larger tasks, plan-reuse (one plan, several executors/sessions), or plan-as-deliverable flows to amortize.
workflow_direct is close to baseline (−15% to −55% cost, once slightly faster wall) — the execution discipline (impact-before-edit, detect_changes-before-commit) is cheap. It produced noticeably more test coverage than baseline for near-equal cost on the feature task.
Quality didn't differentiate because nothing failed. The regime where the workflow should win on resolve rate — cross-module tasks where baselines flail — is the unmeasured cell (cross-module-parse-retry), and the next thing to measure, ideally with --runs 3+ on a free backend.
Caveats: n=1 per cell, one repo, one model; churn numbers from this run predate the intent-to-add/exclude-plans churn fix, so they are not comparable across arms and are omitted above.

Routing implication (to revisit as cells fill in): for tasks up to this size, gitnexus-work direct mode or a plain agent is the cost-optimal route; reserve full gitnexus-plan → gitnexus-work for cross-module work, multi-session execution, or when the plan document itself is a deliverable. If a future run shows the workflow flattering itself here, distrust the run.

Cross-module cell (same day, optimized skills, n=1)

The hardest class — retry-with-backoff across the worker-pool/pipeline seams, transient-vs-deterministic classification:

arm	resolved	cost $	wall	turns	churn
workflow	1/1	18.32	37m	107	4/+373/−17
workflow_direct	1/1	9.53	15m	52	11/+244/−66
baseline	1/1	18.03	34m	98	6/+345/−69

(The workflow_direct row is the clean re-run under clone isolation — the original was contaminated, see the integrity note below.)

This is the cell where the discipline pays. workflow_direct — the execution skill without a planning pass — beat a plain agent by 47% cost and 56% wall time on the hardest class while resolving: impact-first navigation and gated commits prevented the flailing that baseline's 98 turns represent. The full workflow's premium vanished (−1.6% vs baseline; −211%..−333% on smaller classes) — fixed costs amortize here, with a less destructive diff and a durable plan artifact — but it didn't beat direct mode on any measured axis with the plan consumed only once. Resolve rate stayed tied across all cells; the savings story belongs to the execution discipline, and the planning pass is bought for its artifact (multi-session reuse, review, handoff), not for same-session token savings.

Benchmark integrity note (why churn earns its keep): the original workflow_direct cell reported an impossible 28-turn/$4.71 solve with churn byte-identical to the workflow arm — because git worktree add shares the ref namespace, the workflow arm's slug branch survived worktree removal, and the direct arm found and adopted the finished work. Fixed by giving every arm an isolated git clone --no-local --no-hardlinks with no object alternates (agent-created refs and storage die with the clone); the leaked branch was deleted and the cell re-measured. Treat identical churn fingerprints across arms as a contamination alarm.

Optimization re-measurement (same day, commit `830a0459`)

After category-priced plan forms (compact ≤80 lines + mini-pack), category-priced freshness (accept for compact classes), per-category turn budgets, and the work-phase HEAD==pin fast path, the same inv-bug-pdg-note workflow cell re-measured (n=1):

	ground base	optimized	delta
resolved	✅	✅	—
cost $	14.56	11.70	−20%
turns	83	72	−13%
output tokens	59,789	53,345	−11%
cache_read	6.64M	5.07M	−24%
wall	21m	25m	+15%

Verified in-transcript: the compact form fired (115-line plan vs 209 for a simpler task pre-optimization), the plan session dropped 72→49 turns, and NO analyzer rebuild/re-index executed. All savings came from the plan side; this run's work session drew a long test-debugging tail (hence the wall regression) — single-run variance cuts both ways. The optimizations narrow the gap but do not flip the regime: the workflow remains ~3.5× baseline on this task class, so the routing rule above stands unchanged.

Writing good tasks

See tasks.scenarios.yaml. Small enough to finish headless, real enough to require investigation — the workflow's savings come from not re-reading and not re-investigating, which trivial tasks never exercise. Keep verify as a model-visible authored-test quality signal, and add an independent oracle whose source files live under workflow_bench/oracles/. Oracle commands must run only files staged beneath $GITNEXUS_BENCH_ORACLE_ROOT; for Vitest, include the shared vitest.config.mts as an oracle file and pass it explicitly with --config. Prefer verify commands that use the repo's own npm scripts (they carry build pre-hooks).

Relation to the SWE-bench harness

The rest of eval/ benchmarks GitNexus tools inside a litellm agent loop (baseline vs graph-enhanced). This module benchmarks the skill workflow inside the real CLI harness those skills ship for. Different question, same spirit: measure, don't assume.

README.md Unescape Escape