b1d87c1f33
* fix(eval): cut skill-evolution wall clock without shrinking the gate
Reuse matching incumbent/CE cells, sanitize each SHA once, and default
dispatch workers to 3 so weekly review generations finish inside the
EventBridge window. Cap the sweep from leftover instance uptime so a
Friday dispatch still uploads evidence.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(eval): pipeline graph setup and correct the wall-clock cost model
The evolution sweep paid `sanitize` + `analyze --pdg --index-only` for every
unique task SHA on the critical path, one at a time, with nothing overlapping.
`_run_sweep` now starts the next unpaid SHA's clone template and graph snapshot
on a prefetch thread as soon as the current task's cells are dispatched, so
every SHA but the first hides behind a paid session wave. The thread is joined
before that SHA is used and before the trees tempdir is torn down, and a
prefetch failure is recorded against the SHA exactly as an inline failure is.
Tasks whose cells are all reusable comparator rows are not prefetched: they
never build a graph, so priming one would be pure cost.
Adds `measure_evolution_cost.py`, the cost model behind these numbers. It reads
the review corpus, the evolve defaults, and the workflow's workers default —
it does not start a session. Its first version charged `copy_isolated_tree`
once per paid cell, serially. `run_cell` clones inside its own pool worker, so
the clones in a wave overlap and only one is on the critical path per wave;
the model now charges `ceil(cells / workers)` waves.
Estimated review generation at workers=3: cold 21570s, weekly 7710s.
Wall clock is quantised by `ceil(cells_per_task / workers)`. A cold review task
is 9 cells, so workers=4 buys the wall clock of workers=3 and pays host
contention for it. Documented in the workflow's rollout checklist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(eval): price the benchmark against measured cell durations
The cost model assumed every cell runs the 1140s mean. Cells are not uniform:
the 41 rows in Actions run 33912693948's artifact are 826s at the median,
1262s at the mean, 2976s at p90, with two pinned at the 5400s session ceiling.
A wave waits for its slowest cell, so a mean understates every concurrent
schedule — the previous model called workers=3 cold 5.99h when the same
schedule against real durations is 10.33h.
session_durations.json carries the sample in submission order with its
provenance and its caveat: every cell in that run returned unusable evidence,
so the durations are real but a clean run may sit lower. It is the only live
artifact; the 2026-07-22 green run's has expired.
The model now simulates the schedule cell by cell rather than multiplying a
mean by a wave count, averaged over all 41 rotations of the sample so no
single alignment between sample order and cell index decides the answer. It
prices today's barrier (wave_makespan) against a continuously fed pool
(fed_makespan) and reports both, and it charges the proposer session — one
per generation, measured at 344.7s — which it had been omitting entirely.
Measurement only; no runtime behaviour changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(eval): price arms separately and stop inventing setup constants
Two errors in the model, both found by auditing it against the artifact it
claims to describe.
The arms are not interchangeable. `candidate_review` runs 1416s at the mean
against `review`'s 1204s and `ce_review`'s 1176s, and the weekly lane pays the
candidate arm and nothing else — reuse skips both incumbents. Pricing weekly
from a pooled sample charged it for arms it never runs: weekly is 4.59h, not
the 3.65h a pooled sample reported. Cells are also submitted run-major and
arm-minor, so at workers=3 every wave holds one cell of each arm and the
slowest arm sets the wave; the model now builds cells in that order.
The setup constants were invented. GRAPH_ANALYZE_SECONDS=600 and
TEMPLATE_SANITIZE_SECONDS=180 charged 3900s of per-SHA setup for a cold run —
more than the entire non-session time of the source run, which was 2541s for
41 cells and 5 SHAs. `duration_s` is the sum of a cell's Claude sessions
(runner_sessions.py), so that 2541s residual is every clone, graph build,
sandbox and teardown the sweep paid. The model now charges the measured
residual per cell, 62.0s, and no longer credits clone templates or graph
prefetch: both landed after that run and there is no measurement of them yet.
The residual bounds what they can be worth.
Cold 37452s (10.40h), weekly 16541s (4.59h), against a fed pool at 31683s and
16541s. Measurement only; no runtime behaviour changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(eval): charge sweep overhead where more workers cannot dissolve it
Three defects, found by auditing the model against the artifact again.
The overhead was charged inside the schedule. session_durations.json claimed
the residual was charged "per cell and serially - the pessimistic reading",
but task_cells folded it into each cell's duration, where the pool then
divided it by the worker count. The residual mixes per-cell work the pool
really does divide with per-SHA graph setup it cannot, and the artifact cannot
separate them, so it now sits outside the schedule: cold 11.09h, not 10.40h.
Alignment averaging weighted the shortest sample twice. The arm samples are 13,
14 and 14 long and the average ran over max()=14 offsets, so candidate_review's
first cell was counted twice and its last never. Averaging over lcm()=182
offsets weights every arm's sample evenly.
The wall assumed all 54 cells run. Replaying the sample's own error_kind
sequence through today's systemic_outage_streak trips the outage breaker at
cell 5 of 41. The source run executed all 41, so its runner did not break on
that sequence, but the current one would: these numbers price a HEALTHY sweep,
and a sweep with the sample's failure profile never reaches them. Stated on
generation_seconds and recorded next to the sample it qualifies.
Measurement only; no runtime behaviour changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(eval): give the review agent somewhere it can actually write
Every review cell in the last recorded generation returned unusable evidence.
Not some — all 41, across all three arms and all six tasks, at $3653 for the
run. The transcripts say why, 127 times across 35 of 35 sessions:
EROFS: read-only file system,
open '/workspace/review-output.json.tmp.2.90a76e583b0c'
The review arm mounted the artifact as a writable FILE at
/workspace/review-output.json while binding /workspace read-only. The Write
tool writes atomically: it creates `<target>.tmp.<n>.<hex>` beside the target
and renames it. The parent was read-only, so the temp create failed and the
artifact was never written. A writable file inside a read-only directory is
not writable to anything that writes atomically. Agents tried
/proc/self/root/workspace/... and /proc/1/root/workspace/... to get around it;
all 41 artifacts came back 0 bytes.
The artifact now lives in its own writable directory bound at /review-output,
outside the workspace. That is what a rename needs, and it lets the workspace
get stricter rather than looser: the review phase may now change nothing there
at all (enforce_phase_workspace gained allowed_artifact=None), where before it
was entitled to one path inside it. The file is no longer pre-created — the
agent writes it, and absence is now meaningful evidence.
parse_review_output reported every one of these as "review output is not valid
UTF-8 JSON". The file was empty, and its except folded OSError, UnicodeError
and JSONDecodeError into that one string, so a sandbox that made writing
impossible was indistinguishable from an encoding fault. That is why this read
as an agent-quality problem for fifteen consecutive non-green runs. Each cause
now names itself: never written, empty, not valid UTF-8, not valid JSON with
the decoder's position. run_arm also keeps the FIRST error_detail, as it
already did for error_kind, so a phase-boundary violation is no longer buried
under the parse failure it causes.
The test double conflated sandbox.private_root with the clone, which put the
artifact directory inside the workspace and would have hidden the stricter
check. Regression tests pin the mount shape in the generated bwrap argv, the
contract path in the prompt, the four parse diagnostics, and the
untouched-workspace contract.
Verified by unit tests only: this container has unprivileged user namespaces
disabled, so bwrap cannot run here and the mount was not exercised end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(review): close the artifact path in every layer that gates it
Code review of this branch found the relocated review artifact was fixed in the
bwrap mount and nowhere else. Four independent layers decide whether the agent
can write it, and three still named the old location.
Claude Code applies its own filesystem policy to its own tools, and
build_claude_settings listed only /workspace, /tmp and /home/agent under
allowWrite with denyRead ["/"]. The artifact used to live under /workspace, so
this list was correct until it moved. SANDBOX_REVIEW_OUTPUT is now in allowWrite
and allowRead; without it the bwrap bind grants a write the CLI then refuses.
The task corpus still ran `test -s review-output.json` from the workspace, in a
separate sandbox invocation that never sees the artifact mount. Every review
cell would have been stamped verify-failed with resolved=False no matter how
good the review was, which also made those rows permanently unreusable and so
silently disabled this branch's own comparator reuse for review arms. The verify
and hidden-oracle commands now read the location from
GITNEXUS_BENCH_REVIEW_OUTPUT and get the directory bound read-only, mirroring
the mount-plus-env-var shape _run_hidden_oracle already used.
host_text and host_path did not translate the new path, so the host-unsafe
backend told the agent to write somewhere that exists on neither backend.
Adding the mapping exposed a second defect: host_text substituted every
occurrence of a target, and "/review-output" appears twice in
"/review-output/review-output.json" - once as the directory and once inside the
filename. Matching is now anchored to a path boundary.
Comparator reuse had three ways to accept evidence it should have rejected. A
row with no runtime_digest passed the drift lock because the guard only compared
when both sides were bound, and the branch's own test asserted that as correct;
absence is now a mismatch and the test states the rule. materialize_reused_row
overwrote recorded_at with the copy time while the age check read that field, so
a row copied forward each generation refreshed its own clock and never aged out;
the first measurement time is now preserved and aged against. A future-dated
stamp passed a one-sided bound and is now rejected as corrupt.
RUNTIME_DIGEST never reached the runner at all: runner_environment builds a
fixed dict and process_control replaces the child environment wholesale, so the
digest the workflow exports was dropped and the lock it feeds was inert. The
instance-window deadline was also checked only after run_proposer returned,
buying a proposal the generation had no room to benchmark.
The graph prefetch thread was started without copy_context, so it never saw the
cancellation ContextVar the rest of the sweep shares, and the outage breaker
returned without setting cancel_event - together, a tripped breaker would block
on joining a prefetch that was never told to stop. Both fixed, with outage
checked before cancellation at the two exits so an outage keeps exit 1 instead
of becoming a Ctrl-C's 130.
Both bwrap canaries that actually execute a write still bound the pre-fix shape
against a file this branch no longer creates, so they would have errored rather
than caught anything. They now bind the directory and write atomically - temp
file beside the target, then rename - which is the exact operation that failed
with EROFS. A source-text assertion over inspect.getsource(run_arm) was replaced
with one that inspects the real mount, and a wall-clock assertion was pinned to
a fixed monotonic clock.
Not applied, and why: binding task-asset and dependency digests into comparator
reuse needs asset snapshots prepared before the reuse decision rather than
inside the per-task loop, and shipping the comparison without that would add a
guard that silently never fires. Forcing a paid canary cell per incumbent arm
and folding reused rows into the outage streak are behaviour decisions, not
fixes. Clone-template reuse still has no test. The cost model's per-cell
residual still shrinks with arm count, overstating weekly savings by at most the
2541s residual; the docstring now says so rather than inventing a split.
585 eval tests pass, ruff clean, 29 workflow contract tests pass. The two
test_model_gateway.py failures are pre-existing and fail on main.
* fix(review): bind reuse to its environment and keep the health canary real
Applies the five findings the previous review round left open.
Comparator reuse ignored the environment a row was measured in. TaskReuseBinding
carried the task and oracle identity but not the task-asset or sandbox-dependency
digests, and this branch itself changes sandbox_dependencies in the review
corpus - so a reused comparator could be measured against one dependency set and
compared against a candidate built on another, handing the gate a false
comparison. Closing it needed the digests to exist before the reuse decision, so
asset snapshots are now prepared for every task up front instead of lazily
inside the per-task loop. That also removes the concurrent TaskAssetCache.prepare
the prefetch thread could otherwise race, which the file's own "plain dict,
read-then-write race" comment warned about. Both digests fail closed on either
side, matching the runtime digest.
The broken-incumbent canary could not fire when reuse was working. It read
`resolved`, which counts reused rows, so an arm whose cells were all reused
always looked healthy - in precisely the run where a broken environment would go
unnoticed. aggregate now also reports `resolved_fresh` and the canary reads it.
That count would be vacuous if an arm were reused end to end, so the sweep keeps
one paid cell per incumbent arm and says which one it kept.
Reused rows did not participate in the outage streak, so a run of failures could
carry across them and trip on stale history. A reused success now resets the
streak the way a paid success does.
The cost model charged sweep overhead per cell, which credited a weekly
generation for shrinking work it still performs: it pays one arm instead of
three but builds exactly the same graphs. Overhead is charged per SHA now.
Weekly is 5.20h rather than the 4.80h the per-cell rate reported; cold is
10.86h. The residual still cannot be split between per-SHA and per-cell work
from one artifact, so session_durations.json records that assumption and the
direction it errs in, rather than leaving a number nobody can trace.
Clone-template reuse - the branch's core speedup, taken on essentially every
multi-cell sweep - now has a test that builds a real sanitized template, asserts
the cell runs against the copy with the template's HEAD, and fails if run_cell
re-clones. A second test asserting only on a namespace built inside the test was
written and deleted: it exercised nothing, which is the failure this review
round penalised elsewhere.
589 eval tests pass, ruff clean, 29 workflow contract tests pass. The two
test_model_gateway.py failures are pre-existing and fail on main.
* refactor(eval): consolidate duplicated harness logic after the review round
Simplification pass over the branch. Behavior-preserving throughout; three
reviewers, nine findings applied, two skipped.
The review-artifact block in _run_hidden_oracle was unreachable. That function
runs only in run_arm's non-review branch, while the directory it probes for is
created only in the review branch, and each sandbox serves exactly one arm - so
`review_artifact.parent.is_dir()` could never be true. It was added an hour
earlier to make the hidden oracle resolve the moved artifact; the oracle never
runs for review tasks, so the guard was dead on arrival. Deleting it also
removes the duplication it had with the verify-command wiring.
EXCLUDED_ERROR_KINDS is now one definition. runner.py and comparator_reuse.py
each carried the same six-member frozenset, kept in sync by a comment. Only one
direction is possible: runner already imports from comparator_reuse, so the
reverse import fails at module-init with a circular-import error. That is now
stated where the alias lives, so nobody tries it the other way.
ensure_task_graph and prefetch_next_graph shared ten keyword parameters, passed
through two call sites and forwarded whole between them. They now take a
GraphBuildEnv, mirroring TaskCellContext, which already bundles per-cell state
in this file. Its ready_keys() replaces an inline four-set union at the call
site.
Smaller consolidations: _sha256_file's hand-rolled chunk loop becomes
hashlib.file_digest (3.11+, already used in runner_artifacts); _copy_owner_only
reuses task_assets._write_all and COPY_CHUNK_BYTES instead of repeating the
short-write retry; its stat-then-open existence check becomes the O_EXCL failure
it was already relying on, which is atomic rather than merely narrow; and
runner_environment reads the digest through comparator_reuse.current_runtime_digest
instead of re-parsing the environment variable.
Three test docstrings summarised the branch's own history ("the branch's core
speedup", "the regression that produced fifteen runs") rather than the invariant
under test. Rewritten to state the constraint, which is what survives the merge.
Repaired the indentation left behind by the outage-streak edit and flattened the
prefetch dispatch from three nested conditionals to one.
Skipped: consolidating comparator_reuse._real_directory onto proposer_sandbox's
same-named helper - they differ, the sandbox one rejects any symlink in the
resolved path while this one checks only the leaf, so sharing it would tighten
behavior rather than preserve it. That needs a decision about which policy the
reuse path wants, not a simplification.
589 eval tests pass, ruff clean, 29 workflow contract tests pass. Unrelated and
pre-existing: two test_model_gateway.py failures, and
test_process_control.py::test_timeout_kills_term_ignoring_descendants_before_they_write,
which is a TERM-to-KILL timing flake (passes 2 of 3 in isolation) in a file this
branch does not touch.
* refactor(eval): name the reuse directory check for the promise it makes
The simplification pass left one finding open: comparator_reuse and
proposer_sandbox both defined `_real_directory`, same name and same shape, with
different guarantees. The sandbox one rejects every symlink hop in the path; the
reuse one checks only the leaf and resolves through parents. Sharing the name
invites a consolidation that would silently tighten one of them.
They should not be merged, so the name stops claiming they could be.
proposer_sandbox guards a mount root, where a symlink hop changes what an
untrusted session is handed. comparator_reuse guards a data directory whose
contents are already validated one file at a time - reads go through
_regular_file, which lstats and rejects symlinks, and writes through O_NOFOLLOW.
A symlinked parent therefore grants nothing those guards do not already cover,
while refusing one would reject a symlinked artifacts directory or macOS's /var
for no gain.
Renamed to _resolved_directory, with the reasoning recorded at the definition,
and a test that pins both halves: a symlinked parent is accepted and resolved, a
symlinked leaf is still refused. Behavior is unchanged.
591 eval tests pass, ruff clean. The two test_model_gateway.py failures are
pre-existing and fail on main.
* test(eval): measure the sweep scheduler instead of modelling it
measure_evolution_cost predicts wall clock from a model of what
sweep_task_cells does. This runs the real thing - real threads, the real wave
barrier, the real outage breaker - with only the paid agent session replaced by
a sleep, and times it.
Durations are the measured per-arm samples divided by 5000, so a 1416s cell
takes ~0.28s. The shape is kept on purpose: the median cell is 826s against a
5400s ceiling, and that spread is the entire reason a barrier costs anything.
Uniform random sleeps would erase the effect under test. All schedulers consume
one identical seeded plan, so a comparison cannot be an artifact of one of them
drawing luckier cells.
The model survives contact: it tracks real execution within about 10%, and
workers=1 - which runs without a pool at all - sits at 0.95, so the residual
above 1.0 at higher worker counts is per-wave thread overhead rather than a
modelling error. Two structural claims that were arithmetic are now observed.
Weekly is flat from workers=3: 3.59, 3.59, 3.59, 3.60, 3.59, 3.59 across w=3..8.
workers=4 buys nothing over workers=3 on cold, 7.68 against 7.78.
Two prototype schedulers are measured beside it, deliberately before any
production code exists. A continuously fed pool per task is worth more than the
model claimed on cold, -27.3% against a predicted -17.9%, and exactly nothing on
weekly, +0.0%, because a weekly task is one wave with nothing to feed. One pool
across all tasks beats both: -40.7% weekly and -42.9% cold at workers=3, rising
to -65.7% and -63.9% at workers=8. It also subsumes the fed pool, since packing
across tasks is a fed pool.
That reorders the backlog. Cross-task packing moves from second to first: it
dominates on both profiles, and it is the only thing that moves weekly at all.
Raising the worker count is worth nothing until it lands - under the barrier
weekly does not improve from w=3 to w=8, and speedup against serial is 1.58x for
three workers and only 2.40x for eight.
The bound on all of it: sleeping threads do not contend. Real sandboxed sessions
compete for CPU, page cache and disk, and the duration sample was itself
measured at workers=1, so it carries no contention either. These speedups are
upper bounds. The ordering is trustworthy because the schedulers were compared
under identical conditions; the magnitudes are not. The packed prototype is also
a bare ThreadPoolExecutor with no breaker folding, no per-task graph lifecycle
and no reuse binding - which is the actual cost of building it, and is not
measured here.
* test(eval): carry the sweep invariants into the packed prototype
The first packed prototype was a bare ThreadPoolExecutor. It reported -43% and
none of the invariants the shipped scheduler holds, so it priced an idea nobody
could ship. This one carries them: a global submission order continued across
task boundaries, in-order folding, the real outage breaker, and per-task graph
readiness gating behind a serial builder.
The fidelity check first reported the two schedulers tripping on different
cells, 17 against 16. That was my instrumentation, not a divergence -
sweep_task_cells folds an entire wave before it evaluates the breaker, so the
last cell folded is not the cell that tripped. With the harness mirroring the
breaker's own evaluation the two agree exactly, across failures starting at
cell 0, 4 and 12, with overrun inside the workers-1 bound the wave docstring
promises.
Two results worth the exercise.
Head-of-line blocking, not the barrier, is what a naive in-order design pays.
Holding submission to `workers` cells beyond the fold pointer leaves the
faithful scheduler at -8.1% cold and -2.7% weekly: one slow cell stalls the
pointer, the window cannot slide, and it reproduces the wave almost exactly.
That is the number to quote if anyone proposes the obvious implementation.
But the overrun bound turns out to be set by the worker count, not the window.
Only `workers` cells can be running when the breaker trips; everything queued
behind them short-circuits on the halt flag. Overrun is 3 at an unbounded
window exactly as at 6, and the trip cell never moves off 16. So H2 does not
have to trade breaker fidelity for speed - a wide window takes -42% with the
semantics intact. The tension I assumed was there is not, and window=12 already
captures 97% of it.
Still an upper bound: sleeping threads do not contend, and the sample was
measured at workers=1. What this establishes is that the invariants are
affordable, which was the thing blocking H2. Not built here: the trees tempdir
lifecycle, reuse-row binding, and the cancel_event path.
591 eval tests pass, ruff clean.
* test(eval): put the scheduler comparison under real CPU contention
Every Phase 2 number so far came from sleeping threads, which contend for
nothing, against a duration sample measured at workers=1, which contains no
contention either. That was the standing caveat on the whole result, so this
measures it.
A cell now waits for its API share and then burns a fixed number of sha256
rounds in a subprocess. Work-bounded rather than wall-clock bounded, so it takes
longer when cores are busy - that is the effect under test. A subprocess because
Python threads burning Python would measure the GIL rather than the machine.
Calibrated at 519k rounds/s, stable within 2% across three probes.
The first run of this was worthless and is recorded as such: on a 24-core host
with 3 to 6 workers nothing ever contends, since cpu_fraction 0.5 at 6 workers
is about 3 cores of demand out of 24. It measured an absence. Re-run pinned with
taskset to 4 and 2 cores.
The packing advantage survives. It holds between -40% and -47% across every host
size and CPU fraction tested, including a genuinely oversubscribed 2-core box at
cpu_fraction 0.5 with 6 workers.
But contention erodes packing more than it erodes waves, for a structural
reason: packing is what creates the concurrency. Moving from 24 cores to 2 at
cpu 0.5 and 6 workers, the faithful scheduler slows 13% while the wave slows
3.7%, and the gain narrows from 45.0% to 39.8%. Packing and a higher worker
count are therefore not independent wins - packing spends the contention
headroom first, so raising workers has to be re-argued after it lands rather
than added to it.
Three things this still does not measure, and they bound the result. The real
CPU fraction of a benchmark cell is a guess informed by roughly 180 tool calls
per session; nobody has profiled one. The evolution runner's core count decides
which column applies and is unknown here. And the burn is sha256, pure CPU,
while real cells run vitest and analyze, which are memory and IO heavy - so this
is a floor on contention, not a ceiling.
591 eval tests pass, ruff clean.
* perf(eval): add a packed sweep scheduler, and correct the bound I claimed for it
sweep_task_cells finishes one task before starting the next and drains a wave
before refilling it, so a task with fewer cells than workers leaves workers
idle and one slow cell stalls its whole wave. sweep_packed_cells feeds every
task's cells through a single pool instead. Measured against the review corpus
it is worth about 40% of a cold sweep, and it is the only change that moves a
seeded weekly run at all - there a task is three cells and a wave is never full.
The breaker keeps its exact meaning. Cells carry a total submission order
continued across task boundaries, a folder walks results in that order, and
consecutive systemic failures are counted there, so a doomed run aborts on the
same cell it would have under waves. Verified at three failure positions.
This commit also corrects a finding from the Phase 2 prototype. I claimed the
overrun bound was set by the worker count rather than the submission window,
and that packing therefore cost nothing in breaker fidelity. That was derived
from a window sweep that only ever injected failures at one position. Driving
the real function at other positions shows the halt flag does not bound overrun
at all: the folder walks in order, so a slow early cell lets workers race ahead
and the trip is detected after those cells have already paid. An unbounded
queue overran by 11 cells where waves overrun by 2.
So the window is load-bearing and the trade is real, measured at workers=3 with
failures injected at four positions:
window 3 -> -8% wall, overrun 2 (the wave scheduler's own bound)
window 6 -> -27% wall, overrun 4
window 12 -> -42% wall, overrun 9
window 54 -> -44% wall, overrun 11
Overrun is wasted paid sessions at roughly $70 each. The default multiplier is
2, keeping the worst case within twice the wave bound while taking most of the
gain; the curve is in the constant's comment so raising it is an informed
decision rather than a guess.
Not wired in yet: _run_sweep still calls sweep_task_cells per task. Moving the
per-task graph, trees tempdir and reuse binding out of that loop behind
await_ready is the larger and riskier half, and it belongs in its own change.
595 eval tests pass, ruff clean.
* fix(eval): judge harness health on execution, not on how many tasks resolved
broken_incumbent_arms infers "the environment is broken" from an arm resolving
zero tasks. That inference does not hold: a reviewer can be wrong about every
task in a hard corpus while every process, mount and capture worked perfectly.
Actions run 33962002890 is exactly that shape - 51 cells, all resolved=False
with error_kind=oracle-failed, median score 0.212, and a healthy harness.
Someone already knew this, and patched it by excluding review arms at the call
site. That leaves the unsound inference in place for workflow and
workflow_direct, and leaves review arms with no health check at all - so the
run that genuinely was broken, 33912693948, where the mount made an atomic
write impossible and all 41 artifacts came back empty, could not have been
caught here either.
So this replaces the inference rather than adding another exemption. aggregate
now classifies fresh rows into execution failures (the process or its tooling
did not complete), evidence failures (it completed but produced nothing
trustworthy or scoreable), and admissible measurements. An arm is unhealthy
only when it has fresh attempts, zero admissible measurements, and at least one
execution or evidence failure. Resolution count is no longer consulted. Arms
with only reused rows report current health as UNKNOWN rather than good.
With the inference corrected, review arms are checked again, which is what lets
the empty-artifact case be caught at all.
Deliberately unchanged: comparator reuse eligibility, quality denominators,
promotion thresholds, model settings, skill prompts and scheduler behaviour.
Failures that stop being called infrastructure failures still surface in the
counts and reasons - an agent-originated failure must not vanish from reporting
because it was reclassified. broken_incumbent_arms and its tests are left in
place; deleting behaviour belongs in its own change.
Seven regression tests, built from both runs' shapes and labelled as
reconstructed from logged observations, since 33962002890's results.jsonl did
not survive the instance shutdown. They pin: a badly-scoring reviewer is
healthy; an all-zero score is still a valid negative; empty artifacts are
unhealthy; one admissible cell keeps an arm healthy while its failures stay
visible; reused rows alone leave health unknown; reused successes do not mask
fresh failures; and a parseable artifact does not excuse a failed session.
602 eval tests pass, ruff clean.
* fix(eval): pin the health guard below the breaker, and stop calling mixed runs healthy
Two corrections to the health-classification patch.
The regression I wrote could not have proved what it claimed. A fixture of 41
empty artifacts aborts through the outage breaker long before finalization:
review-evidence-invalid is in SYSTEMIC_ERROR_KINDS and the limit is 5, so it
trips at cell 5 through the pre-existing path. It demonstrated failure
detection, not the new guard. The decisive test now uses ONE fresh unusable
cell, asserts the streak stays under the breaker threshold, and only then
requires finalization to abort - leaving the new check as the only thing that
can catch it. Removing the call makes that test fail; restoring it passes.
The accurate defect statement is narrower than the last message claimed. Review
arms were excluded from the final incumbent-health check while the consecutive-
failure breaker gave them separate, partial coverage. They were not unguarded.
Second: "one admissible cell plus two execution failures" was asserted as
healthy. That converts "not wholly unusable" into "ran reliably", which is how
a partly-broken sweep passes review. Arms now report UNKNOWN, OBSERVED_OK,
DEGRADED or UNUSABLE. Only UNUSABLE is fatal, so eligibility and promotion are
untouched - this changes what is reported, not what is allowed.
The guard is extracted as enforce_measurement_health so it can be driven
directly, and it now reports a status line per arm. It names no cause: an empty
artifact establishes that evidence is unusable, not that a mount rejected the
write, so it prints cause=undetermined rather than guessing EROFS. It still
runs after report.md and promotion.json are written, so a failing sweep leaves
its evidence behind.
ce_review is named explicitly at the call site. It is a comparator rather than
a candidate, so it is absent from CANDIDATE_ARMS.values(), and dropping the
review exclusion alone would have left it unclassified.
The wiring test reads _run_sweep's compiled code object for the referenced
global rather than matching source text. It is honest about its limit: it
proves the call exists and would catch its removal, but no test here drives
_run_sweep end to end, which needs bwrap and a sandbox.
broken_incumbent_arms is marked LEGACY and NON-AUTHORITATIVE with removal
tracked. It has no production caller.
608 eval tests pass, ruff clean. The two test_model_gateway.py failures are
test_locked_litellm_translates_messages_to_offline_responses and
test_openai_gateway_never_leaves_proxy_output_on_an_undrained_pipe; both fail
identically on origin/main in this environment, checked directly rather than
carried forward as an inherited label.
* fix(eval): review artifact path, evidence classification, comparator reuse
Extracted from the combined skill-evolution branch. This is the runtime change
set: everything that alters how a sweep executes and what it records. The
packed scheduler and its measurement harness were separated onto
perf/skill-evolution-packed-scheduler, which is purely additive.
Correctness. The review artifact was mounted as a writable FILE inside a
read-only workspace while the agent's Write tool writes atomically - temp file
beside the target, then rename - so the temp create failed EROFS and the
artifact was never written. Four layers gate that path and three named the old
location: the CLI's own allowWrite/allowRead policy, the task corpus verify
command run in its own sandbox invocation, and host_text/host_path for the
host-unsafe backend. Fixing the translator exposed a second defect, since
"/review-output" appears twice in "/review-output/review-output.json"; matching
is now anchored to a path boundary. parse_review_output folded OSError,
UnicodeError and JSONDecodeError into one message, so an artifact that was
never written looked like an encoding fault; each cause now names itself.
Health classification. broken_incumbent_arms inferred a broken environment from
an arm resolving zero tasks, which a reviewer facing a hard corpus falsifies -
Actions run 33962002890 is exactly that shape. Arms are now classified from
fresh execution and evidence outcomes as UNKNOWN, OBSERVED_OK, DEGRADED or
UNUSABLE, and only UNUSABLE aborts. Resolution count is not consulted. The
guard names no cause: an empty artifact establishes unusable evidence, not that
a mount rejected the write.
Comparator reuse. Reuse accepted evidence it should have rejected: a row
without a runtime_digest passed the drift lock, recorded_at was overwritten with
the copy time so a row could outlive its own max_age, and the binding ignored
task-asset and dependency digests although this change alters
sandbox_dependencies in the review corpus. Closing the last one required
preparing asset snapshots before the reuse decision, which also removes the
concurrent TaskAssetCache.prepare the prefetch thread could race.
These three concerns share aggregate() and _run_sweep, which is why they ship
together: separating them further would mean hunk-level surgery on a function
all three modify, and the risk of a silent omission outweighs the reviewability
gain.
592 eval tests pass at this base. The two test_model_gateway.py failures,
test_locked_litellm_translates_messages_to_offline_responses and
test_openai_gateway_never_leaves_proxy_output_on_an_undrained_pipe, fail
identically on origin/main in this environment.
Known gap, and the reason this is not ready to merge: no test drives _run_sweep
end to end. enforce_measurement_health is unit-tested including the
below-breaker unusable case, and the caller wiring is pinned structurally by
reading _run_sweep's compiled code object, but interruption semantics, exit
precedence and persisted artifacts are not exercised through the real path.
* fix(eval): address PR review feedback (#3207)
- aggregate: count admissible rows directly instead of subtracting the
execution and evidence counters, which double-charged a row that is both
a session error and invalid review evidence and could report UNUSABLE for
an arm holding real measurements.
- run_proposer: bound the session timeout by what is left of
--max-runtime-seconds, so clearing the sweep minimum cannot start a
full-length session past the instance window.
- comparator reuse: hold one O_NOFOLLOW descriptor for the size check,
digest and copy, and prove it is the inode that was checked, closing the
swap window a concurrent writer of the reuse directory had.
- Drive the review-artifact mount assertion through run_arm and the
clone-template assertion through run_cell, instead of rebuilding the
expected values in the tests (also removes the CodeQL unnecessary lambda).
- Assert the workflow invokes run-evolution.sh rather than that its YAML
mentions --max-runtime-seconds, which only appears in a comment.
- Correct the parse_review_output failure-mode claim: the fold was empty
artifacts reported as "not valid UTF-8 JSON"; a never-created file raised
FileNotFoundError.
- prettier: wrap the over-long readFileSync call flagged by PR autofix.
Note: pre-existing failure in tests/test_model_gateway.py::test_locked_litellm_translates_messages_to_offline_responses (local LiteLLM proxy never becomes ready in this environment) not addressed by this PR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(eval): address the second round of PR review feedback (#3207)
- Refuse a symlinked `transcripts` component on both sides of comparator
reuse. O_NOFOLLOW guards the leaf only, so a link there redirected the
read or the copy out of the results directory; checked per component as
evolution._require_directory_chain does.
- Base the paid incumbent canary on the cells this sweep PLANS. Reuse
selection accepts any prior run index, so a results directory produced
with more runs left extra keys, the equality never held, and the canary
stopped firing. Extracted as drop_canary_reuse_key and unit-tested.
- Start the runtime clock in main(). --max-runtime-seconds is measured from
/proc/uptime before exec, so parsing, task I/O, preflight and gateway
setup were being handed back to the sweep out of the upload reserve.
- Do not fall back to shutil.copytree when the managed clone copy was
cancelled or timed out; that fallback is for a filesystem that cannot
reflink, and copytree cannot be cancelled.
- Assert the review session's writable mount, not only the verifier's
read-only one: the EROFS bug is about the agent's write.
- Exercise ref isolation in the copy_isolated_tree test rather than
comparing an initial HEAD a shared namespace would also match.
- Point the stale-symlink fixture at the sentinel via os.path.relpath, and
skip the reuse symlink tests where symlink creation needs privilege.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(eval): close the runtime-cap gap and pin the reuse directory
Both were left open on #3207 as approach decisions rather than nits.
Runtime cap: run-evolution.sh computed the budget in its own
`uv run python -c` and passed a number, so the script's remaining
provenance work and the CLI's own startup were spent by nobody and charged
to the sweep — out of the upload reserve the cap exists to protect. The
script now passes --max-runtime-from-instance-window and evolve reads
/proc/uptime itself, on the line after it starts the clock the budget is
measured against, so no interval exists to lose. Also removes an
interpreter start from the script and lets --dry-run print the real argv.
Reuse directory: _real_child_directory lstat-checked `transcripts` and
returned its pathname, so a concurrent writer could rename the directory
and leave a symlink before the name was used again — O_NOFOLLOW guards
only the leaf. Every artifact is now resolved against a held descriptor:
_open_real_directory opens with O_DIRECTORY|O_NOFOLLOW (check and open in
one syscall), and _open_regular / _copy_owner_only take dir_fd. The reuse
path is therefore POSIX-only; _require_openat says so and fails closed,
which the runner already treats as "run a paid cell". _resolved_directory
still tolerates a symlinked reuse root, unchanged and still tested.
evolution._require_directory_chain is still lstat-per-component. It guards
a different surface (candidate overlay reads) that neither review raised,
so it is left alone rather than widened into here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(eval): sample the proposer budget where it is spent, digest what is copied
Four findings against
|
||
|---|---|---|
| .. | ||
| oracles | ||
| review_cases | ||
| __init__.py | ||
| comparator_reuse.py | ||
| evolution.py | ||
| evolve.py | ||
| free-model.litellm.yaml | ||
| gateway_supervisor.py | ||
| learnings.jsonl | ||
| measure_evolution_cost.py | ||
| model_gateway.py | ||
| oracle_assets.py | ||
| process_control.py | ||
| promotion_apply.py | ||
| proposer_sandbox.py | ||
| README.md | ||
| review_scoring.py | ||
| run-evolution.sh | ||
| runner.py | ||
| runner_artifacts.py | ||
| runner_sessions.py | ||
| runner_tasks.py | ||
| runtime_mounts.py | ||
| sanitized_graph.py | ||
| session_durations.json | ||
| simulate_sweep.py | ||
| task_assets.py | ||
| tasks.review.scenarios.yaml | ||
| tasks.scenarios.yaml | ||
Skill benchmark — evolve review quality, measure workflow cost
Measures whether the gitnexus-plan → gitnexus-work engineering workflow
actually saves tokens versus a baseline agent on the same tasks, using real
headless Claude Code sessions. Nothing is estimated: every number comes from
the CLI's own final event in its parent-captured --output-format stream-json
report.
What it compares
| Arm | Sessions | Notes |
|---|---|---|
workflow |
gitnexus-plan on the task, then gitnexus-work on the produced plan |
The skills must be installed (gitnexus setup, or repo-local .claude/skills/) |
candidate_workflow |
same sessions as workflow, with a candidate skill overlay |
Paired with workflow on the same task/ref/model |
workflow_direct |
one gitnexus-work direct-mode session |
The middle option — execution discipline without a planning pass |
candidate_workflow_direct |
same session as workflow_direct, with a candidate skill overlay |
Paired with workflow_direct on the same task/ref/model |
ce_workflow |
ce-plan on the task, then ce-work on the produced plan |
External comparator: the explicitly supplied, pinned compound-engineering plugin's plan→work family |
ce_workflow_direct |
one ce-work direct-mode session |
External comparator paired with workflow_direct |
review |
one gitnexus-review session over an immutable historical PR snapshot |
Emits strict review-output.json; hidden human labels score quality after the session |
candidate_review |
the same review with a gitnexus-review candidate overlay |
Paired with review on the same case/ref/model/runtime |
ce_review |
one pinned ce-code-review session over the same changes |
External comparator paired with both review arms |
baseline |
one session with the identical task text | --disallowedTools Skill so it cannot borrow the workflow; same repo, same MCP tools |
baseline_nomcp |
like baseline, graph tools also disallowed | Separates the workflow-discipline question from the GitNexus-tools question (off by default) |
Every arm runs in a fresh detached git worktree of the task's ref, once per
--runs. The model-visible verify command is recorded as
authored_tests_passed, but cannot certify its own solution: resolved also
requires the task's harness-owned hidden behavioral oracle to pass. Token
savings on a failed task are flagged, not celebrated, and diff churn
(files/+insertions/−deletions vs the starting commit) is recorded as a cheap
over-engineering proxy. Task class labels (trivial → investigation →
cross-module) make the report readable as a routing table: the boundary where
workflow starts beating workflow_direct and baseline is the boundary
lfg's gate and work's direct-mode triage should encode.
Quick start
cd eval
export GITNEXUS_BENCH_ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY"
uv run --locked --extra dev python -m workflow_bench.runner \
--tasks workflow_bench/tasks.scenarios.yaml --runs 3 \
--model claude-sonnet-4-20250514
Scenarios marked expensive: true are skipped unless
--include-expensive is supplied. The report names both selected and skipped
tasks so an omitted cell cannot be mistaken for evidence.
CE comparator arms never discover a user-level plugin. Supply an exact plugin
release explicitly; both flags are mandatory whenever any ce_* arm is
selected:
uv run --locked --extra dev python -m workflow_bench.runner \
--tasks workflow_bench/tasks.scenarios.yaml --runs 3 \
--model claude-sonnet-4-20250514 \
--arms workflow ce_workflow \
--ce-plugin-dir /opt/operator-input/compound-engineering-3.19.0 \
--ce-plugin-version 3.19.0
The runner verifies the manifest version, copies only the plugin manifests, skills, scripts, and assets into a bounded no-symlink snapshot, and mounts that snapshot read-only only for CE arms. Every CE result records its exact plugin version and content-manifest digest.
Output: results/wfbench-<timestamp>/results.jsonl (every run, with session
ids for transcript drill-down) and report.md (medians per task per arm,
plus a savings row: input / cache / output tokens, cost, wall time).
Trust model — fail-closed Linux containment
Task files and candidate prose remain untrusted executable inputs. Every
setup, verifier, incumbent, and candidate cell therefore runs in a
preflighted Bubblewrap boundary with a private home/config/temp, a
self-contained clone, a PID namespace, bounded process-tree ownership, and a
deny-by-default environment. Task-declared dependency roots are mounted
read-only, while graph assets are rebuilt by the harness as described below.
Claude runs in bare,
dontAsk mode with strict clone-local MCP configuration; Bash children do
not inherit the model credential and their network sandbox denies all
domains.
Prebuilt task .gitnexus assets are rejected. For each task commit, the
harness creates the deterministic parentless snapshot first, removes every
analyzer-visible path or stored source reference to the benchmark harness,
neutralizes target-controlled GitNexus config/ignore files, and builds one
fresh PDG index offline with --pdg --index-only --no-stats. It then proves
that neither whole graph nodes nor relationships contain a harness marker and
caches only the bound metadata/database assets for reuse by paired arms.
Each selected task also declares a bounded hidden oracle command and file
set. The harness captures those regular, non-symlink files into an immutable
in-memory snapshot before any arm runs and binds the command, paths, sizes, and
raw bytes into the task digest. Before any task asset or model session, each
disposable clone that contains the benchmark harness is rewritten to a clean,
parentless snapshot without eval/workflow_bench; all original refs, reflogs,
and unreachable Git objects are pruned so git show cannot recover the hidden
bytes. Only after the model exits (and after the authored-test signal is
collected) does the harness materialize the oracle beneath a private host
root, mount it read-only at a random workspace sibling, and supply that mount
through GITNEXUS_BENCH_ORACLE_ROOT. This layout preserves hidden tests'
../gitnexus imports as the credited candidate checkout. Authored and hidden
verifiers run with the complete workspace read-only and networking unshared;
hidden stdout/stderr is never persisted. The harness re-checks every oracle
byte and erases the mountpoint before churn/patch capture. Shipped Vitest
oracles use the staged, digest-bound vitest.config.mts; a candidate cannot
replace repo test config or setup hooks to make the hidden test vacuously pass.
The hidden command invokes the read-only dependency's Vitest binary directly,
without an npx configuration/resolution layer.
Every evaluated repo-local skill root is over-mounted read-only for the full
model session, and an immutable empty user-level skills directory prevents a
writable $HOME skill from shadowing it. Skill-use evidence comes only from
the bounded stream captured directly from Claude stdout by the parent. The
runner parses every event through EOF, requires one final result, correlates
an exact Skill request ID with one later successful result, structurally
redacts the event objects, and stores the canonical redacted JSONL with a
digest. Files written beneath the agent's $HOME are never trusted as
evidence.
Bare mode is deliberately non-interactive: it does not consult a stored
Claude login/keychain or ANTHROPIC_AUTH_TOKEN. Supply an Anthropic API key
through GITNEXUS_BENCH_ANTHROPIC_API_KEY (preferred) or --anthropic-api-key;
the harness maps it to ANTHROPIC_API_KEY only for the trusted Claude parent
and scrubs it from agent-launched tools. GITNEXUS_BENCH_AUTH_TOKEN and
--auth-token remain as aliases. OpenAI keys are not a drop-in
replacement: pass --openai-api-key / GITNEXUS_BENCH_OPENAI_API_KEY with
gpt-* / o* / openai/* model ids and the harness starts a loopback
LiteLLM proxy. The OpenAI key stays on that host process; Claude still sees
only a minted ANTHROPIC_API_KEY plus ANTHROPIC_BASE_URL.
The trusted Claude CLI still needs outbound access to the explicitly supplied
model endpoint. This is not a network broker, so the CLI itself retains that
egress; agent-launched tools do not. Missing Bubblewrap, unsupported hosts,
invalid mounts, or namespace preflight failure stop before model invocation.
Native benchmark execution is therefore Linux/WSL2-only. Evidence assembly
and hand-authored overlay preparation can happen elsewhere, but
--initial-overlay does not bypass containment.
For a local diagnostic inside a container that blocks user namespaces, an operator may explicitly choose the non-containment host backend:
UNSAFE_NO_BWRAP=1 RUNS=1 ./workflow_bench/run-evolution.sh
This mode runs review sessions directly in disposable host worktrees and is
not a security boundary: it does not isolate the network or create a PID
namespace, and a session that can chmod can undo the workspace lock. The
harness drops write bits on the whole clone, with no carve-out, so accidental
npm install / analyze writes cannot invalidate review evidence. The review
artifact is not in the clone at all: it lives in a writable directory bound at
/review-output, outside the workspace.
Sandbox cleanup restores owner write bits before deleting the private TMPDIR,
because a session that copytrees the locked clone would otherwise leave
non-empty 0555 directories that rmtree cannot remove. Historical review
SHAs that gitignore .claude/skills/* are force-added when the harness seeds
or overlays the evaluated gitnexus-review skill.
Treat model and verifier processes as able to access host files and
credentials available to the invoking user. It is restricted to the review
benchmark, forbidden with --apply and whenever CI is set;
promotion-capable and CI runs must use Bubblewrap.
Prompt and skill evolution loop
Prompts age as models and tool harnesses change. Treat the current skills and router thresholds as an incumbent policy, not permanent truth. Candidate changes run offline in the same throwaway clones as the incumbent; production skills never rewrite themselves from a live task.
On the self-hosted evolution box, run-evolution.sh passes
--max-runtime-from-instance-window and the CLI derives its own cap from
/proc/uptime at startup (24h EventBridge window minus a 90-minute upload
reserve), in the same breath as it starts the clock that cap is measured
against — a budget computed anywhere earlier is spent by the seconds between. A workflow_dispatch that lands on an
already-running instance therefore exits in-process instead of vanishing when
the box stops — a cancelled GitHub job skips even if: always(), which is
how run 33962002890 lost 51 finished sessions. Local runs are uncapped.
A review generation is 6 tasks × 3 arms × 3 runs. Serial workers=1 at ~19 minutes per session is a 16-hour job (run 33962002890). Two harness changes cut that without shrinking the gate:
- Comparator reuse.
evolve.pyforwards the seed / prior generation as--reuse-results. Incumbentreviewandce_reviewrows are copied into the newresults.jsonlwhen model, effort, task SHA, prompt digest, oracle bytes, incumbent skill digest, CE plugin digest, and sandbox backend still match. Candidate arms always run. A weekly generation with an unchanged incumbent therefore pays 18 sessions, not 54. A promotion, model change, task-corpus change, or harnessRUNTIME_DIGESTchange invalidates the lock and re-runs the comparators. - Sanitized clone templates. Each unique task SHA is cloned and
sanitized once. Cells copy that parentless snapshot (reflink when the
filesystem allows) instead of
git clone --no-localplus repack/prune/fsck 54 times. Isolation is a private.git, not a second copy of full history.
Dispatch defaults to --workers 3 so those 18 paid cells can overlap. Size
workers to the host: a cell that loses CPU and hits the session ceiling is
an excluded run the gate refuses.
Build an overlay that mirrors only the canonical repo-local skill paths:
/tmp/gn-skill-candidate/
└── .claude/skills/
├── gitnexus-plan/SKILL.md
└── gitnexus-work/SKILL.md
The overlay may contain Markdown files from either of those two skill trees. The runner rejects every other path, including source, test, and MCP configuration files, so a candidate cannot improve its score by changing the task or verifier. Arm selection is derived from the touched skill and must be exact: a plan-only overlay runs the workflow pair; any work overlay runs both workflow and direct-work pairs. Subsets and unrelated extra pairs fail before paid work. For a work overlay:
cd eval
uv run --locked --extra dev python -m workflow_bench.runner \
--tasks workflow_bench/tasks.scenarios.yaml \
--runs 3 --workers 1 --model claude-sonnet-4-20250514 \
--arms workflow candidate_workflow \
workflow_direct candidate_workflow_direct \
--candidate-overlay /tmp/gn-skill-candidate
Candidate runs start from the same task commit, then receive a clean ephemeral
commit containing the overlay. results.jsonl records the named model, task
commit, task-prompt digest, skill digest, overlay digest, hidden-oracle
command/manifest/content digests, immutable dependency content/manifest
digests, separate authored-test and oracle outcomes,
timestamp, local session ids, and digest-bound parent-captured event-stream
artifacts. Those artifacts are the trajectory evidence: cluster failures and
expensive detours, propose one bounded prompt change, and feed it back as the
next overlay.
When candidate arms are present the runner also writes schema-6
promotion.json. It
binds the immutable overlay digest, benchmark model, truthful candidate origin
(a named proposer model or manual-initial-overlay), selected
task definitions, resolved commits, and exact hidden-oracle bytes/commands,
immutable dependency bytes, committed base digest of every apply
destination, exact required arms, thresholds, and evidence expiry. Its default
deterministic gate is deliberately conservative:
Schema 6 binds a separate policy to each required candidate arm and records whether the sweep completed. Apply validates the paired metrics and recomputes each decision. Historical schema 5 reports remain readable; regenerate their benchmark evidence before applying an overlay. Editing a schema number does not supply the missing evidence.
Review candidates optimize weighted F1 with a minimum improvement of 0.01, complete paired evidence on every selected task, and no per-task quality regression. Complete misses score zero. Matching uses maximum cardinality throughout the 100-finding limit. Downgraded findings receive at most their reported severity's weight; only blocking-severity matches count toward blocker recall. Every valid candidate repeat must have the correct verdict, and the minimum blocker recall across repeats must not regress. Clean controls retain their false-positive and verdict safeguards. Implementation candidates retain the efficiency policy below:
- at least 3 paired VALID runs per task, zero excluded runs in either arm (session/infra-error rows therefore block promotion), and a named model;
- a fully measured task that neither arm ever resolves remains reported but is
ungated from the quality comparison — only if its metric was measured in both
arms and no run hit
skill-not-invoked(a skill that never loaded is prompt evidence, not task health). An ungated task still ranks against a looser 100% failed-task regression cap on the promotion metric; - at least half the paired tasks must stay gated, and
promotion.jsondiscloses the gated/ungated split per decision; a set with no gated task at all isinsufficient_evidence; - the candidate must pass the hidden oracle on every valid run of every gated task the incumbent resolves at least once — on a task the incumbent never resolves, partial candidate progress counts as improvement instead of failing the floor, so making some progress is never scored worse than making none;
- no per-task resolution-rate regression (quality is lexicographically first);
- promotion by resolution needs a margin of at least 2 resolved runs — a 1-run difference is noise at this run count and falls through to the efficiency comparison;
- with equal quality, at least 5% median improvement on the promotion metric
(default
cost_usd— the only CLI-reported number that includes subagent spend; token metrics count only the main-loop session and flatter subagent-heavy candidates, so selecting one stamps a warning intopromotion.json); - no individual task may regress the selected efficiency metric by more than 20%.
Tune the efficiency signal with --promotion-metric and the three
--promotion-* thresholds. Applying requires one unique promote decision
for every bound candidate arm. The driver then stages every canonical and
shipped mirror, verifies that all destination bytes still match the bound
bases, replaces them as one compare-and-swap set, verifies byte parity, and
rolls every landed replacement back on failure or interruption.
keep_incumbent and
insufficient_evidence become the next learning queue; their raw
results.jsonl rows carry the session_ids of the trajectories to inspect.
Re-run the paired suite whenever the named model or tool harness changes, and at least every 90 days otherwise. This is prompt-policy optimization using verified agent trajectories as reward evidence; it is intentionally not online model-weight RL. The same records can feed a later offline RL pipeline without weakening today's deterministic promotion boundary.
Closing the loop automatically (evolve.py)
The evolution workflow runs an offline containment preflight with the pinned
Claude Code 2.1.214 binary before starting a paid proposer or benchmark. The
review canary seals the workspace read-only and writes nothing into it: the
artifact directory is bound at /review-output outside the workspace, and the
file itself is deliberately absent until the session creates it, so its absence
distinguishes "never written" from "written badly". Runtime mount placeholders
are prepared in the disposable clone before sealing it; existing config bytes
are preserved.
Any pre-existing result entry, including a symlink, is rejected. Required
canaries fail when their runtime or Bubblewrap is unavailable.
The default outage limit is five consecutive unusable results, across task
boundaries. Invalid review JSON advances this limit even when a skill or session
error was recorded first. A valid zero-quality review resets it. Concurrent
waves can exceed the limit by at most workers - 1 completed cells; no further
wave starts after a trip. Completed rows and redacted diagnostics remain in the
partial report, the runner exits nonzero, and the evolution driver stops without
applying or starting another generation.
SIGINT and SIGTERM propagate one cancellation event through managed commands, including clone, setup, Claude, and verification. Executor submissions copy the run context so indirect subprocess helpers receive the same event. Active process groups or Windows Job Objects are terminated and workers joined before shared assets or the gateway are released. Controlled cancellation tests require cleanup within 15 seconds. Cancellation remains distinct from timeout and quality failure in recorded evidence.
The gateway runs under a private supervisor watching a pipe owned only by the harness. Parent exit, including SIGKILL, closes that pipe and stops the proxy group; Windows also retains kill-on-close Job Object ownership. Keep completed JSONL rows, transcripts, the partial report, and gateway diagnostics when investigating an interrupted run. A subsequent paid comparison needs fresh evidence from all arms under the same dependency lock. LiteLLM pricing comes from that locked release's local cost map; compare no old/new-lock costs as quality evidence.
workflow_bench.evolve automates the three manual arrows — propose,
benchmark, apply — without moving the trust boundary:
cd eval
./workflow_bench/run-evolution.sh # local; no working-tree apply
./workflow_bench/run-evolution.sh --apply # CI; same argv the workflow uses
./workflow_bench/run-evolution.sh --dry-run # print the evolve command
The GitHub skill-evolution job calls this script. Do not invoke
python -m workflow_bench.evolve directly for a full loop. Environment knobs
match the workflow: MODEL, PROPOSER_MODEL, GENERATIONS, RUNS,
WORKERS, PROVIDER, EFFORT, SEED_RESULTS, INCLUDE_EXPENSIVE. The
checked-in production defaults are PROVIDER=openai, MODEL=gpt-5.6-sol,
PROPOSER_MODEL=gpt-5.6-sol, and EFFORT=xhigh.
The scheduled/default profile is read-only review evolution. Set
EVOLUTION_PROFILE=implementation explicitly to run the legacy plan/work
benchmark. Review mode requires CE_PLUGIN_DIR and CE_PLUGIN_VERSION.
Each review generation: a confined proposer session reads only the incumbent
gitnexus-review skill, normalized CE/incumbent/candidate result rows, bounded
review artifacts and session transcripts, and the rejected
proposal.md when available (including a workflow seed from a prior run), and
the learning queue,
then writes ONE bounded candidate overlay plus a reviewer-facing
proposal.md. The proposer's clone is sanitized exactly like an arm's before
its session starts: it authors the artifact the arms are scored with, so
letting it read eval/workflow_bench would hand it the task prompts and the
hidden oracles it is about to be graded against, and a proposal could win the
gate by encoding the expected behavior into a skill rather than by being a
better skill. The overlay is re-validated by candidate_overlay_files
(same boundary: Markdown under gitnexus-review, including exercised
ci-personas/, nothing else), frozen,
and exercised only by its exact required pairs. Task refs are resolved once
before generation zero and the immutable task bindings are forwarded to every
generated runner invocation, so a moving branch cannot change later evidence.
The deterministic quality-first gate rejects blocker-recall regressions,
new false positives on clean controls, and any weighted-score regression.
Repeated evidence (RUNS>=3) is required for promotion; RUNS=1 is
diagnostic-only. CE is the external comparator. Cost and latency are
tiebreakers and never compensate for quality loss. promote stops the loop; with --apply
the authorized frozen bytes
are transactionally applied to the canonical
.claude/skills/ trees and their shipped mirrors as an ordinary
working-tree diff — committing, CI (shipped-skills-sync,
skills-steering), and the PR merge stay human. keep_incumbent feeds that
generation's trajectories to the next proposer. --initial-overlay skips
the generation-0 proposer to benchmark a hand-written candidate;
--proposer-model upgrades only the diagnosis session.
Learning queue. Live skill runs never self-edit (see each
skill's "Skill feedback" section) — instead they may append one-line JSON notes to
workflow_bench/learnings.jsonl (gitignored, machine-local like the
transcripts they complement). The proposer reads the queue as hints, not
ground truth: a learning only reaches a shipped skill by surviving the same
paired benchmark as any other candidate.
For ad-hoc use, run the driver on the existing re-evaluation triggers
(model/harness change or 90-day staleness). The repository workflow runs a
deliberate weekly drift check: dispatch defaults to three concurrent cells
of one task; scheduled concurrency still requires
GITNEXUS_EVOLUTION_WORKERS=3 after a clean proof. --workers is bounded
to 1–8 before paid work starts. --generations remains the only loop bound.
Free-model setup (no paid tokens)
Headless Claude Code honors ANTHROPIC_BASE_URL, and litellm (already an
eval dependency) can proxy its Anthropic-compatible /v1/messages to a model
that costs nothing — a hosted OpenRouter :free variant or a fully local
Ollama model. Config template: free-model.litellm.yaml.
# 1. Choose a proxy master key and start the proxy
# (pick/edit a model route in the yaml first; keep the proxy on loopback —
# anyone who can reach the port with this key can spend the backend quota)
export LITELLM_MASTER_KEY="$(openssl rand -hex 16)"
uv run --locked --with 'litellm[proxy]' litellm --config workflow_bench/free-model.litellm.yaml --port 4000
# 2. Point the benchmark at it
uv run --locked --extra dev python -m workflow_bench.runner \
--tasks workflow_bench/tasks.scenarios.yaml --runs 3 \
--base-url http://localhost:4000 --anthropic-api-key "$LITELLM_MASTER_KEY" --model free-coder
OpenAI API keys
Claude Code still speaks Anthropic /v1/messages. For a paid OpenAI backend,
do not point --anthropic-api-key at an sk-... OpenAI key. Export the OpenAI key
and use OpenAI model ids; the driver starts the proxy itself:
export GITNEXUS_BENCH_OPENAI_API_KEY="$OPENAI_API_KEY"
PROVIDER=openai ./workflow_bench/run-evolution.sh
The GitHub skill-evolution workflow accepts GITNEXUS_BENCH_OPENAI_API_KEY on
the gitnexus-evolution environment. Dispatch with provider=openai to force
that backend even when an Anthropic token is also configured (otherwise auto
keeps using Anthropic whenever that secret exists). Claude default model
inputs are then rewritten to gpt-5.6-sol; every proposer and benchmark
session receives --effort xhigh.
Caveats, honestly:
- Both arms run on the same model, so the comparison stays fair at any quality level — but small free models follow skills less reliably, so expect lower resolve rates and noisier savings than on frontier models. Treat free-model runs as directional; confirm headline numbers with a small paid run.
- Through a proxy
cost_usdreads ~0, and the CLI's token counts are NOT a substitute "real metric": they cover only the main-loop session, so subagent spend is invisible to both. For efficiency ranking, prefer a paid run gated oncost_usd, or sum per-session usage from the transcripts (~/.claude/projects/<cwd-slug>/<session_id>.jsonl, deduplicating events that share onemessage.id). - OpenRouter
:freevariants are rate-limited (~50 req/day on a fresh account); local Ollama has no limits. - Codex users:
codex exec --ossruns local models for free too, but this runner is Claude-Code-first; a codex engine is a straightforward extension (parse its--jsonusage events).
Historical ground base (2026-07-11, Claude Code 2.1.207, unnamed model, n=1/cell)
These figures predate mandatory model provenance and are retained only as historical calibration. They are not eligible promotion evidence and must not be combined with current named-model runs.
Three task classes × three arms, single-repo (GitNexus itself). Every arm resolved every task — at this difficulty, pass/fail quality is saturated and the comparison is pure cost:
| task (class) | arm | resolved | cost $ | wall | turns | vs baseline cost |
|---|---|---|---|---|---|---|
| trivial-version-alias | workflow | 1/1 | 9.16 | 16m | 63 | −333% |
| trivial-version-alias | baseline | 1/1 | 2.11 | 2.8m | 16 | — |
| inv-bug-pdg-note | workflow | 1/1 | 14.56 | 21m | 83 | −331% |
| inv-bug-pdg-note | workflow_direct | 1/1 | 5.23 | 7.5m | 32 | −55% |
| inv-bug-pdg-note | baseline | 1/1 | 3.38 | 4.7m | 22 | — |
| inv-feature-list-repos-filter | workflow | 1/1 | 13.22 | 19m | 84 | −211% |
| inv-feature-list-repos-filter | workflow_direct | 1/1 | 4.87 | 4.8m | 38 | −15% (wall +14% faster) |
| inv-feature-list-repos-filter | baseline | 1/1 | 4.25 | 5.5m | 32 | — |
What the ground base says, honestly:
- The full plan→work workflow never paid for itself at this task scale (tasks a baseline agent finishes in ≤35 turns). Its fixed cost — freshness gate incl. analyzer rebuild + re-index, a full 13-section plan, work-phase re-anchoring — is ~$9–11 per task and needs much larger tasks, plan-reuse (one plan, several executors/sessions), or plan-as-deliverable flows to amortize.
- workflow_direct is close to baseline (−15% to −55% cost, once slightly faster wall) — the execution discipline (impact-before-edit, detect_changes-before-commit) is cheap. It produced noticeably more test coverage than baseline for near-equal cost on the feature task.
- Quality didn't differentiate because nothing failed. The regime where
the workflow should win on resolve rate — cross-module tasks where
baselines flail — is the unmeasured cell (
cross-module-parse-retry), and the next thing to measure, ideally with--runs 3+on a free backend. - Caveats: n=1 per cell, one repo, one model; churn numbers from this run predate the intent-to-add/exclude-plans churn fix, so they are not comparable across arms and are omitted above.
Routing implication (to revisit as cells fill in): for tasks up to this
size, gitnexus-work direct mode or a plain agent is the cost-optimal
route; reserve full gitnexus-plan → gitnexus-work for cross-module work,
multi-session execution, or when the plan document itself is a deliverable.
If a future run shows the workflow flattering itself here, distrust the run.
Cross-module cell (same day, optimized skills, n=1)
The hardest class — retry-with-backoff across the worker-pool/pipeline seams, transient-vs-deterministic classification:
| arm | resolved | cost $ | wall | turns | churn |
|---|---|---|---|---|---|
| workflow | 1/1 | 18.32 | 37m | 107 | 4/+373/−17 |
| workflow_direct | 1/1 | 9.53 | 15m | 52 | 11/+244/−66 |
| baseline | 1/1 | 18.03 | 34m | 98 | 6/+345/−69 |
(The workflow_direct row is the clean re-run under clone isolation — the original was contaminated, see the integrity note below.)
This is the cell where the discipline pays. workflow_direct — the
execution skill without a planning pass — beat a plain agent by 47% cost
and 56% wall time on the hardest class while resolving: impact-first
navigation and gated commits prevented the flailing that baseline's 98
turns represent. The full workflow's premium vanished (−1.6% vs baseline;
−211%..−333% on smaller classes) — fixed costs amortize here, with a less
destructive diff and a durable plan artifact — but it didn't beat direct
mode on any measured axis with the plan consumed only once. Resolve rate
stayed tied across all cells; the savings story belongs to the execution
discipline, and the planning pass is bought for its artifact (multi-session
reuse, review, handoff), not for same-session token savings.
Benchmark integrity note (why churn earns its keep): the original
workflow_direct cell reported an impossible 28-turn/$4.71 solve with churn
byte-identical to the workflow arm — because git worktree add shares the
ref namespace, the workflow arm's slug branch survived worktree removal, and
the direct arm found and adopted the finished work. Fixed by giving every
arm an isolated git clone --no-local --no-hardlinks with no object
alternates (agent-created refs and storage die with the clone);
the leaked branch was deleted and the cell re-measured. Treat identical
churn fingerprints across arms as a contamination alarm.
Optimization re-measurement (same day, commit 830a0459)
After category-priced plan forms (compact ≤80 lines + mini-pack),
category-priced freshness (accept for compact classes), per-category turn
budgets, and the work-phase HEAD==pin fast path, the same
inv-bug-pdg-note workflow cell re-measured (n=1):
| ground base | optimized | delta | |
|---|---|---|---|
| resolved | ✅ | ✅ | — |
| cost $ | 14.56 | 11.70 | −20% |
| turns | 83 | 72 | −13% |
| output tokens | 59,789 | 53,345 | −11% |
| cache_read | 6.64M | 5.07M | −24% |
| wall | 21m | 25m | +15% |
Verified in-transcript: the compact form fired (115-line plan vs 209 for a simpler task pre-optimization), the plan session dropped 72→49 turns, and NO analyzer rebuild/re-index executed. All savings came from the plan side; this run's work session drew a long test-debugging tail (hence the wall regression) — single-run variance cuts both ways. The optimizations narrow the gap but do not flip the regime: the workflow remains ~3.5× baseline on this task class, so the routing rule above stands unchanged.
Writing good tasks
See tasks.scenarios.yaml. Small enough to finish headless, real enough to
require investigation — the workflow's savings come from not re-reading and
not re-investigating, which trivial tasks never exercise. Keep verify as a
model-visible authored-test quality signal, and add an independent oracle
whose source files live under workflow_bench/oracles/. Oracle commands must
run only files staged beneath $GITNEXUS_BENCH_ORACLE_ROOT; for Vitest, include
the shared vitest.config.mts as an oracle file and pass it explicitly with
--config. Prefer verify commands that use the repo's own npm scripts (they
carry build pre-hooks).
Relation to the SWE-bench harness
The rest of eval/ benchmarks GitNexus tools inside a litellm agent loop
(baseline vs graph-enhanced). This module benchmarks the skill workflow
inside the real CLI harness those skills ship for. Different question, same
spirit: measure, don't assume.