From 65c7cfd78cfd93dac2c0aa82a66bf0e0658d22b4 Mon Sep 17 00:00:00 2001 From: Gergo Magyar Date: Sat, 11 Jul 2026 08:46:15 +0000 Subject: [PATCH] docs(eval): record first workflow_bench calibration run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Trivial-task calibration (add -V alias): both arms resolved; workflow arm ~4.3x baseline cost — the documented overhead-dominated regime, recorded so the regime boundary is empirical rather than asserted. Co-Authored-By: Claude Fable 5 --- eval/workflow_bench/README.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/eval/workflow_bench/README.md b/eval/workflow_bench/README.md index b2a9dfb9a..319f874b9 100644 --- a/eval/workflow_bench/README.md +++ b/eval/workflow_bench/README.md @@ -60,6 +60,25 @@ Caveats, honestly: runner is Claude-Code-first; a codex engine is a straightforward extension (parse its `--json` usage events). +## Calibration data point (2026-07-11, Claude Code 2.1.207) + +First live run, deliberately on a *trivial* task ("add `-V` as a `--version` +alias + unit test", 1 run, default model), to calibrate the overhead floor: + +| arm | resolved | cache_create | cache_read | output | cost $ | wall s | turns | +| --- | --- | --- | --- | --- | --- | --- | --- | +| workflow | 1/1 | 184,603 | 3,980,936 | 29,668 | 9.16 | 955 | 63 | +| baseline | 1/1 | 57,113 | 711,184 | 5,222 | 2.11 | 167 | 16 | + +Both arms solved it; the workflow cost ~4.3× more. That is the expected +result for this task class — a two-line alias needs no investigation, so the +workflow's fixed costs (freshness gate incl. analyzer rebuild + re-index, a +209-line plan, context pack, work-phase re-anchoring) are pure overhead. +The savings hypothesis applies to investigation-heavy, multi-file tasks +(see the example tasks) and to plan-once/execute-later flows where the +context pack amortizes. If the benchmark had flattered the workflow on this +task, distrust the benchmark. + ## Writing good tasks See `tasks.example.yaml`. Small enough to finish headless, real enough to