From df37b2fafa34d49e7e44f353c7aab1fad7260f96 Mon Sep 17 00:00:00 2001 From: Owen Date: Sat, 12 Sep 2026 16:55:52 +0100 Subject: [PATCH] docs(benchmarks): point README at the new reproducible XBEN harness Note that the hand-typed XBEN numbers are pending re-verification through benchmarks/xben/, and add a "Reproducing this benchmark" section pointing at its README. Co-Authored-By: Claude Sonnet 5 --- benchmarks/README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/benchmarks/README.md b/benchmarks/README.md index b7d99dece..e67f5d757 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -3,6 +3,23 @@ We use security benchmarks to track Strix's capabilities and improvements over time. We plan to add more benchmarks, both existing ones and our own, to help the community evaluate and compare security agents. +## Reproducing this benchmark + +The numbers below are pending re-verification through an in-repo, +reproducible harness: **[`benchmarks/xben/`](xben/README.md)**. That harness +is the source of truth going forward — it validates committed per-challenge +run records against a JSON Schema, mechanically regenerates the table below +from them, and has a `--check` mode that fails loudly if the committed data +doesn't back up the claim. This closes out the intent of +[usestrix/strix#1263](https://github.com/usestrix/strix/issues/1263) and +[usestrix/strix#1264](https://github.com/usestrix/strix/issues/1264), which +asked for exactly this: the headline number below currently has no committed +data reproducing it in this repository, only the hand-typed table that +follows and a link to an external repo. See `benchmarks/xben/README.md` for +how to run it and why `--check` currently (correctly) fails against the +fixture data checked in so far. + + ## Full Details For the complete benchmark results, evaluation scripts, and run data, see the [usestrix/benchmarks](https://github.com/usestrix/benchmarks) repository. @@ -13,6 +30,11 @@ For the complete benchmark results, evaluation scripts, and run data, see the [u ## Results +> [!NOTE] +> These numbers are hand-typed and **pending re-verification** via +> `benchmarks/xben/` (see above). Treat them as a historical claim, not yet +> as independently reproducible from data committed to this repository. + | Benchmark | Challenges | Success Rate | |-----------|------------|--------------| | [XBEN](https://github.com/usestrix/benchmarks/tree/main/XBEN) | 104 | **96%** |