docs(benchmarks): point README at the new reproducible XBEN harness

Note that the hand-typed XBEN numbers are pending re-verification through
benchmarks/xben/, and add a "Reproducing this benchmark" section pointing
at its README.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Owen 2026-09-12 16:55:52 +01:00
parent 954ad4a163
commit df37b2fafa

View file

@ -3,6 +3,23 @@
We use security benchmarks to track Strix's capabilities and improvements over time. We plan to add more benchmarks, both existing ones and our own, to help the community evaluate and compare security agents.
## Reproducing this benchmark
The numbers below are pending re-verification through an in-repo,
reproducible harness: **[`benchmarks/xben/`](xben/README.md)**. That harness
is the source of truth going forward — it validates committed per-challenge
run records against a JSON Schema, mechanically regenerates the table below
from them, and has a `--check` mode that fails loudly if the committed data
doesn't back up the claim. This closes out the intent of
[usestrix/strix#1263](https://github.com/usestrix/strix/issues/1263) and
[usestrix/strix#1264](https://github.com/usestrix/strix/issues/1264), which
asked for exactly this: the headline number below currently has no committed
data reproducing it in this repository, only the hand-typed table that
follows and a link to an external repo. See `benchmarks/xben/README.md` for
how to run it and why `--check` currently (correctly) fails against the
fixture data checked in so far.
## Full Details
For the complete benchmark results, evaluation scripts, and run data, see the [usestrix/benchmarks](https://github.com/usestrix/benchmarks) repository.
@ -13,6 +30,11 @@ For the complete benchmark results, evaluation scripts, and run data, see the [u
## Results
> [!NOTE]
> These numbers are hand-typed and **pending re-verification** via
> `benchmarks/xben/` (see above). Treat them as a historical claim, not yet
> as independently reproducible from data committed to this repository.
| Benchmark | Challenges | Success Rate |
|-----------|------------|--------------|
| [XBEN](https://github.com/usestrix/benchmarks/tree/main/XBEN) | 104 | **96%** |