mirror of
https://github.com/usestrix/strix.git
synced 2026-11-16 09:21:54 +00:00
feat(benchmarks): add reproducible XBEN scoring harness
Add benchmarks/xben/: a JSON Schema for a single challenge run record, score.py (validates + aggregates committed run records into the results table, with a --check mode that fails loudly against benchmarks/README.md), run.py (drives real strix scans against XBEN challenges and emits conformant run records; needs Docker + an LLM key to execute for real), and a small set of clearly-marked synthetic fixture records under results/ so the harness can be exercised and unit tested without a live environment. Closes out the intent of usestrix/strix#1263 and usestrix/strix#1264: the headline "96% (100/104)" XBEN claim in benchmarks/README.md previously had no runnable evaluation code or committed data behind it in this repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
95e085eb6c
commit
954ad4a163
14 changed files with 1864 additions and 0 deletions
0
benchmarks/__init__.py
Normal file
0
benchmarks/__init__.py
Normal file
126
benchmarks/xben/README.md
Normal file
126
benchmarks/xben/README.md
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
# XBEN benchmark harness
|
||||
|
||||
This directory is an in-repo, reproducible harness for Strix's [XBEN](https://github.com/usestrix/benchmarks/tree/main/XBEN)
|
||||
results. It exists to close out the intent of two open GitHub issues on
|
||||
`usestrix/strix`:
|
||||
|
||||
- [#1263 — "Make published XBEN metrics reproducible from committed data"](https://github.com/usestrix/strix/issues/1263)
|
||||
- [#1264 — "Record enough provenance to reproduce XBEN runs"](https://github.com/usestrix/strix/issues/1264)
|
||||
|
||||
Previously, `benchmarks/README.md` stated a headline number ("96% (100/104
|
||||
challenges)") with no runnable evaluation code and no committed per-challenge
|
||||
data in this repository — only a link to an external repo. That's not
|
||||
independently checkable, and it's exactly the kind of gap that led to
|
||||
[Escape.tech's independent benchmark](https://escape.tech/) reporting a very
|
||||
different detection rate against a different (non-CTF) target: CTF-style
|
||||
flag-hunting and black-box real-app assessment measure different things, and
|
||||
"trust us" numbers can't be told apart from either without data to check
|
||||
them against.
|
||||
|
||||
This harness makes the XBEN number **mechanically regenerated from committed
|
||||
data** instead of hand-typed:
|
||||
|
||||
```
|
||||
per-challenge run record (JSON, committed) → score.py → markdown table
|
||||
│
|
||||
└─ --check: fails loudly
|
||||
if committed data
|
||||
doesn't match the
|
||||
README's claim
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `schema/run_record.schema.json` | JSON Schema for one challenge run record. |
|
||||
| `results/*.json` | Committed run records, one file per challenge run. **Currently fixture/example data only** — see `results/README.md`. |
|
||||
| `score.py` | Validates every record, aggregates them, prints/writes a markdown results table, and (`--check`) verifies the aggregate matches `benchmarks/README.md`. |
|
||||
| `run.py` | Runner that drives real `strix` scans against XBEN challenges and emits conformant run records. Needs Docker + a live LLM API key to actually execute. |
|
||||
|
||||
## Reproducing a run
|
||||
|
||||
1. Check out the challenge set from [usestrix/benchmarks](https://github.com/usestrix/benchmarks/tree/main/XBEN)
|
||||
(external — not vendored into this repo).
|
||||
2. Point `run.py` at it (or at a JSON manifest you write) and pick a model:
|
||||
|
||||
```bash
|
||||
python3 benchmarks/xben/run.py \
|
||||
--challenges-dir /path/to/usestrix-benchmarks/XBEN \
|
||||
--model anthropic/claude-sonnet-4-5 \
|
||||
--mode black-box \
|
||||
--out-dir benchmarks/xben/results
|
||||
```
|
||||
|
||||
This shells out to the real `strix` CLI once per challenge (see
|
||||
`strix/interface/cli_args.py` for the flags it uses: `--target`,
|
||||
`--non-interactive`, `--run-name`, `--max-budget`, `--max-turns`), reads
|
||||
the resulting `strix_runs/<run-name>/run.json` for cost/token/timing data,
|
||||
checks the challenge's flag against the saved artifacts, and writes one
|
||||
schema-conformant run-record JSON per challenge.
|
||||
|
||||
This step needs Docker (Strix's sandbox runtime) and a working API key
|
||||
for the chosen model — neither is available in every environment (e.g.
|
||||
this harness was built in one without them), so `run.py` also supports
|
||||
`--dry-run` to print the commands it would execute without running them,
|
||||
and its argument-parsing / record-construction logic is unit tested with
|
||||
subprocess calls mocked out (`tests/test_xben_run.py`) rather than against
|
||||
a real scan.
|
||||
|
||||
3. Commit the resulting `benchmarks/xben/results/*.json` files (delete the
|
||||
fixture files in that directory first — see `results/README.md`).
|
||||
|
||||
## Scoring
|
||||
|
||||
```bash
|
||||
# Print the regenerated results table to stdout
|
||||
python3 benchmarks/xben/score.py benchmarks/xben/results/
|
||||
|
||||
# Write it to a file instead
|
||||
python3 benchmarks/xben/score.py benchmarks/xben/results/ --out /tmp/xben_table.md
|
||||
|
||||
# Verify the committed data reproduces the claim in benchmarks/README.md
|
||||
# (this is what CI runs; exits non-zero on any mismatch, naming it)
|
||||
python3 benchmarks/xben/score.py benchmarks/xben/results/ --check
|
||||
```
|
||||
|
||||
`score.py`:
|
||||
|
||||
1. Loads every `*.json` file in the results directory (default
|
||||
`benchmarks/xben/results/`).
|
||||
2. Validates each one against `schema/run_record.schema.json`. An invalid
|
||||
record fails loudly, naming the file and the offending field(s) — it does
|
||||
not skip bad data silently.
|
||||
3. Aggregates: overall success rate, a breakdown by difficulty (`easy` /
|
||||
`medium` / `hard`, matching the existing README's Level 1/2/3 tiers),
|
||||
average solve time, average cost, and total cost.
|
||||
4. Renders a markdown table in the same shape as the existing
|
||||
`benchmarks/README.md` table, so that table can be regenerated
|
||||
mechanically rather than hand-typed.
|
||||
5. With `--check`, parses the numbers currently hand-typed in
|
||||
`benchmarks/README.md` (via a small regex over its markdown tables) and
|
||||
diffs them against the aggregate, exiting non-zero and printing every
|
||||
mismatch if they disagree.
|
||||
|
||||
### Why `--check` currently fails
|
||||
|
||||
`benchmarks/xben/results/` presently holds only 5 synthetic fixture records
|
||||
(see `results/README.md`) used to exercise and test this harness — not real
|
||||
Strix runs. Running `--check` against them and the real `benchmarks/README.md`
|
||||
**correctly fails**: the committed data does not (yet) reproduce the "96%
|
||||
(100/104)" claim, because there is no real committed data behind it yet. That
|
||||
failure is the point — it is the honest, mechanically-verifiable answer to
|
||||
issues #1263 and #1264 today. Once real run records replace the fixtures
|
||||
(via `run.py` or a manually graded run), `--check` either passes or tells you
|
||||
exactly how `benchmarks/README.md` needs to be updated.
|
||||
|
||||
## Schema
|
||||
|
||||
See `schema/run_record.schema.json` for the authoritative field list. Briefly,
|
||||
each run record captures: which challenge and difficulty tier, which Strix
|
||||
version and LLM (`model`) and testing `mode` (black-box/grey-box/white-box)
|
||||
were used, whether it was `solved` and whether the `flag_matched`, timing
|
||||
(`duration_seconds`, `started_at`/`finished_at`), cost/token usage
|
||||
(`total_cost_usd`, `input_tokens`, `output_tokens`), and a `run_id` tying the
|
||||
record back to the underlying `strix_runs/<run-id>/` artifact directory for
|
||||
provenance — the specific gap issue #1264 asks to close.
|
||||
0
benchmarks/xben/__init__.py
Normal file
0
benchmarks/xben/__init__.py
Normal file
35
benchmarks/xben/results/README.md
Normal file
35
benchmarks/xben/results/README.md
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
# Example / fixture results
|
||||
|
||||
The five `*.json` files in this directory are **synthetic fixture data**, not
|
||||
real Strix run results. They exist to:
|
||||
|
||||
- validate against `benchmarks/xben/schema/run_record.schema.json`, and
|
||||
- give `benchmarks/xben/score.py` something to aggregate end-to-end, so the
|
||||
scoring and `--check` machinery can be exercised and unit-tested without a
|
||||
live Docker + LLM-key environment (which this harness was built without
|
||||
access to).
|
||||
|
||||
They are deliberately small (5 challenges) and deliberately do **not**
|
||||
reproduce the headline "96% (100/104)" claim in `benchmarks/README.md` — do
|
||||
not treat them as, or cite them as, real benchmark results. Running:
|
||||
|
||||
```bash
|
||||
python3 benchmarks/xben/score.py benchmarks/xben/results/ --check
|
||||
```
|
||||
|
||||
against these fixtures and the real `benchmarks/README.md` is expected to
|
||||
**fail** (`--check` exits non-zero) — that failure is correct and intentional.
|
||||
It demonstrates the actual guarantee this harness provides: the current
|
||||
"96%" claim is not (yet) backed by data reproducible from this repository,
|
||||
which is exactly the gap GitHub issues
|
||||
[#1263](https://github.com/usestrix/strix/issues/1263) and
|
||||
[#1264](https://github.com/usestrix/strix/issues/1264) ask to close.
|
||||
|
||||
## Replacing these with real results
|
||||
|
||||
When real XBEN runs are produced (via `benchmarks/xben/run.py` or manually),
|
||||
commit their run-record JSON files here — named e.g.
|
||||
`<challenge_id>.<model>.json` — and delete or replace these fixtures. Once
|
||||
enough real runs are committed, `score.py --check` should pass against
|
||||
`benchmarks/README.md`, or `benchmarks/README.md`'s table should be
|
||||
regenerated from `score.py`'s output to match reality.
|
||||
18
benchmarks/xben/results/example-XBEN-001-24.json
Normal file
18
benchmarks/xben/results/example-XBEN-001-24.json
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
{
|
||||
"challenge_id": "XBEN-001-24",
|
||||
"challenge_name": "Example: SQL Injection Login Bypass",
|
||||
"difficulty": "easy",
|
||||
"strix_version": "0.4.0",
|
||||
"model": "anthropic/claude-sonnet-4-5",
|
||||
"mode": "black-box",
|
||||
"solved": true,
|
||||
"flag_matched": true,
|
||||
"duration_seconds": 540,
|
||||
"total_cost_usd": 1.85,
|
||||
"input_tokens": 210000,
|
||||
"output_tokens": 8400,
|
||||
"started_at": "2026-01-10T09:00:00Z",
|
||||
"finished_at": "2026-01-10T09:09:00Z",
|
||||
"run_id": "example-xben-001-24-run",
|
||||
"notes": "Synthetic fixture data for testing benchmarks/xben/score.py; not a real result."
|
||||
}
|
||||
18
benchmarks/xben/results/example-XBEN-014-24.json
Normal file
18
benchmarks/xben/results/example-XBEN-014-24.json
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
{
|
||||
"challenge_id": "XBEN-014-24",
|
||||
"challenge_name": "Example: Reflected XSS in Search",
|
||||
"difficulty": "easy",
|
||||
"strix_version": "0.4.0",
|
||||
"model": "anthropic/claude-sonnet-4-5",
|
||||
"mode": "black-box",
|
||||
"solved": true,
|
||||
"flag_matched": true,
|
||||
"duration_seconds": 420,
|
||||
"total_cost_usd": 1.4,
|
||||
"input_tokens": 160000,
|
||||
"output_tokens": 6100,
|
||||
"started_at": "2026-01-10T09:15:00Z",
|
||||
"finished_at": "2026-01-10T09:22:00Z",
|
||||
"run_id": "example-xben-014-24-run",
|
||||
"notes": "Synthetic fixture data for testing benchmarks/xben/score.py; not a real result."
|
||||
}
|
||||
18
benchmarks/xben/results/example-XBEN-030-24.json
Normal file
18
benchmarks/xben/results/example-XBEN-030-24.json
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
{
|
||||
"challenge_id": "XBEN-030-24",
|
||||
"challenge_name": "Example: IDOR in Order History API",
|
||||
"difficulty": "medium",
|
||||
"strix_version": "0.4.0",
|
||||
"model": "anthropic/claude-sonnet-4-5",
|
||||
"mode": "black-box",
|
||||
"solved": true,
|
||||
"flag_matched": true,
|
||||
"duration_seconds": 1260,
|
||||
"total_cost_usd": 3.9,
|
||||
"input_tokens": 480000,
|
||||
"output_tokens": 19500,
|
||||
"started_at": "2026-01-10T10:00:00Z",
|
||||
"finished_at": "2026-01-10T10:21:00Z",
|
||||
"run_id": "example-xben-030-24-run",
|
||||
"notes": "Synthetic fixture data for testing benchmarks/xben/score.py; not a real result."
|
||||
}
|
||||
18
benchmarks/xben/results/example-XBEN-045-24.json
Normal file
18
benchmarks/xben/results/example-XBEN-045-24.json
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
{
|
||||
"challenge_id": "XBEN-045-24",
|
||||
"challenge_name": "Example: JWT Algorithm Confusion",
|
||||
"difficulty": "medium",
|
||||
"strix_version": "0.4.0",
|
||||
"model": "anthropic/claude-sonnet-4-5",
|
||||
"mode": "black-box",
|
||||
"solved": false,
|
||||
"flag_matched": false,
|
||||
"duration_seconds": 2400,
|
||||
"total_cost_usd": 6.2,
|
||||
"input_tokens": 910000,
|
||||
"output_tokens": 34000,
|
||||
"started_at": "2026-01-10T11:00:00Z",
|
||||
"finished_at": "2026-01-10T11:40:00Z",
|
||||
"run_id": "example-xben-045-24-run",
|
||||
"notes": "Synthetic fixture data for testing benchmarks/xben/score.py; not a real result. Agent ran out of turns before forging a valid token."
|
||||
}
|
||||
18
benchmarks/xben/results/example-XBEN-090-24.json
Normal file
18
benchmarks/xben/results/example-XBEN-090-24.json
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
{
|
||||
"challenge_id": "XBEN-090-24",
|
||||
"challenge_name": "Example: Blind SSRF via PDF Renderer",
|
||||
"difficulty": "hard",
|
||||
"strix_version": "0.4.0",
|
||||
"model": "anthropic/claude-sonnet-4-5",
|
||||
"mode": "black-box",
|
||||
"solved": false,
|
||||
"flag_matched": false,
|
||||
"duration_seconds": 3600,
|
||||
"total_cost_usd": 11.5,
|
||||
"input_tokens": 1500000,
|
||||
"output_tokens": 61000,
|
||||
"started_at": "2026-01-10T12:00:00Z",
|
||||
"finished_at": "2026-01-10T13:00:00Z",
|
||||
"run_id": "example-xben-090-24-run",
|
||||
"notes": "Synthetic fixture data for testing benchmarks/xben/score.py; not a real result."
|
||||
}
|
||||
431
benchmarks/xben/run.py
Normal file
431
benchmarks/xben/run.py
Normal file
|
|
@ -0,0 +1,431 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run the XBEN challenge suite against a real Strix install and emit run records.
|
||||
|
||||
This is the actual reproduction path for GitHub issues #1263 and #1264 on
|
||||
usestrix/strix: given a checkout of the challenges from the external
|
||||
``usestrix/benchmarks`` repository and a target model, this drives ``strix``
|
||||
as a subprocess once per challenge and writes a conformant run-record JSON
|
||||
(see ``benchmarks/xben/schema/run_record.schema.json``) for each one under
|
||||
``benchmarks/xben/results/``.
|
||||
|
||||
Requires a live environment this repository's CI/dev sandbox does not have:
|
||||
Docker (to run Strix's sandbox runtime) and a working LLM API key for the
|
||||
selected model. Nothing in this module is exercised end-to-end without that
|
||||
environment; the argument-parsing and record-construction logic is unit
|
||||
tested with mocked subprocess calls (see ``tests/test_xben_run.py``), but
|
||||
actually launching a challenge and getting a real flag match is not faked
|
||||
here.
|
||||
|
||||
Challenge discovery
|
||||
--------------------
|
||||
The XBEN challenge set lives in the external ``usestrix/benchmarks``
|
||||
repository, not in this one. This script does not know that repository's
|
||||
exact on-disk layout, so it supports two ways to tell it about challenges:
|
||||
|
||||
1. ``--manifest PATH``: a JSON file you provide explicitly, shaped as a list
|
||||
of objects: ``{"challenge_id", "challenge_name", "difficulty", "target",
|
||||
"flag"}`` (``flag`` is optional; when omitted, ``flag_matched`` in the
|
||||
emitted record is always ``false`` and must be graded manually).
|
||||
2. ``--challenges-dir PATH`` (no manifest): every immediate subdirectory of
|
||||
``PATH`` containing a ``challenge.json`` file (with the same shape as one
|
||||
manifest entry, minus ``challenge_id`` which defaults to the directory
|
||||
name) is treated as one challenge. Adjust ``discover_challenges`` here if
|
||||
the real ``usestrix/benchmarks`` layout differs once you have it checked
|
||||
out.
|
||||
|
||||
Usage:
|
||||
python3 benchmarks/xben/run.py --manifest challenges.json \\
|
||||
--model anthropic/claude-sonnet-4-5 --mode black-box \\
|
||||
--out-dir benchmarks/xben/results
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from importlib.metadata import version
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from rich.console import Console
|
||||
|
||||
from strix.core.paths import run_dir_for
|
||||
|
||||
|
||||
console = Console()
|
||||
error_console = Console(stderr=True)
|
||||
|
||||
_DEFAULT_OUT_DIR = Path(__file__).parent / "results"
|
||||
_VALID_DIFFICULTIES = ("easy", "medium", "hard")
|
||||
_VALID_MODES = ("black-box", "grey-box", "white-box")
|
||||
|
||||
|
||||
class ChallengeSpecError(Exception):
|
||||
"""A challenge manifest/directory entry is missing required fields."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChallengeSpec:
|
||||
challenge_id: str
|
||||
challenge_name: str
|
||||
difficulty: str
|
||||
target: str
|
||||
flag: str | None = None
|
||||
|
||||
|
||||
def _challenge_from_dict(
|
||||
data: dict[str, Any], *, default_id: str | None, source: str
|
||||
) -> ChallengeSpec:
|
||||
challenge_id = str(data.get("challenge_id") or default_id or "")
|
||||
if not challenge_id:
|
||||
raise ChallengeSpecError(f"{source}: missing 'challenge_id'")
|
||||
|
||||
difficulty = str(data.get("difficulty", ""))
|
||||
if difficulty not in _VALID_DIFFICULTIES:
|
||||
raise ChallengeSpecError(
|
||||
f"{source}: 'difficulty' must be one of {_VALID_DIFFICULTIES}, got {difficulty!r}",
|
||||
)
|
||||
|
||||
target = data.get("target")
|
||||
if not target or not isinstance(target, str):
|
||||
raise ChallengeSpecError(f"{source}: missing or invalid 'target'")
|
||||
|
||||
flag = data.get("flag")
|
||||
return ChallengeSpec(
|
||||
challenge_id=challenge_id,
|
||||
challenge_name=str(data.get("challenge_name") or challenge_id),
|
||||
difficulty=difficulty,
|
||||
target=target,
|
||||
flag=str(flag) if flag else None,
|
||||
)
|
||||
|
||||
|
||||
def load_manifest(manifest_path: Path) -> list[ChallengeSpec]:
|
||||
raw: Any = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
if not isinstance(raw, list):
|
||||
raise ChallengeSpecError(f"{manifest_path}: manifest must be a JSON array")
|
||||
return [
|
||||
_challenge_from_dict(entry, default_id=None, source=f"{manifest_path}[{i}]")
|
||||
for i, entry in enumerate(raw)
|
||||
if isinstance(entry, dict)
|
||||
]
|
||||
|
||||
|
||||
def discover_challenges(challenges_dir: Path) -> list[ChallengeSpec]:
|
||||
"""Discover challenges from ``<challenges_dir>/<name>/challenge.json`` entries.
|
||||
|
||||
This assumes each challenge has a ``challenge.json`` describing its target
|
||||
and difficulty. Adjust this function once the real usestrix/benchmarks
|
||||
XBEN layout is available to run against — this repository was built
|
||||
without network access to check it out.
|
||||
"""
|
||||
if not challenges_dir.is_dir():
|
||||
raise FileNotFoundError(f"challenges directory not found: {challenges_dir}")
|
||||
|
||||
specs: list[ChallengeSpec] = []
|
||||
for child in sorted(challenges_dir.iterdir()):
|
||||
manifest_file = child / "challenge.json"
|
||||
if not child.is_dir() or not manifest_file.is_file():
|
||||
continue
|
||||
raw: Any = json.loads(manifest_file.read_text(encoding="utf-8"))
|
||||
if not isinstance(raw, dict):
|
||||
raise ChallengeSpecError(f"{manifest_file}: must be a JSON object")
|
||||
specs.append(_challenge_from_dict(raw, default_id=child.name, source=str(manifest_file)))
|
||||
|
||||
if not specs:
|
||||
raise ChallengeSpecError(
|
||||
f"no challenge.json files found under {challenges_dir} "
|
||||
"(pass --manifest instead if the challenge set uses a different layout)",
|
||||
)
|
||||
return specs
|
||||
|
||||
|
||||
def build_strix_command(
|
||||
*,
|
||||
strix_bin: str,
|
||||
target: str,
|
||||
run_name: str,
|
||||
max_budget_usd: float | None,
|
||||
max_turns: int | None,
|
||||
) -> list[str]:
|
||||
"""Build the ``strix`` CLI invocation for one challenge run.
|
||||
|
||||
Mirrors the flags documented in ``strix/interface/cli_args.py``:
|
||||
``--target``/``-t``, ``--non-interactive``/``-n`` (required for a
|
||||
subprocess run — the interactive TUI would hang), and the optional
|
||||
``--max-budget`` / ``--max-turns`` guards so a single stuck challenge
|
||||
can't burn the whole benchmark's budget.
|
||||
"""
|
||||
command = [
|
||||
strix_bin,
|
||||
"--target",
|
||||
target,
|
||||
"--non-interactive",
|
||||
"--run-name",
|
||||
run_name,
|
||||
]
|
||||
if max_budget_usd is not None:
|
||||
command += ["--max-budget", str(max_budget_usd)]
|
||||
if max_turns is not None:
|
||||
command += ["--max-turns", str(max_turns)]
|
||||
return command
|
||||
|
||||
|
||||
def run_strix_subprocess(
|
||||
command: list[str], *, env: dict[str, str]
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
"""Invoke the ``strix`` CLI as a subprocess and capture its output.
|
||||
|
||||
Requires Docker and a working LLM API key in ``env`` — this is the part
|
||||
of the pipeline that cannot be exercised in a sandbox without either.
|
||||
"""
|
||||
return subprocess.run( # noqa: S603
|
||||
command,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def read_run_json(run_dir: Path) -> dict[str, Any]:
|
||||
run_json_path = run_dir / "run.json"
|
||||
if not run_json_path.is_file():
|
||||
raise FileNotFoundError(f"no run.json produced under {run_dir}")
|
||||
data: Any = json.loads(run_json_path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
raise TypeError(f"{run_json_path} is not a JSON object")
|
||||
return data
|
||||
|
||||
|
||||
def flag_was_matched(run_dir: Path, flag: str | None) -> bool:
|
||||
"""Best-effort check for the challenge flag in the run's saved artifacts.
|
||||
|
||||
Searches ``vulnerabilities.json`` and the executive report markdown for
|
||||
the literal flag string. A real grading pass may want something more
|
||||
precise (e.g. the agent's own "finish" tool call payload); this is
|
||||
intentionally simple and documented as best-effort.
|
||||
"""
|
||||
if not flag:
|
||||
return False
|
||||
candidates = [run_dir / "vulnerabilities.json", run_dir / "penetration_test_report.md"]
|
||||
return any(path.is_file() and flag in path.read_text(encoding="utf-8") for path in candidates)
|
||||
|
||||
|
||||
def get_strix_version() -> str:
|
||||
try:
|
||||
return version("strix-agent")
|
||||
except Exception: # noqa: BLE001
|
||||
return "unknown"
|
||||
|
||||
|
||||
def build_run_record(
|
||||
*,
|
||||
challenge: ChallengeSpec,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
model: str,
|
||||
mode: str,
|
||||
started_at: datetime,
|
||||
finished_at: datetime,
|
||||
) -> dict[str, Any]:
|
||||
"""Construct a schema-conformant run record from a completed run's artifacts."""
|
||||
run_json = read_run_json(run_dir)
|
||||
llm_usage = run_json.get("llm_usage") or {}
|
||||
status = run_json.get("status")
|
||||
|
||||
matched = flag_was_matched(run_dir, challenge.flag)
|
||||
solved = matched if challenge.flag else status == "completed"
|
||||
|
||||
return {
|
||||
"challenge_id": challenge.challenge_id,
|
||||
"challenge_name": challenge.challenge_name,
|
||||
"difficulty": challenge.difficulty,
|
||||
"strix_version": get_strix_version(),
|
||||
"model": model,
|
||||
"mode": mode,
|
||||
"solved": bool(solved),
|
||||
"flag_matched": bool(matched),
|
||||
"duration_seconds": (finished_at - started_at).total_seconds(),
|
||||
"total_cost_usd": float(llm_usage.get("cost") or 0.0),
|
||||
"input_tokens": int(llm_usage.get("input_tokens") or 0),
|
||||
"output_tokens": int(llm_usage.get("output_tokens") or 0),
|
||||
"started_at": started_at.isoformat(),
|
||||
"finished_at": finished_at.isoformat(),
|
||||
"run_id": run_id,
|
||||
}
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
source = parser.add_mutually_exclusive_group(required=True)
|
||||
source.add_argument(
|
||||
"--manifest",
|
||||
type=Path,
|
||||
help="JSON file listing challenges explicitly (see module docstring for shape).",
|
||||
)
|
||||
source.add_argument(
|
||||
"--challenges-dir",
|
||||
type=Path,
|
||||
help="Directory of <challenge>/challenge.json entries (see module docstring).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
required=True,
|
||||
help="LLM identifier to run with, passed through as the STRIX_LLM env var "
|
||||
"(litellm-style, e.g. 'anthropic/claude-sonnet-4-5').",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mode",
|
||||
choices=_VALID_MODES,
|
||||
default="black-box",
|
||||
help="Testing mode recorded in the run record (default: black-box).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--out-dir",
|
||||
type=Path,
|
||||
default=_DEFAULT_OUT_DIR,
|
||||
help=f"Directory to write run-record JSON files to (default: {_DEFAULT_OUT_DIR}).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--strix-bin",
|
||||
default="strix",
|
||||
help="Path to the strix executable to invoke (default: 'strix' on PATH).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-budget",
|
||||
type=float,
|
||||
default=None,
|
||||
help="Per-challenge --max-budget passed through to strix, in USD.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-turns",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Per-challenge --max-turns passed through to strix.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--challenge-id",
|
||||
action="append",
|
||||
dest="challenge_ids",
|
||||
default=None,
|
||||
help="Only run this challenge id (repeatable). Default: run every discovered challenge.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Print the strix commands that would run, without executing them.",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def _select_challenges(
|
||||
challenges: list[ChallengeSpec], challenge_ids: list[str] | None
|
||||
) -> list[ChallengeSpec]:
|
||||
if not challenge_ids:
|
||||
return challenges
|
||||
wanted = set(challenge_ids)
|
||||
selected = [c for c in challenges if c.challenge_id in wanted]
|
||||
missing = wanted - {c.challenge_id for c in selected}
|
||||
if missing:
|
||||
raise ChallengeSpecError(f"unknown --challenge-id value(s): {sorted(missing)}")
|
||||
return selected
|
||||
|
||||
|
||||
def run_one_challenge(
|
||||
challenge: ChallengeSpec,
|
||||
*,
|
||||
args: argparse.Namespace,
|
||||
env: dict[str, str],
|
||||
run_dir_for_name: Any,
|
||||
) -> dict[str, Any]:
|
||||
"""Run a single challenge end-to-end and return its run record."""
|
||||
run_name = f"xben-{challenge.challenge_id}-{int(datetime.now(tz=UTC).timestamp())}"
|
||||
command = build_strix_command(
|
||||
strix_bin=args.strix_bin,
|
||||
target=challenge.target,
|
||||
run_name=run_name,
|
||||
max_budget_usd=args.max_budget,
|
||||
max_turns=args.max_turns,
|
||||
)
|
||||
|
||||
started_at = datetime.now(tz=UTC)
|
||||
result = run_strix_subprocess(command, env=env)
|
||||
finished_at = datetime.now(tz=UTC)
|
||||
|
||||
if result.returncode != 0:
|
||||
error_console.print(
|
||||
f"[bold red]strix exited {result.returncode} for {challenge.challenge_id}[/bold red]",
|
||||
)
|
||||
error_console.print(result.stderr)
|
||||
|
||||
run_dir = run_dir_for_name(run_name)
|
||||
return build_run_record(
|
||||
challenge=challenge,
|
||||
run_dir=run_dir,
|
||||
run_id=run_name,
|
||||
model=args.model,
|
||||
mode=args.mode,
|
||||
started_at=started_at,
|
||||
finished_at=finished_at,
|
||||
)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
|
||||
try:
|
||||
challenges = (
|
||||
load_manifest(args.manifest)
|
||||
if args.manifest
|
||||
else discover_challenges(args.challenges_dir)
|
||||
)
|
||||
challenges = _select_challenges(challenges, args.challenge_ids)
|
||||
except (ChallengeSpecError, FileNotFoundError, json.JSONDecodeError) as exc:
|
||||
error_console.print(f"[bold red]error:[/bold red] {exc}")
|
||||
return 1
|
||||
|
||||
if args.dry_run:
|
||||
for challenge in challenges:
|
||||
command = build_strix_command(
|
||||
strix_bin=args.strix_bin,
|
||||
target=challenge.target,
|
||||
run_name=f"xben-{challenge.challenge_id}-<timestamp>",
|
||||
max_budget_usd=args.max_budget,
|
||||
max_turns=args.max_turns,
|
||||
)
|
||||
console.print(f"[{challenge.challenge_id}] {' '.join(command)}")
|
||||
return 0
|
||||
|
||||
env = dict(os.environ)
|
||||
env["STRIX_LLM"] = args.model
|
||||
|
||||
args.out_dir.mkdir(parents=True, exist_ok=True)
|
||||
exit_code = 0
|
||||
for challenge in challenges:
|
||||
try:
|
||||
record = run_one_challenge(
|
||||
challenge,
|
||||
args=args,
|
||||
env=env,
|
||||
run_dir_for_name=run_dir_for,
|
||||
)
|
||||
except (FileNotFoundError, TypeError) as exc:
|
||||
error_console.print(f"[bold red]{challenge.challenge_id} failed:[/bold red] {exc}")
|
||||
exit_code = 1
|
||||
continue
|
||||
|
||||
out_path = args.out_dir / f"{challenge.challenge_id}.{args.model.replace('/', '_')}.json"
|
||||
out_path.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
|
||||
console.print(f"[green]wrote[/green] {out_path}")
|
||||
|
||||
return exit_code
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
104
benchmarks/xben/schema/run_record.schema.json
Normal file
104
benchmarks/xben/schema/run_record.schema.json
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://github.com/usestrix/strix/blob/main/benchmarks/xben/schema/run_record.schema.json",
|
||||
"title": "XBEN Run Record",
|
||||
"description": "A single Strix run against a single XBEN challenge. One JSON file per challenge run, committed under benchmarks/xben/results/, is the unit of provenance the benchmark's aggregate numbers (benchmarks/xben/score.py) are computed from. See benchmarks/xben/README.md, and GitHub issues #1263 and #1264 on usestrix/strix, for why this file exists.",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"challenge_id",
|
||||
"challenge_name",
|
||||
"difficulty",
|
||||
"strix_version",
|
||||
"model",
|
||||
"mode",
|
||||
"solved",
|
||||
"flag_matched",
|
||||
"duration_seconds",
|
||||
"total_cost_usd",
|
||||
"input_tokens",
|
||||
"output_tokens",
|
||||
"started_at",
|
||||
"finished_at",
|
||||
"run_id"
|
||||
],
|
||||
"properties": {
|
||||
"challenge_id": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Stable identifier of the XBEN challenge, as named in the usestrix/benchmarks XBEN challenge set (e.g. 'XBEN-001-24')."
|
||||
},
|
||||
"challenge_name": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Human-readable challenge title."
|
||||
},
|
||||
"difficulty": {
|
||||
"type": "string",
|
||||
"enum": ["easy", "medium", "hard"],
|
||||
"description": "Challenge difficulty tier. Maps to the existing benchmarks/README.md table as easy=Level 1, medium=Level 2, hard=Level 3."
|
||||
},
|
||||
"strix_version": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Strix version that produced this run, e.g. '0.4.0' (matches `strix --version` / the strix-agent package version)."
|
||||
},
|
||||
"model": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "LLM identifier used for the run, e.g. the value passed via the STRIX_LLM environment variable (litellm-style model string such as 'anthropic/claude-sonnet-4-5')."
|
||||
},
|
||||
"mode": {
|
||||
"type": "string",
|
||||
"enum": ["black-box", "grey-box", "white-box"],
|
||||
"description": "Testing mode: black-box (URL/deployed target only), grey-box (target + partial knowledge), or white-box (source code access)."
|
||||
},
|
||||
"solved": {
|
||||
"type": "boolean",
|
||||
"description": "Whether the agent was judged to have solved the challenge overall (normally identical to flag_matched for XBEN's CTF-style scoring, but kept distinct so a run can be recorded even when solved-ness is determined by something other than an exact flag match)."
|
||||
},
|
||||
"flag_matched": {
|
||||
"type": "boolean",
|
||||
"description": "Whether the agent recovered and reported the exact challenge flag."
|
||||
},
|
||||
"duration_seconds": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"description": "Wall-clock time for the run, in seconds."
|
||||
},
|
||||
"total_cost_usd": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"description": "Total LLM cost for the run in USD, as reported by Strix's usage accounting (run.json llm_usage.cost)."
|
||||
},
|
||||
"input_tokens": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"description": "Total input/prompt tokens consumed across all LLM calls in the run."
|
||||
},
|
||||
"output_tokens": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"description": "Total output/completion tokens produced across all LLM calls in the run."
|
||||
},
|
||||
"started_at": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "ISO 8601 timestamp (UTC) when the run started."
|
||||
},
|
||||
"finished_at": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "ISO 8601 timestamp (UTC) when the run finished."
|
||||
},
|
||||
"run_id": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Provenance link back to the underlying Strix run artifact directory, i.e. the '<run-name>' in 'strix_runs/<run-name>/' that this record was derived from."
|
||||
},
|
||||
"notes": {
|
||||
"type": "string",
|
||||
"description": "Optional free-text notes (e.g. manual grading remarks, anomalies, retries)."
|
||||
}
|
||||
}
|
||||
}
|
||||
451
benchmarks/xben/score.py
Normal file
451
benchmarks/xben/score.py
Normal file
|
|
@ -0,0 +1,451 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Aggregate and score committed XBEN run records.
|
||||
|
||||
Reads a directory of run-record JSON files (see
|
||||
``benchmarks/xben/schema/run_record.schema.json``), validates each one, and
|
||||
computes the same headline numbers hand-typed into ``benchmarks/README.md``:
|
||||
overall success rate, a breakdown by difficulty, average solve time, and
|
||||
total/average cost.
|
||||
|
||||
This is the "reproducible from committed data" half of closing out GitHub
|
||||
issues #1263 ("Make published XBEN metrics reproducible from committed
|
||||
data") and #1264 ("Record enough provenance to reproduce XBEN runs") on
|
||||
usestrix/strix: instead of a hand-typed table with no backing data, the
|
||||
table is regenerated mechanically from JSON files committed under
|
||||
``benchmarks/xben/results/``.
|
||||
|
||||
Usage:
|
||||
python3 benchmarks/xben/score.py [RESULTS_DIR] [--out PATH] [--check]
|
||||
|
||||
See ``benchmarks/xben/README.md`` for the full walkthrough.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from rich.console import Console
|
||||
|
||||
|
||||
_DEFAULT_RESULTS_DIR = Path(__file__).parent / "results"
|
||||
_DEFAULT_SCHEMA_PATH = Path(__file__).parent / "schema" / "run_record.schema.json"
|
||||
_DEFAULT_README_PATH = Path(__file__).parent.parent / "README.md"
|
||||
|
||||
_DIFFICULTY_LABELS: dict[str, str] = {
|
||||
"easy": "Level 1 (Easy)",
|
||||
"medium": "Level 2 (Medium)",
|
||||
"hard": "Level 3 (Hard)",
|
||||
}
|
||||
_DIFFICULTY_ORDER: tuple[str, ...] = ("easy", "medium", "hard")
|
||||
|
||||
console = Console()
|
||||
error_console = Console(stderr=True)
|
||||
|
||||
|
||||
class SchemaValidationError(Exception):
|
||||
"""A run-record JSON file failed validation against the XBEN schema."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Minimal JSON Schema validation.
|
||||
#
|
||||
# ``jsonschema`` is not a declared dependency of strix-agent (it only ever
|
||||
# shows up transitively via other packages' requirements), so rather than
|
||||
# depend on an undeclared package we implement just the subset of JSON
|
||||
# Schema (Draft 2020-12) that benchmarks/xben/schema/run_record.schema.json
|
||||
# actually uses: object/type/required/properties/additionalProperties,
|
||||
# enum, minimum, and the "date-time" format.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_TYPE_MAP: dict[str, tuple[type, ...]] = {
|
||||
"object": (dict,),
|
||||
"string": (str,),
|
||||
"boolean": (bool,),
|
||||
"number": (int, float),
|
||||
"integer": (int,),
|
||||
}
|
||||
|
||||
|
||||
def _check_type(value: Any, expected: str, path: str, errors: list[str]) -> None:
|
||||
python_types = _TYPE_MAP.get(expected)
|
||||
if python_types is None:
|
||||
return
|
||||
# bool is a subclass of int in Python; JSON Schema treats them as distinct.
|
||||
if expected in {"number", "integer"} and isinstance(value, bool):
|
||||
errors.append(f"{path}: expected {expected}, got boolean")
|
||||
return
|
||||
if not isinstance(value, python_types):
|
||||
errors.append(f"{path}: expected {expected}, got {type(value).__name__}")
|
||||
|
||||
|
||||
def _check_format(value: str, fmt: str, path: str, errors: list[str]) -> None:
|
||||
if fmt != "date-time":
|
||||
return
|
||||
try:
|
||||
datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
errors.append(f"{path}: '{value}' is not a valid ISO 8601 date-time")
|
||||
|
||||
|
||||
def _validate_property(
|
||||
value: Any, prop_schema: dict[str, Any], path: str, errors: list[str]
|
||||
) -> None:
|
||||
expected_type = prop_schema.get("type")
|
||||
if isinstance(expected_type, str):
|
||||
_check_type(value, expected_type, path, errors)
|
||||
|
||||
enum = prop_schema.get("enum")
|
||||
if enum is not None and value not in enum:
|
||||
errors.append(f"{path}: {value!r} is not one of {enum!r}")
|
||||
|
||||
minimum = prop_schema.get("minimum")
|
||||
if minimum is not None and isinstance(value, int | float) and value < minimum:
|
||||
errors.append(f"{path}: {value} is less than the minimum {minimum}")
|
||||
|
||||
min_length = prop_schema.get("minLength")
|
||||
if min_length is not None and isinstance(value, str) and len(value) < min_length:
|
||||
errors.append(f"{path}: string is shorter than minLength {min_length}")
|
||||
|
||||
fmt = prop_schema.get("format")
|
||||
if fmt is not None and isinstance(value, str):
|
||||
_check_format(value, fmt, path, errors)
|
||||
|
||||
|
||||
def validate_record(record: Any, schema: dict[str, Any], *, source: str) -> None:
|
||||
"""Validate ``record`` against ``schema``, raising with file+field context on failure."""
|
||||
errors: list[str] = []
|
||||
|
||||
if not isinstance(record, dict):
|
||||
raise SchemaValidationError(f"{source}: top-level value must be a JSON object")
|
||||
|
||||
required = schema.get("required", [])
|
||||
errors.extend(
|
||||
f"missing required field '{field_name}'"
|
||||
for field_name in required
|
||||
if field_name not in record
|
||||
)
|
||||
|
||||
properties: dict[str, Any] = schema.get("properties", {})
|
||||
for key, value in record.items():
|
||||
if key not in properties:
|
||||
if schema.get("additionalProperties") is False:
|
||||
errors.append(f"unexpected field '{key}' (not in schema)")
|
||||
continue
|
||||
_validate_property(value, properties[key], key, errors)
|
||||
|
||||
if errors:
|
||||
formatted = "; ".join(errors)
|
||||
raise SchemaValidationError(f"{source}: {formatted}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Loading and aggregation.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def load_schema(schema_path: Path) -> dict[str, Any]:
|
||||
data: Any = json.loads(schema_path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
raise SchemaValidationError(f"{schema_path}: schema file is not a JSON object")
|
||||
return data
|
||||
|
||||
|
||||
def load_results(results_dir: Path, schema: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""Load and validate every ``*.json`` run record under ``results_dir``.
|
||||
|
||||
Raises ``SchemaValidationError`` naming the offending file and field on the
|
||||
first invalid record — validation failures are meant to be loud, not
|
||||
silently skipped.
|
||||
"""
|
||||
if not results_dir.is_dir():
|
||||
raise FileNotFoundError(f"results directory not found: {results_dir}")
|
||||
|
||||
records: list[dict[str, Any]] = []
|
||||
for path in sorted(results_dir.glob("*.json")):
|
||||
try:
|
||||
raw: Any = json.loads(path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError as exc:
|
||||
raise SchemaValidationError(f"{path.name}: invalid JSON ({exc})") from exc
|
||||
validate_record(raw, schema, source=path.name)
|
||||
records.append(raw)
|
||||
|
||||
if not records:
|
||||
raise SchemaValidationError(f"no run-record *.json files found under {results_dir}")
|
||||
|
||||
return records
|
||||
|
||||
|
||||
@dataclass
|
||||
class DifficultyStats:
|
||||
solved: int = 0
|
||||
total: int = 0
|
||||
|
||||
@property
|
||||
def success_rate(self) -> float:
|
||||
return (self.solved / self.total * 100) if self.total else 0.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class AggregateResult:
|
||||
total_challenges: int = 0
|
||||
solved_challenges: int = 0
|
||||
total_duration_seconds: float = 0.0
|
||||
total_cost_usd: float = 0.0
|
||||
by_difficulty: dict[str, DifficultyStats] = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def success_rate(self) -> float:
|
||||
return (
|
||||
(self.solved_challenges / self.total_challenges * 100) if self.total_challenges else 0.0
|
||||
)
|
||||
|
||||
@property
|
||||
def avg_duration_seconds(self) -> float:
|
||||
return self.total_duration_seconds / self.total_challenges if self.total_challenges else 0.0
|
||||
|
||||
@property
|
||||
def avg_cost_usd(self) -> float:
|
||||
return self.total_cost_usd / self.total_challenges if self.total_challenges else 0.0
|
||||
|
||||
|
||||
def aggregate(records: list[dict[str, Any]]) -> AggregateResult:
|
||||
result = AggregateResult()
|
||||
for record in records:
|
||||
result.total_challenges += 1
|
||||
solved = bool(record["solved"])
|
||||
if solved:
|
||||
result.solved_challenges += 1
|
||||
result.total_duration_seconds += float(record["duration_seconds"])
|
||||
result.total_cost_usd += float(record["total_cost_usd"])
|
||||
|
||||
difficulty = str(record["difficulty"])
|
||||
stats = result.by_difficulty.setdefault(difficulty, DifficultyStats())
|
||||
stats.total += 1
|
||||
if solved:
|
||||
stats.solved += 1
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Markdown rendering, matching the shape of the existing benchmarks/README.md
|
||||
# tables so that file's tables can be replaced wholesale by this output.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _format_minutes(seconds: float) -> str:
|
||||
return f"~{round(seconds / 60)} minutes"
|
||||
|
||||
|
||||
def _format_cost(usd: float) -> str:
|
||||
return f"~${usd:,.0f}"
|
||||
|
||||
|
||||
def render_markdown(agg: AggregateResult) -> str:
|
||||
lines: list[str] = []
|
||||
lines.append("| Benchmark | Challenges | Success Rate |")
|
||||
lines.append("|-----------|------------|--------------|")
|
||||
lines.append(
|
||||
f"| XBEN | {agg.total_challenges} | **{agg.success_rate:.0f}%** |",
|
||||
)
|
||||
lines.append("")
|
||||
lines.append("**Performance by Difficulty:**")
|
||||
lines.append("")
|
||||
lines.append("| Difficulty | Solved | Success Rate |")
|
||||
lines.append("|------------|--------|--------------|")
|
||||
for key in _DIFFICULTY_ORDER:
|
||||
stats = agg.by_difficulty.get(key)
|
||||
if stats is None:
|
||||
continue
|
||||
label = _DIFFICULTY_LABELS[key]
|
||||
lines.append(f"| {label} | {stats.solved}/{stats.total} | {stats.success_rate:.0f}% |")
|
||||
lines.append("")
|
||||
lines.append("**Resource Usage:**")
|
||||
lines.append(f"- Average solve time: {_format_minutes(agg.avg_duration_seconds)}")
|
||||
lines.append(
|
||||
f"- Total cost: {_format_cost(agg.total_cost_usd)} for {agg.total_challenges} challenges",
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# --check: parse the numbers currently hand-typed in benchmarks/README.md and
|
||||
# compare them against what's computed from committed data.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReadmeClaims:
|
||||
total_challenges: int
|
||||
success_rate: float
|
||||
by_difficulty: dict[str, DifficultyStats]
|
||||
|
||||
|
||||
_OVERALL_ROW_RE = re.compile(
|
||||
r"\|\s*\[?XBEN\]?[^|]*\|\s*(\d+)\s*\|\s*\*\*(\d+(?:\.\d+)?)%\*\*\s*\|",
|
||||
)
|
||||
_DIFFICULTY_ROW_RE = re.compile(
|
||||
r"\|\s*Level\s+(\d+)\s*\(([A-Za-z]+)\)\s*\|\s*(\d+)/(\d+)\s*\|\s*(\d+(?:\.\d+)?)%\s*\|",
|
||||
)
|
||||
|
||||
|
||||
def parse_readme_claims(readme_text: str) -> ReadmeClaims:
|
||||
"""Extract the headline numbers from the existing hand-typed README table."""
|
||||
overall_match = _OVERALL_ROW_RE.search(readme_text)
|
||||
if not overall_match:
|
||||
raise SchemaValidationError(
|
||||
"could not find the XBEN results row (e.g. '| XBEN | 104 | **96%** |') in the README",
|
||||
)
|
||||
total_challenges = int(overall_match.group(1))
|
||||
success_rate = float(overall_match.group(2))
|
||||
|
||||
by_difficulty: dict[str, DifficultyStats] = {}
|
||||
for match in _DIFFICULTY_ROW_RE.finditer(readme_text):
|
||||
level_name = match.group(2).lower()
|
||||
difficulty = next((d for d in _DIFFICULTY_ORDER if d == level_name), None)
|
||||
if difficulty is None:
|
||||
continue
|
||||
solved, total = int(match.group(3)), int(match.group(4))
|
||||
by_difficulty[difficulty] = DifficultyStats(solved=solved, total=total)
|
||||
|
||||
return ReadmeClaims(
|
||||
total_challenges=total_challenges,
|
||||
success_rate=success_rate,
|
||||
by_difficulty=by_difficulty,
|
||||
)
|
||||
|
||||
|
||||
def diff_against_claims(agg: AggregateResult, claims: ReadmeClaims) -> list[str]:
|
||||
"""Return a list of human-readable mismatches between ``agg`` and ``claims``."""
|
||||
mismatches: list[str] = []
|
||||
|
||||
if agg.total_challenges != claims.total_challenges:
|
||||
mismatches.append(
|
||||
f"total challenges: README claims {claims.total_challenges}, "
|
||||
f"committed data has {agg.total_challenges}",
|
||||
)
|
||||
if round(agg.success_rate) != round(claims.success_rate):
|
||||
mismatches.append(
|
||||
f"overall success rate: README claims {claims.success_rate:.0f}%, "
|
||||
f"committed data computes {agg.success_rate:.0f}%",
|
||||
)
|
||||
|
||||
for difficulty in _DIFFICULTY_ORDER:
|
||||
# A difficulty absent from either side is equivalent to a zero-challenge
|
||||
# DifficultyStats(0, 0) for that side, so a table row with "0/0" and no
|
||||
# committed challenges of that difficulty compare as equal rather than
|
||||
# as "present on one side only".
|
||||
claimed = claims.by_difficulty.get(difficulty, DifficultyStats())
|
||||
actual = agg.by_difficulty.get(difficulty, DifficultyStats())
|
||||
if claimed.solved != actual.solved or claimed.total != actual.total:
|
||||
mismatches.append(
|
||||
f"difficulty '{difficulty}': README claims {claimed.solved}/{claimed.total}, "
|
||||
f"committed data has {actual.solved}/{actual.total}",
|
||||
)
|
||||
|
||||
return mismatches
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Validate and aggregate committed XBEN run records into the "
|
||||
"benchmarks/README.md results table."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"results_dir",
|
||||
type=Path,
|
||||
nargs="?",
|
||||
default=_DEFAULT_RESULTS_DIR,
|
||||
help=f"Directory of run-record *.json files (default: {_DEFAULT_RESULTS_DIR}).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--schema",
|
||||
type=Path,
|
||||
default=_DEFAULT_SCHEMA_PATH,
|
||||
help=f"Path to the run-record JSON Schema (default: {_DEFAULT_SCHEMA_PATH}).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--out",
|
||||
type=Path,
|
||||
default=None,
|
||||
help="Write the markdown table to this file instead of stdout.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--check",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Exit non-zero if the aggregated numbers don't match what's currently "
|
||||
"hand-typed in the README (this is what CI runs)."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--readme",
|
||||
type=Path,
|
||||
default=_DEFAULT_README_PATH,
|
||||
help=f"README to check against with --check (default: {_DEFAULT_README_PATH}).",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
|
||||
try:
|
||||
schema = load_schema(args.schema)
|
||||
records = load_results(args.results_dir, schema)
|
||||
except (SchemaValidationError, FileNotFoundError) as exc:
|
||||
error_console.print(f"[bold red]error:[/bold red] {exc}")
|
||||
return 1
|
||||
|
||||
agg = aggregate(records)
|
||||
markdown = render_markdown(agg)
|
||||
|
||||
if args.out:
|
||||
args.out.write_text(markdown, encoding="utf-8")
|
||||
console.print(f"Wrote markdown table to {args.out}")
|
||||
else:
|
||||
console.print(markdown, highlight=False, soft_wrap=True)
|
||||
|
||||
if args.check:
|
||||
if not args.readme.is_file():
|
||||
error_console.print(f"[bold red]error:[/bold red] README not found: {args.readme}")
|
||||
return 1
|
||||
readme_text = args.readme.read_text(encoding="utf-8")
|
||||
try:
|
||||
claims = parse_readme_claims(readme_text)
|
||||
except SchemaValidationError as exc:
|
||||
error_console.print(f"[bold red]error:[/bold red] {exc}")
|
||||
return 1
|
||||
|
||||
mismatches = diff_against_claims(agg, claims)
|
||||
if mismatches:
|
||||
error_console.print(
|
||||
f"[bold red]--check failed:[/bold red] aggregated numbers from "
|
||||
f"{args.results_dir} do not match {args.readme}:",
|
||||
)
|
||||
for mismatch in mismatches:
|
||||
error_console.print(f" - {mismatch}")
|
||||
return 1
|
||||
console.print(
|
||||
f"[bold green]--check passed:[/bold green] {args.readme} matches committed data."
|
||||
)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
331
tests/test_xben_run.py
Normal file
331
tests/test_xben_run.py
Normal file
|
|
@ -0,0 +1,331 @@
|
|||
"""Tests for the XBEN benchmark runner (benchmarks/xben/run.py).
|
||||
|
||||
This module drives a real ``strix`` subprocess against a live Docker + LLM
|
||||
environment, neither of which is available here. Every test below mocks
|
||||
``subprocess.run`` (via ``run.run_strix_subprocess``) and the filesystem
|
||||
artifacts a real run would produce, so only the argument-parsing and
|
||||
run-record-construction logic is exercised -- never a real scan.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from benchmarks.xben import run
|
||||
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _manifest_entry(**overrides: Any) -> dict[str, Any]:
|
||||
base: dict[str, Any] = {
|
||||
"challenge_id": "XBEN-001-24",
|
||||
"challenge_name": "Test Challenge",
|
||||
"difficulty": "easy",
|
||||
"target": "https://target.example.com",
|
||||
"flag": "FLAG{test}",
|
||||
}
|
||||
base.update(overrides)
|
||||
return base
|
||||
|
||||
|
||||
class TestChallengeFromDict:
|
||||
def test_valid_entry(self) -> None:
|
||||
spec = run._challenge_from_dict(_manifest_entry(), default_id=None, source="test")
|
||||
assert spec.challenge_id == "XBEN-001-24"
|
||||
assert spec.difficulty == "easy"
|
||||
assert spec.target == "https://target.example.com"
|
||||
assert spec.flag == "FLAG{test}"
|
||||
|
||||
def test_default_id_used_when_missing(self) -> None:
|
||||
entry = _manifest_entry()
|
||||
del entry["challenge_id"]
|
||||
spec = run._challenge_from_dict(entry, default_id="dir-name", source="test")
|
||||
assert spec.challenge_id == "dir-name"
|
||||
|
||||
def test_missing_id_raises(self) -> None:
|
||||
entry = _manifest_entry()
|
||||
del entry["challenge_id"]
|
||||
with pytest.raises(run.ChallengeSpecError, match="challenge_id"):
|
||||
run._challenge_from_dict(entry, default_id=None, source="test")
|
||||
|
||||
def test_invalid_difficulty_raises(self) -> None:
|
||||
with pytest.raises(run.ChallengeSpecError, match="difficulty"):
|
||||
run._challenge_from_dict(
|
||||
_manifest_entry(difficulty="impossible"), default_id=None, source="test"
|
||||
)
|
||||
|
||||
def test_missing_target_raises(self) -> None:
|
||||
entry = _manifest_entry()
|
||||
del entry["target"]
|
||||
with pytest.raises(run.ChallengeSpecError, match="target"):
|
||||
run._challenge_from_dict(entry, default_id=None, source="test")
|
||||
|
||||
def test_missing_flag_is_allowed(self) -> None:
|
||||
entry = _manifest_entry()
|
||||
del entry["flag"]
|
||||
spec = run._challenge_from_dict(entry, default_id=None, source="test")
|
||||
assert spec.flag is None
|
||||
|
||||
|
||||
class TestLoadManifest:
|
||||
def test_loads_list_of_entries(self, tmp_path: Path) -> None:
|
||||
manifest = tmp_path / "manifest.json"
|
||||
manifest.write_text(json.dumps([_manifest_entry(), _manifest_entry(challenge_id="B")]))
|
||||
specs = run.load_manifest(manifest)
|
||||
assert [s.challenge_id for s in specs] == ["XBEN-001-24", "B"]
|
||||
|
||||
def test_non_list_manifest_raises(self, tmp_path: Path) -> None:
|
||||
manifest = tmp_path / "manifest.json"
|
||||
manifest.write_text(json.dumps({"not": "a list"}))
|
||||
with pytest.raises(run.ChallengeSpecError, match="must be a JSON array"):
|
||||
run.load_manifest(manifest)
|
||||
|
||||
|
||||
class TestDiscoverChallenges:
|
||||
def test_discovers_challenge_json_per_subdirectory(self, tmp_path: Path) -> None:
|
||||
chal_dir = tmp_path / "chal-1"
|
||||
chal_dir.mkdir()
|
||||
entry = _manifest_entry()
|
||||
del entry["challenge_id"]
|
||||
(chal_dir / "challenge.json").write_text(json.dumps(entry))
|
||||
|
||||
specs = run.discover_challenges(tmp_path)
|
||||
assert len(specs) == 1
|
||||
assert specs[0].challenge_id == "chal-1"
|
||||
|
||||
def test_missing_directory_raises(self, tmp_path: Path) -> None:
|
||||
with pytest.raises(FileNotFoundError):
|
||||
run.discover_challenges(tmp_path / "missing")
|
||||
|
||||
def test_no_challenges_found_raises(self, tmp_path: Path) -> None:
|
||||
with pytest.raises(run.ChallengeSpecError, match=r"no challenge\.json"):
|
||||
run.discover_challenges(tmp_path)
|
||||
|
||||
|
||||
class TestBuildStrixCommand:
|
||||
def test_basic_command_shape(self) -> None:
|
||||
command = run.build_strix_command(
|
||||
strix_bin="strix",
|
||||
target="https://target.example.com",
|
||||
run_name="xben-run-1",
|
||||
max_budget_usd=None,
|
||||
max_turns=None,
|
||||
)
|
||||
assert command == [
|
||||
"strix",
|
||||
"--target",
|
||||
"https://target.example.com",
|
||||
"--non-interactive",
|
||||
"--run-name",
|
||||
"xben-run-1",
|
||||
]
|
||||
|
||||
def test_includes_optional_budget_and_turns(self) -> None:
|
||||
command = run.build_strix_command(
|
||||
strix_bin="strix",
|
||||
target="https://target.example.com",
|
||||
run_name="xben-run-1",
|
||||
max_budget_usd=5.0,
|
||||
max_turns=100,
|
||||
)
|
||||
assert "--max-budget" in command
|
||||
assert command[command.index("--max-budget") + 1] == "5.0"
|
||||
assert "--max-turns" in command
|
||||
assert command[command.index("--max-turns") + 1] == "100"
|
||||
|
||||
|
||||
class TestFlagWasMatched:
|
||||
def test_no_flag_means_not_matched(self, tmp_path: Path) -> None:
|
||||
assert run.flag_was_matched(tmp_path, None) is False
|
||||
|
||||
def test_flag_found_in_vulnerabilities_json(self, tmp_path: Path) -> None:
|
||||
(tmp_path / "vulnerabilities.json").write_text('[{"evidence": "FLAG{found-it}"}]')
|
||||
assert run.flag_was_matched(tmp_path, "FLAG{found-it}") is True
|
||||
|
||||
def test_flag_found_in_report_markdown(self, tmp_path: Path) -> None:
|
||||
(tmp_path / "penetration_test_report.md").write_text("We recovered FLAG{in-report}.")
|
||||
assert run.flag_was_matched(tmp_path, "FLAG{in-report}") is True
|
||||
|
||||
def test_flag_not_present_anywhere(self, tmp_path: Path) -> None:
|
||||
(tmp_path / "vulnerabilities.json").write_text("[]")
|
||||
assert run.flag_was_matched(tmp_path, "FLAG{missing}") is False
|
||||
|
||||
|
||||
class TestBuildRunRecord:
|
||||
def test_builds_conformant_record_from_run_json(self, tmp_path: Path) -> None:
|
||||
run_dir = tmp_path / "strix_runs" / "xben-run-1"
|
||||
run_dir.mkdir(parents=True)
|
||||
(run_dir / "run.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"status": "completed",
|
||||
"llm_usage": {"cost": 2.5, "input_tokens": 1000, "output_tokens": 200},
|
||||
},
|
||||
),
|
||||
)
|
||||
(run_dir / "vulnerabilities.json").write_text('[{"evidence": "FLAG{yes}"}]')
|
||||
|
||||
challenge = run.ChallengeSpec(
|
||||
challenge_id="XBEN-001-24",
|
||||
challenge_name="Test",
|
||||
difficulty="easy",
|
||||
target="https://target.example.com",
|
||||
flag="FLAG{yes}",
|
||||
)
|
||||
started = datetime(2026, 1, 1, tzinfo=UTC)
|
||||
finished = datetime(2026, 1, 1, 0, 5, tzinfo=UTC)
|
||||
|
||||
record = run.build_run_record(
|
||||
challenge=challenge,
|
||||
run_dir=run_dir,
|
||||
run_id="xben-run-1",
|
||||
model="anthropic/claude-sonnet-4-5",
|
||||
mode="black-box",
|
||||
started_at=started,
|
||||
finished_at=finished,
|
||||
)
|
||||
|
||||
assert record["challenge_id"] == "XBEN-001-24"
|
||||
assert record["difficulty"] == "easy"
|
||||
assert record["model"] == "anthropic/claude-sonnet-4-5"
|
||||
assert record["mode"] == "black-box"
|
||||
assert record["flag_matched"] is True
|
||||
assert record["solved"] is True
|
||||
assert record["duration_seconds"] == pytest.approx(300.0)
|
||||
assert record["total_cost_usd"] == pytest.approx(2.5)
|
||||
assert record["input_tokens"] == 1000
|
||||
assert record["output_tokens"] == 200
|
||||
assert record["run_id"] == "xben-run-1"
|
||||
|
||||
def test_missing_run_json_raises(self, tmp_path: Path) -> None:
|
||||
challenge = run.ChallengeSpec(
|
||||
challenge_id="X",
|
||||
challenge_name="X",
|
||||
difficulty="easy",
|
||||
target="https://target.example.com",
|
||||
)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
run.build_run_record(
|
||||
challenge=challenge,
|
||||
run_dir=tmp_path,
|
||||
run_id="run-1",
|
||||
model="m",
|
||||
mode="black-box",
|
||||
started_at=datetime.now(tz=UTC),
|
||||
finished_at=datetime.now(tz=UTC),
|
||||
)
|
||||
|
||||
def test_unsolved_without_flag_uses_status(self, tmp_path: Path) -> None:
|
||||
run_dir = tmp_path
|
||||
(run_dir / "run.json").write_text(json.dumps({"status": "completed", "llm_usage": {}}))
|
||||
challenge = run.ChallengeSpec(
|
||||
challenge_id="X",
|
||||
challenge_name="X",
|
||||
difficulty="easy",
|
||||
target="https://target.example.com",
|
||||
flag=None,
|
||||
)
|
||||
record = run.build_run_record(
|
||||
challenge=challenge,
|
||||
run_dir=run_dir,
|
||||
run_id="run-1",
|
||||
model="m",
|
||||
mode="black-box",
|
||||
started_at=datetime.now(tz=UTC),
|
||||
finished_at=datetime.now(tz=UTC),
|
||||
)
|
||||
assert record["flag_matched"] is False
|
||||
assert record["solved"] is True # falls back to run status when there's no flag to check
|
||||
|
||||
|
||||
class TestRunOneChallengeWithMockedSubprocess:
|
||||
def test_invokes_subprocess_and_builds_record(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
challenge = run.ChallengeSpec(
|
||||
challenge_id="XBEN-001-24",
|
||||
challenge_name="Test",
|
||||
difficulty="easy",
|
||||
target="https://target.example.com",
|
||||
flag="FLAG{yes}",
|
||||
)
|
||||
|
||||
fake_result = subprocess.CompletedProcess(
|
||||
args=["strix"], returncode=0, stdout="", stderr=""
|
||||
)
|
||||
mock_run = MagicMock(return_value=fake_result)
|
||||
monkeypatch.setattr(run, "run_strix_subprocess", mock_run)
|
||||
|
||||
def fake_run_dir_for(run_name: str) -> Path:
|
||||
run_dir = tmp_path / run_name
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
(run_dir / "run.json").write_text(
|
||||
json.dumps({"status": "completed", "llm_usage": {"cost": 1.0}}),
|
||||
)
|
||||
(run_dir / "vulnerabilities.json").write_text('[{"evidence": "FLAG{yes}"}]')
|
||||
return run_dir
|
||||
|
||||
args = run.build_parser().parse_args(
|
||||
["--manifest", "unused.json", "--model", "test-model"],
|
||||
)
|
||||
|
||||
record = run.run_one_challenge(
|
||||
challenge,
|
||||
args=args,
|
||||
env={},
|
||||
run_dir_for_name=fake_run_dir_for,
|
||||
)
|
||||
|
||||
mock_run.assert_called_once()
|
||||
called_command = mock_run.call_args.args[0]
|
||||
assert called_command[:3] == ["strix", "--target", "https://target.example.com"]
|
||||
assert record["challenge_id"] == "XBEN-001-24"
|
||||
assert record["flag_matched"] is True
|
||||
|
||||
|
||||
class TestMainDryRun:
|
||||
def test_dry_run_prints_commands_without_running_strix(
|
||||
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
manifest = tmp_path / "manifest.json"
|
||||
manifest.write_text(json.dumps([_manifest_entry()]))
|
||||
|
||||
mock_run = MagicMock()
|
||||
monkeypatch.setattr(run, "run_strix_subprocess", mock_run)
|
||||
|
||||
exit_code = run.main(["--manifest", str(manifest), "--model", "test-model", "--dry-run"])
|
||||
|
||||
assert exit_code == 0
|
||||
mock_run.assert_not_called()
|
||||
captured = capsys.readouterr()
|
||||
assert "XBEN-001-24" in captured.out
|
||||
assert "--target" in captured.out
|
||||
|
||||
def test_unknown_challenge_id_filter_errors(
|
||||
self, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
manifest = tmp_path / "manifest.json"
|
||||
manifest.write_text(json.dumps([_manifest_entry()]))
|
||||
|
||||
exit_code = run.main(
|
||||
[
|
||||
"--manifest",
|
||||
str(manifest),
|
||||
"--model",
|
||||
"test-model",
|
||||
"--challenge-id",
|
||||
"does-not-exist",
|
||||
"--dry-run",
|
||||
],
|
||||
)
|
||||
assert exit_code == 1
|
||||
captured = capsys.readouterr()
|
||||
assert "does-not-exist" in captured.err
|
||||
296
tests/test_xben_score.py
Normal file
296
tests/test_xben_score.py
Normal file
|
|
@ -0,0 +1,296 @@
|
|||
"""Tests for the XBEN benchmark scorer (benchmarks/xben/score.py)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pytest
|
||||
|
||||
from benchmarks.xben import score
|
||||
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _schema() -> dict[str, Any]:
|
||||
return score.load_schema(score._DEFAULT_SCHEMA_PATH)
|
||||
|
||||
|
||||
def _record(**overrides: Any) -> dict[str, Any]:
|
||||
base: dict[str, Any] = {
|
||||
"challenge_id": "XBEN-001-24",
|
||||
"challenge_name": "Test Challenge",
|
||||
"difficulty": "easy",
|
||||
"strix_version": "0.4.0",
|
||||
"model": "anthropic/claude-sonnet-4-5",
|
||||
"mode": "black-box",
|
||||
"solved": True,
|
||||
"flag_matched": True,
|
||||
"duration_seconds": 300,
|
||||
"total_cost_usd": 1.5,
|
||||
"input_tokens": 1000,
|
||||
"output_tokens": 100,
|
||||
"started_at": "2026-01-01T00:00:00Z",
|
||||
"finished_at": "2026-01-01T00:05:00Z",
|
||||
"run_id": "run-1",
|
||||
}
|
||||
base.update(overrides)
|
||||
return base
|
||||
|
||||
|
||||
class TestValidateRecord:
|
||||
def test_valid_record_passes(self) -> None:
|
||||
score.validate_record(_record(), _schema(), source="test.json")
|
||||
|
||||
def test_missing_required_field_is_reported_by_name(self) -> None:
|
||||
record = _record()
|
||||
del record["challenge_id"]
|
||||
with pytest.raises(score.SchemaValidationError) as exc_info:
|
||||
score.validate_record(record, _schema(), source="bad.json")
|
||||
assert "bad.json" in str(exc_info.value)
|
||||
assert "challenge_id" in str(exc_info.value)
|
||||
|
||||
def test_invalid_enum_value_is_reported(self) -> None:
|
||||
record = _record(difficulty="impossible")
|
||||
with pytest.raises(score.SchemaValidationError) as exc_info:
|
||||
score.validate_record(record, _schema(), source="bad.json")
|
||||
assert "difficulty" in str(exc_info.value)
|
||||
|
||||
def test_wrong_type_is_reported(self) -> None:
|
||||
record = _record(solved="yes")
|
||||
with pytest.raises(score.SchemaValidationError) as exc_info:
|
||||
score.validate_record(record, _schema(), source="bad.json")
|
||||
assert "solved" in str(exc_info.value)
|
||||
|
||||
def test_negative_number_below_minimum_is_reported(self) -> None:
|
||||
record = _record(duration_seconds=-1)
|
||||
with pytest.raises(score.SchemaValidationError) as exc_info:
|
||||
score.validate_record(record, _schema(), source="bad.json")
|
||||
assert "duration_seconds" in str(exc_info.value)
|
||||
|
||||
def test_invalid_date_time_format_is_reported(self) -> None:
|
||||
record = _record(started_at="not-a-date")
|
||||
with pytest.raises(score.SchemaValidationError) as exc_info:
|
||||
score.validate_record(record, _schema(), source="bad.json")
|
||||
assert "started_at" in str(exc_info.value)
|
||||
|
||||
def test_unexpected_field_is_reported(self) -> None:
|
||||
record = _record(unknown_field="oops")
|
||||
with pytest.raises(score.SchemaValidationError) as exc_info:
|
||||
score.validate_record(record, _schema(), source="bad.json")
|
||||
assert "unknown_field" in str(exc_info.value)
|
||||
|
||||
def test_optional_notes_field_is_allowed(self) -> None:
|
||||
score.validate_record(_record(notes="all good"), _schema(), source="test.json")
|
||||
|
||||
def test_non_object_record_is_rejected(self) -> None:
|
||||
with pytest.raises(score.SchemaValidationError):
|
||||
score.validate_record(["not", "an", "object"], _schema(), source="bad.json")
|
||||
|
||||
|
||||
class TestLoadResults:
|
||||
def test_loads_valid_records(self, tmp_path: Path) -> None:
|
||||
(tmp_path / "a.json").write_text(json.dumps(_record(challenge_id="A")), encoding="utf-8")
|
||||
(tmp_path / "b.json").write_text(json.dumps(_record(challenge_id="B")), encoding="utf-8")
|
||||
records = score.load_results(tmp_path, _schema())
|
||||
assert {r["challenge_id"] for r in records} == {"A", "B"}
|
||||
|
||||
def test_missing_directory_raises(self, tmp_path: Path) -> None:
|
||||
with pytest.raises(FileNotFoundError):
|
||||
score.load_results(tmp_path / "does-not-exist", _schema())
|
||||
|
||||
def test_empty_directory_raises(self, tmp_path: Path) -> None:
|
||||
with pytest.raises(score.SchemaValidationError, match="no run-record"):
|
||||
score.load_results(tmp_path, _schema())
|
||||
|
||||
def test_invalid_json_names_the_file(self, tmp_path: Path) -> None:
|
||||
(tmp_path / "broken.json").write_text("{not valid json", encoding="utf-8")
|
||||
with pytest.raises(score.SchemaValidationError, match=r"broken\.json"):
|
||||
score.load_results(tmp_path, _schema())
|
||||
|
||||
def test_invalid_record_stops_loading_and_names_file(self, tmp_path: Path) -> None:
|
||||
(tmp_path / "good.json").write_text(json.dumps(_record()), encoding="utf-8")
|
||||
bad = _record()
|
||||
del bad["model"]
|
||||
(tmp_path / "bad.json").write_text(json.dumps(bad), encoding="utf-8")
|
||||
with pytest.raises(score.SchemaValidationError, match=r"bad\.json"):
|
||||
score.load_results(tmp_path, _schema())
|
||||
|
||||
def test_fixture_results_directory_is_valid(self) -> None:
|
||||
# The committed example fixtures under benchmarks/xben/results/ must
|
||||
# themselves validate cleanly -- this is what score.py --help's
|
||||
# documented default invocation runs against.
|
||||
records = score.load_results(score._DEFAULT_RESULTS_DIR, _schema())
|
||||
assert len(records) >= 1
|
||||
|
||||
|
||||
class TestAggregate:
|
||||
def test_overall_success_rate(self) -> None:
|
||||
records = [
|
||||
_record(challenge_id="1", difficulty="easy", solved=True),
|
||||
_record(challenge_id="2", difficulty="easy", solved=False),
|
||||
]
|
||||
agg = score.aggregate(records)
|
||||
assert agg.total_challenges == 2
|
||||
assert agg.solved_challenges == 1
|
||||
assert agg.success_rate == pytest.approx(50.0)
|
||||
|
||||
def test_breakdown_by_difficulty(self) -> None:
|
||||
records = [
|
||||
_record(challenge_id="1", difficulty="easy", solved=True),
|
||||
_record(challenge_id="2", difficulty="easy", solved=True),
|
||||
_record(challenge_id="3", difficulty="hard", solved=False),
|
||||
]
|
||||
agg = score.aggregate(records)
|
||||
assert agg.by_difficulty["easy"].solved == 2
|
||||
assert agg.by_difficulty["easy"].total == 2
|
||||
assert agg.by_difficulty["hard"].solved == 0
|
||||
assert agg.by_difficulty["hard"].total == 1
|
||||
|
||||
def test_average_duration_and_cost(self) -> None:
|
||||
records = [
|
||||
_record(challenge_id="1", duration_seconds=100, total_cost_usd=1.0),
|
||||
_record(challenge_id="2", duration_seconds=300, total_cost_usd=3.0),
|
||||
]
|
||||
agg = score.aggregate(records)
|
||||
assert agg.avg_duration_seconds == pytest.approx(200.0)
|
||||
assert agg.total_cost_usd == pytest.approx(4.0)
|
||||
assert agg.avg_cost_usd == pytest.approx(2.0)
|
||||
|
||||
def test_empty_aggregate_does_not_divide_by_zero(self) -> None:
|
||||
agg = score.aggregate([])
|
||||
assert agg.success_rate == 0.0
|
||||
assert agg.avg_duration_seconds == 0.0
|
||||
assert agg.avg_cost_usd == 0.0
|
||||
|
||||
|
||||
class TestRenderMarkdown:
|
||||
def test_contains_expected_headers_and_rows(self) -> None:
|
||||
agg = score.aggregate(
|
||||
[
|
||||
_record(challenge_id="1", difficulty="easy", solved=True),
|
||||
_record(challenge_id="2", difficulty="hard", solved=False),
|
||||
],
|
||||
)
|
||||
markdown = score.render_markdown(agg)
|
||||
assert "| Benchmark | Challenges | Success Rate |" in markdown
|
||||
assert "| Difficulty | Solved | Success Rate |" in markdown
|
||||
assert "Level 1 (Easy)" in markdown
|
||||
assert "Level 3 (Hard)" in markdown
|
||||
assert "Average solve time" in markdown
|
||||
assert "Total cost" in markdown
|
||||
|
||||
|
||||
class TestReadmeCheck:
|
||||
_README_TEMPLATE = """\
|
||||
## Results
|
||||
|
||||
| Benchmark | Challenges | Success Rate |
|
||||
|-----------|------------|--------------|
|
||||
| XBEN | {total} | **{rate:.0f}%** |
|
||||
|
||||
**Performance by Difficulty:**
|
||||
|
||||
| Difficulty | Solved | Success Rate |
|
||||
|------------|--------|--------------|
|
||||
| Level 1 (Easy) | {easy_solved}/{easy_total} | {easy_rate:.0f}% |
|
||||
| Level 2 (Medium) | {medium_solved}/{medium_total} | {medium_rate:.0f}% |
|
||||
| Level 3 (Hard) | {hard_solved}/{hard_total} | {hard_rate:.0f}% |
|
||||
"""
|
||||
|
||||
def _readme_matching(self, agg: score.AggregateResult) -> str:
|
||||
easy = agg.by_difficulty.get("easy", score.DifficultyStats())
|
||||
medium = agg.by_difficulty.get("medium", score.DifficultyStats())
|
||||
hard = agg.by_difficulty.get("hard", score.DifficultyStats())
|
||||
return self._README_TEMPLATE.format(
|
||||
total=agg.total_challenges,
|
||||
rate=agg.success_rate,
|
||||
easy_solved=easy.solved,
|
||||
easy_total=easy.total,
|
||||
easy_rate=easy.success_rate,
|
||||
medium_solved=medium.solved,
|
||||
medium_total=medium.total,
|
||||
medium_rate=medium.success_rate,
|
||||
hard_solved=hard.solved,
|
||||
hard_total=hard.total,
|
||||
hard_rate=hard.success_rate,
|
||||
)
|
||||
|
||||
def test_matching_readme_has_no_mismatches(self) -> None:
|
||||
records = [
|
||||
_record(challenge_id="1", difficulty="easy", solved=True),
|
||||
_record(challenge_id="2", difficulty="medium", solved=True),
|
||||
_record(challenge_id="3", difficulty="hard", solved=False),
|
||||
]
|
||||
agg = score.aggregate(records)
|
||||
readme_text = self._readme_matching(agg)
|
||||
claims = score.parse_readme_claims(readme_text)
|
||||
assert score.diff_against_claims(agg, claims) == []
|
||||
|
||||
def test_mismatched_readme_is_reported(self) -> None:
|
||||
records = [_record(challenge_id="1", difficulty="easy", solved=True)]
|
||||
agg = score.aggregate(records)
|
||||
readme_text = self._readme_matching(agg)
|
||||
# Now aggregate a different set of records that disagrees.
|
||||
other_records = [
|
||||
_record(challenge_id="1", difficulty="easy", solved=True),
|
||||
_record(challenge_id="2", difficulty="easy", solved=False),
|
||||
]
|
||||
other_agg = score.aggregate(other_records)
|
||||
claims = score.parse_readme_claims(readme_text)
|
||||
mismatches = score.diff_against_claims(other_agg, claims)
|
||||
assert mismatches
|
||||
assert any("total challenges" in m for m in mismatches)
|
||||
|
||||
def test_real_readme_and_fixtures_are_out_of_sync(self) -> None:
|
||||
# This is the exact situation documented in benchmarks/xben/results/README.md:
|
||||
# the fixture data is synthetic and intentionally does not reproduce the
|
||||
# real README's headline claim, so --check against it must fail.
|
||||
agg = score.aggregate(score.load_results(score._DEFAULT_RESULTS_DIR, _schema()))
|
||||
readme_text = score._DEFAULT_README_PATH.read_text(encoding="utf-8")
|
||||
claims = score.parse_readme_claims(readme_text)
|
||||
assert score.diff_against_claims(agg, claims) != []
|
||||
|
||||
|
||||
class TestCli:
|
||||
def test_main_prints_table_to_stdout(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||
exit_code = score.main([str(score._DEFAULT_RESULTS_DIR)])
|
||||
assert exit_code == 0
|
||||
captured = capsys.readouterr()
|
||||
assert "Benchmark" in captured.out
|
||||
|
||||
def test_main_writes_table_to_out_file(self, tmp_path: Path) -> None:
|
||||
out_path = tmp_path / "table.md"
|
||||
exit_code = score.main([str(score._DEFAULT_RESULTS_DIR), "--out", str(out_path)])
|
||||
assert exit_code == 0
|
||||
assert "Benchmark" in out_path.read_text(encoding="utf-8")
|
||||
|
||||
def test_main_check_fails_against_real_readme_with_fixture_data(self) -> None:
|
||||
exit_code = score.main([str(score._DEFAULT_RESULTS_DIR), "--check"])
|
||||
assert exit_code == 1
|
||||
|
||||
def test_main_check_passes_with_matching_readme(self, tmp_path: Path) -> None:
|
||||
(tmp_path / "a.json").write_text(
|
||||
json.dumps(_record(challenge_id="A", difficulty="easy", solved=True)),
|
||||
encoding="utf-8",
|
||||
)
|
||||
agg = score.aggregate([_record(challenge_id="A", difficulty="easy", solved=True)])
|
||||
readme_path = tmp_path / "README.md"
|
||||
readme_path.write_text(TestReadmeCheck()._readme_matching(agg), encoding="utf-8")
|
||||
exit_code = score.main(
|
||||
[str(tmp_path), "--check", "--readme", str(readme_path)],
|
||||
)
|
||||
assert exit_code == 0
|
||||
|
||||
def test_main_reports_invalid_record_and_exits_nonzero(
|
||||
self, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
bad = _record()
|
||||
del bad["run_id"]
|
||||
(tmp_path / "bad.json").write_text(json.dumps(bad), encoding="utf-8")
|
||||
exit_code = score.main([str(tmp_path)])
|
||||
assert exit_code == 1
|
||||
captured = capsys.readouterr()
|
||||
assert "bad.json" in captured.err
|
||||
Loading…
Add table
Reference in a new issue