Bryan Helmkamp
c592c681ff
Persist fabro_run_dir in scoreboard instances.jsonl
...
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 16:43:25 -04:00
Bryan Helmkamp
241be62cde
Reduce default concurrency to 150, increase default timeout to 20min
...
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 12:32:18 -04:00
Bryan Helmkamp
942054c43a
Record timeout and sandbox resources in scoreboard metadata
...
Adds --timeout, --sandbox-cpu, --sandbox-memory flags to record_results.py.
Re-recorded both existing runs with the new fields.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 11:24:56 -04:00
Bryan Helmkamp
0431ad1a05
Add scoreboard system and record Haiku 4.5 baseline: 54.0% on SWE-Bench-Lite
...
record_results.py combines generation + eval results into a git-tracked
scoreboard. Per run: README, meta.json, instances.jsonl. Auto-generates
leaderboard.json ranked by resolve rate.
Haiku 4.5 baseline: 162/300 (54.0%), $26.13 total ($0.087/instance).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 09:56:43 -04:00
