Bryan Helmkamp
f3efcc931e
Record GPT-5.4 20min timeout: 65.7% on SWE-Bench-Lite ($718.62)
...
Best resolve rate at 197/300 but 3.3x more expensive than Opus.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 20:55:59 -04:00
Bryan Helmkamp
dfcece9a0d
Record Opus 4.6 20min timeout: 58.0% on SWE-Bench-Lite ($218.65)
...
100% patch rate, 0 timeouts. Only +1 instance over Sonnet at 4x the cost.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 14:43:20 -04:00
Bryan Helmkamp
165742d7fa
Record Sonnet 4.6 20min timeout: 57.7% on SWE-Bench-Lite ($55.22)
...
20min timeout vs 10min: 173 vs 167 resolved (+6), patch rate 99% vs 94%.
Also fixes: revert to v4 snapshots, concurrency default to 100, preflight
uses actual 4 CPU per sandbox.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 13:36:26 -04:00
Bryan Helmkamp
942054c43a
Record timeout and sandbox resources in scoreboard metadata
...
Adds --timeout, --sandbox-cpu, --sandbox-memory flags to record_results.py.
Re-recorded both existing runs with the new fields.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 11:24:56 -04:00
Bryan Helmkamp
5eccf9701b
Record Sonnet 4.6 baseline: 55.7% on SWE-Bench-Lite ($39.78)
...
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 11:23:23 -04:00
Bryan Helmkamp
0431ad1a05
Add scoreboard system and record Haiku 4.5 baseline: 54.0% on SWE-Bench-Lite
...
record_results.py combines generation + eval results into a git-tracked
scoreboard. Per run: README, meta.json, instances.jsonl. Auto-generates
leaderboard.json ranked by resolve rate.
Haiku 4.5 baseline: 162/300 (54.0%), $26.13 total ($0.087/instance).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 09:56:43 -04:00
