20min timeout vs 10min: 173 vs 167 resolved (+6), patch rate 99% vs 94%.
Also fixes: revert to v4 snapshots, concurrency default to 100, preflight
uses actual 4 CPU per sandbox.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Daytona bakes CPU/memory at snapshot creation time. v4 snapshots have
4 CPU / 8 GB. Preflight now checks against 4 CPU per sandbox.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Checks running sandboxes against the 500 CPU org limit with 20% buffer.
Exits with a suggested --max-workers value if capacity is insufficient.
Default concurrency set to 200 (safe with 2 CPU per sandbox).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds --timeout, --sandbox-cpu, --sandbox-memory flags to record_results.py.
Re-recorded both existing runs with the new fields.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On timeout, finds the orphaned sandbox via fabro ps --label and deletes
it. Non-fatal if cleanup fails. Also adds [pull_request] enabled=false
to generated workflow.toml configs to prevent eval runs from opening PRs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On restart, reads existing output JSONL files to find completed instance
IDs, skips them, and appends new results. Final summary recomputes from
the full results file so it reflects all runs combined.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
evaluate_daytona.py runs the swebench test harness on Daytona sandboxes
instead of local Docker. Reuses the same snapshots from the generation
phase. Applies model patch + test patch, runs tests, grades with
swebench's log parsers. No local Docker needed.
Also bumps default --max-workers from 20 to 100 in run_eval.py.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When --output-dir is relative and fabro runs from /tmp, generated
workflow.toml paths were unresolvable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Python scripts for running SWE-bench Lite evals against Fabro agent
in Daytona sandboxes: instance orchestration, Dockerfile generation,
and result evaluation via the official swebench harness.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
