On timeout, finds the orphaned sandbox via fabro ps --label and deletes
it. Non-fatal if cleanup fails. Also adds [pull_request] enabled=false
to generated workflow.toml configs to prevent eval runs from opening PRs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On restart, reads existing output JSONL files to find completed instance
IDs, skips them, and appends new results. Final summary recomputes from
the full results file so it reflects all runs combined.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
evaluate_daytona.py runs the swebench test harness on Daytona sandboxes
instead of local Docker. Reuses the same snapshots from the generation
phase. Applies model patch + test patch, runs tests, grades with
swebench's log parsers. No local Docker needed.
Also bumps default --max-workers from 20 to 100 in run_eval.py.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When --output-dir is relative and fabro runs from /tmp, generated
workflow.toml paths were unresolvable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Python scripts for running SWE-bench Lite evals against Fabro agent
in Daytona sandboxes: instance orchestration, Dockerfile generation,
and result evaluation via the official swebench harness.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
