Commit graph

10 commits

Author SHA1 Message Date
Bryan Helmkamp
5fc9157017
refactor(workflow): remove retro stage (#230)
## Summary

Removes Fabro's automatic retro generation stage so workflow runs go
directly from execution to finalization and optional PR creation. This
drops the retro-specific crate, events, projection fields, config/API
knobs, and user-facing docs in favor of the existing durable run
observability surfaces.

## What Changed

- Deleted the `fabro-retro` crate and the workflow `retro` pipeline
phase, with finalization now consuming `Executed` state directly.
- Removed retro configuration and API surface area, including
`--no-retro`, `[run.execution].retros`, manifest `no_retro`,
`features.retros`, and run projection `retro*` fields.
- Retired typed `retro.*` events while keeping historical event logs
readable by deserializing retired retro event names as `Unknown`.
- Stopped appending retro sections to generated PR bodies and updated
docs, marketing copy, screenshots, and navigation to point users toward
observability/event-stream inspection.

## Testing

Not run during PR creation; this branch already contained the
implementation commit.

---

[![Compound
Engineering](https://img.shields.io/badge/Compound_Engineering-6366f1)](https://github.com/EveryInc/compound-engineering-plugin)
🤖 Generated with GPT-5 (unknown context, reasoning unspecified) via
[Codex](https://openai.com/codex)
2026-05-09 10:18:20 -04:00
Bryan Helmkamp
d8155a305c Reduce default patch generation concurrency to 75
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 16:05:24 -04:00
Bryan Helmkamp
0db08b937c Fix snapshot CPU mismatch: use v4 snapshots (4 CPU), reduce concurrency to 100
Daytona bakes CPU/memory at snapshot creation time. v4 snapshots have
4 CPU / 8 GB. Preflight now checks against 4 CPU per sandbox.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 12:44:14 -04:00
Bryan Helmkamp
241be62cde Reduce default concurrency to 150, increase default timeout to 20min
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 12:32:18 -04:00
Bryan Helmkamp
adf6594b97 Add Daytona CPU preflight check before starting eval runs
Checks running sandboxes against the 500 CPU org limit with 20% buffer.
Exits with a suggested --max-workers value if capacity is insufficient.
Default concurrency set to 200 (safe with 2 CPU per sandbox).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 12:20:36 -04:00
Bryan Helmkamp
e26286d966 Add best-effort Daytona sandbox cleanup on timeout and disable PR creation
On timeout, finds the orphaned sandbox via fabro ps --label and deletes
it. Non-fatal if cleanup fails. Also adds [pull_request] enabled=false
to generated workflow.toml configs to prevent eval runs from opening PRs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 10:42:47 -04:00
Bryan Helmkamp
b491542e95 Make SWE-bench patch generation and evaluation resumable
On restart, reads existing output JSONL files to find completed instance
IDs, skips them, and appends new results. Final summary recomputes from
the full results file so it reflects all runs combined.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 10:33:06 -04:00
Bryan Helmkamp
fe1f4f346c Add Daytona-based SWE-bench evaluator and bump default concurrency to 100
evaluate_daytona.py runs the swebench test harness on Daytona sandboxes
instead of local Docker. Reuses the same snapshots from the generation
phase. Applies model patch + test patch, runs tests, grades with
swebench's log parsers. No local Docker needed.

Also bumps default --max-workers from 20 to 100 in run_eval.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 09:13:09 -04:00
Bryan Helmkamp
0ef79bfd3e Fix run_eval.py: resolve output-dir to absolute path
When --output-dir is relative and fabro runs from /tmp, generated
workflow.toml paths were unresolvable.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 08:39:24 -04:00
Bryan Helmkamp
e246b7f4bb Add SWE-bench evaluation harness
Python scripts for running SWE-bench Lite evals against Fabro agent
in Daytona sandboxes: instance orchestration, Dockerfile generation,
and result evaluation via the official swebench harness.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 08:37:24 -04:00