Commit graph

660 commits

Author SHA1 Message Date
Bryan Helmkamp
3eef9ee928
Model Test Bounded Concurrency Implementation Plan (#204)
This change makes bulk `fabro model test` run configured model checks
concurrently instead of serially. A new `--jobs/-j` flag (defaulting to
4, minimum 1) controls the concurrency bound; the single-model path
(`--model <MODEL>`) is unaffected. Under the hood, the serial `for` loop
over configured models is replaced with a
`futures::stream::buffer_unordered(jobs)` pipeline that clones the
shared-state `Client` per request. Completed results carry their
original list index and are sorted before rendering, so final stdout
table rows and JSON output remain in listing order regardless of which
requests finish first.

Three new integration tests verify the concurrency behavior using an
inline Axum harness with a `ConcurrencyGate` barrier. The gate holds all
in-flight requests until the expected number arrive simultaneously, then
releases them, letting tests assert `max_in_flight` exactly rather than
relying on timing. The ordering test goes further by assigning
reverse-listing response delays so the last-listed model always finishes
first; if the index sort were dropped, the JSON result order would
invert and the assertion would fail. A 15-second gate timeout ensures a
regression to serial execution surfaces as a clear `max_in_flight == 1`
failure rather than a hung test.

Existing behavior is fully preserved: unconfigured models are still
skipped without a POST, a configured model returning `skip` after
listing is still a failure, `--deep` uses the same `--jobs` value, and
`--jobs 1` reproduces the previous serial behavior for users hitting
provider rate limits.

### Fabro Details

<details>
<summary>Ran 9 stages in 30m 52s for $19.61</summary>

| Stage | Duration | Cost | Retries |
|---|---|---|---|
| start | 0s | – | 0 |
| toolchain | 1s | – | 0 |
| preflight_compile | 2m 8s | – | 0 |
| preflight_lint | 2m 21s | – | 0 |
| implement | 8m 56s | $3.87 | 0 |
| simplify_opus | 7m 55s | $1.43 | 0 |
| simplify_gpt | 6m 58s | $14.32 | 0 |
| verify | 1m 49s | – | 0 |
| fmt | 2s | – | 0 |
| **Total** | **30m 52s** | **$19.61** | **0** |

</details>

<details>
<summary>Ran <code>ImplementPlan.fabro</code> (12 nodes and 15
edges)</summary>

```dot
digraph ImplementPlan {
    graph [
        goal="Implement and simplify",
        model_stylesheet="
            * { model: claude-opus-4-7; }
        "
    ]
    rankdir=LR

    start [shape=Mdiamond, label="Start"]
    exit  [shape=Msquare, label="Exit"]

    toolchain         [label="Toolchain", shape=parallelogram, script="command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1", max_retries=0]
    preflight_compile [label="Preflight Compile", shape=parallelogram, script="cargo check -q --workspace 2>&1", max_retries=0]
    preflight_lint    [label="Preflight Lint", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1", max_retries=0]
    fix_lints         [label="Fix Lints", prompt="The preflight lint step failed. Read the build output from context and fix all clippy lint warnings.", max_visits=3]
    implement         [label="Implement", prompt="Read the plan file referenced in the goal and implement every step. Make all the code changes described in the plan. Use red/green TDD."]
    simplify_opus     [label="Simplify (Opus)", prompt="@prompts/simplify.md"]
    simplify_gpt      [label="Simplify (GPT-55)", prompt="@prompts/simplify.md", model="gpt-55"]
    verify            [label="Verify", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --cargo-quiet --workspace --status-level fail 2>&1 && cargo dev docs refresh 2>&1 && cargo dev docs check 2>&1", goal_gate=true, retry_target="fixup"]
    fixup             [label="Fixup", prompt="The verify step failed. Read the build output from context and fix all clippy lint warnings, test failures, and generated docs errors.", max_visits=3]
    fmt               [label="Format", shape=parallelogram, script="cargo +nightly-2026-04-14 fmt --all 2>&1", max_retries=0]

    start -> toolchain
    toolchain -> preflight_compile [condition="outcome=succeeded"]
    toolchain -> exit
    preflight_compile -> preflight_lint [condition="outcome=succeeded"]
    preflight_compile -> exit
    preflight_lint -> implement [condition="outcome=succeeded"]
    preflight_lint -> fix_lints
    fix_lints -> preflight_lint
    implement -> simplify_opus -> simplify_gpt -> verify
    verify -> fmt   [condition="outcome=succeeded"]
    verify -> fixup
    fixup -> verify
    fmt -> exit
}

```

</details>

⚒️ Generated with [Fabro](https://fabro.sh)

---------

Co-authored-by: Fabro <noreply@fabro.sh>
2026-05-04 13:46:44 -04:00
Bryan Helmkamp
92cfcbde71
chore: update plan 2026-05-04 12:55:19 -04:00
Bryan Helmkamp
2ef34a228e
docs: sync public docs to recent runtime changes 2026-05-04 11:43:03 -04:00
Bryan Helmkamp
31cbdb5c31
docs(changelog): refresh recent product changes 2026-05-04 11:38:55 -04:00
Bryan Helmkamp
f4f5416db8
docs: clarify chain-rendering boundary in error strategy
`thiserror`-derived `Display` does not walk `#[source]`, so `format!("{err}")`
and `format!("{err:#}")` on a typed error silently produce only the
top-level message — the same format string changes meaning when migrating
from `anyhow::Result` to a typed `Result`. Point at
`fabro_util::error::collect_chain` as the canonical helper and broaden
the test guidance to cover typed errors.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 11:28:28 -04:00
Bryan Helmkamp
e0594a86bd
chore: add plan 2026-05-04 11:12:40 -04:00
Bryan Helmkamp
be0b5829b9
chore: add plan 2026-05-04 11:12:40 -04:00
Bryan Helmkamp
95eb13750a
fix(billing): render non-LLM run stages
Include completed stages without LLM usage in run billing responses so command-only runs still show runtime rows. Keep token and model aggregates scoped to billed LLM usage, and render placeholder values in the web billing table.
2026-05-03 22:49:33 -04:00
Bryan Helmkamp
6780dff63f
fix(workflow): retain exec output tails on failures
Keep sandbox exec failures structured until event emission so git push, checkpoint, notice, and retro failures can expose redacted output tails without expanding their terse error strings.

Also add log rendering that appends sanitized tail content for exec-backed errors while preserving the existing safe Display behavior.
2026-05-03 21:30:34 -04:00
Bryan Helmkamp
446c7cc065
fix(sandbox): surface snapshot lifecycle progress
Emit snapshot slow-path events only when Docker or Daytona actually performs image or snapshot work, replace retired completion markers with snapshot.ready, and render the lifecycle in attach/log output.
2026-05-03 20:44:16 -04:00
Bryan Helmkamp
649b57c185
feat(api): expose web_url on run responses and run.created event
So that CLI and other API consumers can surface a clickable link to the
run's web UI page instead of guessing route shapes or probing settings.
The server populates `web_url` from `server.web.enabled` and
`server.web.url`, returns it on `RunStatusResponse` (create plus all
lifecycle transitions), and persists it on the `run.created` event so
attach replays the same link without re-deriving it.

CLI: prints `Web UI: <url>` as a run-header info line, driven off the
replayed event so fresh runs and `attach` share one code path. Absent
when the UI is disabled.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 18:18:19 -04:00
Bryan Helmkamp
e5c5feaa8d
fix(workflows): pin clippy and fmt to nightly-2026-04-14
The smoke and implement-plan workflows ran cargo clippy without a
toolchain prefix, so on the Daytona snapshot they fell through to the
baked-in stable toolchain. clippy.toml now uses allow-unwrap-types
(added in clippy 1.95), which the stable in fabro-v7 doesn't recognize.
Pin every fmt and clippy invocation to nightly-2026-04-14 so they match
.github/workflows/rust.yml. Also update the public repl-handoff example
to keep the documented template consistent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 17:32:46 -04:00
Bryan Helmkamp
5c60fe8182
chore: plans 2026-05-03 13:10:04 -04:00
Bryan Helmkamp
bfb6bdb25c
fix(cli): bound server readiness probes
Use HTTP health checks with short deadlines for managed server readiness and add finite control-plane request timeouts for CLI/server clients. Keep stream bodies uncapped so SSE attach flows can remain long-lived.
2026-05-02 20:13:18 -04:00
Bryan Helmkamp
ae5ccb5ce2
refactor(workflow): split event module by responsibility
Keep fabro_workflow::event as the public facade while moving event conversion, names, redaction, sink, emitter, stored-field helpers, and StageScope into focused modules. Co-locate the existing event tests with the moved code and update the events strategy docs for the new module layout.
2026-05-02 14:52:10 -04:00
Bryan Helmkamp
43f03087e6
Merge remote-tracking branch 'origin/main' 2026-05-02 10:06:25 -04:00
Bryan Helmkamp
f6b8d1acdb
Fix principal auth gap regressions 2026-05-02 10:02:12 -04:00
Bryan Helmkamp
29c45498b0
Fix run principal attribution gaps 2026-05-02 09:14:02 -04:00
Bryan Helmkamp
f628b91c04
Merge origin/main
Reconcile parallel refactors that converged on stage vocabulary from
opposite directions. Origin's vocabulary wins everywhere it conflicts:
StageProjection (struct), StageState (status enum, kept), StageCompletion
(record). Local's artifact-retries feature, OpenAPI typing of JSON-blob
fields, parallel awaits in build_conclusion_from_store, and shared
RETRY_SEGMENT_PREFIX constant carry forward on top.

Origin's stage_at_visit reducer helpers (8 arms) supersede the local
stage_entry_with_current_visit (5 arms); same intent, broader coverage.
Origin's iter_stages_mut and first_event_seq helper unification are
preserved unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 22:05:41 -04:00
Bryan Helmkamp
8d7b9a804a
Unify run event principals 2026-05-01 21:56:47 -04:00
Bryan Helmkamp
cc81f538f4
refactor(api): type StageState JSON-blob fields so the TS client stops emitting any
provider_used, script_invocation, and script_timing become object | null;
parallel_results becomes Array<object> | null. The Rust StageState type is
unaffected because fabro-api/build.rs replaces it with fabro_types::StageState.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 21:16:56 -04:00
Bryan Helmkamp
95dae5afac
feat(run): separate stage state and artifact retries 2026-05-01 20:10:47 -04:00
Bryan Helmkamp
cea1fa739d
refactor(run-projection): use stage vocabulary 2026-05-01 19:56:22 -04:00
Bryan Helmkamp
b3e2b818c3
chore: docs 2026-05-01 19:35:02 -04:00
Bryan Helmkamp
e13a7e5506
docs(error): document error handling strategy 2026-05-01 15:05:07 -04:00
Bryan Helmkamp
1b30a2803a
fix(cli): remove dev-token env auth override 2026-05-01 15:04:01 -04:00
Bryan Helmkamp
155d0a5a28
Merge remote-tracking branch 'origin/main' 2026-05-01 09:43:39 -04:00
Bryan Helmkamp
5ef4b87878
Merge remote-tracking branch 'origin/main' 2026-05-01 09:34:44 -04:00
Bryan Helmkamp
b96647314c
docs(deployment): split into local vs self-hosted, drop Render/Fly/DO
Reframe the deployment docs around the actual product story: Fabro
runs as a server, and the only deployment question is where that
server runs (laptop vs self-hosted Docker). Drop the Render, Fly.io,
and DigitalOcean guides and their config files; keep Railway as the
managed shortcut.

- New: administration/deployment.mdx (overview, two-mode framing)
- New: administration/self-host-docker.mdx (compose-first how-to)
- Move: administration/deploy-server.mdx -> reference/server-operations.mdx
  (it was operational reference, not deploy guidance)
- Delete: deploy-render.mdx, deploy-fly-io.mdx, deploy-digital-ocean.mdx
- Delete: render.yaml, fly.toml, railway.toml, Dockerfile.deploy
- docker-compose.yaml: load .env if present so users can drive the
  stack from a single env file end-to-end
- Update internal links and the docs-test that pinned the old path
2026-05-01 09:26:35 -04:00
Bryan Helmkamp
2f7aeba417
fix(workflow): preserve exec failure diagnostics
Add bounded redacted exec output tails to failure events while keeping tracing log-safe. Centralize tail projection on ExecResult and thread diagnostics through metadata, setup, devcontainer, and CLI install failures.
2026-05-01 08:45:50 -04:00
Bryan Helmkamp
93908de46c
refactor(retro): share run dump hydration
Move RunDump into fabro-dump so CLI export and retro uploads share the same hydrated run layout. Drop the legacy artifact file-ref parser, add best-effort run.log retrieval for retro, and update retro prompts/docs to use events.jsonl and checkpoints.
2026-04-30 23:31:12 -04:00
Bryan Helmkamp
e50df2b58b
feat(command): distinguish cancelled commands from timeouts
Represent command termination explicitly across sandbox results, events,
run projections, API types, and the run stage UI. This removes the fake
-1 exit code path for timeout/cancel and lets consumers tell cancelled
commands apart from timed-out commands.
2026-04-30 22:45:37 -04:00
Bryan Helmkamp
8ac400df1c
feat(command): stream command logs from CAS-backed storage
Persist command stdout/stderr through scratch logs and finalized CAS refs, expose byte-offset tailing through the API, and render separate streaming panels in the web run view.

Resolve command output blob refs for execution-time consumers such as edge routing and retros, and make Docker streaming timeout/cancel drain output before returning.
2026-04-30 22:45:37 -04:00
Bryan Helmkamp
7e62dae28b
chore: add plans 2026-04-30 15:04:44 -04:00
Bryan Helmkamp
015992208a
feat(models): expose configured status
Add configured to the model API contract and server responses so clients can see whether provider credential material exists before testing. Use that signal in bulk model tests to skip unconfigured providers before printing progress and treat post-list skips as race failures.
2026-04-30 09:15:02 -04:00
Bryan Helmkamp
b5f200d701
chore(web): namespace static images under /images and skip in HTTP logs
Move favicon, logo, logotype, and PNG icons from /public/ root to
/public/images/ so the HTTP log middleware can drop them by path
prefix. Extends the existing /assets/ skip in http_log_middleware to
cover /images/ as well, removing favicon/logo entries from the server
log without filtering by extension (which would risk muting future
extension-suffixed API routes).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 08:57:14 -04:00
Bryan Helmkamp
fb51c1485b
docs(plans): add command output streaming and type unification plans
Stage the two 2026-04-30 plan documents: command output streaming with
CAS log storage, and the duplicate-type unification rollup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 08:36:00 -04:00
Bryan Helmkamp
157ec03a18
docs: sync public docs to recent runtime changes
Reflect Docker as the default sandbox provider, add `skip_clone` for
clone-based providers, document the `[run.sandbox.docker]` config
table, and update tutorial command lines from `files-internal/...` to
`docs/internal/...`. Bump the docs skill watermark to the latest synced
commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 08:35:53 -04:00
Bryan Helmkamp
d92ee47fe3
chore(demo): move simplify prompt to prompts/ alongside graphs
Rename `files-internal/prompts/simplify.md` to `prompts/simplify.md`
adjacent to the .fabro files that reference it, and update the
plan-implement and simplify demos plus the plan-implement test fixture
to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 08:35:48 -04:00
Bryan Helmkamp
c8185eaee6
docs(changelog): refresh recent product changes 2026-04-30 08:11:08 -04:00
Bryan Helmkamp
d65c1fa635
refactor(types): remove stage status compatibility 2026-04-30 06:48:47 -04:00
Bryan Helmkamp
1862024b61
refactor(api): tighten shared type schemas 2026-04-30 06:20:05 -04:00
Bryan Helmkamp
78a2f8638b
refactor(api): reuse billing and model domain types 2026-04-30 06:15:18 -04:00
Bryan Helmkamp
f16391485b
refactor(workflow): update stage outcome semantics 2026-04-30 06:06:51 -04:00
Bryan Helmkamp
71ec9fdee6
Merge remote-tracking branch 'origin/main'
# Conflicts:
#	lib/crates/fabro-types/src/lib.rs
2026-04-29 20:43:12 -04:00
Bryan Helmkamp
6a8329b376
refactor(api): align run projection types 2026-04-29 20:32:50 -04:00
Bryan Helmkamp
4a4f0f8548
feat(workflow): emit metadata snapshot events
Add typed metadata snapshot events around init, checkpoint, and finalize archive writes so run logs expose durable metadata timing and failures. Include snapshot accounting, CLI rendering with compatibility-notice suppression, and event documentation.
2026-04-29 19:31:13 -04:00
Bryan Helmkamp
7feef1c6b2
Merge remote-tracking branch 'origin/main' 2026-04-29 13:01:43 -04:00
Bryan Helmkamp
6156f65799
fix(run-files): normalize degraded file diffs
Return degraded run files with the same FileDiff[] shape as live responses, using nullable contents and per-file unified patches so the web sidebar and deep links work consistently.
2026-04-29 13:00:55 -04:00
Bryan Helmkamp
890fbb8fa4
feat(server): correlate API errors with request ids
Generate a fresh UUIDv4 per request, attach it to response headers,
JSON error bodies, and HTTP response logs so client-visible failures can be
matched to server logs without trusting inbound request id headers.
2026-04-29 12:20:56 -04:00