The release, docker, and Homebrew jobs ran on ubuntu-latest, which
migrates across Ubuntu major versions on GitHub's schedule. Pin to
ubuntu-24.04, the image ubuntu-latest resolved to in the last green
release run, matching the explicit runner labels used elsewhere.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
setup-bun installed the latest Bun at run time, so every job floated to
new Bun releases the day they shipped. Bun bundles the SPA embedded in
release binaries, so an unvetted Bun release could break or silently
change shipped artifacts. Pin to 1.3.14, the version the last green
nightly used, and hold off on the day-old 1.4.0 until it has soaked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rust 1.98.0 (released 2026-08-20) passes --fix-cortex-a53-843419 to the
linker for aarch64-unknown-linux-musl, which the zig cc wrapper used by
cargo-zigbuild rejects, breaking the release build for that target. Pin
all workflows that installed unpinned stable to 1.97.1 until the zig
toolchain handles the new flag. The nightly-2026-04-14 fmt/clippy
toolchains are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
## What
Fixes the `openai_twin_*` parity-matrix failures that have been on
`main` since #449: every multi-turn scenario whose scripted response
includes text fails on its second turn with 400 `"message input items
require supported content"`.
## Root cause
Two twin behaviors collided (bisected: passes at #447, fails at #449):
1. **The twin's streaming `response.output_item.done` for message items
omitted the `content` array** (`test/twin/openai/src/sse.rs`) — it sent
only `id`/`type`/`status`/`role`, where the real API sends the completed
item in full. The openai adapter preserves message output items verbatim
(`ContentPart::Other { kind: OPENAI_MESSAGE }`) and replays them as
assistant history on the next turn — required so reasoning items keep
their "required following item" in Responses round-trips. So the replay
arrived content-less.
2. **#449 tightened the twin's input validation** to also validate
explicit `type: "message"` items (previously only type-less items were
validated as messages; anything with an explicit type was accepted
unchecked). The twin started rejecting its own round-tripped output.
The new validation caught a real infidelity in the emitter — the emit
side is what's wrong.
Nobody noticed because **CI never runs the twin e2e suites**: `rust.yml`
runs `--profile ci` without `--run-ignored`, so the parity matrix only
runs when someone invokes the e2e profile locally.
## Fix
- The streamed message `output_item.done` now carries its `output_text`
content, matching the real API and the twin's own non-streaming
`responses_json()`.
- The input validator accepts `output_text` parts on **assistant**
message items (the real API allows these; the twin's non-streaming
responses already require it for faithful replay). Non-assistant
`output_text` parts get a dedicated rejection message.
## Tests
- New contract test
`responses_stream_message_item_done_round_trips_as_input`: streams a
response, asserts the completed message item carries its `output_text`
content, and replays the item verbatim as assistant-history input,
asserting the twin accepts its own output.
- `cargo nextest run -p twin-openai` — 56 passed
- `cargo nextest run -p fabro-agent -E 'test(parity)' --run-ignored
only` — **91/91 passed** (was 7 failing)
- `cargo nextest run -p fabro-llm --run-ignored only` — 10 passed
- `cargo nextest run --workspace` — green apart from two pre-existing
env-dependent `fabro-workflow` failures that reproduce on clean `main`
in shells with provider API keys exported (unrelated; CI is green on
them because it has no such keys)
- clippy `-D warnings` / fmt — clean
Found while reviewing #481 (whose parity runs surfaced this); #481
itself is unaffected — it doesn't touch the openai adapter or the twin,
and the failures exist on its merge-base.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## CI (separate commit, drop if unwanted)
`ci: run twin-mode e2e suites on Linux` adds a step to the existing
Linux test job running the ignored twin-mode suites for the packages
that are fully green today (`fabro-agent`, `fabro-llm`, `twin-openai`) —
104 tests, ~1s on a warm build, no secrets needed (live-only tests
self-skip in twin mode). This is what would have caught the #449
regression. The remaining ignored suites (fabro-cli twin tests,
Docker/Daytona sandbox tests, fabro-spa asset test) need their own fixes
before joining; widen the `-E` filter as they're cleaned up. Note the
step deliberately avoids the `e2e` nextest profile, since
`NEXTEST_PROFILE=e2e` implies strict mode, which fails on missing
secrets.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
Enforce Cargo lockfile use across CI and release automation so jobs fail
on stale `Cargo.lock` state instead of resolving dependencies
implicitly. This adds `--locked` to Rust CI, release builds/tests,
nightly release tagging, the TypeScript workflow's embedded Rust build,
and helper-owned Cargo calls in `fabro-dev`.
This also fixes the Linux CI flake exposed by the PR: canceling a
durably blocked in-process run could take the abort path while the
workflow was still unwinding a human-input gate, causing
`run.failed(cancelled)` to be followed by `run.unblocked`. That invalid
event order broke projection rebuilds and made `GET /runs/{id}` return
404. Cancellation now uses the durable lifecycle status when selecting
the in-process blocked-run path, so the pending interview is cancelled
before the terminal event is emitted.
The release command's intentional `cargo update --workspace` step is
unchanged, because that step updates `Cargo.lock` after bumping the
workspace version.
## Testing
- `cargo nextest run --locked -p fabro-dev --features dev -E
'test(dry_run_computes_stable_version_from_date) |
test(dry_run_prints_equivalent_build_commands)'`
- `cargo --locked dev release --dry-run --skip-tests --release-date
2026-01-01`
- `cargo --locked dev docs check`
- `cargo nextest run --locked -p fabro-server --features test-support
cancel_durably_blocked_in_process_run_cancels_pending_interview_without_abort_signal
--status-level fail --final-status-level fail --show-progress none`
- `cargo nextest run --locked -p fabro-server --features test-support
--test it scenario::lifecycle --profile ci --status-level fail
--final-status-level fail --show-progress none --no-fail-fast`
- Linux Docker stress reproduction: `cargo nextest run --locked -p
fabro-server --features test-support --test it
scenario::lifecycle::full_http_lifecycle_cancel --profile ci
--stress-count 200 --status-level fail --final-status-level fail
--show-progress none`
- `cargo +nightly-2026-04-14 fmt --check --all`
- `cargo +nightly-2026-04-14 clippy --locked -p fabro-dev --features dev
--all-targets -- -D warnings`
- `cargo +nightly-2026-04-14 clippy --locked -p fabro-server --features
test-support --all-targets -- -D warnings`
- `git diff --check`
Full `cargo nextest run --locked -p fabro-dev --features dev` currently
has two unrelated policy-test failures:
`policy::catalog_builtin_references_stay_in_allowlist` and
`policy::workflow_template_rendering_call_sites_stay_in_allowlist`.
---
[](https://github.com/EveryInc/compound-engineering-plugin)
🤖 Generated with GPT-5 via [Codex](https://openai.com/codex)
---------
Co-authored-by: Release Repro <release-repro@example.com>
The ubuntu-*-arm-32-cores runner images don't ship with unzip, so
oven-sh/setup-bun fails when extracting the bun release zip. x86
runner images include it, which is why only the aarch64-unknown-linux
compile jobs failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Relocate the Mintlify tree to docs/public and consolidate internal docs under docs/internal. Update build scripts, tests, CI filters, README references, and local docs skills to follow the new layout.
Keeps the repo root tidy. The staged Linux musl binaries used by
the Dockerfile and the release pipeline now live at
tmp/docker-context/<arch>/fabro instead of docker-context/<arch>/fabro.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a settings reference generator backed by OptionsMetadata on the sparse config layer structs. The generated user-configuration page is fenced and checked in CI alongside the CLI reference.
Add a cargo dev generator for the CLI reference and gate the generated docs in CI. The generator reads the fabro clap command tree through a narrow public reference surface so CLI docs drift is caught without exposing runtime command internals.
clippy.toml already bans std::env::{set_var,remove_var} via
disallowed_methods, and every existing call site carries a scoped
#[expect(clippy::disallowed_methods, reason = "...")]. The shell grep
is redundant and forced a second, less granular allowlist.
Also update server-secrets-strategy.md to describe clippy as the
enforcement mechanism.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move server-only settings reads out of user-facing CLI commands into a
dedicated local_server module, the install/uninstall exceptions, and the
worker subcommand. Adds bin/dev/check-boundary.sh to prevent regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Prevents silent bun.lock drift across CI runs (root cause of the
nightly fabro-spa staleness failure) and re-runs typescript.yml when
nightly.yml itself changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Stacked cleanup of the `canonicalize blocked run status` work (local
commit `d13cdf374`) plus reconciliation with origin's `canonicalize
paginated run list responses` (origin commit `8ab689da7`). Both efforts
ran in parallel and diverged on the column name (`blocked` vs `waiting`)
and on how the board response is shaped — this PR converges them,
keeping `blocked` as the canonical column id while adopting origin's
`column` field on `RunListItem` and `StoreRunSummary` shape.
Also fixes a production-worker regression introduced by the
canonicalization: the worker's start-precondition only accepted
`Submitted | Starting`, so once runs started transitioning through
`Queued` on the way to `Starting`, every subprocess-worker run failed
with `Precondition failed: cannot start run: status is Queued`. That
cascaded into ~90 failing CLI/server integration tests locally.
## Commits
1. `f65843168` refactor(runs): simplify blocked status follow-ups
2. `1492d956c` chore: resolve clippy warnings
3. `676fd9f44` first merge of origin/main
4. `23fc92a2f` **fix(runs): allow Queued status in start precondition**
← the cascade-fix
5. `36b507a83` refactor: simplify pause/unpause + dedupe web status
tables
6. `8d8d27748` refactor(workflow): encapsulate BlockedStateTracker
inside HumanHandler
7. `1c17fda35` second merge of origin/main — resolves waiting vs blocked
8. `4cd3ef7b1` refactor(workflow): Mutex<usize> → AtomicUsize
9. `2e5a58e8a` fix(demo): align run-4 lifecycle status with Blocked
board column
## Test plan
- [x] fmt, clippy, build, doctests all clean
- [x] `cargo nextest run --workspace` — **4092/4092 pass**
- [x] `bun test` — **26/26 pass**, typecheck + production build clean
- [x] Manual CLI repro of the Queued-precondition fix
- [x] Browser smoke test: all 5 columns render with correct
labels/colors, demo run-4 appears in Blocked lane with question text
intact
## Known follow-up (not blocking)
A "paused-while-blocked" run (status `Paused` + `blocked_reason: Some`)
lands in the `running` column because the visible status chooses
`Paused` over `Blocked`. The pending question is not prominent on the
board. Addressing it would require `board_column()` to branch on
`(status, blocked_reason)` rather than just `status` — worth a separate
ticket.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The release workflow now builds musl artifacts with cargo-zigbuild, but
x86_64 musl tests still run through plain cargo test via nextest. Restore
musl-tools and the target-specific compiler/linker env for that test path
so fabro-proc's build.rs can compile its C helper again.
The shipped aarch64-unknown-linux-musl binary segfaulted at startup on
every arm64 runtime (Apple Silicon, Graviton, Ampere, Docker arm64).
Root cause: a glibc-vs-musl .init_array calling-convention mismatch --
a C static library in the dep graph has an __attribute__((constructor))
that expects (argc, argv, envp) per glibc, but musl on aarch64 calls
it with no args, so register garbage propagates into pointer arithmetic
and faults before main runs.
Switch the musl compile steps to cargo-zigbuild (zig 0.13.0). Zig's
bundled cc + lld produce working static-PIE binaries for both musl
targets, sidestepping Ubuntu musl-tools' -no-pie quirk and the
init_array ordering that triggered the crash. Drop the CARGO_TARGET_*
linker overrides and the musl-tools apt install -- zig handles both.
bin/dev/docker-build.sh mirrors the same toolchain so the local Docker
image build matches CI.
Verified by running fabro version from the resulting arm64 image on
ghcr.io/fabro-sh/dhi-alpine-base:3.23-dev, alpine:3.22, and
debian:stable-slim -- all print the version banner with exit 0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Today's nightly published an arm64 image that segfaults on any
invocation (`fabro version` → SIGSEGV). The docker job only built and
pushed; the binary was never executed inside the final image layout,
so the broken arm64 manifest reached ghcr.io undetected.
Before the multi-arch push, build each platform single-arch with
load: true and run `fabro version` in the loaded image. A segfault,
missing binary, or broken entrypoint now fails the job instead of
shipping a broken image. The subsequent multi-arch push reuses buildx
cache from the per-platform builds, so the net cost is ~one short
`docker run` per arch.
Node.js 20 actions are deprecated on GitHub Actions runners; updating to
the latest majors silences the warning and keeps the release pipeline
working past the September 2026 removal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tarballs in each compile matrix and the multi-arch ghcr image now get
Sigstore-signed provenance attestations via GitHub's attest-build-provenance
action. Users can verify with `gh attestation verify` — covered in a new
docs/reference/verifying-releases.mdx.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The aarch64-musl binary SIGSEGVs at startup on the ubuntu-24.04-arm
runner (empty stdout/stderr, non-zero exit), so every test that
spawns 'fabro server start' fails. The shipped binary runs natively
on Alpine via the Docker image, so skip the test step here and rely
on x86_64-musl + both gnu targets for test coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The aarch64 Linux compile job fails on ubuntu-22.04-arm-32-cores
because openssl-sys can't find pkg-config or OpenSSL headers.
build-essential alone doesn't pull them in on this image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Release run 24608869678 failed on both aarch64 Linux compiles with
"linker `cc` not found" after the dtolnay/rust-toolchain fix got us
past rustup. The ubuntu-*-arm-32-cores images don't ship build tools
preinstalled (the x86_64 variants do). Add a Linux-only step that
installs build-essential so `cc` is available for cargo's build
scripts, and drop the now-redundant apt-get update from the musl
toolchain step since it runs right after.
v3.1.0+ of actions/create-github-app-token deprecates `app-id` in
favor of the GitHub App's Client ID. Reads from the new
FABRO_RELEASES_APP_CLIENT_ID variable in the nightly environment.
Release run 24607436574 failed on both aarch64 Linux compiles with
"rustup: command not found" — the ubuntu-*-arm-32-cores runner images
don't ship with rustup preinstalled, while the x86_64 variants do. Our
rust.yml and typescript.yml already use dtolnay/rust-toolchain@stable;
switch release.yml and nightly.yml to the same action so rustup is
bootstrapped regardless of runner image. Targets are passed via the
action's `targets:` input instead of a manual `rustup target add`.
Matches the faster runner release.yml already uses for release-mode
nextest + build work, cutting Tag nightly wall time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v2.2.2 runs on Node.js 20, which GitHub is forcing to Node.js 24 on
June 2, 2026 and removing entirely on September 16, 2026. Bump to
v3.1.1 which runs on Node.js 24. The `app-id`/`private-key` inputs we
use are unchanged (`app-id` is deprecated in favor of `client-id`, but
still accepted).
Now that the release workflow publishes musl binaries, the runtime
image can drop the debian:trixie-slim base for alpine:3.22. The
image shrinks from ~287 MB to ~96 MB (66% smaller) with a smaller
attack surface.
- Dockerfile: alpine:3.22 base, apk packages (ca-certificates git
tini su-exec), BusyBox adduser/addgroup, tini at /sbin/tini.
- entrypoint.sh: replace runuser with su-exec, Alpine's idiomatic
drop-privileges helper.
- release.yml docker job: pull the two linux-musl artifacts instead
of linux-gnu. The docker image and the Alpine install.sh path now
ship the same binary.
- bin/dev/docker-build.sh: compile fabro-cli for the host's musl
target in rust:1-bookworm with musl-tools, the matching CC/LINKER
env vars, and LIBZ_SYS_STATIC=1. Same pattern as CI.
Verified locally on aarch64: Alpine image builds, server binds on
$PORT (default 32276), endpoints return 200, fabro server process
runs as unprivileged UID 1000 under tini.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cuts the release pipeline's critical path (aarch64-apple-darwin) from
~72m to an expected ~35m, with similar wins on the four Linux targets.
- macOS aarch64: macos-15 -> macos-15-xlarge (3 -> 6 vCPU M1)
- Linux x86 gnu/musl: ubuntu-latest/24.04 -> ubuntu-24.04-x86-32-cores
- Linux arm gnu: ubuntu-22.04-arm -> ubuntu-22.04-arm-32-cores
- Linux arm musl: ubuntu-24.04-arm -> ubuntu-24.04-arm-32-cores
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Start every workflow with permissions: {} and grant the minimum
required per job, following Astral's defense-in-depth pattern so a
newly added job can't silently inherit repo read access.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extend the release matrix to two statically-linked musl variants so
Alpine and other musl-based Linux hosts can install without glibc.
Homebrew and the Docker image remain glibc-only.
- release.yml: add x86_64-unknown-linux-musl (ubuntu-24.04) and
aarch64-unknown-linux-musl (ubuntu-24.04-arm) matrix rows with
musl-tools, CC_*_musl, CARGO_TARGET_*_LINKER, and LIBZ_SYS_STATIC
- Cargo.toml: enable git2 vendored-libgit2 so libgit2 compiles from
source for every target (needed because musl cannot link against
Ubuntu's glibc-built libgit2-dev)
- install.sh: check `ldd --version` for "musl" and rewrite the target
from -gnu to -musl so Alpine users get the right tarball
- upgrade.rs: add detect_linux_libc() / parse_ldd_libc() helper and
route detect_target() Linux arms through it, with unit tests
covering glibc, musl, empty, and unknown output
- tests/it: extend target regex in the dry-run snapshot filter
Ubuntu 24.04 is required for the musl runner: 22.04 ships musl 1.2.2
which SIGSEGVs statically-linked x86_64 test binaries at startup.
Confirmed against graphviz-sys CI before landing here.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the docker/Dockerfile-based api+web compose setup with a
single root Dockerfile that runs the fabro server with the embedded
web UI on port 80, persists state under /storage, and drops to a
non-root fabro user with CAP_NET_BIND_SERVICE.
The release workflow stages the prebuilt Linux binaries from the
compile job into a buildx context and publishes multi-arch images
to ghcr.io/fabro-sh/fabro as :<version> (always), :latest (stable
tags only), and :nightly (nightly tags only).
Also address zizmor findings in nightly.yml (pinned
create-github-app-token, persist-credentials: false with explicit
remote URL setup) and release.yml (no-cache on tag-triggered
setup-bun to close the cache-poisoning path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nextest with --workspace pulls in dev-dependencies that enable extra
features, forcing Cargo to recompile the whole graph. Running tests
first warms the cache; the final release binary build for fabro-cli
reuses those artifacts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New .github/workflows/nightly.yml that fires daily at 09:00 UTC (plus
on-demand via workflow_dispatch) and runs bin/dev/release.sh nightly to
tag a fresh pre-release. Tag shape: v0.X.Y-nightly.N. Skips cleanly when
HEAD is already the commit referenced by the newest v*-nightly.* tag.
The job runs in the `nightly` environment, which scopes the
`FABRO_RELEASES_APP_PRIVATE_KEY` secret and `FABRO_RELEASES_APP_ID`
variable to just this workflow and restricts deployments to `main`.
`actions/create-github-app-token` mints an installation token on the
`Fabro Releases` GitHub App; that token authenticates both the checkout
(so the bump commit + tag can push back to main) and any downstream git
operations release.sh performs. Commits are attributed to
`fabro-releases[bot]`.
Pre-tag testing stays in release.sh's verify_release_tests, so a broken
main fails the nightly workflow before a dud version bump commit lands.
The tag push then triggers the existing release.yml, which publishes
the GH prerelease and updates Formula/fabro-nightly.rb in the tap.
Prerequisites (all done manually):
- GitHub App `Fabro Releases` installed on fabro + homebrew-tap
- `nightly` environment with the app credentials
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
