From df303d29b31e8427d7aa60ee08db722f44343965 Mon Sep 17 00:00:00 2001 From: Fabro Date: Wed, 27 May 2026 21:05:11 -0400 Subject: [PATCH] =?UTF-8?q?checkpoint=20=E2=9A=92=EF=B8=8F=20Generated=20w?= =?UTF-8?q?ith=20[Fabro](https://fabro.sh)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- run.json | 576 ++- stages/008-verify@1/diff.patch | 4955 ++++++++++++++++++++++++ stages/008-verify@1/output.log | 1 + stages/008-verify@1/script_timing.json | 8 + stages/008-verify@1/status.json | 6 + stages/009-fixup@1/prompt.md | 443 +++ stages/009-fixup@1/provider_used.json | 5 + stages/009-fixup@1/response.md | 7 + 8 files changed, 5987 insertions(+), 14 deletions(-) create mode 100644 stages/008-verify@1/diff.patch create mode 100644 stages/008-verify@1/output.log create mode 100644 stages/008-verify@1/script_timing.json create mode 100644 stages/008-verify@1/status.json create mode 100644 stages/009-fixup@1/prompt.md create mode 100644 stages/009-fixup@1/provider_used.json create mode 100644 stages/009-fixup@1/response.md diff --git a/run.json b/run.json index 02cde3f64..a6da68818 100644 --- a/run.json +++ b/run.json @@ -503,7 +503,7 @@ "kind": "running" }, "status_updated_at": "2026-05-28T00:13:58.936868Z", - "last_event_at": "2026-05-28T01:02:40.663242Z", + "last_event_at": "2026-05-28T01:05:10.798008Z", "pending_control": null, "checkpoints": [ { @@ -1350,9 +1350,9 @@ } }, { - "seq": 0, + "seq": 1298, "checkpoint": { - "timestamp": "2026-05-28T01:02:42.286337Z", + "timestamp": "2026-05-28T01:02:46.629278Z", "current_node": "verify", "completed_nodes": [ "start", @@ -1365,24 +1365,282 @@ "verify" ], "node_retries": {}, + "context_values": { + "internal.retry_count.preflight_compile": 0, + "internal.retry_count.simplify_gpt": 0, + "response.simplify_opus": "## Review Summary\n\nAll three review agents came back essentially clean.\n\n- **Code Reuse Review** (e7156f0c): Clean. All new code follows existing local patterns (parse + `expect_err`, `body.get(x).is_none()`, `RouterOptions` destructure). No missed utilities.\n- **Code Quality Review** (59feb42b): Clean. Symbol audit confirms full removal of `IpAllowlistConfig`, `IpAllowlist`, `IpAllowEntry`, `LiteralIpAllowEntry`, `ip_allowlist_middleware`, `GitHubMetaResolver`, `github_meta_hooks`, `trusted_proxy_count`, `ServerIpAllowlistSettings`, `ServerIpAllowlistOverrideSettings`, `extract_client_ip`, `GitHubMetaCache`, and frontend `ip_allowlist`/`Count`/`plural` wiring. `deny_unknown_fields` enforces hard removal at the TOML boundary. No stubs, dead imports, or vestigial parameters from the refactor.\n- **Efficiency Review** (51c928ad): Clean. Per-request IP allowlist middleware (two layers — main app and webhook) gone. Startup GitHub `/meta` fetch + on-disk cache gone. `ConnectInfo` plumbing removed from `serve.rs` (both Tcp and Unix variants) and from `tests/it/api/tcp.rs`. No new hot-path work introduced anywhere.\n\n**One out-of-scope finding skipped:** the efficiency agent flagged `install.rs:710` as still using `into_make_service_with_connect_info::()` despite nothing in `install.rs` extracting `ConnectInfo` (the `detect_remote_ip` helper only reads the `x-forwarded-for` header). I verified this is true — it's the last `ConnectInfo` caller in the crate — but it's pre-existing code in the install flow, not introduced by this refactor, and unrelated to IP allowlisting. The agent rated it low-severity cleanup. Per the scope discipline of this refactor (hard removal of IP allowlisting only, no scope creep) and the system prompt's \"only make changes that are directly requested or clearly necessary,\" I'm leaving it alone.\n\nNo fixes applied — the diff was already clean.", + "internal.retry_count.verify": 0, + "last_stage": "simplify_gpt", + "thread.simplify_opus.current_node": "simplify_gpt", + "thread.implement.current_node": "simplify_opus", + "response.implement": "Implemented the full IP allowlisting removal refactor.\n\nSummary:\n- Removed inbound source-IP allowlist config from dense settings and sparse config layers:\n - `[server.ip_allowlist]`\n - `[server.integrations.github.webhooks.ip_allowlist]`\n - `IpAllowEntry` and related settings/types.\n- Deleted server runtime enforcement:\n - Removed `ip_allowlist` middleware/module.\n - Removed `IpAllowlistConfig`, client-IP extraction, trusted proxy handling, and GitHub `/meta` hook range resolution.\n - Simplified router/startup wiring and removed `ConnectInfo` serving requirements that only existed for allowlisting.\n- Preserved unrelated allowlists:\n - GitHub username allowlists remain.\n - Sandbox/Daytona CIDR allow lists remain.\n- Updated OpenAPI and generated clients:\n - Removed allowlist schemas/fields from `fabro-api.yaml`.\n - Regenerated Rust API and TypeScript API client.\n - Deleted stale generated TS allowlist model files.\n- Updated web UI:\n - Removed Settings > Security network allowlist display.\n - Adjusted Security page/nav copy.\n- Updated tests/docs:\n - Added unknown-field tests proving stale IP allowlist config is now rejected.\n - Added settings API assertions that removed fields are absent.\n - Updated security docs and changelog to direct operators to upstream network controls.\n- Updated dependency graph:\n - Removed direct `ipnet` deps from `fabro-types` and `fabro-server`; retained transitive/use in `fabro-config` for sandbox CIDR validation.\n\nRed/green TDD:\n- Added stale-config rejection tests and verified they failed before schema removal, then passed after removal.\n\nValidation run:\n- `cargo build -p fabro-api`\n- `cd lib/packages/fabro-api-client && bun run generate`\n- `cargo build -p fabro-cli --bin fabro`\n- `cargo nextest run -p fabro-config -p fabro-types -p fabro-api -p fabro-server --no-fail-fast`\n- `cargo +nightly-2026-04-14 fmt --check --all`\n- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings`\n- `git diff --check`\n- Stale-symbol search confirmed remaining `ip_allowlist`/IP allowlist references are only intentional docs/tests/assertions.\n\nNo commits were created.", + "internal.run_id": "01KSNYVQCKXPCSCEGWMZ1Q574Z", + "internal.retry_count.start": 0, + "graph.model_stylesheet": "\n * { model: claude-opus-4-7; }\n ", + "response.simplify_gpt": "Review complete.\n\nWhat I checked:\n- Diff against merge base `a992a7d76...`\n- Remaining references to removed IP allowlist symbols/settings outside archival/build output\n- Key config/server/API/web/docs changes for cleanup issues\n- Parallel review agents were launched for reuse, quality, and efficiency review; no actionable cleanup changes were needed.\n\nNo code changes were made during this review.\n\nVerification passed:\n- `cargo nextest run -p fabro-config -p fabro-types -p fabro-api -p fabro-server`\n - 1372 passed\n- `cd apps/fabro-web && bun run typecheck`\n - passed\n- Cleanup search:\n - `rg` found no remaining non-archival references to `ip_allowlist`, `trusted_proxy_count`, `github_meta_hooks`, `IpAllow`, `GitHubMetaResolver`, etc. excluding `target/` and archival docs.\n\nWorking tree is clean.", + "thread.preflight_compile.current_node": "preflight_lint", + "internal.fidelity": "compact", + "internal.retry_count.toolchain": 0, + "thread.simplify_gpt.current_node": "verify", + "internal.retry_count.preflight_lint": 0, + "graph.rankdir": "LR", + "internal.retry_count.simplify_opus": 0, + "graph.goal": "---\ntitle: \"refactor: Remove IP allowlisting\"\ntype: refactor\nstatus: active\ndate: 2026-05-27\n---\n\n# refactor: Remove IP allowlisting\n\n## Overview\n\nRemove Fabro's inbound server IP allowlisting feature completely. This removes the\nserver-wide `[server.ip_allowlist]` setting, the GitHub webhook-specific\n`[server.integrations.github.webhooks.ip_allowlist]` overlay, request middleware,\nclient-IP extraction, GitHub `/meta` hook-range expansion, settings API exposure,\ngenerated API client types, and the Settings > Security UI display.\n\nExisting configs that still contain removed IP allowlist keys should fail as\nunknown fields. This is an intentional hard removal, not a compatibility\ndeprecation.\n\n## Problem Frame\n\nThe feature is being removed from Fabro rather than maintained as an in-process\nnetwork access-control layer. Network source restrictions should be handled\noutside Fabro by reverse proxies, firewalls, VPNs, Tailscale, platform ingress,\nor other deployment-layer controls.\n\n## Requirements Trace\n\n- R1. Remove all runtime enforcement of inbound source-IP allowlisting from web,\n API, static asset, and GitHub webhook routes.\n- R2. Remove the server settings schema for global and GitHub webhook IP\n allowlists.\n- R3. Remove `IpAllowEntry` and related API schema/client types from public\n settings payloads.\n- R4. Keep unrelated allowlists intact: GitHub username allowlists, sandbox\n egress CIDR allow lists, and internal policy/test allowlists.\n- R5. Preserve existing authentication, GitHub webhook HMAC verification,\n routing, health checks, request logging, and settings hot-reload behavior.\n- R6. Treat old IP allowlist config as invalid after removal.\n- R7. Update operator-facing docs/changelog so users know to move source-IP\n restrictions upstream.\n\n## Scope Boundaries\n\n- Do not remove `[server.auth.github].allowed_usernames`.\n- Do not remove Daytona/sandbox `cidr_allow_list` network policy support.\n- Do not remove generic uses of \"allowlist\" in policy tests, markdown rendering,\n model controls, or other unrelated domains.\n- Do not add a migration, warning-only parser, or fallback compatibility path for\n old IP allowlist settings.\n- Do not change GitHub webhook signature verification or webhook route strategy.\n\n## Context & Research\n\n### Relevant Code and Patterns\n\n- Settings use sparse config layers in `lib/crates/fabro-config/src/layers/`\n and dense resolved types in `lib/crates/fabro-types/src/settings/`.\n- The OpenAPI spec at `docs/public/api-reference/fabro-api.yaml` is the source\n of truth for API wire shape; `cargo build -p fabro-api` regenerates Rust API\n code, and `cd lib/packages/fabro-api-client && bun run generate` regenerates\n the TypeScript client.\n- `lib/crates/fabro-server/src/server.rs` builds the router and currently wires\n IP allowlist middleware before other route dispatch behavior.\n- `lib/crates/fabro-server/src/serve.rs` currently resolves server and webhook\n allowlist configs at startup and creates a GitHub `/meta` resolver.\n- `apps/fabro-web/app/routes/settings-security.tsx` displays the settings API's\n `server.ip_allowlist` state.\n\n### Main Removal Surfaces\n\n- Config/types: `fabro-types`, `fabro-config`, config tests.\n- Server runtime: `fabro-server/src/ip_allowlist.rs`, router wiring, startup\n wiring, test helpers, routing/TCP tests.\n- API/contracts: OpenAPI schema, `fabro-api` type replacements/exports, generated\n TypeScript API client models.\n- Frontend/docs: Settings > Security copy, settings navigation copy, changelog.\n\n## Key Technical Decisions\n\n- **Hard removal:** Removed config keys should fail through existing\n `deny_unknown_fields` behavior. This keeps the change simple and makes stale\n deployment config visible immediately.\n- **Delete, do not stub:** Remove `IpAllowlistConfig` and middleware parameters\n instead of passing default empty configs through the router. Empty stubs would\n keep the feature shape alive and make future cleanup harder.\n- **Keep webhook auth unchanged:** GitHub webhook source-IP filtering goes away,\n but HMAC signature verification remains the security boundary for webhook\n payload authenticity.\n- **Generated clients follow OpenAPI:** Update the OpenAPI schemas first, then\n regenerate Rust and TypeScript API outputs rather than hand-editing generated\n client files except as a short-term cleanup if generation leaves stale exports.\n- **Docs mention upstream controls:** The changelog and security guidance should\n direct operators to network-layer controls, not a replacement Fabro setting.\n\n## Implementation Units\n\n- [ ] **Unit 1: Remove config schema and resolved types**\n\n**Goal:** Delete the IP allowlist settings contract from config parsing and dense\nserver settings.\n\n**Requirements:** R2, R3, R4, R6\n\n**Dependencies:** None\n\n**Files:**\n- Modify: `lib/crates/fabro-types/src/settings/server.rs`\n- Modify: `lib/crates/fabro-types/src/settings/mod.rs`\n- Modify: `lib/crates/fabro-types/Cargo.toml`\n- Modify: `lib/crates/fabro-config/src/layers/server.rs`\n- Modify: `lib/crates/fabro-config/src/layers/mod.rs`\n- Modify: `lib/crates/fabro-config/src/lib.rs`\n- Modify: `lib/crates/fabro-config/src/resolve/server.rs`\n- Modify: `lib/crates/fabro-config/src/tests/resolve_server.rs`\n\n**Approach:**\n- Remove `ServerNamespace.ip_allowlist` and its `test_default()` population.\n- Delete `ServerIpAllowlistSettings`, `ServerIpAllowlistOverrideSettings`, and\n `IpAllowEntry`.\n- Remove `ServerLayer.ip_allowlist`, `ServerIpAllowlistLayer`,\n `ServerIpAllowlistOverrideLayer`, and `IntegrationWebhooksLayer.ip_allowlist`.\n- Remove resolver functions and validation for global and webhook IP allowlists,\n including `github_meta_hooks` parsing and Unix socket trusted-proxy checks.\n- Remove `ipnet` from `fabro-types`. Keep `ipnet` in `fabro-config` because\n `resolve/environment.rs` still validates sandbox CIDR policy.\n- Replace old positive/negative IP allowlist config tests with unknown-field\n tests for `[server.ip_allowlist]` and\n `[server.integrations.github.webhooks.ip_allowlist]`.\n\n**Patterns to follow:**\n- Existing unknown-field tests in `lib/crates/fabro-config/src/tests/resolve_server.rs`\n for retired server settings.\n- Existing `ServerLayer`/`ServerNamespace` layer-to-resolved pattern for removing\n a server subdomain cleanly.\n\n**Test scenarios:**\n- Error path: TOML with `[server.ip_allowlist]` fails parsing/resolution with an\n unknown-field diagnostic mentioning `ip_allowlist`.\n- Error path: TOML with `[server.integrations.github.webhooks.ip_allowlist]`\n fails with an unknown-field diagnostic mentioning `ip_allowlist`.\n- Happy path: minimal valid server settings still resolve without any\n `ip_allowlist` field.\n- Integration: serialized `ServerSettings` JSON no longer includes\n `server.ip_allowlist`.\n\n**Verification:**\n- `fabro-config` and `fabro-types` compile without removed IP allowlist symbols.\n- Config tests prove stale allowlist settings are rejected.\n\n- [ ] **Unit 2: Remove server middleware and startup resolution**\n\n**Goal:** Remove all runtime source-IP filtering and GitHub `/meta` range\nresolution from `fabro-server`.\n\n**Requirements:** R1, R4, R5\n\n**Dependencies:** Unit 1\n\n**Files:**\n- Delete: `lib/crates/fabro-server/src/ip_allowlist.rs`\n- Modify: `lib/crates/fabro-server/src/lib.rs`\n- Modify: `lib/crates/fabro-server/src/server.rs`\n- Modify: `lib/crates/fabro-server/src/serve.rs`\n- Modify: `lib/crates/fabro-server/src/test_support.rs`\n- Modify: `lib/crates/fabro-server/Cargo.toml`\n- Modify: `lib/crates/fabro-server/src/auth/translate.rs`\n- Modify: `lib/crates/fabro-server/src/web_auth.rs`\n- Modify: `lib/crates/fabro-cli/tests/it/support/auth_harness.rs`\n\n**Approach:**\n- Remove the public `ip_allowlist` module export.\n- Delete `IpAllowlistConfig`, `IpAllowlist`, `GitHubMetaResolver`, client-IP\n extraction, middleware, and GitHub meta cache helpers.\n- Simplify `build_router_with_options` by removing its `Arc`\n parameter and removing `RouterOptions.github_webhook_ip_allowlist`.\n- Remove the global allowlist middleware layer from the main app router.\n- Simplify `github_webhook_routes` so it only receives the webhook secret and no\n route-specific allowlist config.\n- Remove startup creation of `GitHubMetaResolver`, `default_ip_allowlist`,\n `resolve_github_webhook_ip_allowlist`, and\n `resolve_startup_github_webhook_ip_allowlist`.\n- Replace all test/helper call sites that pass `IpAllowlistConfig::default()`\n with the simplified router signature.\n- Remove `ipnet` and any now-unused HTTP mocking/test-only dependencies from\n `fabro-server` if they are only used by the deleted module.\n- Consider whether `serve.rs` still needs `SocketAddr` for TCP\n `ConnectInfo`; if it was only present for allowlisting tests, remove the\n connect-info service wrapper and imports.\n\n**Patterns to follow:**\n- Keep router layers ordered as they are after the allowlist layer is removed:\n auth translation, demo routing, auth extension, canonical host, security\n headers, HTTP logging, request ID.\n- Existing webhook route HMAC tests and auth tests should remain the source of\n truth for webhook behavior.\n\n**Test scenarios:**\n- Happy path: API requests from any TCP remote address route normally when auth\n requirements are satisfied.\n- Happy path: `/health` remains accessible.\n- Integration: GitHub webhook route still rejects missing/invalid signatures and\n accepts valid signatures exactly as before.\n- Cleanup: there are no references to `IpAllowlistConfig`, `ip_allowlist_middleware`,\n `GitHubMetaResolver`, `github_meta_hooks`, or `github-meta-hooks.json`.\n\n**Verification:**\n- `fabro-server` and CLI auth harness tests compile against the simplified router\n API.\n- Runtime startup no longer performs any GitHub `/meta` fetch for webhook IP\n ranges.\n\n- [ ] **Unit 3: Update OpenAPI and generated API clients**\n\n**Goal:** Remove IP allowlist fields and schemas from public settings API\ncontracts and regenerated clients.\n\n**Requirements:** R3, R4\n\n**Dependencies:** Units 1 and 2\n\n**Files:**\n- Modify: `docs/public/api-reference/fabro-api.yaml`\n- Modify: `lib/crates/fabro-api/build.rs`\n- Modify: `lib/crates/fabro-api/src/lib.rs`\n- Modify: `lib/crates/fabro-api/tests/server_settings_round_trip.rs`\n- Modify/generated: `lib/packages/fabro-api-client/src/models/*`\n\n**Approach:**\n- Remove `ip_allowlist` from `ServerNamespace.required` and\n `ServerNamespace.properties`.\n- Remove `ServerIpAllowlistSettings`,\n `ServerIpAllowlistOverrideSettings`, `IpAllowEntry`,\n `LiteralIpAllowEntry`, and `GitHubMetaHooksEntry` schemas.\n- Remove `IntegrationWebhooksSettings.ip_allowlist` from required/properties.\n- Remove `with_replacement` entries and public re-exports for removed types in\n `fabro-api`.\n- Regenerate Rust API code by building `fabro-api`.\n- Regenerate TypeScript Axios client and ensure stale allowlist model files and\n index exports are gone.\n- Strengthen round-trip tests to assert settings JSON omits\n `server.ip_allowlist` and webhook `ip_allowlist`.\n\n**Patterns to follow:**\n- Existing OpenAPI-first workflow in `AGENTS.md`.\n- Existing `server_settings_family_reuses_domain_types` assertions for shared\n domain type identity.\n\n**Test scenarios:**\n- Contract: `ServerSettings` round-trips through API types without any IP\n allowlist field.\n- Contract: OpenAPI-generated TypeScript `ServerNamespace` has no\n `ip_allowlist` property.\n- Contract: `IntegrationWebhooksSettings` has only webhook strategy fields after\n removal.\n\n**Verification:**\n- Generated Rust and TypeScript clients match the updated OpenAPI spec.\n- `rg \"ServerIpAllowlist|IpAllowEntry|server-ip-allowlist|literal-ip-allow-entry\"`\n finds no remaining generated/public API references.\n\n- [ ] **Unit 4: Update web settings UI**\n\n**Goal:** Remove IP allowlist display from Settings > Security and align copy\nwith the new settings shape.\n\n**Requirements:** R3, R7\n\n**Dependencies:** Unit 3\n\n**Files:**\n- Modify: `apps/fabro-web/app/routes/settings-security.tsx`\n- Modify: `apps/fabro-web/app/routes/settings.tsx`\n\n**Approach:**\n- Change the Security page description from \"Authentication methods and network\n allowlist\" to authentication-only wording.\n- Remove destructuring and rendering of `settings.server.ip_allowlist`.\n- Remove the unused `Count` and `plural` imports if no longer used.\n- Update the Settings nav description for Security from \"Authentication and\n network allowlist\" to authentication-focused copy.\n\n**Patterns to follow:**\n- Existing `settings-panel` row layout for Auth methods and Allowed usernames.\n\n**Test scenarios:**\n- Typecheck: the route compiles against the regenerated API client with no\n `server.ip_allowlist` property.\n- UI behavior: Security page still renders auth methods and allowed usernames.\n- Cleanup: no frontend references to `ip_allowlist`, `IP allowlist`, or\n `trusted_proxy_count` remain.\n\n**Verification:**\n- `apps/fabro-web` typecheck passes against the new generated client.\n\n- [ ] **Unit 5: Update tests, docs, changelog, and dependency lockfile**\n\n**Goal:** Remove stale references and document the operator-facing behavior\nchange.\n\n**Requirements:** R4, R6, R7\n\n**Dependencies:** Units 1 through 4\n\n**Files:**\n- Modify: `lib/crates/fabro-server/tests/it/api/routing.rs`\n- Modify: `lib/crates/fabro-server/tests/it/api/tcp.rs`\n- Modify: `lib/crates/fabro-server/tests/it/api/settings.rs`\n- Modify: `docs/public/administration/security.mdx`\n- Create or modify: `docs/public/changelog/2026-05-27.mdx`\n- Modify: `Cargo.lock` if dependency graph changes\n\n**Approach:**\n- Delete route/TCP tests that only prove IP allowlist enforcement or\n `ConnectInfo` behavior.\n- Adjust any router setup helpers after the signature simplification.\n- Add a settings API assertion that `server.ip_allowlist` is absent from\n `/api/v1/settings`.\n- Update security docs to explicitly state that Fabro does not provide inbound\n source-IP allowlisting and operators should use upstream network controls.\n- Add a changelog entry dated 2026-05-27 describing the hard removal and the\n expected replacement at the deployment layer.\n- Run dependency resolution after removing direct `ipnet`/test dependencies;\n keep transitive or unrelated `ipnet` entries needed by sandbox CIDR validation.\n\n**Patterns to follow:**\n- Existing changelog style in `docs/public/changelog/2026-05-26.mdx`.\n- Existing settings API integration test style in\n `lib/crates/fabro-server/tests/it/api/settings.rs`.\n\n**Test scenarios:**\n- Settings API: response contains server auth, listen, storage, scheduler, and\n integrations fields, but not `server.ip_allowlist`.\n- Config compatibility: old IP allowlist TOML fails before startup rather than\n being silently ignored.\n- Docs validation: security docs no longer imply Fabro can restrict inbound\n source IPs internally.\n\n**Verification:**\n- `rg -n \"ip_allowlist|trusted_proxy_count|github_meta_hooks|IP allowlist|ip allowlist\"`\n returns only historical archived plan/brainstorm/spec references or unrelated\n non-server allowlist text.\n\n## System-Wide Impact\n\n- **Public API:** `GET /api/v1/settings` response shape changes by removing\n `server.ip_allowlist` and webhook `ip_allowlist`.\n- **Config compatibility:** Existing `settings.toml` files containing the removed\n keys become invalid. This is intentional.\n- **Runtime security posture:** Fabro no longer blocks requests based on source\n IP. Operators must enforce network source restrictions upstream.\n- **Webhook handling:** GitHub webhook HMAC verification remains unchanged; only\n optional source-IP filtering is removed.\n- **Generated clients:** Downstream TypeScript/Rust consumers that read\n `server.ip_allowlist` must update.\n\n## Risks & Mitigations\n\n| Risk | Mitigation |\n|------|------------|\n| Operators accidentally expose a server that previously relied on Fabro IP allowlisting | Changelog and security docs explicitly call out the removal and direct operators to upstream controls. |\n| Generated API clients retain stale types | Update OpenAPI first, regenerate both Rust and TypeScript clients, and run search checks for stale symbols. |\n| Unrelated allowlist functionality is removed by broad search/replace | Scope searches to `IpAllow`, `ip_allowlist`, `trusted_proxy_count`, and `github_meta_hooks`; preserve sandbox CIDR and GitHub username allowlists. |\n| Router signature cleanup breaks many tests | Update shared test helpers first, then compile-driven cleanup of remaining call sites. |\n| `ipnet` is removed where still needed | Keep `fabro-config` dependency if environment CIDR validation still imports `ipnet::IpNet`. |\n\n## Test Plan\n\n- `cargo build -p fabro-api`\n- `cd lib/packages/fabro-api-client && bun run generate`\n- `cargo nextest run -p fabro-config -p fabro-types -p fabro-api -p fabro-server`\n- `cd apps/fabro-web && bun run typecheck`\n- `cd apps/fabro-web && bun test`\n- `cargo +nightly-2026-04-14 fmt --check --all`\n- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings`\n\n## Assumptions\n\n- The accepted compatibility policy is hard removal: no warning-only parser, no\n migration, and no custom compatibility error path.\n- Public API consumers can tolerate a breaking settings shape change for this\n removed feature.\n- Historical documents in `docs/plans/`, `docs/brainstorms/`, and\n `docs/superpowers/` are archival and do not need rewriting.\n\n## Sources & References\n\n- Previous feature requirements:\n `docs/brainstorms/2026-04-15-ip-whitelist-requirements.md`\n- Previous implementation plan:\n `docs/plans/2026-04-15-002-feat-ip-allowlist-plan.md`\n- API workflow guidance: `AGENTS.md`\n- OpenAPI source of truth: `docs/public/api-reference/fabro-api.yaml`\n", + "last_response": "Review complete.\n\nWhat I checked:\n- Diff against merge base `a992a7d76...`\n- Remaining references to removed IP allowlist symbols/settings outside archival/build output\n- Key config/server/API/web/doc", + "thread.start.current_node": "toolchain", + "command.output": "blob://sha256/ba184126dad761158723a25e15818b2cb0ac7327ad06fa7513f7a7ce9377eee6", + "thread.toolchain.current_node": "preflight_compile", + "internal.node_visit_count": 1, + "failure_signature": "verify|deterministic|script failed with exit code: ## output from https://github.com/fabro-sh/fabro * branch main -> fetch_head .. main -> origin/main auto-merging cargo.lock auto-merging lib/crates/fabro-api/build.rs auto-merging lib/crates/fabro", + "outcome": "failed", + "failure_class": "deterministic", + "current_node": "verify", + "thread.preflight_lint.current_node": "implement", + "internal.work_dir": "/home/daytona/workspace/fabro", + "internal.retry_count.implement": 0, + "internal.thread_id": "simplify_gpt" + }, + "node_outcomes": { + "toolchain": { + "status": "succeeded", + "context_updates": { + "command.output": "blob://sha256/fc14b2ba2d770e5cd3169df7a29525c962adfc4cfa3097b9098c63ebd61a748c" + }, + "notes": "Script completed: command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1", + "usage": null, + "timing": { + "wall_time_ms": 0, + "inference_time_ms": 0, + "tool_time_ms": 1482, + "active_time_ms": 1482 + } + }, + "simplify_opus": { + "status": "succeeded", + "context_updates": { + "response.simplify_opus": "## Review Summary\n\nAll three review agents came back essentially clean.\n\n- **Code Reuse Review** (e7156f0c): Clean. All new code follows existing local patterns (parse + `expect_err`, `body.get(x).is_none()`, `RouterOptions` destructure). No missed utilities.\n- **Code Quality Review** (59feb42b): Clean. Symbol audit confirms full removal of `IpAllowlistConfig`, `IpAllowlist`, `IpAllowEntry`, `LiteralIpAllowEntry`, `ip_allowlist_middleware`, `GitHubMetaResolver`, `github_meta_hooks`, `trusted_proxy_count`, `ServerIpAllowlistSettings`, `ServerIpAllowlistOverrideSettings`, `extract_client_ip`, `GitHubMetaCache`, and frontend `ip_allowlist`/`Count`/`plural` wiring. `deny_unknown_fields` enforces hard removal at the TOML boundary. No stubs, dead imports, or vestigial parameters from the refactor.\n- **Efficiency Review** (51c928ad): Clean. Per-request IP allowlist middleware (two layers — main app and webhook) gone. Startup GitHub `/meta` fetch + on-disk cache gone. `ConnectInfo` plumbing removed from `serve.rs` (both Tcp and Unix variants) and from `tests/it/api/tcp.rs`. No new hot-path work introduced anywhere.\n\n**One out-of-scope finding skipped:** the efficiency agent flagged `install.rs:710` as still using `into_make_service_with_connect_info::()` despite nothing in `install.rs` extracting `ConnectInfo` (the `detect_remote_ip` helper only reads the `x-forwarded-for` header). I verified this is true — it's the last `ConnectInfo` caller in the crate — but it's pre-existing code in the install flow, not introduced by this refactor, and unrelated to IP allowlisting. The agent rated it low-severity cleanup. Per the scope discipline of this refactor (hard removal of IP allowlisting only, no scope creep) and the system prompt's \"only make changes that are directly requested or clearly necessary,\" I'm leaving it alone.\n\nNo fixes applied — the diff was already clean.", + "last_response": "## Review Summary\n\nAll three review agents came back essentially clean.\n\n- **Code Reuse Review** (e7156f0c): Clean. All new code follows existing local patterns (parse + `expect_err`, `body.get(x).is_", + "last_stage": "simplify_opus" + }, + "notes": "Stage completed: simplify_opus", + "usage": { + "input": { + "usage": { + "model": { + "provider": "anthropic", + "model_id": "claude-opus-4-7" + }, + "tokens": { + "input_tokens": 21129, + "output_tokens": 5543, + "reasoning_tokens": 0, + "cache_read_tokens": 251482, + "cache_write_tokens": 61150 + } + }, + "facts": { + "algorithm": "anthropic", + "cache_write_5m_tokens": 61150, + "cache_write_1h_tokens": 0 + } + }, + "total_usd_micros": 752148 + }, + "timing": { + "wall_time_ms": 0, + "inference_time_ms": 83152, + "tool_time_ms": 271255, + "active_time_ms": 354407 + } + }, + "preflight_lint": { + "status": "succeeded", + "context_updates": { + "command.output": "blob://sha256/12ae32cb1ec02d01eda3581b127c1fee3b0dc53572ed6baf239721a03d82e126" + }, + "notes": "Script completed: cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1", + "usage": null, + "timing": { + "wall_time_ms": 0, + "inference_time_ms": 0, + "tool_time_ms": 142983, + "active_time_ms": 142983 + } + }, + "simplify_gpt": { + "status": "succeeded", + "context_updates": { + "last_response": "Review complete.\n\nWhat I checked:\n- Diff against merge base `a992a7d76...`\n- Remaining references to removed IP allowlist symbols/settings outside archival/build output\n- Key config/server/API/web/doc", + "last_stage": "simplify_gpt", + "response.simplify_gpt": "Review complete.\n\nWhat I checked:\n- Diff against merge base `a992a7d76...`\n- Remaining references to removed IP allowlist symbols/settings outside archival/build output\n- Key config/server/API/web/docs changes for cleanup issues\n- Parallel review agents were launched for reuse, quality, and efficiency review; no actionable cleanup changes were needed.\n\nNo code changes were made during this review.\n\nVerification passed:\n- `cargo nextest run -p fabro-config -p fabro-types -p fabro-api -p fabro-server`\n - 1372 passed\n- `cd apps/fabro-web && bun run typecheck`\n - passed\n- Cleanup search:\n - `rg` found no remaining non-archival references to `ip_allowlist`, `trusted_proxy_count`, `github_meta_hooks`, `IpAllow`, `GitHubMetaResolver`, etc. excluding `target/` and archival docs.\n\nWorking tree is clean." + }, + "notes": "Stage completed: simplify_gpt", + "usage": { + "input": { + "usage": { + "model": { + "provider": "openai", + "model_id": "gpt-5.5" + }, + "tokens": { + "input_tokens": 445377, + "output_tokens": 3232, + "reasoning_tokens": 1131, + "cache_read_tokens": 909312, + "cache_write_tokens": 0 + } + }, + "facts": { + "algorithm": "openai" + } + }, + "total_usd_micros": 2812431 + }, + "timing": { + "wall_time_ms": 0, + "inference_time_ms": 172391, + "tool_time_ms": 42057, + "active_time_ms": 214448 + } + }, + "verify": { + "status": "failed", + "context_updates": { + "command.output": "blob://sha256/ba184126dad761158723a25e15818b2cb0ac7327ad06fa7513f7a7ce9377eee6" + }, + "failure": { + "message": "Script failed with exit code: 1\n\n## output\nFrom https://github.com/fabro-sh/fabro\n * branch main -> FETCH_HEAD\n a992a7d76..475b4ab65 main -> origin/main\nAuto-merging Cargo.lock\nAuto-merging lib/crates/fabro-api/build.rs\nAuto-merging lib/crates/fabro-api/src/lib.rs\nAuto-merging lib/crates/fabro-server/Cargo.toml\nAuto-merging lib/crates/fabro-server/src/lib.rs\nAuto-merging lib/crates/fabro-server/src/serve.rs\nAuto-merging lib/crates/fabro-server/src/server.rs\nAuto-merging lib/crates/fabro-server/src/server/tests.rs\nAuto-merging lib/crates/fabro-server/src/test_support.rs\nCONFLICT (content): Merge conflict in lib/crates/fabro-server/src/test_support.rs\nAutomatic merge failed; fix conflicts and then commit the result.\n", + "category": "deterministic" + }, + "usage": null, + "timing": { + "wall_time_ms": 0, + "inference_time_ms": 0, + "tool_time_ms": 1604, + "active_time_ms": 1604 + } + }, + "preflight_compile": { + "status": "succeeded", + "context_updates": { + "command.output": "blob://sha256/12ae32cb1ec02d01eda3581b127c1fee3b0dc53572ed6baf239721a03d82e126" + }, + "notes": "Script completed: cargo check -q --workspace 2>&1", + "usage": null, + "timing": { + "wall_time_ms": 0, + "inference_time_ms": 0, + "tool_time_ms": 135614, + "active_time_ms": 135614 + } + }, + "implement": { + "status": "succeeded", + "context_updates": { + "last_stage": "implement", + "last_response": "Implemented the full IP allowlisting removal refactor.\n\nSummary:\n- Removed inbound source-IP allowlist config from dense settings and sparse config layers:\n - `[server.ip_allowlist]`\n - `[server.int", + "response.implement": "Implemented the full IP allowlisting removal refactor.\n\nSummary:\n- Removed inbound source-IP allowlist config from dense settings and sparse config layers:\n - `[server.ip_allowlist]`\n - `[server.integrations.github.webhooks.ip_allowlist]`\n - `IpAllowEntry` and related settings/types.\n- Deleted server runtime enforcement:\n - Removed `ip_allowlist` middleware/module.\n - Removed `IpAllowlistConfig`, client-IP extraction, trusted proxy handling, and GitHub `/meta` hook range resolution.\n - Simplified router/startup wiring and removed `ConnectInfo` serving requirements that only existed for allowlisting.\n- Preserved unrelated allowlists:\n - GitHub username allowlists remain.\n - Sandbox/Daytona CIDR allow lists remain.\n- Updated OpenAPI and generated clients:\n - Removed allowlist schemas/fields from `fabro-api.yaml`.\n - Regenerated Rust API and TypeScript API client.\n - Deleted stale generated TS allowlist model files.\n- Updated web UI:\n - Removed Settings > Security network allowlist display.\n - Adjusted Security page/nav copy.\n- Updated tests/docs:\n - Added unknown-field tests proving stale IP allowlist config is now rejected.\n - Added settings API assertions that removed fields are absent.\n - Updated security docs and changelog to direct operators to upstream network controls.\n- Updated dependency graph:\n - Removed direct `ipnet` deps from `fabro-types` and `fabro-server`; retained transitive/use in `fabro-config` for sandbox CIDR validation.\n\nRed/green TDD:\n- Added stale-config rejection tests and verified they failed before schema removal, then passed after removal.\n\nValidation run:\n- `cargo build -p fabro-api`\n- `cd lib/packages/fabro-api-client && bun run generate`\n- `cargo build -p fabro-cli --bin fabro`\n- `cargo nextest run -p fabro-config -p fabro-types -p fabro-api -p fabro-server --no-fail-fast`\n- `cargo +nightly-2026-04-14 fmt --check --all`\n- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings`\n- `git diff --check`\n- Stale-symbol search confirmed remaining `ip_allowlist`/IP allowlist references are only intentional docs/tests/assertions.\n\nNo commits were created." + }, + "notes": "Stage completed: implement", + "usage": { + "input": { + "usage": { + "model": { + "provider": "openai", + "model_id": "gpt-5.5" + }, + "tokens": { + "input_tokens": 2556897, + "output_tokens": 26084, + "reasoning_tokens": 9280, + "cache_read_tokens": 17923072, + "cache_write_tokens": 0 + } + }, + "facts": { + "algorithm": "openai" + } + }, + "total_usd_micros": 22806941 + }, + "files_touched": [ + "/home/daytona/workspace/fabro/docs/public/changelog/2026-05-27.mdx" + ], + "timing": { + "wall_time_ms": 0, + "inference_time_ms": 1238818, + "tool_time_ms": 594726, + "active_time_ms": 1833544 + } + }, + "start": { + "status": "succeeded", + "usage": null + } + }, + "next_node_id": "fixup", + "git_commit_sha": "2a5a13ae2b22206fbb4b114a90e8b3b84cc0182f", + "loop_failure_signatures": { + "verify|deterministic|script failed with exit code: ## output from https://github.com/fabro-sh/fabro * branch main -> fetch_head .. main -> origin/main auto-merging cargo.lock auto-merging lib/crates/fabro-api/build.rs auto-merging lib/crates/fabro": 1 + }, + "node_visits": { + "preflight_compile": 1, + "implement": 1, + "preflight_lint": 1, + "toolchain": 1, + "verify": 1, + "start": 1, + "simplify_opus": 1, + "simplify_gpt": 1 + } + }, + "diff": { + "patch": "diff --git a/Cargo.lock b/Cargo.lock\nindex 7365db53d..2ef51ead4 100644\n--- a/Cargo.lock\n+++ b/Cargo.lock\n@@ -1889,6 +1889,7 @@ dependencies = [\n \"tempfile\",\n \"thiserror 2.0.18\",\n \"tokio\",\n+ \"tokio-tungstenite 0.26.2\",\n \"tokio-util\",\n \"toml 0.8.23\",\n \"toml_edit\",\n@@ -2407,6 +2408,7 @@ dependencies = [\n \"thiserror 2.0.18\",\n \"tokio\",\n \"tokio-stream\",\n+ \"tokio-tungstenite 0.26.2\",\n \"tokio-util\",\n \"toml 0.8.23\",\n \"toml_edit\",\ndiff --git a/lib/crates/fabro-api/build.rs b/lib/crates/fabro-api/build.rs\nindex 88d465128..3ef344399 100644\n--- a/lib/crates/fabro-api/build.rs\n+++ b/lib/crates/fabro-api/build.rs\n@@ -617,6 +617,7 @@ fn main() {\n (\"SandboxTimestamps\", \"fabro_types::SandboxTimestamps\", &[]),\n (\"AskFabro\", \"fabro_types::AskFabro\", &[]),\n (\"Automation\", \"fabro_automation::Automation\", &[]),\n+ (\"AutomationRef\", \"fabro_types::AutomationRef\", &[]),\n (\"AutomationTarget\", \"fabro_automation::AutomationTarget\", &[\n ]),\n (\ndiff --git a/lib/crates/fabro-api/src/lib.rs b/lib/crates/fabro-api/src/lib.rs\nindex 3837275f0..28903a67d 100644\n--- a/lib/crates/fabro-api/src/lib.rs\n+++ b/lib/crates/fabro-api/src/lib.rs\n@@ -39,32 +39,33 @@ pub mod types {\n pub use fabro_types::{\n ActivatedSkill, AgentMcpToolSummary, AgentSkillActivationSource, AgentSkillSummary,\n AgentToolCategory, AgentToolSource, AgentToolSummary, AgentToolsAvailableProps, AskFabro,\n- AuthMethod, BilledTokenCounts, CommandTermination, Conclusion, CreateVariableRequest,\n- DiffStats, DiffSummary, DirtyStatus, EventEnvelope, ExecOutputTail, FailureCategory,\n- FailureDetail, FailureSignature, GitContext, IdpIdentity, IntegrationConnectionKind,\n- IntegrationConnectionState, IntegrationConnectionStatus, IntegrationProvider,\n- IntegrationStatus, InterviewOption, InterviewQuestionRecord, McpServerProjection,\n- McpServerStatus, PairId, PairMessageId, PairMessageRecord, PairMessageRequest, PairRecord,\n- PairStartRequest, PairStatus, PairTarget, PairTranscriptEntry, PairTranscriptResponse,\n- PendingInterviewRecord, PermissionLevel, PreRunPushOutcome, Principal, PullRequest,\n- PullRequestDetails, PullRequestDetailsStatus, PullRequestDetailsUnavailableReason,\n- PullRequestLink, PullRequestMeta, PullRequestResponse, QuestionType, RepositoryRef, Run,\n- RunApproval, RunApprovalState, RunClientProvenance, RunEvent, RunEventDetailContentKind,\n- RunEventDetailResponse, RunFailure, RunPairStatusResponse, RunProjection, RunProvenance,\n- RunRunnableSource, RunSandbox, RunSandboxFailure, RunSandboxInstance, RunSandboxKind,\n- RunSandboxPlan, RunSandboxRuntime, RunServerProvenance, RunSize, SandboxDetails,\n- SandboxInfo, SandboxListMeta, SandboxListResponse, SandboxNetwork, SandboxNetworkPolicy,\n- SandboxNetworkPolicyMode, SandboxProviderKind, SandboxProviderLookupError,\n- SandboxResources, SandboxService, SandboxServiceListResponse, SandboxState,\n- SandboxTimestamps, SecretMetadata, SecretType, ServerSettings, SessionDetail, SessionId,\n- SessionMessage, SessionRecord, SessionStatus, SessionSummary, SessionTurn,\n- SkillsProjection, StageCompletion, StageContextWindow, StageContextWindowBreakdownItem,\n- StageContextWindowCategory, StageContextWindowCountMethod, StageContextWindowProjection,\n- StageContextWindowStaleness, StageContextWindowUnavailableReason,\n- StageContextWindowWarning, StageHandler, StageModelUsage, StageOutcome, StageProjection,\n- StageState, SubAgentProjection, SubAgentStatus, SystemActorKind, SystemIntegrationStatus,\n- SystemIntegrationsResponse, TodoListProjection, TurnId, UpdateVariableRequest,\n- UserPrincipal, Variable, VariableListResponse, WorkflowSettings,\n+ AuthMethod, AutomationRef, BilledTokenCounts, CommandTermination, Conclusion,\n+ CreateVariableRequest, DiffStats, DiffSummary, DirtyStatus, EventEnvelope, ExecOutputTail,\n+ FailureCategory, FailureDetail, FailureSignature, GitContext, IdpIdentity,\n+ IntegrationConnectionKind, IntegrationConnectionState, IntegrationConnectionStatus,\n+ IntegrationProvider, IntegrationStatus, InterviewOption, InterviewQuestionRecord,\n+ McpServerProjection, McpServerStatus, PairId, PairMessageId, PairMessageRecord,\n+ PairMessageRequest, PairRecord, PairStartRequest, PairStatus, PairTarget,\n+ PairTranscriptEntry, PairTranscriptResponse, PendingInterviewRecord, PermissionLevel,\n+ PreRunPushOutcome, Principal, PullRequest, PullRequestDetails, PullRequestDetailsStatus,\n+ PullRequestDetailsUnavailableReason, PullRequestLink, PullRequestMeta, PullRequestResponse,\n+ QuestionType, RepositoryRef, Run, RunApproval, RunApprovalState, RunClientProvenance,\n+ RunEvent, RunEventDetailContentKind, RunEventDetailResponse, RunFailure,\n+ RunPairStatusResponse, RunProjection, RunProvenance, RunRunnableSource, RunSandbox,\n+ RunSandboxFailure, RunSandboxInstance, RunSandboxKind, RunSandboxPlan, RunSandboxRuntime,\n+ RunServerProvenance, RunSize, SandboxDetails, SandboxInfo, SandboxListMeta,\n+ SandboxListResponse, SandboxNetwork, SandboxNetworkPolicy, SandboxNetworkPolicyMode,\n+ SandboxProviderKind, SandboxProviderLookupError, SandboxResources, SandboxService,\n+ SandboxServiceListResponse, SandboxState, SandboxTimestamps, SecretMetadata, SecretType,\n+ ServerSettings, SessionDetail, SessionId, SessionMessage, SessionRecord, SessionStatus,\n+ SessionSummary, SessionTurn, SkillsProjection, StageCompletion, StageContextWindow,\n+ StageContextWindowBreakdownItem, StageContextWindowCategory, StageContextWindowCountMethod,\n+ StageContextWindowProjection, StageContextWindowStaleness,\n+ StageContextWindowUnavailableReason, StageContextWindowWarning, StageHandler,\n+ StageModelUsage, StageOutcome, StageProjection, StageState, SubAgentProjection,\n+ SubAgentStatus, SystemActorKind, SystemIntegrationStatus, SystemIntegrationsResponse,\n+ TodoListProjection, TurnId, UpdateVariableRequest, UserPrincipal, Variable,\n+ VariableListResponse, WorkflowSettings,\n };\n \n pub use crate::generated::types::*;\ndiff --git a/lib/crates/fabro-api/tests/run_summary_round_trip.rs b/lib/crates/fabro-api/tests/run_summary_round_trip.rs\nindex 0c5a70fb5..798448941 100644\n--- a/lib/crates/fabro-api/tests/run_summary_round_trip.rs\n+++ b/lib/crates/fabro-api/tests/run_summary_round_trip.rs\n@@ -3,9 +3,9 @@ use std::collections::HashMap;\n \n use chrono::{TimeZone, Utc};\n use fabro_api::types::{\n- RepositoryRef as ApiRepositoryRef, Run as ApiRun, RunApproval as ApiRunApproval,\n- RunApprovalState as ApiRunApprovalState, RunRunnableSource as ApiRunRunnableSource,\n- RunSize as ApiRunSize,\n+ AutomationRef as ApiAutomationRef, RepositoryRef as ApiRepositoryRef, Run as ApiRun,\n+ RunApproval as ApiRunApproval, RunApprovalState as ApiRunApprovalState,\n+ RunRunnableSource as ApiRunRunnableSource, RunSize as ApiRunSize,\n };\n use fabro_types::status::{RunStatus, SuccessReason};\n use fabro_types::{\n@@ -24,6 +24,7 @@ fn run_summary_reuses_domain_types() {\n assert_same_type::();\n assert_same_type::();\n assert_same_type::();\n+ assert_same_type::();\n }\n \n #[test]\ndiff --git a/lib/crates/fabro-cli/Cargo.toml b/lib/crates/fabro-cli/Cargo.toml\nindex a4d15b5b5..d0b8b7373 100644\n--- a/lib/crates/fabro-cli/Cargo.toml\n+++ b/lib/crates/fabro-cli/Cargo.toml\n@@ -61,6 +61,7 @@ anyhow.workspace = true\n miette.workspace = true\n dotenvy.workspace = true\n tokio.workspace = true\n+tokio-tungstenite.workspace = true\n tracing.workspace = true\n tracing-subscriber.workspace = true\n tracing-appender.workspace = true\n@@ -128,3 +129,4 @@ fabro-test = { workspace = true }\n fabro-macros = { path = \"../fabro-macros\" }\n hkdf.workspace = true\n reqwest = { workspace = true, features = [\"cookies\"] }\n+tokio = { workspace = true, features = [\"test-util\", \"macros\"] }\ndiff --git a/lib/crates/fabro-cli/src/commands/run/runner.rs b/lib/crates/fabro-cli/src/commands/run/runner.rs\nindex fb3b31d43..ac1a2bbd0 100644\n--- a/lib/crates/fabro-cli/src/commands/run/runner.rs\n+++ b/lib/crates/fabro-cli/src/commands/run/runner.rs\n@@ -1,10 +1,4 @@\n-#![expect(\n- clippy::disallowed_types,\n- reason = \"sync CLI `run` subprocess wrapper: reads server subprocess stdout line-by-line via \\\n- std::io::BufReader; not on a Tokio path\"\n-)]\n-\n-use std::io::{BufRead as StdBufRead, BufReader as StdBufReader};\n+use std::collections::{HashSet, VecDeque};\n use std::path::{Path, PathBuf};\n use std::sync::Arc;\n use std::time::Duration;\n@@ -12,10 +6,14 @@ use std::time::Duration;\n use anyhow::{Context, Result, anyhow};\n use async_trait::async_trait;\n use fabro_api::types::RunManifest;\n+use fabro_client::ServerTarget;\n use fabro_config::user::active_settings_path;\n use fabro_config::{ServerSettingsBuilder, Storage, load_llm_catalog_settings};\n use fabro_interview::{\n- AnswerSubmission, ControlInterviewer, WorkerControlEnvelope, WorkerControlMessage,\n+ AnswerSubmission, ControlInterviewer, WORKER_CONTROL_INVALID_CURSOR_REASON,\n+ WORKER_CONTROL_PONG_TIMEOUT_REASON, WORKER_CONTROL_WS_LIVENESS_TIMEOUT,\n+ WORKER_CONTROL_WS_PING_INTERVAL, WorkerControlDeliveryFrame, WorkerControlEnvelope,\n+ WorkerControlMessage,\n };\n use fabro_model::Catalog;\n use fabro_server::run_tool_manifest;\n@@ -34,11 +32,23 @@ use fabro_workflow::operations::{self, StartServices};\n use fabro_workflow::run_control::RunControlState;\n use fabro_workflow::runtime_store::{RunStoreBackend, RunStoreHandle};\n use fabro_workflow::services::FabroRunToolServices;\n+use futures::{SinkExt, StreamExt};\n use jsonwebtoken::dangerous::insecure_decode;\n+#[cfg(test)]\n+use tokio::io::DuplexStream;\n+use tokio::net::TcpStream;\n+#[cfg(unix)]\n+use tokio::net::UnixStream;\n #[cfg(unix)]\n use tokio::signal::unix::{SignalKind, signal};\n-use tokio::sync::{Mutex, RwLock as AsyncRwLock, mpsc};\n-use tokio::time::sleep;\n+use tokio::sync::{Mutex, RwLock as AsyncRwLock, oneshot};\n+use tokio::task::JoinHandle;\n+use tokio::time::{self, Instant, MissedTickBehavior};\n+use tokio_tungstenite::tungstenite::client::IntoClientRequest;\n+use tokio_tungstenite::tungstenite::http::{HeaderValue, Request as WebSocketRequest, header};\n+use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode;\n+use tokio_tungstenite::tungstenite::protocol::{self, Message as WebSocketMessage};\n+use tokio_tungstenite::{MaybeTlsStream, WebSocketStream, connect_async, tungstenite};\n use tokio_util::sync::CancellationToken;\n \n use crate::args::RunWorkerMode;\n@@ -75,7 +85,7 @@ pub(crate) async fn execute(\n let _ = fabro_proc::title_init();\n set_worker_title(&run_id, initial_worker_title_phase(mode));\n \n- let target = server.parse::()?;\n+ let target = server.parse::()?;\n let client = server_client::connect_server_target_with_bearer(&target, worker_token).await?;\n let run_store = HttpRunStore::connect(run_id, client.clone_for_reuse()).await?;\n let run_state = run_store\n@@ -110,13 +120,24 @@ pub(crate) async fn execute(\n let cancel_token = CancellationToken::new();\n let emitter = Arc::new(Emitter::new(run_id));\n let steering_hub = Arc::new(fabro_workflow::SteeringHub::new(Arc::clone(&emitter)));\n- spawn_worker_control_stream(\n- Arc::clone(&interviewer),\n- cancel_token.clone(),\n- Arc::clone(&steering_hub),\n- )?;\n let run_control = RunControlState::new();\n install_signal_handlers(Arc::clone(&run_control), cancel_token.clone())?;\n+ let mut control_manager = if run_state.status.is_terminal() {\n+ None\n+ } else {\n+ Some(spawn_worker_control_manager(\n+ target.clone(),\n+ run_id,\n+ worker_token.to_owned(),\n+ Arc::clone(&interviewer),\n+ cancel_token.clone(),\n+ Arc::clone(&steering_hub),\n+ Arc::clone(&run_control),\n+ ))\n+ };\n+ if let Some(control_manager) = &mut control_manager {\n+ control_manager.wait_for_first_connection().await?;\n+ }\n let vault = load_worker_vault(storage_dir.as_deref())?;\n let github_app = {\n let vault_guard = match &vault {\n@@ -158,13 +179,26 @@ pub(crate) async fn execute(\n fabro_run_tools,\n };\n \n- match mode {\n- RunWorkerMode::Start => {\n- operations::start(&run_dir, services).await?;\n+ let execution = async {\n+ match mode {\n+ RunWorkerMode::Start => operations::start(&run_dir, services).await,\n+ RunWorkerMode::Resume => operations::resume(&run_dir, services).await,\n }\n- RunWorkerMode::Resume => {\n- operations::resume(&run_dir, services).await?;\n+ };\n+\n+ if let Some(mut control_manager) = control_manager {\n+ tokio::select! {\n+ result = execution => {\n+ control_manager.finish();\n+ result?;\n+ }\n+ fatal = control_manager.fatal_control_loss() => {\n+ control_manager.finish();\n+ return Err(fatal);\n+ }\n }\n+ } else {\n+ execution.await?;\n }\n \n Ok(())\n@@ -254,96 +288,510 @@ fn load_worker_vault(storage_dir: Option<&Path>) -> Result,\n- cancel_token: CancellationToken,\n- steering_hub: Arc,\n-) -> Result<()> {\n- let (event_tx, event_rx) = mpsc::unbounded_channel();\n- tokio::spawn(handle_worker_control_stream_events(\n- interviewer,\n- cancel_token,\n- steering_hub,\n- event_rx,\n- ));\n- std::thread::Builder::new()\n- .name(\"fabro-worker-control\".to_string())\n- .spawn(move || {\n- read_worker_control_stream_blocking(StdBufReader::new(std::io::stdin()), &event_tx);\n- })\n- .context(\"failed to spawn worker control reader thread\")?;\n- Ok(())\n+#[derive(Default)]\n+struct AppliedWorkerControlDeliveryIds {\n+ last: Option,\n+ order: VecDeque,\n+ recent: HashSet,\n }\n \n-fn read_worker_control_stream_blocking(\n- mut reader: R,\n- event_tx: &mpsc::UnboundedSender,\n-) where\n- R: StdBufRead,\n-{\n- let mut line = String::new();\n- loop {\n- line.clear();\n- match reader.read_line(&mut line) {\n- Ok(0) | Err(_) => {\n- let _ = event_tx.send(WorkerControlStreamEvent::Eof);\n- break;\n- }\n- Ok(_) => {\n- let line = line.trim_end_matches(['\\r', '\\n']).to_string();\n- if event_tx.send(WorkerControlStreamEvent::Line(line)).is_err() {\n- break;\n- }\n+impl AppliedWorkerControlDeliveryIds {\n+ fn last_applied_id(&self) -> Option<&str> {\n+ self.last.as_deref()\n+ }\n+\n+ fn contains(&self, id: &str) -> bool {\n+ self.recent.contains(id)\n+ }\n+\n+ fn record(&mut self, id: String) {\n+ if !self.recent.insert(id.clone()) {\n+ self.last = Some(id);\n+ return;\n+ }\n+ self.order.push_back(id.clone());\n+ self.last = Some(id);\n+ while self.order.len() > WORKER_CONTROL_APPLIED_ID_DEDUPE_CAPACITY {\n+ if let Some(evicted) = self.order.pop_front() {\n+ self.recent.remove(&evicted);\n }\n }\n }\n }\n \n-async fn handle_worker_control_stream_events(\n+struct WorkerControlManagerHandle {\n+ first_connection: Option>>,\n+ fatal: Option>,\n+ done: CancellationToken,\n+ task: JoinHandle<()>,\n+}\n+\n+impl WorkerControlManagerHandle {\n+ async fn wait_for_first_connection(&mut self) -> Result<()> {\n+ let receiver = self\n+ .first_connection\n+ .take()\n+ .context(\"worker control first-connection receiver missing\")?;\n+ receiver\n+ .await\n+ .context(\"worker control manager stopped before first connection\")?\n+ }\n+\n+ async fn fatal_control_loss(&mut self) -> anyhow::Error {\n+ let Some(receiver) = self.fatal.take() else {\n+ return anyhow!(\"worker control fatal receiver missing\");\n+ };\n+ receiver\n+ .await\n+ .unwrap_or_else(|_| anyhow!(\"worker control manager stopped before workflow completed\"))\n+ }\n+\n+ fn finish(&self) {\n+ self.done.cancel();\n+ self.task.abort();\n+ }\n+}\n+\n+#[derive(Debug)]\n+struct WorkerControlStreamConnectRequest {\n+ request: WebSocketRequest<()>,\n+ unix_socket_path: Option,\n+}\n+\n+impl WorkerControlStreamConnectRequest {\n+ fn request_for_tungstenite(&self) -> WebSocketRequest<()> {\n+ self.request.clone()\n+ }\n+\n+ #[cfg(test)]\n+ fn uri(&self) -> String {\n+ self.request.uri().to_string()\n+ }\n+\n+ #[cfg(test)]\n+ fn authorization(&self) -> Option<&str> {\n+ self.request\n+ .headers()\n+ .get(header::AUTHORIZATION)\n+ .and_then(|value| value.to_str().ok())\n+ }\n+}\n+\n+enum WorkerControlSocket {\n+ Tcp(Box>>),\n+ #[cfg(unix)]\n+ Unix(Box>),\n+ #[cfg(test)]\n+ Test(Box>),\n+}\n+\n+impl WorkerControlSocket {\n+ async fn send(&mut self, message: WebSocketMessage) -> Result<(), tungstenite::Error> {\n+ match self {\n+ Self::Tcp(socket) => socket.send(message).await,\n+ #[cfg(unix)]\n+ Self::Unix(socket) => socket.send(message).await,\n+ #[cfg(test)]\n+ Self::Test(socket) => socket.send(message).await,\n+ }\n+ }\n+\n+ async fn next(&mut self) -> Option> {\n+ match self {\n+ Self::Tcp(socket) => socket.next().await,\n+ #[cfg(unix)]\n+ Self::Unix(socket) => socket.next().await,\n+ #[cfg(test)]\n+ Self::Test(socket) => socket.next().await,\n+ }\n+ }\n+}\n+\n+#[derive(Debug)]\n+enum WorkerControlConnectError {\n+ InvalidCursor,\n+ Other(anyhow::Error),\n+}\n+\n+fn spawn_worker_control_manager(\n+ target: ServerTarget,\n+ run_id: RunId,\n+ worker_token: String,\n+ interviewer: Arc,\n+ cancel_token: CancellationToken,\n+ steering_hub: Arc,\n+ run_control: Arc,\n+) -> WorkerControlManagerHandle {\n+ let (first_tx, first_rx) = oneshot::channel();\n+ let (fatal_tx, fatal_rx) = oneshot::channel();\n+ let done = CancellationToken::new();\n+ let task_done = done.clone();\n+ let task = tokio::spawn(async move {\n+ run_worker_control_manager(\n+ target,\n+ run_id,\n+ worker_token,\n+ interviewer,\n+ cancel_token,\n+ steering_hub,\n+ run_control,\n+ task_done,\n+ first_tx,\n+ fatal_tx,\n+ )\n+ .await;\n+ });\n+ WorkerControlManagerHandle {\n+ first_connection: Some(first_rx),\n+ fatal: Some(fatal_rx),\n+ done,\n+ task,\n+ }\n+}\n+\n+#[allow(\n+ clippy::too_many_arguments,\n+ reason = \"Worker control manager owns the worker-side control dependencies.\"\n+)]\n+async fn run_worker_control_manager(\n+ target: ServerTarget,\n+ run_id: RunId,\n+ worker_token: String,\n interviewer: Arc,\n cancel_token: CancellationToken,\n steering_hub: Arc,\n- mut event_rx: mpsc::UnboundedReceiver,\n+ run_control: Arc,\n+ done: CancellationToken,\n+ first_tx: oneshot::Sender>,\n+ fatal_tx: oneshot::Sender,\n ) {\n- while let Some(event) = event_rx.recv().await {\n- match event {\n- WorkerControlStreamEvent::Line(line) => {\n- apply_worker_control_line(&interviewer, &cancel_token, &steering_hub, &line).await;\n+ let mut first_tx = Some(first_tx);\n+ let mut fatal_tx = Some(fatal_tx);\n+ let mut backoff = WORKER_CONTROL_RECONNECT_INITIAL_BACKOFF;\n+ let mut applied_ids = AppliedWorkerControlDeliveryIds::default();\n+\n+ while !done.is_cancelled() {\n+ let request = match build_worker_control_stream_request(\n+ &target,\n+ &run_id,\n+ &worker_token,\n+ applied_ids.last_applied_id(),\n+ ) {\n+ Ok(request) => request,\n+ Err(err) => {\n+ report_fatal_control_loss(\n+ &interviewer,\n+ &cancel_token,\n+ &mut first_tx,\n+ &mut fatal_tx,\n+ format!(\"failed to build worker control stream request: {err:#}\"),\n+ )\n+ .await;\n+ return;\n+ }\n+ };\n+\n+ match connect_worker_control_stream(request).await {\n+ Ok(mut socket) => {\n+ if let Some(first_tx) = first_tx.take() {\n+ let _ = first_tx.send(Ok(()));\n+ }\n+ backoff = WORKER_CONTROL_RECONNECT_INITIAL_BACKOFF;\n+ match handle_worker_control_socket(\n+ &mut socket,\n+ &interviewer,\n+ &cancel_token,\n+ &steering_hub,\n+ &run_control,\n+ &mut applied_ids,\n+ &done,\n+ )\n+ .await\n+ {\n+ Ok(()) => {}\n+ Err(WorkerControlConnectError::InvalidCursor) => {\n+ report_fatal_control_loss(\n+ &interviewer,\n+ &cancel_token,\n+ &mut first_tx,\n+ &mut fatal_tx,\n+ \"worker control stream replay cursor is invalid\".to_string(),\n+ )\n+ .await;\n+ return;\n+ }\n+ Err(WorkerControlConnectError::Other(err)) => {\n+ tracing::debug!(error = %err, \"Worker control stream disconnected\");\n+ }\n+ }\n }\n- WorkerControlStreamEvent::Eof => {\n- interviewer.interrupt_all().await;\n+ Err(WorkerControlConnectError::InvalidCursor) => {\n+ report_fatal_control_loss(\n+ &interviewer,\n+ &cancel_token,\n+ &mut first_tx,\n+ &mut fatal_tx,\n+ \"worker control stream replay cursor is invalid\".to_string(),\n+ )\n+ .await;\n return;\n }\n+ Err(WorkerControlConnectError::Other(err)) => {\n+ tracing::debug!(error = %err, \"Worker control stream connection failed\");\n+ }\n }\n+\n+ sleep_or_done(&done, backoff).await;\n+ backoff = next_worker_control_reconnect_backoff(backoff);\n }\n+}\n \n+async fn report_fatal_control_loss(\n+ interviewer: &ControlInterviewer,\n+ cancel_token: &CancellationToken,\n+ first_tx: &mut Option>>,\n+ fatal_tx: &mut Option>,\n+ detail: String,\n+) {\n+ let message = format!(\"worker control channel lost: {detail}\");\n interviewer.interrupt_all().await;\n+ if let Some(first_tx) = first_tx.take() {\n+ let _ = first_tx.send(Err(anyhow!(message.clone())));\n+ }\n+ if let Some(fatal_tx) = fatal_tx.take() {\n+ let _ = fatal_tx.send(anyhow!(message));\n+ }\n+ cancel_token.cancel();\n+}\n+\n+async fn sleep_or_done(done: &CancellationToken, delay: Duration) {\n+ tokio::select! {\n+ () = done.cancelled() => {}\n+ () = time::sleep(delay) => {}\n+ }\n+}\n+\n+fn next_worker_control_reconnect_backoff(current: Duration) -> Duration {\n+ current\n+ .saturating_mul(2)\n+ .min(WORKER_CONTROL_RECONNECT_MAX_BACKOFF)\n+}\n+\n+fn build_worker_control_stream_request(\n+ target: &ServerTarget,\n+ run_id: &RunId,\n+ worker_token: &str,\n+ after: Option<&str>,\n+) -> Result {\n+ let (url, unix_socket_path) = match target {\n+ ServerTarget::HttpUrl(_) => {\n+ let base = target\n+ .as_http_url()\n+ .context(\"HTTP server target missing URL\")?;\n+ let websocket_base = if let Some(rest) = base.strip_prefix(\"http://\") {\n+ format!(\"ws://{rest}\")\n+ } else if let Some(rest) = base.strip_prefix(\"https://\") {\n+ format!(\"wss://{rest}\")\n+ } else {\n+ anyhow::bail!(\"unsupported server URL scheme\");\n+ };\n+ (\n+ worker_control_stream_url(&websocket_base, run_id, after),\n+ None,\n+ )\n+ }\n+ ServerTarget::UnixSocket(path) => {\n+ let url = worker_control_stream_url(\"ws://fabro\", run_id, after);\n+ (url, Some(path.as_path().to_path_buf()))\n+ }\n+ };\n+ let mut request = url\n+ .as_str()\n+ .into_client_request()\n+ .context(\"failed to build worker control stream request\")?;\n+ request.headers_mut().insert(\n+ header::AUTHORIZATION,\n+ HeaderValue::from_str(&format!(\"Bearer {worker_token}\"))\n+ .context(\"failed to build worker control stream authorization header\")?,\n+ );\n+ Ok(WorkerControlStreamConnectRequest {\n+ request,\n+ unix_socket_path,\n+ })\n }\n \n-async fn apply_worker_control_line(\n+fn worker_control_stream_url(base: &str, run_id: &RunId, after: Option<&str>) -> String {\n+ let mut url = format!(\"{base}/api/v1/runs/{run_id}/worker/control-stream\");\n+ if let Some(after) = after {\n+ url.push_str(\"?after=\");\n+ url.push_str(after);\n+ }\n+ url\n+}\n+\n+async fn connect_worker_control_stream(\n+ request: WorkerControlStreamConnectRequest,\n+) -> Result {\n+ if let Some(path) = request.unix_socket_path.as_ref() {\n+ #[cfg(unix)]\n+ {\n+ let ws_request = request.request_for_tungstenite();\n+ let stream = UnixStream::connect(path)\n+ .await\n+ .map_err(|err| WorkerControlConnectError::Other(anyhow::Error::new(err)))?;\n+ let (socket, _) = tokio_tungstenite::client_async(ws_request, stream)\n+ .await\n+ .map_err(classify_tungstenite_error)?;\n+ Ok(WorkerControlSocket::Unix(Box::new(socket)))\n+ }\n+ #[cfg(not(unix))]\n+ {\n+ let _ = path;\n+ Err(WorkerControlConnectError::Other(anyhow!(\n+ \"Unix-socket worker control stream is not supported on this platform\"\n+ )))\n+ }\n+ } else {\n+ let (socket, _) = connect_async(request.request)\n+ .await\n+ .map_err(classify_tungstenite_error)?;\n+ Ok(WorkerControlSocket::Tcp(Box::new(socket)))\n+ }\n+}\n+\n+fn classify_tungstenite_error(error: tungstenite::Error) -> WorkerControlConnectError {\n+ if let tungstenite::Error::Http(response) = &error {\n+ if response.status().as_u16() == 410 {\n+ return WorkerControlConnectError::InvalidCursor;\n+ }\n+ }\n+ WorkerControlConnectError::Other(anyhow::Error::new(error))\n+}\n+\n+async fn handle_worker_control_socket(\n+ socket: &mut WorkerControlSocket,\n interviewer: &ControlInterviewer,\n cancel_token: &CancellationToken,\n steering_hub: &fabro_workflow::SteeringHub,\n- line: &str,\n-) {\n- if line.trim().is_empty() {\n- return;\n+ run_control: &RunControlState,\n+ applied_ids: &mut AppliedWorkerControlDeliveryIds,\n+ done: &CancellationToken,\n+) -> Result<(), WorkerControlConnectError> {\n+ let mut ping_interval = time::interval(WORKER_CONTROL_WS_PING_INTERVAL);\n+ ping_interval.set_missed_tick_behavior(MissedTickBehavior::Delay);\n+ let mut last_liveness = Instant::now();\n+ let liveness_timeout = time::sleep_until(last_liveness + WORKER_CONTROL_WS_LIVENESS_TIMEOUT);\n+ tokio::pin!(liveness_timeout);\n+\n+ loop {\n+ liveness_timeout\n+ .as_mut()\n+ .reset(last_liveness + WORKER_CONTROL_WS_LIVENESS_TIMEOUT);\n+\n+ tokio::select! {\n+ () = done.cancelled() => return Ok(()),\n+ _ = ping_interval.tick() => {\n+ socket\n+ .send(WebSocketMessage::Ping(Vec::new().into()))\n+ .await\n+ .map_err(|err| WorkerControlConnectError::Other(anyhow::Error::new(err)))?;\n+ }\n+ () = &mut liveness_timeout => {\n+ let _ = socket\n+ .send(WebSocketMessage::Close(Some(protocol::CloseFrame {\n+ code: CloseCode::Away,\n+ reason: WORKER_CONTROL_PONG_TIMEOUT_REASON.into(),\n+ })))\n+ .await;\n+ return Err(WorkerControlConnectError::Other(anyhow!(\n+ \"worker control WebSocket liveness timed out\"\n+ )));\n+ }\n+ message = socket.next() => {\n+ let Some(message) = message else {\n+ return Ok(());\n+ };\n+ match message {\n+ Ok(WebSocketMessage::Text(text)) => {\n+ last_liveness = Instant::now();\n+ let frame = serde_json::from_str::(text.as_str())\n+ .map_err(|err| WorkerControlConnectError::Other(anyhow::Error::new(err)))?;\n+ apply_worker_control_delivery_frame(\n+ interviewer,\n+ cancel_token,\n+ steering_hub,\n+ run_control,\n+ applied_ids,\n+ frame,\n+ )\n+ .await;\n+ }\n+ Ok(WebSocketMessage::Ping(payload)) => {\n+ last_liveness = Instant::now();\n+ socket\n+ .send(WebSocketMessage::Pong(payload))\n+ .await\n+ .map_err(|err| WorkerControlConnectError::Other(anyhow::Error::new(err)))?;\n+ }\n+ Ok(WebSocketMessage::Pong(_) | WebSocketMessage::Binary(_)) => {\n+ last_liveness = Instant::now();\n+ }\n+ Ok(WebSocketMessage::Close(frame)) => {\n+ if frame.as_ref().is_some_and(|frame| {\n+ frame.reason.as_str() == WORKER_CONTROL_INVALID_CURSOR_REASON\n+ }) {\n+ return Err(WorkerControlConnectError::InvalidCursor);\n+ }\n+ return Ok(());\n+ }\n+ Ok(WebSocketMessage::Frame(_)) => {}\n+ Err(err) => {\n+ return Err(WorkerControlConnectError::Other(anyhow::Error::new(err)));\n+ }\n+ }\n+ }\n+ }\n }\n+}\n \n- let Ok(message) = serde_json::from_str::(line) else {\n- return;\n- };\n+async fn apply_worker_control_delivery_frame(\n+ interviewer: &ControlInterviewer,\n+ cancel_token: &CancellationToken,\n+ steering_hub: &fabro_workflow::SteeringHub,\n+ run_control: &RunControlState,\n+ applied_ids: &mut AppliedWorkerControlDeliveryIds,\n+ frame: WorkerControlDeliveryFrame,\n+) -> bool {\n+ // Duplicate ids cannot reach us under normal operation: the server replays\n+ // strictly after the last applied id. Guard against a server-side bug or\n+ // reconnect race by ignoring recently-applied delivery ids.\n+ if applied_ids.contains(&frame.id) {\n+ return false;\n+ }\n+ let frame_id = frame.id;\n+ apply_worker_control_message(\n+ interviewer,\n+ cancel_token,\n+ steering_hub,\n+ run_control,\n+ frame.envelope,\n+ )\n+ .await;\n+ applied_ids.record(frame_id);\n+ true\n+}\n \n+async fn apply_worker_control_message(\n+ interviewer: &ControlInterviewer,\n+ cancel_token: &CancellationToken,\n+ steering_hub: &fabro_workflow::SteeringHub,\n+ run_control: &RunControlState,\n+ message: WorkerControlEnvelope,\n+) {\n match message.message {\n WorkerControlMessage::InterviewAnswer { qid, answer, actor } => {\n let _ = interviewer\n@@ -354,6 +802,12 @@ async fn apply_worker_control_line(\n cancel_token.cancel();\n interviewer.interrupt_all().await;\n }\n+ WorkerControlMessage::RunPause => {\n+ run_control.request_pause();\n+ }\n+ WorkerControlMessage::RunUnpause => {\n+ run_control.request_unpause();\n+ }\n WorkerControlMessage::Steer { text, actor } => {\n steering_hub.deliver_steer(text, Some(actor));\n }\n@@ -485,7 +939,7 @@ impl HttpRunStore {\n Err(err) => last_error = Some(err),\n }\n if let Some(delay) = RUN_STORE_RETRY_DELAYS.get(attempt) {\n- sleep(*delay).await;\n+ time::sleep(*delay).await;\n }\n }\n Err(last_error\n@@ -757,10 +1211,14 @@ fn install_signal_handlers(\n )]\n mod tests {\n use std::sync::Arc;\n+ use std::time::Duration;\n \n use chrono::Utc;\n+ use fabro_client::ServerTarget;\n use fabro_config::Storage;\n- use fabro_interview::{AnswerValue, ControlInterviewer, Interviewer, Question};\n+ use fabro_interview::{\n+ AnswerValue, ControlInterviewer, Interviewer, Question, WorkerControlEnvelope,\n+ };\n use fabro_types::run_event::{\n InterviewCompletedProps, InterviewStartedProps, RunCompletedProps, RunControlEffectProps,\n RunFailedProps, RunStatusTransitionProps,\n@@ -771,12 +1229,17 @@ mod tests {\n };\n use fabro_vault::{SecretType, Vault};\n use fabro_workflow::event::RunEventSink;\n+ use fabro_workflow::run_control::RunControlState;\n+ use tokio::time;\n+ use tokio_tungstenite::tungstenite::protocol::{Message as TestWebSocketMessage, Role};\n use tokio_util::sync::CancellationToken;\n \n use super::{\n- WorkerControlStreamEvent, WorkerTitlePhase, apply_worker_control_line,\n- handle_worker_control_stream_events, initial_worker_title_phase, load_worker_vault,\n- read_worker_control_stream_blocking, stamp_system_worker, worker_title,\n+ AppliedWorkerControlDeliveryIds, WorkerControlConnectError, WorkerControlSocket,\n+ WorkerTitlePhase, apply_worker_control_delivery_frame, apply_worker_control_message,\n+ build_worker_control_stream_request, connect_worker_control_stream,\n+ handle_worker_control_socket, initial_worker_title_phase, load_worker_vault,\n+ next_worker_control_reconnect_backoff, stamp_system_worker, worker_title,\n worker_title_phase_for_event,\n };\n use crate::args::RunWorkerMode;\n@@ -1018,20 +1481,28 @@ mod tests {\n }\n \n #[tokio::test]\n- async fn worker_control_line_routes_answer_by_question_id() {\n+ async fn worker_control_routes_answer_by_question_id() {\n let interviewer = Arc::new(ControlInterviewer::new());\n let cancel_token = CancellationToken::new();\n+ let run_control = RunControlState::new();\n let mut question = Question::new(\"Approve?\", QuestionType::YesNo);\n question.id = \"q-1\".to_string();\n let ask_interviewer = Arc::clone(&interviewer);\n let answer_task = tokio::spawn(async move { ask_interviewer.ask(question).await });\n \n let hub = test_steering_hub();\n- apply_worker_control_line(\n+ apply_worker_control_message(\n &interviewer,\n &cancel_token,\n &hub,\n- r#\"{\"v\":1,\"type\":\"interview.answer\",\"qid\":\"q-1\",\"answer\":{\"kind\":\"yes\"},\"actor\":{\"kind\":\"system\",\"system_kind\":\"engine\"}}\"#,\n+ &run_control,\n+ WorkerControlEnvelope::interview_answer(\n+ \"q-1\",\n+ fabro_interview::AnswerSubmission::system(\n+ fabro_interview::Answer::yes(),\n+ fabro_types::SystemActorKind::Engine,\n+ ),\n+ ),\n )\n .await;\n \n@@ -1041,9 +1512,10 @@ mod tests {\n }\n \n #[tokio::test]\n- async fn worker_control_line_cancel_sets_cancel_token_and_interrupts_pending_interviews() {\n+ async fn worker_control_cancel_sets_cancel_token_and_interrupts_pending_interviews() {\n let interviewer = Arc::new(ControlInterviewer::new());\n let cancel_token = CancellationToken::new();\n+ let run_control = RunControlState::new();\n let mut question = Question::new(\"Approve?\", QuestionType::YesNo);\n question.id = \"q-1\".to_string();\n let ask_interviewer = Arc::clone(&interviewer);\n@@ -1051,11 +1523,12 @@ mod tests {\n tokio::task::yield_now().await;\n \n let hub = test_steering_hub();\n- apply_worker_control_line(\n+ apply_worker_control_message(\n &interviewer,\n &cancel_token,\n &hub,\n- r#\"{\"v\":1,\"type\":\"run.cancel\"}\"#,\n+ &run_control,\n+ WorkerControlEnvelope::cancel_run(),\n )\n .await;\n \n@@ -1065,57 +1538,206 @@ mod tests {\n }\n \n #[tokio::test]\n- async fn blocking_worker_control_stream_emits_lines_and_eof() {\n- let (event_tx, mut event_rx) = tokio::sync::mpsc::unbounded_channel();\n+ async fn worker_control_pause_and_unpause_route_to_run_control() {\n+ let interviewer = Arc::new(ControlInterviewer::new());\n+ let cancel_token = CancellationToken::new();\n+ let run_control = RunControlState::new();\n+ let hub = test_steering_hub();\n \n- read_worker_control_stream_blocking(\n- std::io::Cursor::new(\n- b\"{\\\"v\\\":1,\\\"type\\\":\\\"run.cancel\\\"}\\n{\\\"v\\\":1,\\\"type\\\":\\\"interview.answer\\\",\\\"qid\\\":\\\"q-1\\\",\\\"answer\\\":{\\\"kind\\\":\\\"yes\\\"},\\\"actor\\\":{\\\"kind\\\":\\\"system\\\",\\\"system_kind\\\":\\\"engine\\\"}}\\n\",\n- ),\n- &event_tx,\n+ apply_worker_control_message(\n+ &interviewer,\n+ &cancel_token,\n+ &hub,\n+ &run_control,\n+ WorkerControlEnvelope::pause_run(),\n+ )\n+ .await;\n+ assert!(run_control.pause_requested());\n+\n+ apply_worker_control_message(\n+ &interviewer,\n+ &cancel_token,\n+ &hub,\n+ &run_control,\n+ WorkerControlEnvelope::unpause_run(),\n+ )\n+ .await;\n+ assert!(!run_control.pause_requested());\n+ }\n+\n+ #[tokio::test]\n+ async fn duplicate_delivery_ids_are_not_applied_twice() {\n+ let interviewer = Arc::new(ControlInterviewer::new());\n+ let cancel_token = CancellationToken::new();\n+ let run_control = RunControlState::new();\n+ let hub = test_steering_hub();\n+ let mut applied_ids = AppliedWorkerControlDeliveryIds::default();\n+ let frame = fabro_interview::WorkerControlDeliveryFrame {\n+ id: \"local:1\".to_string(),\n+ envelope: WorkerControlEnvelope::pause_run(),\n+ };\n+\n+ assert!(\n+ apply_worker_control_delivery_frame(\n+ &interviewer,\n+ &cancel_token,\n+ &hub,\n+ &run_control,\n+ &mut applied_ids,\n+ frame.clone(),\n+ )\n+ .await\n+ );\n+ assert!(\n+ !apply_worker_control_delivery_frame(\n+ &interviewer,\n+ &cancel_token,\n+ &hub,\n+ &run_control,\n+ &mut applied_ids,\n+ frame,\n+ )\n+ .await\n+ );\n+\n+ assert_eq!(applied_ids.last_applied_id(), Some(\"local:1\"));\n+ }\n+\n+ #[test]\n+ fn worker_control_request_construction_for_http_targets() {\n+ let run_id = fixtures::RUN_1;\n+ let request = build_worker_control_stream_request(\n+ &ServerTarget::http_url(\"http://example.com:3000\").unwrap(),\n+ &run_id,\n+ \"worker-token\",\n+ None,\n+ )\n+ .unwrap();\n+ assert_eq!(\n+ request.uri(),\n+ format!(\"ws://example.com:3000/api/v1/runs/{run_id}/worker/control-stream\")\n+ );\n+ assert_eq!(request.authorization(), Some(\"Bearer worker-token\"));\n+\n+ let reconnect = build_worker_control_stream_request(\n+ &ServerTarget::http_url(\"https://example.com\").unwrap(),\n+ &run_id,\n+ \"worker-token\",\n+ Some(\"local:42\"),\n+ )\n+ .unwrap();\n+ assert_eq!(\n+ reconnect.uri(),\n+ format!(\"wss://example.com/api/v1/runs/{run_id}/worker/control-stream?after=local:42\")\n );\n+ }\n+\n+ #[cfg(unix)]\n+ #[test]\n+ fn worker_control_request_construction_for_unix_socket_targets() {\n+ let run_id = fixtures::RUN_1;\n+ let request = build_worker_control_stream_request(\n+ &ServerTarget::unix_socket_path(\"/tmp/fabro.sock\").unwrap(),\n+ &run_id,\n+ \"worker-token\",\n+ Some(\"local:42\"),\n+ )\n+ .unwrap();\n \n assert_eq!(\n- event_rx.try_recv(),\n- Ok(WorkerControlStreamEvent::Line(\n- r#\"{\"v\":1,\"type\":\"run.cancel\"}\"#.to_string()\n- ))\n+ request.uri(),\n+ format!(\"ws://fabro/api/v1/runs/{run_id}/worker/control-stream?after=local:42\")\n );\n assert_eq!(\n- event_rx.try_recv(),\n- Ok(WorkerControlStreamEvent::Line(\n- r#\"{\"v\":1,\"type\":\"interview.answer\",\"qid\":\"q-1\",\"answer\":{\"kind\":\"yes\"},\"actor\":{\"kind\":\"system\",\"system_kind\":\"engine\"}}\"#\n- .to_string()\n- ))\n+ request.unix_socket_path.as_deref(),\n+ Some(std::path::Path::new(\"/tmp/fabro.sock\"))\n );\n- assert_eq!(event_rx.try_recv(), Ok(WorkerControlStreamEvent::Eof));\n }\n \n- #[tokio::test]\n- async fn worker_control_event_loop_eof_interrupts_pending_interviews() {\n+ #[test]\n+ fn worker_control_reconnect_backoff_is_bounded() {\n+ assert_eq!(\n+ next_worker_control_reconnect_backoff(Duration::from_millis(100)),\n+ Duration::from_millis(200)\n+ );\n+ assert_eq!(\n+ next_worker_control_reconnect_backoff(Duration::from_secs(4)),\n+ Duration::from_secs(5)\n+ );\n+ assert_eq!(\n+ next_worker_control_reconnect_backoff(Duration::from_secs(5)),\n+ Duration::from_secs(5)\n+ );\n+ }\n+\n+ #[tokio::test(start_paused = true)]\n+ async fn worker_control_socket_times_out_without_liveness() {\n+ let (worker_io, _server_io) = tokio::io::duplex(1024);\n+ let worker_ws =\n+ tokio_tungstenite::WebSocketStream::from_raw_socket(worker_io, Role::Client, None)\n+ .await;\n+ let mut socket = WorkerControlSocket::Test(Box::new(worker_ws));\n let interviewer = Arc::new(ControlInterviewer::new());\n let cancel_token = CancellationToken::new();\n- let mut question = Question::new(\"Approve?\", QuestionType::YesNo);\n- question.id = \"q-1\".to_string();\n- let ask_interviewer = Arc::clone(&interviewer);\n- let answer_task = tokio::spawn(async move { ask_interviewer.ask(question).await });\n- let (event_tx, event_rx) = tokio::sync::mpsc::unbounded_channel();\n+ let hub = test_steering_hub();\n+ let run_control = RunControlState::new();\n+ let mut applied_ids = AppliedWorkerControlDeliveryIds::default();\n+ let done = CancellationToken::new();\n+\n+ let task = tokio::spawn(async move {\n+ handle_worker_control_socket(\n+ &mut socket,\n+ &interviewer,\n+ &cancel_token,\n+ &hub,\n+ &run_control,\n+ &mut applied_ids,\n+ &done,\n+ )\n+ .await\n+ });\n \n- event_tx.send(WorkerControlStreamEvent::Eof).unwrap();\n- drop(event_tx);\n+ tokio::task::yield_now().await;\n+ time::advance(Duration::from_secs(46)).await;\n+ let result = task.await.unwrap();\n+ assert!(matches!(result, Err(WorkerControlConnectError::Other(_))));\n+ }\n \n- let hub = test_steering_hub();\n- handle_worker_control_stream_events(\n- Arc::clone(&interviewer),\n- cancel_token.clone(),\n- hub,\n- event_rx,\n+ #[cfg(unix)]\n+ #[tokio::test]\n+ async fn worker_control_unix_socket_handshake_completes() {\n+ let temp = tempfile::tempdir().unwrap();\n+ let socket_path = temp.path().join(\"fabro.sock\");\n+ let listener = tokio::net::UnixListener::bind(&socket_path).unwrap();\n+ let server = tokio::spawn(async move {\n+ let (stream, _) = listener.accept().await.unwrap();\n+ let mut socket = tokio_tungstenite::accept_async(stream).await.unwrap();\n+ while let Some(message) = futures::StreamExt::next(&mut socket).await {\n+ match message.unwrap() {\n+ TestWebSocketMessage::Close(_) => break,\n+ TestWebSocketMessage::Ping(payload) => {\n+ futures::SinkExt::send(&mut socket, TestWebSocketMessage::Pong(payload))\n+ .await\n+ .unwrap();\n+ }\n+ _ => {}\n+ }\n+ }\n+ });\n+ let request = build_worker_control_stream_request(\n+ &ServerTarget::unix_socket_path(&socket_path).unwrap(),\n+ &fixtures::RUN_1,\n+ \"worker-token\",\n+ None,\n )\n- .await;\n+ .unwrap();\n \n- let answer = answer_task.await.unwrap().answer;\n- assert_eq!(answer.value, AnswerValue::Interrupted);\n- assert!(!cancel_token.is_cancelled());\n+ let mut socket = connect_worker_control_stream(request).await.unwrap();\n+ socket\n+ .send(TestWebSocketMessage::Close(None))\n+ .await\n+ .unwrap();\n+ server.await.unwrap();\n }\n \n #[tokio::test]\ndiff --git a/lib/crates/fabro-cli/tests/it/cmd/runner.rs b/lib/crates/fabro-cli/tests/it/cmd/runner.rs\nindex d7d87fb93..d5b307d35 100644\n--- a/lib/crates/fabro-cli/tests/it/cmd/runner.rs\n+++ b/lib/crates/fabro-cli/tests/it/cmd/runner.rs\n@@ -794,6 +794,66 @@ fn detached_run_answers_pending_question_without_interview_scratch_files() {\n )));\n }\n \n+#[test]\n+fn detached_run_cancel_reaches_worker_over_control_websocket() {\n+ let context = auth_context();\n+ let run_id = unique_run_id();\n+ let workflow_path = context.temp_dir.join(\"cancel-over-control-websocket.fabro\");\n+ let _gate = write_gated_workflow(\n+ &workflow_path,\n+ \"cancel_over_control_websocket\",\n+ \"Wait for cancellation\",\n+ );\n+\n+ let output = context\n+ .command()\n+ .args([\n+ \"run\",\n+ \"--detach\",\n+ \"--run-id\",\n+ run_id.as_str(),\n+ \"--environment\",\n+ \"local\",\n+ workflow_path.to_str().unwrap(),\n+ ])\n+ .timeout(SHARED_DAEMON_TIMEOUT)\n+ .output()\n+ .expect(\"detached run should execute\");\n+ assert!(\n+ output.status.success(),\n+ \"detached run failed:\\nstdout:\\n{}\\nstderr:\\n{}\",\n+ String::from_utf8_lossy(&output.stdout),\n+ String::from_utf8_lossy(&output.stderr)\n+ );\n+\n+ let run_dir = context.find_run_dir(&run_id);\n+ wait_for_event_names(&run_dir, &[\"run.running\"]);\n+ tokio::runtime::Runtime::new()\n+ .expect(\"test runtime should build\")\n+ .block_on(async {\n+ let (client, base_url) =\n+ server_endpoint(&context.storage_dir).expect(\"server endpoint should exist\");\n+ let response = client\n+ .post(format!(\"{base_url}/api/v1/runs/{run_id}/cancel\"))\n+ .send()\n+ .await\n+ .expect(\"cancel request should succeed\");\n+ assert_reqwest_status(\n+ response,\n+ fabro_http::StatusCode::OK,\n+ format!(\"POST /api/v1/runs/{run_id}/cancel\"),\n+ )\n+ .await;\n+ });\n+\n+ wait_for_status(&run_dir, &[\"failed\"]);\n+ let events = stored_worker_events(&run_dir);\n+ assert!(events.iter().any(|event| matches!(\n+ &event.body,\n+ EventBody::RunFailed(props) if props.failure.reason == FailureReason::Cancelled\n+ )));\n+}\n+\n #[cfg(unix)]\n #[test]\n fn worker_exits_after_sigterm_cancel_even_when_stdin_stays_open() {\ndiff --git a/lib/crates/fabro-interview/src/control_protocol.rs b/lib/crates/fabro-interview/src/control_protocol.rs\nindex cdc820216..f76b61121 100644\n--- a/lib/crates/fabro-interview/src/control_protocol.rs\n+++ b/lib/crates/fabro-interview/src/control_protocol.rs\n@@ -1,3 +1,5 @@\n+use std::time::Duration;\n+\n use fabro_types::{PairId, PairMessageId, PairTarget, Principal, RunId};\n use serde::{Deserialize, Serialize};\n \n@@ -5,6 +7,25 @@ use crate::{Answer, AnswerSubmission, AnswerValue};\n \n pub const WORKER_CONTROL_PROTOCOL_VERSION: u8 = 1;\n \n+/// Interval between worker-control WebSocket ping frames.\n+///\n+/// Server and worker both initiate pings at this cadence; either side that\n+/// fails to observe inbound traffic for [`WORKER_CONTROL_WS_LIVENESS_TIMEOUT`]\n+/// closes the WebSocket.\n+pub const WORKER_CONTROL_WS_PING_INTERVAL: Duration = Duration::from_secs(15);\n+\n+/// Maximum quiet time allowed on a worker-control WebSocket before either side\n+/// declares the connection dead.\n+pub const WORKER_CONTROL_WS_LIVENESS_TIMEOUT: Duration = Duration::from_secs(45);\n+\n+/// WebSocket close-frame reason used when the server can no longer prove\n+/// replay correctness for the requested cursor. Workers must treat this as\n+/// fatal control-channel loss.\n+pub const WORKER_CONTROL_INVALID_CURSOR_REASON: &str = \"invalid_cursor\";\n+\n+/// WebSocket close-frame reason used when the ping/pong watchdog fires.\n+pub const WORKER_CONTROL_PONG_TIMEOUT_REASON: &str = \"pong_timeout\";\n+\n #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]\n pub struct WorkerControlEnvelope {\n pub v: u8,\n@@ -33,6 +54,22 @@ impl WorkerControlEnvelope {\n }\n }\n \n+ #[must_use]\n+ pub fn pause_run() -> Self {\n+ Self {\n+ v: WORKER_CONTROL_PROTOCOL_VERSION,\n+ message: WorkerControlMessage::RunPause,\n+ }\n+ }\n+\n+ #[must_use]\n+ pub fn unpause_run() -> Self {\n+ Self {\n+ v: WORKER_CONTROL_PROTOCOL_VERSION,\n+ message: WorkerControlMessage::RunUnpause,\n+ }\n+ }\n+\n #[must_use]\n pub fn steer(text: impl Into, actor: Principal) -> Self {\n Self {\n@@ -121,6 +158,10 @@ pub enum WorkerControlMessage {\n },\n #[serde(rename = \"run.cancel\")]\n RunCancel,\n+ #[serde(rename = \"run.pause\")]\n+ RunPause,\n+ #[serde(rename = \"run.unpause\")]\n+ RunUnpause,\n #[serde(rename = \"run.steer\")]\n Steer { text: String, actor: Principal },\n #[serde(rename = \"run.interrupt\")]\n@@ -147,6 +188,12 @@ pub enum WorkerControlMessage {\n PairEnd { pair_id: PairId, actor: Principal },\n }\n \n+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]\n+pub struct WorkerControlDeliveryFrame {\n+ pub id: String,\n+ pub envelope: WorkerControlEnvelope,\n+}\n+\n #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]\n #[serde(tag = \"kind\", rename_all = \"snake_case\")]\n pub enum WorkerControlAnswer {\n@@ -229,6 +276,26 @@ mod tests {\n assert_eq!(parsed, envelope);\n }\n \n+ #[test]\n+ fn pause_run_round_trips_through_json() {\n+ let envelope = WorkerControlEnvelope::pause_run();\n+ let json = serde_json::to_string(&envelope).unwrap();\n+ assert_eq!(json, r#\"{\"v\":1,\"type\":\"run.pause\"}\"#);\n+\n+ let parsed: WorkerControlEnvelope = serde_json::from_str(&json).unwrap();\n+ assert_eq!(parsed, envelope);\n+ }\n+\n+ #[test]\n+ fn unpause_run_round_trips_through_json() {\n+ let envelope = WorkerControlEnvelope::unpause_run();\n+ let json = serde_json::to_string(&envelope).unwrap();\n+ assert_eq!(json, r#\"{\"v\":1,\"type\":\"run.unpause\"}\"#);\n+\n+ let parsed: WorkerControlEnvelope = serde_json::from_str(&json).unwrap();\n+ assert_eq!(parsed, envelope);\n+ }\n+\n #[test]\n fn steer_append_round_trips_through_json() {\n let envelope = WorkerControlEnvelope::steer(\"try again\", Principal::System {\n@@ -326,4 +393,20 @@ mod tests {\n serde_json::from_str(&serde_json::to_string(&end).unwrap()).unwrap();\n assert_eq!(parsed, end);\n }\n+\n+ #[test]\n+ fn delivery_frame_round_trips_through_json() {\n+ let frame = WorkerControlDeliveryFrame {\n+ id: \"local:42\".to_string(),\n+ envelope: WorkerControlEnvelope::cancel_run(),\n+ };\n+ let json = serde_json::to_string(&frame).unwrap();\n+ assert_eq!(\n+ json,\n+ r#\"{\"id\":\"local:42\",\"envelope\":{\"v\":1,\"type\":\"run.cancel\"}}\"#\n+ );\n+\n+ let parsed: WorkerControlDeliveryFrame = serde_json::from_str(&json).unwrap();\n+ assert_eq!(parsed, frame);\n+ }\n }\ndiff --git a/lib/crates/fabro-interview/src/lib.rs b/lib/crates/fabro-interview/src/lib.rs\nindex 4c050c4f7..6d8c414a1 100644\n--- a/lib/crates/fabro-interview/src/lib.rs\n+++ b/lib/crates/fabro-interview/src/lib.rs\n@@ -222,8 +222,10 @@ pub use callback::CallbackInterviewer;\n pub use console::ConsoleInterviewer;\n pub use control::{ControlInterviewer, SubmitError};\n pub use control_protocol::{\n- WORKER_CONTROL_PROTOCOL_VERSION, WorkerControlAnswer, WorkerControlEnvelope,\n- WorkerControlMessage,\n+ WORKER_CONTROL_INVALID_CURSOR_REASON, WORKER_CONTROL_PONG_TIMEOUT_REASON,\n+ WORKER_CONTROL_PROTOCOL_VERSION, WORKER_CONTROL_WS_LIVENESS_TIMEOUT,\n+ WORKER_CONTROL_WS_PING_INTERVAL, WorkerControlAnswer, WorkerControlDeliveryFrame,\n+ WorkerControlEnvelope, WorkerControlMessage,\n };\n pub use queue::QueueInterviewer;\n pub use recording::RecordingInterviewer;\ndiff --git a/lib/crates/fabro-server/Cargo.toml b/lib/crates/fabro-server/Cargo.toml\nindex c36eee94a..dfde73f66 100644\n--- a/lib/crates/fabro-server/Cargo.toml\n+++ b/lib/crates/fabro-server/Cargo.toml\n@@ -69,6 +69,7 @@ serde.workspace = true\n serde_json.workspace = true\n serde_yaml = \"0.9\"\n anyhow.workspace = true\n+async-trait.workspace = true\n clap.workspace = true\n toml.workspace = true\n toml_edit.workspace = true\n@@ -109,8 +110,8 @@ http-body-util = \"0.1\"\n httpmock = \"0.8\"\n serde_yaml = \"0.9\"\n tracing-subscriber.workspace = true\n-async-trait.workspace = true\n tokio-util.workspace = true\n+tokio-tungstenite.workspace = true\n fabro-macros = { path = \"../fabro-macros\" }\n fabro-sandbox = { path = \"../fabro-sandbox\", features = [\"test-support\"] }\n fabro-test = { workspace = true }\ndiff --git a/lib/crates/fabro-server/src/automation_materializer.rs b/lib/crates/fabro-server/src/automation_materializer.rs\nnew file mode 100644\nindex 000000000..74c9c2677\n--- /dev/null\n+++ b/lib/crates/fabro-server/src/automation_materializer.rs\n@@ -0,0 +1,797 @@\n+use std::path::{Path, PathBuf};\n+use std::time::Duration;\n+\n+use anyhow::Context as _;\n+use async_trait::async_trait;\n+use base64::Engine as _;\n+use base64::engine::general_purpose::STANDARD as BASE64_STANDARD;\n+use fabro_api::types::RunManifest;\n+use fabro_automation::{AutomationId, AutomationTarget};\n+use fabro_manifest::ManifestBuildInput;\n+use fabro_types::{DirtyStatus, GitContext, PreRunPushOutcome, RunId};\n+use fabro_util::error::collect_chain;\n+use tokio::process::Command;\n+use tokio::{fs, task, time};\n+\n+const GIT_CLONE_TIMEOUT: Duration = Duration::from_mins(2);\n+const GIT_FETCH_TIMEOUT: Duration = Duration::from_mins(1);\n+const GIT_CHECKOUT_TIMEOUT: Duration = Duration::from_secs(30);\n+const GIT_REV_PARSE_TIMEOUT: Duration = Duration::from_secs(10);\n+\n+#[derive(Debug, Clone, PartialEq, Eq)]\n+pub(crate) struct AutomationRunMaterializeInput {\n+ pub automation_id: AutomationId,\n+ pub target: AutomationTarget,\n+ pub run_id: RunId,\n+ pub user_settings_path: PathBuf,\n+ pub temp_root: PathBuf,\n+}\n+\n+#[derive(Debug, Clone)]\n+pub(crate) struct AutomationRunMaterialized {\n+ pub manifest: RunManifest,\n+ pub submitted_manifest_bytes: Vec,\n+}\n+\n+#[derive(thiserror::Error, Debug, Clone, PartialEq, Eq)]\n+pub(crate) enum AutomationRunMaterializeError {\n+ #[error(\"invalid automation target: {0}\")]\n+ InvalidTarget(String),\n+ #[error(\"failed to clone automation repository: {0}\")]\n+ CloneFailed(String),\n+ #[error(\"failed to resolve automation workflow: {0}\")]\n+ WorkflowNotFound(String),\n+ #[error(\"failed to build run manifest: {0}\")]\n+ Manifest(String),\n+}\n+\n+#[async_trait]\n+pub(crate) trait AutomationRunMaterializer: Send + Sync {\n+ async fn materialize(\n+ &self,\n+ input: AutomationRunMaterializeInput,\n+ ) -> Result;\n+}\n+\n+#[derive(Clone)]\n+pub(crate) struct ProductionAutomationRunMaterializer {\n+ github_credentials: Option,\n+ github_api_base_url: String,\n+ http_client: Option,\n+}\n+\n+impl ProductionAutomationRunMaterializer {\n+ pub(crate) fn new(\n+ github_credentials: Option,\n+ github_api_base_url: String,\n+ http_client: Option,\n+ ) -> Self {\n+ Self {\n+ github_credentials,\n+ github_api_base_url,\n+ http_client,\n+ }\n+ }\n+}\n+\n+#[async_trait]\n+impl AutomationRunMaterializer for ProductionAutomationRunMaterializer {\n+ async fn materialize(\n+ &self,\n+ input: AutomationRunMaterializeInput,\n+ ) -> Result {\n+ let repo = parse_github_repository_slug(&input.target.repository)?;\n+ fs::create_dir_all(&input.temp_root).await.map_err(|err| {\n+ AutomationRunMaterializeError::CloneFailed(format!(\n+ \"failed to create temp root {}: {err}\",\n+ input.temp_root.display()\n+ ))\n+ })?;\n+ let temp_dir = tempfile::Builder::new()\n+ .prefix(&format!(\n+ \"automation-{}-{}-\",\n+ input.automation_id.as_str(),\n+ input.run_id\n+ ))\n+ .tempdir_in(&input.temp_root)\n+ .map_err(|err| {\n+ AutomationRunMaterializeError::CloneFailed(format!(\n+ \"failed to create per-run temp directory under {}: {err}\",\n+ input.temp_root.display()\n+ ))\n+ })?;\n+ let checkout_dir = temp_dir.path().join(\"repo\");\n+ let clone_url = github_clone_url(&repo);\n+ let auth = resolve_git_auth_config(\n+ self.github_credentials.as_ref(),\n+ &repo,\n+ &self.github_api_base_url,\n+ self.http_client.clone(),\n+ )\n+ .await\n+ .map_err(|err| {\n+ AutomationRunMaterializeError::CloneFailed(render_error_chain(err.as_ref()))\n+ })?;\n+\n+ run_git_plan(build_clone_plan(&clone_url, &checkout_dir, auth.as_ref())).await?;\n+ run_git_plan(build_fetch_ref_plan(\n+ &clone_url,\n+ &checkout_dir,\n+ &input.target.ref_selector,\n+ auth.as_ref(),\n+ ))\n+ .await?;\n+ run_git_plan(build_checkout_ref_plan(&checkout_dir)).await?;\n+ let checked_out_sha = run_git_plan(build_rev_parse_head_plan(&checkout_dir))\n+ .await\n+ .map(|stdout| String::from_utf8_lossy(&stdout).trim().to_string())?;\n+\n+ let manifest_input = ManifestFromCheckoutInput {\n+ input,\n+ checkout_dir,\n+ repo,\n+ checked_out_sha: Some(checked_out_sha),\n+ };\n+ task::spawn_blocking(move || build_manifest_from_checkout(manifest_input))\n+ .await\n+ .map_err(|err| {\n+ AutomationRunMaterializeError::Manifest(format!(\n+ \"manifest build task failed: {err}\"\n+ ))\n+ })?\n+ }\n+}\n+\n+#[derive(Debug, Clone, PartialEq, Eq)]\n+pub(crate) struct GithubRepository {\n+ owner: String,\n+ name: String,\n+}\n+\n+fn parse_github_repository_slug(\n+ value: &str,\n+) -> Result {\n+ let Some((owner, repo)) = value.split_once('/') else {\n+ return Err(AutomationRunMaterializeError::InvalidTarget(format!(\n+ \"repository must be a GitHub owner/repo slug: {value}\"\n+ )));\n+ };\n+ if repo.contains('/') || !valid_github_owner(owner) || !valid_github_repo(repo) {\n+ return Err(AutomationRunMaterializeError::InvalidTarget(format!(\n+ \"repository must be a GitHub owner/repo slug: {value}\"\n+ )));\n+ }\n+ Ok(GithubRepository {\n+ owner: owner.to_string(),\n+ name: repo.to_string(),\n+ })\n+}\n+\n+fn valid_github_owner(value: &str) -> bool {\n+ if value.is_empty() || value.len() > 39 {\n+ return false;\n+ }\n+ let bytes = value.as_bytes();\n+ let first = bytes[0];\n+ let last = bytes[bytes.len() - 1];\n+ (first.is_ascii_alphanumeric() && last.is_ascii_alphanumeric())\n+ && bytes\n+ .iter()\n+ .all(|byte| byte.is_ascii_alphanumeric() || *byte == b'-')\n+}\n+\n+fn valid_github_repo(value: &str) -> bool {\n+ !value.is_empty()\n+ && value.len() <= 100\n+ && value != \".\"\n+ && value != \"..\"\n+ && !value.starts_with('.')\n+ && value\n+ .bytes()\n+ .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'_' | b'-'))\n+}\n+\n+fn github_clone_url(repo: &GithubRepository) -> String {\n+ format!(\"https://github.com/{}/{}.git\", repo.owner, repo.name)\n+}\n+\n+fn github_metadata_url(repo: &GithubRepository) -> String {\n+ format!(\"https://github.com/{}/{}\", repo.owner, repo.name)\n+}\n+\n+#[derive(Clone, Debug, PartialEq, Eq)]\n+pub(crate) struct GitAuthConfig {\n+ extraheader: Option,\n+ sensitive_values: Vec,\n+}\n+\n+impl GitAuthConfig {\n+ fn new(username: Option, password: Option) -> Self {\n+ let Some(password) = password.filter(|value| !value.is_empty()) else {\n+ return Self {\n+ extraheader: None,\n+ sensitive_values: Vec::new(),\n+ };\n+ };\n+ let username = username\n+ .filter(|value| !value.is_empty())\n+ .unwrap_or_else(|| \"x-access-token\".to_string());\n+ let encoded_credentials = BASE64_STANDARD.encode(format!(\"{username}:{password}\"));\n+ let extraheader = basic_auth_header_from_encoded(&encoded_credentials);\n+ Self {\n+ sensitive_values: vec![password, encoded_credentials, extraheader.clone()],\n+ extraheader: Some(extraheader),\n+ }\n+ }\n+\n+ fn git_env(&self, clone_url: &str) -> Vec<(String, String)> {\n+ let Some(extraheader) = self.extraheader.as_ref() else {\n+ return Vec::new();\n+ };\n+ vec![\n+ (\"GIT_CONFIG_COUNT\".to_string(), \"1\".to_string()),\n+ (\n+ \"GIT_CONFIG_KEY_0\".to_string(),\n+ format!(\"http.{clone_url}.extraheader\"),\n+ ),\n+ (\"GIT_CONFIG_VALUE_0\".to_string(), extraheader.clone()),\n+ ]\n+ }\n+\n+ fn sensitive_values(&self) -> &[String] {\n+ &self.sensitive_values\n+ }\n+}\n+\n+async fn resolve_git_auth_config(\n+ credentials: Option<&fabro_github::GitHubCredentials>,\n+ repo: &GithubRepository,\n+ github_api_base_url: &str,\n+ http_client: Option,\n+) -> anyhow::Result> {\n+ let Some(credentials) = credentials else {\n+ return Ok(None);\n+ };\n+ let context = match http_client {\n+ Some(client) => {\n+ fabro_github::GitHubContext::with_http_client(credentials, github_api_base_url, client)\n+ }\n+ None => fabro_github::GitHubContext::new(credentials, github_api_base_url),\n+ };\n+ let (username, password) =\n+ fabro_github::resolve_clone_credentials(&context, &repo.owner, &repo.name).await?;\n+ Ok(Some(GitAuthConfig::new(username, password)))\n+}\n+\n+fn basic_auth_header(username: &str, password: &str) -> String {\n+ basic_auth_header_from_encoded(&BASE64_STANDARD.encode(format!(\"{username}:{password}\")))\n+}\n+\n+fn basic_auth_header_from_encoded(encoded_credentials: &str) -> String {\n+ format!(\"AUTHORIZATION: basic {encoded_credentials}\")\n+}\n+\n+#[derive(Debug, Clone, PartialEq, Eq)]\n+pub(crate) struct GitCommandPlan {\n+ program: String,\n+ args: Vec,\n+ env: Vec<(String, String)>,\n+ current_dir: Option,\n+ timeout: Duration,\n+ sensitive_values: Vec,\n+}\n+\n+impl GitCommandPlan {\n+ fn new(args: impl IntoIterator>, timeout: Duration) -> Self {\n+ Self {\n+ program: \"git\".to_string(),\n+ args: args.into_iter().map(Into::into).collect(),\n+ env: vec![(\"GIT_TERMINAL_PROMPT\".to_string(), \"0\".to_string())],\n+ current_dir: None,\n+ timeout,\n+ sensitive_values: Vec::new(),\n+ }\n+ }\n+\n+ fn current_dir(mut self, current_dir: impl Into) -> Self {\n+ self.current_dir = Some(current_dir.into());\n+ self\n+ }\n+\n+ fn with_auth(mut self, clone_url: &str, auth: Option<&GitAuthConfig>) -> Self {\n+ if let Some(auth) = auth {\n+ self.env.extend(auth.git_env(clone_url));\n+ self.sensitive_values\n+ .extend(auth.sensitive_values().iter().cloned());\n+ }\n+ self\n+ }\n+\n+ fn env_value(&self, name: &str) -> Option<&str> {\n+ self.env\n+ .iter()\n+ .find(|(key, _)| key == name)\n+ .map(|(_, value)| value.as_str())\n+ }\n+}\n+\n+fn build_clone_plan(\n+ clone_url: &str,\n+ checkout_dir: &Path,\n+ auth: Option<&GitAuthConfig>,\n+) -> GitCommandPlan {\n+ GitCommandPlan::new(\n+ [\n+ \"clone\".to_string(),\n+ \"--depth\".to_string(),\n+ \"1\".to_string(),\n+ \"--no-checkout\".to_string(),\n+ clone_url.to_string(),\n+ checkout_dir.display().to_string(),\n+ ],\n+ GIT_CLONE_TIMEOUT,\n+ )\n+ .with_auth(clone_url, auth)\n+}\n+\n+fn build_fetch_ref_plan(\n+ clone_url: &str,\n+ checkout_dir: &Path,\n+ ref_selector: &str,\n+ auth: Option<&GitAuthConfig>,\n+) -> GitCommandPlan {\n+ GitCommandPlan::new(\n+ [\n+ \"fetch\".to_string(),\n+ \"--depth\".to_string(),\n+ \"1\".to_string(),\n+ \"origin\".to_string(),\n+ \"--\".to_string(),\n+ ref_selector.to_string(),\n+ ],\n+ GIT_FETCH_TIMEOUT,\n+ )\n+ .current_dir(checkout_dir)\n+ .with_auth(clone_url, auth)\n+}\n+\n+fn build_checkout_ref_plan(checkout_dir: &Path) -> GitCommandPlan {\n+ GitCommandPlan::new(\n+ [\"checkout\", \"--force\", \"--detach\", \"FETCH_HEAD\"],\n+ GIT_CHECKOUT_TIMEOUT,\n+ )\n+ .current_dir(checkout_dir)\n+}\n+\n+fn build_rev_parse_head_plan(checkout_dir: &Path) -> GitCommandPlan {\n+ GitCommandPlan::new([\"rev-parse\", \"HEAD\"], GIT_REV_PARSE_TIMEOUT).current_dir(checkout_dir)\n+}\n+\n+async fn run_git_plan(plan: GitCommandPlan) -> Result, AutomationRunMaterializeError> {\n+ let mut command = Command::new(&plan.program);\n+ command.args(&plan.args);\n+ command.envs(plan.env.iter().map(|(key, value)| (key, value)));\n+ if let Some(current_dir) = plan.current_dir.as_ref() {\n+ command.current_dir(current_dir);\n+ }\n+ command.kill_on_drop(true);\n+\n+ let output = time::timeout(plan.timeout, command.output())\n+ .await\n+ .map_err(|_| {\n+ AutomationRunMaterializeError::CloneFailed(format!(\n+ \"{} timed out after {}s\",\n+ safe_command_label(&plan),\n+ plan.timeout.as_secs()\n+ ))\n+ })?\n+ .map_err(|err| {\n+ AutomationRunMaterializeError::CloneFailed(format!(\n+ \"failed to run {}: {err}\",\n+ safe_command_label(&plan)\n+ ))\n+ })?;\n+\n+ if output.status.success() {\n+ return Ok(output.stdout);\n+ }\n+\n+ let stdout = String::from_utf8_lossy(&output.stdout);\n+ let stderr = String::from_utf8_lossy(&output.stderr);\n+ let mut message = format!(\n+ \"{} exited with status {}\",\n+ safe_command_label(&plan),\n+ output.status\n+ );\n+ if !stderr.trim().is_empty() {\n+ message.push_str(\": \");\n+ message.push_str(stderr.trim());\n+ } else if !stdout.trim().is_empty() {\n+ message.push_str(\": \");\n+ message.push_str(stdout.trim());\n+ }\n+ Err(AutomationRunMaterializeError::CloneFailed(\n+ redact_git_output(&message, &plan.sensitive_values),\n+ ))\n+}\n+\n+fn safe_command_label(plan: &GitCommandPlan) -> String {\n+ if plan.args.is_empty() {\n+ plan.program.clone()\n+ } else {\n+ format!(\"{} {}\", plan.program, plan.args.join(\" \"))\n+ }\n+}\n+\n+fn redact_git_output(text: &str, sensitive_values: &[String]) -> String {\n+ let mut redacted = fabro_redact::redact_string(text);\n+ for value in sensitive_values\n+ .iter()\n+ .map(String::as_str)\n+ .filter(|value| !value.is_empty())\n+ {\n+ redacted = redacted.replace(value, \"REDACTED\");\n+ }\n+ redacted\n+}\n+\n+fn render_error_chain(error: &(dyn std::error::Error + 'static)) -> String {\n+ collect_chain(error).join(\": \")\n+}\n+\n+#[derive(Debug)]\n+pub(crate) struct ManifestFromCheckoutInput {\n+ input: AutomationRunMaterializeInput,\n+ checkout_dir: PathBuf,\n+ repo: GithubRepository,\n+ checked_out_sha: Option,\n+}\n+\n+fn build_manifest_from_checkout(\n+ args: ManifestFromCheckoutInput,\n+) -> Result {\n+ let ManifestFromCheckoutInput {\n+ input,\n+ checkout_dir,\n+ repo,\n+ checked_out_sha,\n+ } = args;\n+ let built = fabro_manifest::build_run_manifest(ManifestBuildInput {\n+ workflow: input.target.workflow.as_str().into(),\n+ cwd: checkout_dir,\n+ run_id: Some(input.run_id),\n+ user_settings_path: Some(input.user_settings_path),\n+ ..ManifestBuildInput::default()\n+ })\n+ .map_err(|err| manifest_build_error(&err))?;\n+\n+ let mut manifest = built.manifest;\n+ manifest.git = Some(GitContext {\n+ origin_url: github_metadata_url(&repo),\n+ branch: input.target.ref_selector,\n+ sha: checked_out_sha,\n+ dirty: DirtyStatus::Clean,\n+ push_outcome: PreRunPushOutcome::NotAttempted,\n+ });\n+ let submitted_manifest_bytes = serde_json::to_vec(&manifest)\n+ .with_context(|| {\n+ format!(\n+ \"failed to serialize materialized manifest for automation {}\",\n+ input.automation_id.as_str()\n+ )\n+ })\n+ .map_err(|err| AutomationRunMaterializeError::Manifest(err.to_string()))?;\n+ Ok(AutomationRunMaterialized {\n+ manifest,\n+ submitted_manifest_bytes,\n+ })\n+}\n+\n+fn manifest_build_error(error: &anyhow::Error) -> AutomationRunMaterializeError {\n+ if error.chain().any(|source| {\n+ source\n+ .downcast_ref::()\n+ .is_some_and(|err| matches!(err, fabro_config::Error::WorkflowNotFound(_)))\n+ }) {\n+ AutomationRunMaterializeError::WorkflowNotFound(render_error_chain(error.as_ref()))\n+ } else {\n+ AutomationRunMaterializeError::Manifest(render_error_chain(error.as_ref()))\n+ }\n+}\n+\n+#[cfg(any(test, feature = \"test-support\"))]\n+#[derive(Clone)]\n+pub struct TestAutomationRunMaterializer {\n+ inner: std::sync::Arc>,\n+}\n+\n+#[cfg(any(test, feature = \"test-support\"))]\n+struct TestAutomationRunMaterializerState {\n+ captured_inputs: Vec,\n+ response: Result,\n+}\n+\n+#[cfg(any(test, feature = \"test-support\"))]\n+impl TestAutomationRunMaterializer {\n+ pub fn succeed(manifest: RunManifest, submitted_manifest_bytes: Vec) -> Self {\n+ Self::new(Ok(AutomationRunMaterialized {\n+ manifest,\n+ submitted_manifest_bytes,\n+ }))\n+ }\n+\n+ pub fn fail_invalid_target(message: impl Into) -> Self {\n+ Self::new(Err(AutomationRunMaterializeError::InvalidTarget(\n+ message.into(),\n+ )))\n+ }\n+\n+ fn new(response: Result) -> Self {\n+ Self {\n+ inner: std::sync::Arc::new(std::sync::Mutex::new(TestAutomationRunMaterializerState {\n+ captured_inputs: Vec::new(),\n+ response,\n+ })),\n+ }\n+ }\n+\n+ pub(crate) fn captured_inputs(&self) -> Vec {\n+ self.inner\n+ .lock()\n+ .expect(\"test automation materializer lock poisoned\")\n+ .captured_inputs\n+ .clone()\n+ }\n+\n+ pub(crate) fn into_materializer(self) -> std::sync::Arc {\n+ std::sync::Arc::new(self)\n+ }\n+}\n+\n+#[cfg(any(test, feature = \"test-support\"))]\n+#[async_trait]\n+impl AutomationRunMaterializer for TestAutomationRunMaterializer {\n+ async fn materialize(\n+ &self,\n+ input: AutomationRunMaterializeInput,\n+ ) -> Result {\n+ let mut guard = self\n+ .inner\n+ .lock()\n+ .expect(\"test automation materializer lock poisoned\");\n+ guard.captured_inputs.push(input);\n+ guard.response.clone()\n+ }\n+}\n+\n+#[cfg(test)]\n+mod tests {\n+ #![expect(\n+ clippy::disallowed_methods,\n+ reason = \"Materializer unit tests write small temporary workflow fixtures synchronously.\"\n+ )]\n+\n+ use std::fs;\n+ use std::path::Path;\n+\n+ use fabro_automation::{AutomationId, AutomationTarget};\n+ use fabro_types::{DirtyStatus, PreRunPushOutcome, RunId};\n+ use tempfile::TempDir;\n+\n+ use super::*;\n+\n+ fn target(repository: &str, ref_selector: &str, workflow: &str) -> AutomationTarget {\n+ AutomationTarget {\n+ repository: repository.to_string(),\n+ ref_selector: ref_selector.to_string(),\n+ workflow: workflow.to_string(),\n+ }\n+ }\n+\n+ #[test]\n+ fn target_repository_urls_are_github_metadata_urls_without_credentials() {\n+ let repo = parse_github_repository_slug(\"fabro-sh/fabro\").expect(\"slug should parse\");\n+\n+ assert_eq!(repo.owner, \"fabro-sh\");\n+ assert_eq!(repo.name, \"fabro\");\n+ assert_eq!(\n+ github_clone_url(&repo),\n+ \"https://github.com/fabro-sh/fabro.git\"\n+ );\n+ assert_eq!(\n+ github_metadata_url(&repo),\n+ \"https://github.com/fabro-sh/fabro\"\n+ );\n+ assert!(!github_clone_url(&repo).contains('@'));\n+ }\n+\n+ #[test]\n+ fn target_repository_validation_rejects_non_github_owner_repo_shapes() {\n+ for value in [\n+ \"fabro-sh\",\n+ \"https://github.com/fabro-sh/fabro\",\n+ \"fabro-sh/fabro/extra\",\n+ \"-owner/repo\",\n+ \"owner/.git\",\n+ ] {\n+ let error = parse_github_repository_slug(value).expect_err(\"invalid slug should fail\");\n+ assert!(\n+ error.to_string().contains(\"invalid automation target\"),\n+ \"unexpected error for {value}: {error}\"\n+ );\n+ }\n+ }\n+\n+ #[test]\n+ fn ref_checkout_command_plans_use_argv_prompt_disable_and_timeouts() {\n+ let repo = parse_github_repository_slug(\"fabro-sh/fabro\").unwrap();\n+ let clone_url = github_clone_url(&repo);\n+ let temp = TempDir::new().unwrap();\n+ let checkout_dir = temp.path().join(\"repo\");\n+\n+ let clone = build_clone_plan(&clone_url, &checkout_dir, None);\n+ assert_eq!(clone.program, \"git\");\n+ assert_eq!(clone.args, vec![\n+ \"clone\",\n+ \"--depth\",\n+ \"1\",\n+ \"--no-checkout\",\n+ \"https://github.com/fabro-sh/fabro.git\",\n+ checkout_dir.to_str().unwrap(),\n+ ]);\n+ assert_eq!(clone.timeout, Duration::from_mins(2));\n+ assert_eq!(clone.env_value(\"GIT_TERMINAL_PROMPT\"), Some(\"0\"));\n+\n+ let fetch = build_fetch_ref_plan(&clone_url, &checkout_dir, \"feature/materialize\", None);\n+ assert_eq!(fetch.args, vec![\n+ \"fetch\",\n+ \"--depth\",\n+ \"1\",\n+ \"origin\",\n+ \"--\",\n+ \"feature/materialize\",\n+ ]);\n+ assert_eq!(fetch.current_dir.as_deref(), Some(checkout_dir.as_path()));\n+ assert_eq!(fetch.timeout, Duration::from_mins(1));\n+ assert_eq!(fetch.env_value(\"GIT_TERMINAL_PROMPT\"), Some(\"0\"));\n+\n+ let checkout = build_checkout_ref_plan(&checkout_dir);\n+ assert_eq!(checkout.args, vec![\n+ \"checkout\",\n+ \"--force\",\n+ \"--detach\",\n+ \"FETCH_HEAD\"\n+ ]);\n+ assert_eq!(\n+ checkout.current_dir.as_deref(),\n+ Some(checkout_dir.as_path())\n+ );\n+ assert_eq!(checkout.timeout, Duration::from_secs(30));\n+ assert_eq!(checkout.env_value(\"GIT_TERMINAL_PROMPT\"), Some(\"0\"));\n+ }\n+\n+ #[test]\n+ fn credential_redaction_removes_tokens_and_basic_auth_headers() {\n+ let secret = \"ghu_materializer_secret\";\n+ let basic = basic_auth_header(\"x-access-token\", secret);\n+ let message = format!(\n+ \"fatal: could not read Username for https://github.com/fabro-sh/fabro.git; token={secret}; header={basic}\"\n+ );\n+\n+ let redacted = redact_git_output(&message, &[secret.to_string(), basic.clone()]);\n+\n+ assert!(!redacted.contains(secret), \"token leaked: {redacted}\");\n+ assert!(\n+ !redacted.contains(&basic),\n+ \"basic header leaked: {redacted}\"\n+ );\n+ let encoded_secret = BASE64_STANDARD.encode(format!(\"x-access-token:{secret}\"));\n+ assert!(\n+ !redacted.contains(&encoded_secret),\n+ \"encoded credential leaked: {redacted}\"\n+ );\n+ assert!(\n+ redacted.contains(\"REDACTED\"),\n+ \"expected redaction marker: {redacted}\"\n+ );\n+ }\n+\n+ #[test]\n+ fn credential_config_env_keeps_clone_url_uncredentialed() {\n+ let repo = parse_github_repository_slug(\"fabro-sh/fabro\").unwrap();\n+ let clone_url = github_clone_url(&repo);\n+ let auth = GitAuthConfig::new(\n+ Some(\"x-access-token\".to_string()),\n+ Some(\"ghu_secret\".to_string()),\n+ );\n+ let plan = build_clone_plan(&clone_url, Path::new(\"/tmp/fabro-checkout\"), Some(&auth));\n+\n+ assert!(\n+ plan.args\n+ .iter()\n+ .any(|arg| arg == \"https://github.com/fabro-sh/fabro.git\")\n+ );\n+ assert!(plan.args.iter().all(|arg| !arg.contains(\"ghu_secret\")));\n+ assert_eq!(plan.env_value(\"GIT_CONFIG_COUNT\"), Some(\"1\"));\n+ assert_eq!(\n+ plan.env_value(\"GIT_CONFIG_KEY_0\"),\n+ Some(\"http.https://github.com/fabro-sh/fabro.git.extraheader\")\n+ );\n+ assert!(\n+ plan.env_value(\"GIT_CONFIG_VALUE_0\")\n+ .is_some_and(|value| value.starts_with(\"AUTHORIZATION: basic \"))\n+ );\n+ }\n+\n+ #[test]\n+ fn workflow_path_resolution_builds_manifest_from_checkout_directory() {\n+ let temp = TempDir::new().unwrap();\n+ let checkout = temp.path().join(\"checkout\");\n+ let workflow_dir = checkout.join(\".fabro/workflows/demo\");\n+ fs::create_dir_all(&workflow_dir).unwrap();\n+ fs::write(checkout.join(\".fabro/project.toml\"), \"_version = 1\\n\").unwrap();\n+ fs::write(\n+ workflow_dir.join(\"workflow.fabro\"),\n+ r#\"digraph Demo { graph [goal=\"Ship automation\"] start [shape=Mdiamond] exit [shape=Msquare] start -> exit }\"#,\n+ )\n+ .unwrap();\n+ fs::write(\n+ workflow_dir.join(\"workflow.toml\"),\n+ \"_version = 1\\n[workflow]\\ngraph = \\\"workflow.fabro\\\"\\n\",\n+ )\n+ .unwrap();\n+ let user_settings_path = temp.path().join(\"settings.toml\");\n+ fs::write(&user_settings_path, \"_version = 1\\n\").unwrap();\n+ let run_id = RunId::new();\n+ let repo = parse_github_repository_slug(\"fabro-sh/fabro\").unwrap();\n+ let sha = \"0123456789abcdef0123456789abcdef01234567\".to_string();\n+\n+ let materialized = build_manifest_from_checkout(ManifestFromCheckoutInput {\n+ input: AutomationRunMaterializeInput {\n+ automation_id: AutomationId::new(\"nightly\").unwrap(),\n+ target: target(\"fabro-sh/fabro\", \"main\", \"demo\"),\n+ run_id,\n+ user_settings_path: user_settings_path.clone(),\n+ temp_root: temp.path().to_path_buf(),\n+ },\n+ checkout_dir: checkout.clone(),\n+ repo,\n+ checked_out_sha: Some(sha.clone()),\n+ })\n+ .expect(\"manifest should build from checkout\");\n+\n+ assert_eq!(\n+ materialized.manifest.run_id.as_deref(),\n+ Some(run_id.to_string().as_str())\n+ );\n+ assert_eq!(materialized.manifest.cwd, checkout.display().to_string());\n+ assert_eq!(\n+ materialized.manifest.target.path,\n+ \".fabro/workflows/demo/workflow.fabro\"\n+ );\n+ assert!(\n+ materialized\n+ .manifest\n+ .configs\n+ .iter()\n+ .any(|config| config.path.as_deref() == Some(user_settings_path.to_str().unwrap()))\n+ );\n+ let git = materialized\n+ .manifest\n+ .git\n+ .as_ref()\n+ .expect(\"git context should be set\");\n+ assert_eq!(git.origin_url, \"https://github.com/fabro-sh/fabro\");\n+ assert_eq!(git.branch, \"main\");\n+ assert_eq!(git.sha.as_deref(), Some(sha.as_str()));\n+ assert_eq!(git.dirty, DirtyStatus::Clean);\n+ assert_eq!(git.push_outcome, PreRunPushOutcome::NotAttempted);\n+ let submitted_manifest: serde_json::Value =\n+ serde_json::from_slice(&materialized.submitted_manifest_bytes)\n+ .expect(\"submitted bytes should be a manifest\");\n+ assert_eq!(\n+ submitted_manifest,\n+ serde_json::to_value(&materialized.manifest).unwrap()\n+ );\n+ }\n+}\ndiff --git a/lib/crates/fabro-server/src/lib.rs b/lib/crates/fabro-server/src/lib.rs\nindex 443300c09..76805781c 100644\n--- a/lib/crates/fabro-server/src/lib.rs\n+++ b/lib/crates/fabro-server/src/lib.rs\n@@ -9,6 +9,11 @@\n )]\n \n pub mod auth;\n+#[allow(\n+ dead_code,\n+ reason = \"Automation scheduler wiring will call the materializer; issue #398 adds the shared core first.\"\n+)]\n+mod automation_materializer;\n mod canonical_host;\n mod canonical_origin;\n pub mod csp;\n@@ -43,6 +48,7 @@ pub mod static_files;\n #[cfg(any(test, feature = \"test-support\"))]\n pub mod test_support;\n pub mod web_auth;\n+mod worker_control;\n mod worker_token;\n \n pub use error::{ApiError, Error, Result};\ndiff --git a/lib/crates/fabro-server/src/principal_middleware.rs b/lib/crates/fabro-server/src/principal_middleware.rs\nindex acc9eb250..1b037a6ab 100644\n--- a/lib/crates/fabro-server/src/principal_middleware.rs\n+++ b/lib/crates/fabro-server/src/principal_middleware.rs\n@@ -58,6 +58,7 @@ pub(crate) struct RequestAuth(pub(crate) AuthContextSlot);\n pub(crate) struct RequiredUser(pub(crate) UserPrincipal);\n pub(crate) struct RequiredRunManagementActor(pub(crate) Principal);\n pub(crate) struct RequireRunScoped(pub(crate) RunId);\n+pub(crate) struct RequireWorkerRunScoped(pub(crate) RunId);\n pub(crate) struct RequireRunManagementTarget(pub(crate) RunId, pub(crate) Principal);\n pub(crate) struct RequireRunBlob(pub(crate) RunId, pub(crate) RunBlobId);\n pub(crate) struct RequireRunStageScoped(pub(crate) RunId, pub(crate) String);\n@@ -245,6 +246,23 @@ impl FromRequestParts> for RequireRunScoped {\n }\n }\n \n+impl FromRequestParts> for RequireWorkerRunScoped {\n+ type Rejection = Response;\n+\n+ async fn from_request_parts(\n+ parts: &mut Parts,\n+ state: &Arc,\n+ ) -> Result {\n+ let Path(id): Path = Path::from_request_parts(parts, state)\n+ .await\n+ .map_err(IntoResponse::into_response)?;\n+ let run_id = parse_run_id_path(&id)?;\n+ require_worker_for_run(&auth_slot_from_parts(parts), &run_id)\n+ .map_err(IntoResponse::into_response)?;\n+ Ok(Self(run_id))\n+ }\n+}\n+\n impl FromRequestParts> for RequireRunManagementTarget {\n type Rejection = Response;\n \n@@ -418,6 +436,15 @@ fn require_worker_or_user_for_run(\n }\n }\n \n+fn require_worker_for_run(slot: &AuthContextSlot, route_run_id: &RunId) -> Result<(), ApiError> {\n+ let context = slot.0.lock().expect(\"auth context lock poisoned\");\n+ match &context.principal {\n+ Principal::Worker { run_id } if run_id == route_run_id => Ok(()),\n+ Principal::Worker { .. } | Principal::User(_) => Err(ApiError::forbidden()),\n+ _ => Err(auth_rejection(context.auth_status, context.auth_error_code)),\n+ }\n+}\n+\n fn require_run_management_target(\n slot: &AuthContextSlot,\n route_run_id: &RunId,\ndiff --git a/lib/crates/fabro-server/src/serve.rs b/lib/crates/fabro-server/src/serve.rs\nindex f4a3c5089..3042cbae1 100644\n--- a/lib/crates/fabro-server/src/serve.rs\n+++ b/lib/crates/fabro-server/src/serve.rs\n@@ -776,6 +776,10 @@ where\n http_client: None,\n sandbox_provider_registry: None,\n shutdown: shutdown.clone(),\n+ #[cfg(test)]\n+ worker_control_bus: None,\n+ #[cfg(any(test, feature = \"test-support\"))]\n+ automation_materializer_override: None,\n })?;\n let reconciled = reconcile_incomplete_runs_on_startup(&state).await?;\n if reconciled > 0 {\ndiff --git a/lib/crates/fabro-server/src/server.rs b/lib/crates/fabro-server/src/server.rs\nindex 529a374dd..88260152c 100644\n--- a/lib/crates/fabro-server/src/server.rs\n+++ b/lib/crates/fabro-server/src/server.rs\n@@ -124,7 +124,7 @@ use sha2::{Digest, Sha256};\n use tempfile::NamedTempFile;\n use tokio::fs;\n use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};\n-use tokio::process::{ChildStderr, ChildStdin, Command};\n+use tokio::process::{ChildStderr, Command};\n use tokio::sync::broadcast::error::RecvError;\n use tokio::sync::{\n Mutex as AsyncMutex, Notify, OwnedMutexGuard, RwLock as AsyncRwLock, Semaphore, broadcast,\n@@ -140,6 +140,10 @@ use tracing::{Instrument, debug, error, info, warn};\n use ulid::Ulid;\n \n use crate::auth::{self, GithubEndpoints, auth_translation_middleware, demo_routing_middleware};\n+use crate::automation_materializer::{\n+ AutomationRunMaterializeError, AutomationRunMaterializeInput, AutomationRunMaterialized,\n+ AutomationRunMaterializer, ProductionAutomationRunMaterializer,\n+};\n use crate::canonical_origin::resolve_canonical_origin;\n use crate::error::ApiError;\n use crate::github_webhooks::{\n@@ -148,14 +152,15 @@ use crate::github_webhooks::{\n use crate::jwt_auth::{self, AuthMode};\n use crate::principal_middleware::{\n AuthContextSlot, RequestAuth, RequestAuthContext, RequireRunBlob, RequireRunManagementTarget,\n- RequireRunScoped, RequireRunStageScoped, RequireStageArtifact, RequiredUser,\n- principal_middleware,\n+ RequireRunScoped, RequireRunStageScoped, RequireStageArtifact, RequireWorkerRunScoped,\n+ RequiredUser, principal_middleware,\n };\n use crate::request_id::{self, RequestId};\n use crate::run_files::{FilesInFlight, new_files_in_flight};\n use crate::server_secrets::{LlmClientResult, ServerSecrets};\n use crate::spawn_env::{apply_render_graph_env, apply_worker_env};\n use crate::startup::load_startup_vault;\n+use crate::worker_control::{LocalWorkerControlBus, WorkerControlBus, WorkerControlBusError};\n use crate::worker_token::{WorkerScopeSet, WorkerTokenKeys, issue_worker_token_with_scopes};\n use crate::{\n canonical_host, demo, diagnostics, run_manifest, security_headers, static_files, web_auth,\n@@ -267,7 +272,6 @@ enum ExecutionResult {\n \n const WORKER_CANCEL_GRACE: Duration = Duration::from_secs(5);\n const TERMINAL_DELETE_WORKER_GRACE: Duration = Duration::from_millis(50);\n-const WORKER_CONTROL_QUEUE_CAPACITY: usize = 8;\n const WORKER_CONTROL_ENQUEUE_TIMEOUT: Duration = Duration::from_secs(1);\n /// Per-model billing totals.\n #[derive(Default)]\n@@ -289,8 +293,9 @@ pub(crate) type RegistryFactoryOverride =\n \n #[derive(Clone)]\n enum RunAnswerTransport {\n- Subprocess {\n- control_tx: mpsc::Sender,\n+ Worker {\n+ run_id: RunId,\n+ bus: Arc,\n },\n InProcess {\n interviewer: Arc,\n@@ -312,18 +317,46 @@ enum PairTransportError {\n }\n \n impl RunAnswerTransport {\n+ async fn publish_worker_control(\n+ run_id: RunId,\n+ bus: &Arc,\n+ message: WorkerControlEnvelope,\n+ ) -> Result<(), WorkerControlBusError> {\n+ timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, bus.publish(run_id, message))\n+ .await\n+ .map_err(|_| WorkerControlBusError::PublishTimeout)?\n+ .map(|_| ())\n+ }\n+\n+ fn answer_error_from_bus(error: &WorkerControlBusError) -> AnswerTransportError {\n+ match error {\n+ WorkerControlBusError::PublishTimeout => AnswerTransportError::Timeout,\n+ WorkerControlBusError::Closed\n+ | WorkerControlBusError::Unavailable\n+ | WorkerControlBusError::InvalidCursor { .. } => AnswerTransportError::Closed,\n+ }\n+ }\n+\n+ fn pair_error_from_bus(error: &WorkerControlBusError) -> PairTransportError {\n+ match error {\n+ WorkerControlBusError::PublishTimeout => PairTransportError::Timeout,\n+ WorkerControlBusError::Closed\n+ | WorkerControlBusError::Unavailable\n+ | WorkerControlBusError::InvalidCursor { .. } => PairTransportError::Closed,\n+ }\n+ }\n+\n async fn submit(\n &self,\n qid: &str,\n submission: AnswerSubmission,\n ) -> Result<(), AnswerTransportError> {\n match self {\n- Self::Subprocess { control_tx } => {\n+ Self::Worker { run_id, bus } => {\n let message = WorkerControlEnvelope::interview_answer(qid.to_string(), submission);\n- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message))\n+ Self::publish_worker_control(*run_id, bus, message)\n .await\n- .map_err(|_| AnswerTransportError::Timeout)?\n- .map_err(|_| AnswerTransportError::Closed)\n+ .map_err(|err| Self::answer_error_from_bus(&err))\n }\n Self::InProcess { interviewer, .. } => interviewer\n .submit(qid, submission)\n@@ -334,12 +367,11 @@ impl RunAnswerTransport {\n \n async fn cancel_run(&self) -> Result<(), AnswerTransportError> {\n match self {\n- Self::Subprocess { control_tx } => {\n+ Self::Worker { run_id, bus } => {\n let message = WorkerControlEnvelope::cancel_run();\n- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message))\n+ Self::publish_worker_control(*run_id, bus, message)\n .await\n- .map_err(|_| AnswerTransportError::Timeout)?\n- .map_err(|_| AnswerTransportError::Closed)\n+ .map_err(|err| Self::answer_error_from_bus(&err))\n }\n Self::InProcess { interviewer, .. } => {\n interviewer.cancel_all().await;\n@@ -352,12 +384,11 @@ impl RunAnswerTransport {\n /// in-process steering hub.\n async fn steer(&self, text: String, actor: Principal) -> Result<(), AnswerTransportError> {\n match self {\n- Self::Subprocess { control_tx } => {\n+ Self::Worker { run_id, bus } => {\n let message = WorkerControlEnvelope::steer(text, actor);\n- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message))\n+ Self::publish_worker_control(*run_id, bus, message)\n .await\n- .map_err(|_| AnswerTransportError::Timeout)?\n- .map_err(|_| AnswerTransportError::Closed)\n+ .map_err(|err| Self::answer_error_from_bus(&err))\n }\n Self::InProcess { steering_hub, .. } => {\n steering_hub.deliver_steer(text, Some(actor));\n@@ -368,12 +399,11 @@ impl RunAnswerTransport {\n \n async fn interrupt(&self, actor: Principal) -> Result<(), AnswerTransportError> {\n match self {\n- Self::Subprocess { control_tx } => {\n+ Self::Worker { run_id, bus } => {\n let message = WorkerControlEnvelope::interrupt(actor);\n- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message))\n+ Self::publish_worker_control(*run_id, bus, message)\n .await\n- .map_err(|_| AnswerTransportError::Timeout)?\n- .map_err(|_| AnswerTransportError::Closed)\n+ .map_err(|err| Self::answer_error_from_bus(&err))\n }\n Self::InProcess { steering_hub, .. } => {\n steering_hub.interrupt(Some(&actor));\n@@ -388,12 +418,11 @@ impl RunAnswerTransport {\n actor: Principal,\n ) -> Result<(), AnswerTransportError> {\n match self {\n- Self::Subprocess { control_tx } => {\n+ Self::Worker { run_id, bus } => {\n let message = WorkerControlEnvelope::interrupt_then_steer(text, actor);\n- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message))\n+ Self::publish_worker_control(*run_id, bus, message)\n .await\n- .map_err(|_| AnswerTransportError::Timeout)?\n- .map_err(|_| AnswerTransportError::Closed)\n+ .map_err(|err| Self::answer_error_from_bus(&err))\n }\n Self::InProcess { steering_hub, .. } => {\n steering_hub.interrupt_then_steer(&text, Some(&actor));\n@@ -410,12 +439,14 @@ impl RunAnswerTransport {\n actor: Principal,\n ) -> Result<(), PairTransportError> {\n match self {\n- Self::Subprocess { control_tx } => {\n+ Self::Worker {\n+ run_id: worker_run_id,\n+ bus,\n+ } => {\n let message = WorkerControlEnvelope::start_pair(run_id, pair_id, target, actor);\n- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message))\n+ Self::publish_worker_control(*worker_run_id, bus, message)\n .await\n- .map_err(|_| PairTransportError::Timeout)?\n- .map_err(|_| PairTransportError::Closed)\n+ .map_err(|err| Self::pair_error_from_bus(&err))\n }\n Self::InProcess { steering_hub, .. } => steering_hub\n .start_pair(run_id, pair_id, target, Some(actor))\n@@ -433,7 +464,7 @@ impl RunAnswerTransport {\n actor: Principal,\n ) -> Result<(), PairTransportError> {\n match self {\n- Self::Subprocess { control_tx } => {\n+ Self::Worker { run_id, bus } => {\n let message = WorkerControlEnvelope::pair_message(\n pair_id,\n message_id,\n@@ -441,10 +472,9 @@ impl RunAnswerTransport {\n client_message_id.clone(),\n actor,\n );\n- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message))\n+ Self::publish_worker_control(*run_id, bus, message)\n .await\n- .map_err(|_| PairTransportError::Timeout)?\n- .map_err(|_| PairTransportError::Closed)\n+ .map_err(|err| Self::pair_error_from_bus(&err))\n }\n Self::InProcess { steering_hub, .. } => steering_hub\n .send_pair_message(pair_id, message_id, text, client_message_id, Some(actor))\n@@ -455,12 +485,11 @@ impl RunAnswerTransport {\n \n async fn end_pair(&self, pair_id: PairId, actor: Principal) -> Result<(), PairTransportError> {\n match self {\n- Self::Subprocess { control_tx } => {\n+ Self::Worker { run_id, bus } => {\n let message = WorkerControlEnvelope::end_pair(pair_id, actor);\n- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message))\n+ Self::publish_worker_control(*run_id, bus, message)\n .await\n- .map_err(|_| PairTransportError::Timeout)?\n- .map_err(|_| PairTransportError::Closed)\n+ .map_err(|err| Self::pair_error_from_bus(&err))\n }\n Self::InProcess { steering_hub, .. } => steering_hub\n .end_pair(pair_id, Some(actor))\n@@ -468,6 +497,30 @@ impl RunAnswerTransport {\n .map_err(PairTransportError::Control),\n }\n }\n+\n+ async fn pause_run(&self) -> Result<(), AnswerTransportError> {\n+ match self {\n+ Self::Worker { run_id, bus } => {\n+ let message = WorkerControlEnvelope::pause_run();\n+ Self::publish_worker_control(*run_id, bus, message)\n+ .await\n+ .map_err(|err| Self::answer_error_from_bus(&err))\n+ }\n+ Self::InProcess { .. } => Err(AnswerTransportError::Closed),\n+ }\n+ }\n+\n+ async fn unpause_run(&self) -> Result<(), AnswerTransportError> {\n+ match self {\n+ Self::Worker { run_id, bus } => {\n+ let message = WorkerControlEnvelope::unpause_run();\n+ Self::publish_worker_control(*run_id, bus, message)\n+ .await\n+ .map_err(|err| Self::answer_error_from_bus(&err))\n+ }\n+ Self::InProcess { .. } => Err(AnswerTransportError::Closed),\n+ }\n+ }\n }\n \n #[derive(Debug, Clone)]\n@@ -1008,10 +1061,13 @@ pub struct AppState {\n session_runtimes: SessionRuntimeManager,\n artifact_store: ArtifactStore,\n automation_store: Arc,\n+ #[cfg(any(test, feature = \"test-support\"))]\n+ automation_materializer_override: Option>,\n worker_tokens: WorkerTokenKeys,\n started_at: Instant,\n resource_sampler: resource_sampler::ResourceSampler,\n max_concurrent_runs: usize,\n+ pub(crate) worker_control_bus: Arc,\n scheduler_notify: Notify,\n global_event_tx: broadcast::Sender,\n /// Per-run coalescing registry for `GET /runs/{id}/files`. Concurrent\n@@ -1048,6 +1104,33 @@ impl AppState {\n pub(crate) fn automation_store(&self) -> &AutomationStore {\n &self.automation_store\n }\n+\n+ #[allow(\n+ dead_code,\n+ reason = \"Automation scheduler wiring will call this after issue #398's materialization core.\"\n+ )]\n+ pub(crate) async fn materialize_automation_run(\n+ &self,\n+ input: AutomationRunMaterializeInput,\n+ ) -> Result {\n+ #[cfg(any(test, feature = \"test-support\"))]\n+ if let Some(materializer) = self.automation_materializer_override.as_ref() {\n+ return materializer.materialize(input).await;\n+ }\n+\n+ let settings = self.server_settings();\n+ let credentials = self\n+ .github_credentials(&settings.server.integrations.github)\n+ .ok()\n+ .flatten();\n+ ProductionAutomationRunMaterializer::new(\n+ credentials,\n+ self.github_api_base_url.clone(),\n+ self.http_client.clone(),\n+ )\n+ .materialize(input)\n+ .await\n+ }\n }\n \n pub(crate) struct AskFabroReadiness {\n@@ -1129,21 +1212,25 @@ async fn lock_pull_request_create(\n }\n \n pub(crate) struct AppStateConfig {\n- pub(crate) resolved_settings: ResolvedAppStateSettings,\n+ pub(crate) resolved_settings: ResolvedAppStateSettings,\n pub(crate) registry_factory_override: Option>,\n- pub(crate) max_concurrent_runs: usize,\n- pub(crate) store: Arc,\n- pub(crate) artifact_store: ArtifactStore,\n- pub(crate) vault_path: PathBuf,\n- pub(crate) variables_path: PathBuf,\n- pub(crate) preloaded_vault: Option,\n- pub(crate) server_secrets: ServerSecrets,\n- pub(crate) env_lookup: EnvLookup,\n- pub(crate) github_api_base_url: Option,\n- pub(crate) active_config_path: PathBuf,\n- pub(crate) http_client: Option,\n+ pub(crate) max_concurrent_runs: usize,\n+ pub(crate) store: Arc,\n+ pub(crate) artifact_store: ArtifactStore,\n+ pub(crate) vault_path: PathBuf,\n+ pub(crate) variables_path: PathBuf,\n+ pub(crate) preloaded_vault: Option,\n+ pub(crate) server_secrets: ServerSecrets,\n+ pub(crate) env_lookup: EnvLookup,\n+ pub(crate) github_api_base_url: Option,\n+ pub(crate) active_config_path: PathBuf,\n+ pub(crate) http_client: Option,\n pub(crate) sandbox_provider_registry: Option,\n- pub(crate) shutdown: CancellationToken,\n+ pub(crate) shutdown: CancellationToken,\n+ #[cfg(test)]\n+ pub(crate) worker_control_bus: Option>,\n+ #[cfg(any(test, feature = \"test-support\"))]\n+ pub(crate) automation_materializer_override: Option>,\n }\n \n #[derive(Clone)]\n@@ -2177,6 +2264,10 @@ pub(crate) fn build_app_state(config: AppStateConfig) -> anyhow::Result anyhow::Result = {\n+ #[cfg(test)]\n+ {\n+ worker_control_bus.unwrap_or_else(|| Arc::new(LocalWorkerControlBus::new()))\n+ }\n+ #[cfg(not(test))]\n+ {\n+ Arc::new(LocalWorkerControlBus::new())\n+ }\n+ };\n Ok(Arc::new(AppState {\n runs: Mutex::new(HashMap::new()),\n aggregate_billing: Mutex::new(BillingAccumulator::default()),\n@@ -2267,10 +2368,13 @@ pub(crate) fn build_app_state(config: AppStateConfig) -> anyhow::Result {\n@@ -2966,6 +3077,7 @@ async fn finish_cancelled_run_before_execution(state: &Arc, run_id: Ru\n clear_live_run_state(managed_run);\n }\n drop(runs);\n+ cleanup_worker_control_bus_for_run(state.as_ref(), run_id);\n state.scheduler_notify.notify_one();\n }\n \n@@ -3101,6 +3213,7 @@ fn fail_managed_run(state: &Arc, run_id: RunId, reason: FailureReason,\n managed_run.error = Some(message);\n clear_live_run_state(managed_run);\n }\n+ cleanup_worker_control_bus_for_run(state.as_ref(), run_id);\n }\n \n fn update_live_run_from_event(state: &AppState, run_id: RunId, event: &RunEvent) {\n@@ -3176,6 +3289,7 @@ fn update_live_run_from_event(state: &AppState, run_id: RunId, event: &RunEvent)\n managed_run.active_api_targets.clear();\n managed_run.active_steerable_stages.clear();\n managed_run.active_non_steerable_stages.clear();\n+ cleanup_worker_control_bus_for_run(state, run_id);\n }\n EventBody::RunFailed(props) => {\n managed_run.status = RunStatus::Failed {\n@@ -3188,6 +3302,7 @@ fn update_live_run_from_event(state: &AppState, run_id: RunId, event: &RunEvent)\n managed_run.active_api_targets.clear();\n managed_run.active_steerable_stages.clear();\n managed_run.active_non_steerable_stages.clear();\n+ cleanup_worker_control_bus_for_run(state, run_id);\n }\n // Track active agent sessions by steerability. Activated/deactivated\n // are leased by session id so stale deactivations cannot clear a newer\n@@ -3272,20 +3387,6 @@ async fn drain_worker_stderr(run_id: RunId, stderr: ChildStderr) -> anyhow::Resu\n Ok(())\n }\n \n-async fn pump_worker_control_jsonl(\n- mut stdin: ChildStdin,\n- mut control_rx: mpsc::Receiver,\n-) -> anyhow::Result<()> {\n- while let Some(message) = control_rx.recv().await {\n- let mut line = serde_json::to_vec(&message)?;\n- line.push(b'\\n');\n- stdin.write_all(&line).await?;\n- stdin.flush().await?;\n- }\n-\n- Ok(())\n-}\n-\n async fn append_worker_exit_failure(\n run_store: &fabro_store::RunDatabase,\n run_id: RunId,\n@@ -3369,7 +3470,7 @@ fn worker_command(\n .arg(run_id.to_string())\n .arg(\"--mode\")\n .arg(worker_mode_arg(mode))\n- .stdin(Stdio::piped())\n+ .stdin(Stdio::null())\n .stdout(worker_stdout)\n .stderr(Stdio::piped());\n \n@@ -4048,25 +4149,6 @@ async fn execute_run_subprocess(state: Arc, run_id: RunId) {\n }\n }\n \n- let Some(stdin) = child.stdin.take() else {\n- let message = \"Worker stdin pipe was unavailable\".to_string();\n- tracing::error!(run_id = %run_id, \"{message}\");\n- let _ = child.start_kill();\n- let failure_event = workflow_event::Event::workflow_run_failed_from_error(\n- &WorkflowError::engine(message.clone()),\n- fabro_types::RunTiming::default(),\n- FailureReason::LaunchFailed,\n- None,\n- None,\n- None,\n- None,\n- );\n- let _ = workflow_event::append_event(&run_store, &run_id, &failure_event).await;\n- fail_managed_run(&state, run_id, FailureReason::LaunchFailed, message);\n- state.scheduler_notify.notify_one();\n- return;\n- };\n-\n let Some(stderr) = child.stderr.take() else {\n let message = \"Worker stderr pipe was unavailable\".to_string();\n tracing::error!(run_id = %run_id, \"{message}\");\n@@ -4086,15 +4168,16 @@ async fn execute_run_subprocess(state: Arc, run_id: RunId) {\n return;\n };\n \n- let (control_tx, control_rx) = mpsc::channel(WORKER_CONTROL_QUEUE_CAPACITY);\n {\n let mut runs = state.runs.lock().expect(\"runs lock poisoned\");\n if let Some(managed_run) = runs.get_mut(&run_id) {\n- managed_run.answer_transport = Some(RunAnswerTransport::Subprocess { control_tx });\n+ managed_run.answer_transport = Some(RunAnswerTransport::Worker {\n+ run_id,\n+ bus: Arc::clone(&state.worker_control_bus),\n+ });\n }\n }\n \n- let control_task = tokio::spawn(pump_worker_control_jsonl(stdin, control_rx));\n let stderr_task = tokio::spawn(drain_worker_stderr(run_id, stderr));\n \n let wait_status = match child.wait().await {\n@@ -4119,9 +4202,6 @@ async fn execute_run_subprocess(state: Arc, run_id: RunId) {\n }\n };\n \n- control_task.abort();\n- let _ = control_task.await;\n-\n match stderr_task.await {\n Ok(Ok(())) => {}\n Ok(Err(err)) => {\ndiff --git a/lib/crates/fabro-server/src/server/handler/lifecycle.rs b/lib/crates/fabro-server/src/server/handler/lifecycle.rs\nindex 0c2697133..d8a2589ac 100644\n--- a/lib/crates/fabro-server/src/server/handler/lifecycle.rs\n+++ b/lib/crates/fabro-server/src/server/handler/lifecycle.rs\n@@ -387,10 +387,6 @@ async fn cancel_run(\n | RunStatus::Running\n | RunStatus::Blocked { .. }\n | RunStatus::Paused { .. } => {\n- let use_cancel_signal = !matches!(\n- managed_run.answer_transport,\n- Some(RunAnswerTransport::InProcess { .. })\n- );\n let persist_cancelled_status = matches!(\n managed_run.status,\n RunStatus::Submitted | RunStatus::Pending { .. } | RunStatus::Runnable\n@@ -404,9 +400,7 @@ async fn cancel_run(\n persist_cancelled_status,\n managed_run.answer_transport.clone(),\n managed_run.cancel_token.clone(),\n- use_cancel_signal\n- .then(|| managed_run.cancel_tx.take())\n- .flatten(),\n+ managed_run.cancel_tx.take(),\n managed_run.worker_pid,\n ))\n }\n@@ -436,22 +430,29 @@ async fn cancel_run(\n if let Some(token) = &cancel_token {\n token.cancel();\n }\n- let sent_cancel_signal = if let Some(cancel_tx) = cancel_tx {\n+ let sent_in_process_cancel = if let Some(cancel_tx) = cancel_tx {\n let _ = cancel_tx.send(());\n true\n } else {\n false\n };\n- if let Some(answer_transport) = answer_transport {\n- if !(sent_cancel_signal && matches!(answer_transport, RunAnswerTransport::InProcess { .. }))\n+ let delivered_control = if let Some(answer_transport) = answer_transport {\n+ if sent_in_process_cancel\n+ && matches!(answer_transport, RunAnswerTransport::InProcess { .. })\n {\n- let _ = answer_transport.cancel_run().await;\n+ true\n+ } else {\n+ answer_transport.cancel_run().await.is_ok()\n+ }\n+ } else {\n+ false\n+ };\n+ if !delivered_control {\n+ if let Some(worker_pid) = worker_pid {\n+ #[cfg(unix)]\n+ fabro_proc::sigterm(worker_pid);\n+ schedule_worker_kill(Arc::clone(&state), id, worker_pid);\n }\n- }\n- if let Some(worker_pid) = worker_pid {\n- #[cfg(unix)]\n- fabro_proc::sigterm(worker_pid);\n- schedule_worker_kill(Arc::clone(&state), id, worker_pid);\n }\n \n if persist_cancelled_status {\n@@ -504,9 +505,9 @@ async fn unmanaged_cancel_response(\n /// How `pause_run` should enact the transition, chosen from the current run\n /// status.\n enum PauseMode {\n- /// Worker is running; ask it to pause via SIGUSR1. Status flips to\n- /// `Paused` once the worker acknowledges.\n- Signal { worker_pid: u32 },\n+ /// Worker is running; ask it to pause via the worker control bus. Status\n+ /// flips to `Paused` once the worker acknowledges.\n+ Transport { transport: RunAnswerTransport },\n /// Worker is blocked on a human gate; flip to `Paused` directly by\n /// appending `RunPaused` ourselves.\n AppendEvent,\n@@ -514,8 +515,9 @@ enum PauseMode {\n \n /// How `unpause_run` should enact the transition.\n enum UnpauseMode {\n- /// No outstanding block; ask the worker to resume via SIGUSR2.\n- Signal { worker_pid: u32 },\n+ /// No outstanding block; ask the worker to resume via the worker control\n+ /// bus.\n+ Transport { transport: RunAnswerTransport },\n /// Was paused while blocked; append `RunUnpaused` and let the reducer\n /// restore the underlying blocked state from `Paused { prior_block }`.\n AppendEvent,\n@@ -544,11 +546,11 @@ async fn pause_run(\n let runs = state.runs.lock().expect(\"runs lock poisoned\");\n match runs.get(&id) {\n Some(managed_run) if managed_run.status == RunStatus::Running => {\n- let Some(worker_pid) = managed_run.worker_pid else {\n+ let Some(transport) = managed_run.answer_transport.clone() else {\n return ApiError::new(StatusCode::CONFLICT, \"Run worker is not available.\")\n .into_response();\n };\n- PauseMode::Signal { worker_pid }\n+ PauseMode::Transport { transport }\n }\n Some(managed_run) if matches!(managed_run.status, RunStatus::Blocked { .. }) => {\n PauseMode::AppendEvent\n@@ -578,11 +580,14 @@ async fn pause_run(\n return ApiError::new(StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response();\n }\n match mode {\n- PauseMode::Signal { worker_pid } => {\n- #[cfg(unix)]\n- fabro_proc::sigusr1(worker_pid);\n- #[cfg(not(unix))]\n- let _ = worker_pid;\n+ PauseMode::Transport { transport } => {\n+ if transport.pause_run().await.is_err() {\n+ return ApiError::new(\n+ StatusCode::SERVICE_UNAVAILABLE,\n+ \"Failed to deliver pause request to the active run.\",\n+ )\n+ .into_response();\n+ }\n }\n PauseMode::AppendEvent => {\n if let Some(response) = synchronous_transition(state.as_ref(), id, |events| {\n@@ -625,11 +630,11 @@ async fn unpause_run(\n prior_block: Some(_),\n } => UnpauseMode::AppendEvent,\n RunStatus::Paused { prior_block: None } => {\n- let Some(worker_pid) = managed_run.worker_pid else {\n+ let Some(transport) = managed_run.answer_transport.clone() else {\n return ApiError::new(StatusCode::CONFLICT, \"Run worker is not available.\")\n .into_response();\n };\n- UnpauseMode::Signal { worker_pid }\n+ UnpauseMode::Transport { transport }\n }\n _ => {\n return ApiError::new(StatusCode::CONFLICT, \"Run is not paused.\")\n@@ -658,11 +663,14 @@ async fn unpause_run(\n return ApiError::new(StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response();\n }\n match mode {\n- UnpauseMode::Signal { worker_pid } => {\n- #[cfg(unix)]\n- fabro_proc::sigusr2(worker_pid);\n- #[cfg(not(unix))]\n- let _ = worker_pid;\n+ UnpauseMode::Transport { transport } => {\n+ if transport.unpause_run().await.is_err() {\n+ return ApiError::new(\n+ StatusCode::SERVICE_UNAVAILABLE,\n+ \"Failed to deliver unpause request to the active run.\",\n+ )\n+ .into_response();\n+ }\n }\n UnpauseMode::AppendEvent => {\n if let Some(response) = synchronous_transition(state.as_ref(), id, |events| {\ndiff --git a/lib/crates/fabro-server/src/server/handler/mod.rs b/lib/crates/fabro-server/src/server/handler/mod.rs\nindex 0804cccb3..33c6aa8eb 100644\n--- a/lib/crates/fabro-server/src/server/handler/mod.rs\n+++ b/lib/crates/fabro-server/src/server/handler/mod.rs\n@@ -15,7 +15,7 @@ mod lifecycle;\n mod models;\n mod pair;\n mod pull_requests;\n-mod runs;\n+pub(in crate::server) mod runs;\n mod sandbox;\n mod sandboxes;\n mod secrets;\n@@ -23,6 +23,7 @@ mod sessions;\n mod steer;\n pub(in crate::server) mod system;\n mod variables;\n+mod worker_control;\n \n pub(super) use system::{health, openapi_spec};\n \n@@ -167,6 +168,7 @@ pub(super) fn real_routes() -> Router> {\n .merge(models::routes())\n .merge(secrets::routes())\n .merge(variables::routes())\n+ .merge(worker_control::routes())\n .merge(sessions::routes())\n .merge(system::routes())\n .merge(completions::routes())\ndiff --git a/lib/crates/fabro-server/src/server/handler/runs.rs b/lib/crates/fabro-server/src/server/handler/runs.rs\nindex c4928d261..3e83bddbb 100644\n--- a/lib/crates/fabro-server/src/server/handler/runs.rs\n+++ b/lib/crates/fabro-server/src/server/handler/runs.rs\n@@ -21,9 +21,10 @@ use fabro_interview::AnswerSubmission;\n use fabro_llm::client::Client as LlmClient;\n use fabro_types::settings::ResolveEnvError;\n use fabro_types::{\n- Principal, RunClientProvenance, RunId, RunProvenance, RunServerProvenance, StageContextWindow,\n- StageContextWindowStaleness, StageContextWindowUnavailableReason, StageHandler,\n- StageModelUsage, StageProjection, SystemActorKind, WorkflowSettings, parse_blob_ref,\n+ AutomationRef, Principal, RunClientProvenance, RunId, RunProvenance, RunServerProvenance,\n+ StageContextWindow, StageContextWindowStaleness, StageContextWindowUnavailableReason,\n+ StageHandler, StageModelUsage, StageProjection, SystemActorKind, WorkflowSettings,\n+ parse_blob_ref,\n };\n use fabro_util::version::FABRO_VERSION;\n use fabro_workflow::command_log::{command_log_path, read_json_string_blob, read_log_slice};\n@@ -598,12 +599,50 @@ async fn create_run(\n Err(err) => return ApiError::bad_request(err.to_string()).into_response(),\n };\n let explicit_title_supplied = req.title.is_some();\n+ Box::pin(create_run_from_manifest(\n+ state,\n+ CreateRunFromManifestRequest {\n+ manifest: req,\n+ submitted_manifest_bytes: body.to_vec(),\n+ explicit_run_id: None,\n+ explicit_title_supplied,\n+ actor,\n+ headers,\n+ automation: None,\n+ },\n+ ))\n+ .await\n+}\n+\n+pub(crate) struct CreateRunFromManifestRequest {\n+ pub(crate) manifest: RunManifest,\n+ pub(crate) submitted_manifest_bytes: Vec,\n+ pub(crate) explicit_run_id: Option,\n+ pub(crate) explicit_title_supplied: bool,\n+ pub(crate) actor: Principal,\n+ pub(crate) headers: HeaderMap,\n+ pub(crate) automation: Option,\n+}\n+\n+pub(crate) async fn create_run_from_manifest(\n+ state: Arc,\n+ request: CreateRunFromManifestRequest,\n+) -> Response {\n+ let CreateRunFromManifestRequest {\n+ manifest,\n+ submitted_manifest_bytes,\n+ explicit_run_id,\n+ explicit_title_supplied,\n+ actor,\n+ headers,\n+ automation,\n+ } = request;\n let manifest_run_defaults = state.manifest_run_defaults();\n let manifest_environment_defaults = state.manifest_environment_defaults();\n let mut prepared = match run_manifest::prepare_manifest_with_environment_defaults(\n manifest_run_defaults.as_ref(),\n manifest_environment_defaults.as_ref(),\n- &req,\n+ &manifest,\n ) {\n Ok(prepared) => prepared,\n Err(err) => return ApiError::bad_request(err.to_string()).into_response(),\n@@ -612,7 +651,9 @@ async fn create_run(\n return ApiError::bad_request(format!(\"Run config variable interpolation failed: {err}\"))\n .into_response();\n }\n- let run_id = prepared.run_id.unwrap_or_else(RunId::new);\n+ let run_id = explicit_run_id\n+ .or(prepared.run_id)\n+ .unwrap_or_else(RunId::new);\n let provider = run_manifest::effective_sandbox_provider(&prepared.settings.run);\n if let Some(error) =\n run_manifest::sandbox_provider_policy_error(&state.server_settings(), provider)\n@@ -653,7 +694,8 @@ async fn create_run(\n );\n create_input.run_id = Some(run_id);\n create_input.provenance = Some(run_provenance(&headers, &actor));\n- create_input.submitted_manifest_bytes = Some(body.to_vec());\n+ create_input.submitted_manifest_bytes = Some(submitted_manifest_bytes);\n+ create_input.automation = automation;\n \n let storage_root = match resolve_interp_string(&state.server_settings().server.storage.root) {\n Ok(path) => PathBuf::from(path),\ndiff --git a/lib/crates/fabro-server/src/server/handler/worker_control.rs b/lib/crates/fabro-server/src/server/handler/worker_control.rs\nnew file mode 100644\nindex 000000000..d643a904e\n--- /dev/null\n+++ b/lib/crates/fabro-server/src/server/handler/worker_control.rs\n@@ -0,0 +1,166 @@\n+use std::sync::Arc;\n+\n+use axum::extract::ws::{\n+ CloseFrame, Message as WsMessage, WebSocket, WebSocketUpgrade, close_code,\n+};\n+use fabro_interview::{\n+ WORKER_CONTROL_INVALID_CURSOR_REASON, WORKER_CONTROL_PONG_TIMEOUT_REASON,\n+ WORKER_CONTROL_WS_LIVENESS_TIMEOUT, WORKER_CONTROL_WS_PING_INTERVAL,\n+ WorkerControlDeliveryFrame,\n+};\n+use futures_util::{SinkExt, StreamExt};\n+use tokio::time::{self, Instant, MissedTickBehavior};\n+\n+use super::super::{\n+ ApiError, AppState, IntoResponse, Query, RequireWorkerRunScoped, Response, Router, State,\n+ StatusCode, get,\n+};\n+use crate::worker_control::{WorkerControlBusError, WorkerControlCursor, WorkerControlReceiver};\n+\n+#[derive(Debug, serde::Deserialize)]\n+struct WorkerControlStreamQuery {\n+ after: Option,\n+}\n+\n+pub(super) fn routes() -> Router> {\n+ Router::new().route(\n+ \"/runs/{id}/worker/control-stream\",\n+ get(worker_control_stream),\n+ )\n+}\n+\n+async fn worker_control_stream(\n+ RequireWorkerRunScoped(id): RequireWorkerRunScoped,\n+ State(state): State>,\n+ Query(query): Query,\n+ ws: WebSocketUpgrade,\n+) -> Response {\n+ let cached = match state.store.get_cached_run(&id).await {\n+ Ok(Some(cached)) => cached,\n+ Ok(None) => return ApiError::not_found(\"Run not found.\").into_response(),\n+ Err(err) => {\n+ return ApiError::new(StatusCode::INTERNAL_SERVER_ERROR, err.to_string())\n+ .into_response();\n+ }\n+ };\n+ if cached.projection.archived_at.is_some() {\n+ return ApiError::new(StatusCode::CONFLICT, \"Run is archived.\").into_response();\n+ }\n+ if cached.projection.status.is_terminal() {\n+ return ApiError::new(\n+ StatusCode::CONFLICT,\n+ \"Run is terminal and cannot accept worker control streams.\",\n+ )\n+ .into_response();\n+ }\n+\n+ let cursor = match WorkerControlCursor::from_after_query(query.after.as_deref()) {\n+ Ok(cursor) => cursor,\n+ Err(err @ WorkerControlBusError::InvalidCursor { .. }) => {\n+ return ApiError::new(StatusCode::GONE, err.to_string()).into_response();\n+ }\n+ Err(err) => return worker_control_bus_error_response(&err),\n+ };\n+ let receiver = match state.worker_control_bus.subscribe(id, cursor).await {\n+ Ok(receiver) => receiver,\n+ Err(err @ WorkerControlBusError::InvalidCursor { .. }) => {\n+ return ApiError::new(StatusCode::GONE, err.to_string()).into_response();\n+ }\n+ Err(err) => return worker_control_bus_error_response(&err),\n+ };\n+\n+ ws.on_upgrade(move |socket| worker_control_websocket(socket, receiver))\n+}\n+\n+fn worker_control_bus_error_response(err: &WorkerControlBusError) -> Response {\n+ let status = match err {\n+ WorkerControlBusError::Closed | WorkerControlBusError::Unavailable => {\n+ StatusCode::SERVICE_UNAVAILABLE\n+ }\n+ WorkerControlBusError::PublishTimeout => StatusCode::SERVICE_UNAVAILABLE,\n+ WorkerControlBusError::InvalidCursor { .. } => StatusCode::GONE,\n+ };\n+ ApiError::new(status, err.to_string()).into_response()\n+}\n+\n+async fn worker_control_websocket(socket: WebSocket, mut receiver: WorkerControlReceiver) {\n+ let (mut sender, mut receiver_ws) = socket.split();\n+ let mut ping_interval = time::interval(WORKER_CONTROL_WS_PING_INTERVAL);\n+ ping_interval.set_missed_tick_behavior(MissedTickBehavior::Delay);\n+ let mut last_liveness = Instant::now();\n+ let liveness_timeout = time::sleep_until(last_liveness + WORKER_CONTROL_WS_LIVENESS_TIMEOUT);\n+ tokio::pin!(liveness_timeout);\n+\n+ loop {\n+ liveness_timeout\n+ .as_mut()\n+ .reset(last_liveness + WORKER_CONTROL_WS_LIVENESS_TIMEOUT);\n+\n+ tokio::select! {\n+ delivery = receiver.recv() => {\n+ let Some(delivery) = delivery else {\n+ let _ = sender.send(WsMessage::Close(None)).await;\n+ return;\n+ };\n+ let delivery = match delivery {\n+ Ok(delivery) => delivery,\n+ Err(WorkerControlBusError::InvalidCursor { .. }) => {\n+ let _ = sender.send(invalid_cursor_close_message()).await;\n+ return;\n+ }\n+ Err(_) => {\n+ let _ = sender.send(WsMessage::Close(None)).await;\n+ return;\n+ }\n+ };\n+ let frame = WorkerControlDeliveryFrame {\n+ id: delivery.id.to_string(),\n+ envelope: delivery.envelope,\n+ };\n+ let Ok(text) = serde_json::to_string(&frame) else {\n+ let _ = sender.send(WsMessage::Close(None)).await;\n+ return;\n+ };\n+ if sender.send(WsMessage::Text(text.into())).await.is_err() {\n+ return;\n+ }\n+ }\n+ message = receiver_ws.next() => {\n+ let Some(message) = message else {\n+ return;\n+ };\n+ match message {\n+ Ok(WsMessage::Ping(payload)) => {\n+ last_liveness = Instant::now();\n+ if sender.send(WsMessage::Pong(payload)).await.is_err() {\n+ return;\n+ }\n+ }\n+ Ok(WsMessage::Pong(_) | WsMessage::Text(_) | WsMessage::Binary(_)) => {\n+ last_liveness = Instant::now();\n+ }\n+ Ok(WsMessage::Close(_)) | Err(_) => return,\n+ }\n+ }\n+ _ = ping_interval.tick() => {\n+ if sender.send(WsMessage::Ping(Vec::new().into())).await.is_err() {\n+ return;\n+ }\n+ }\n+ () = &mut liveness_timeout => {\n+ let _ = sender.send(WsMessage::Close(Some(CloseFrame {\n+ code: close_code::AWAY,\n+ reason: WORKER_CONTROL_PONG_TIMEOUT_REASON.into(),\n+ }))).await;\n+ return;\n+ }\n+ }\n+ }\n+}\n+\n+fn invalid_cursor_close_message() -> WsMessage {\n+ WsMessage::Close(Some(CloseFrame {\n+ code: close_code::POLICY,\n+ reason: WORKER_CONTROL_INVALID_CURSOR_REASON.into(),\n+ }))\n+}\ndiff --git a/lib/crates/fabro-server/src/server/tests.rs b/lib/crates/fabro-server/src/server/tests.rs\nindex df5d26152..866db6757 100644\n--- a/lib/crates/fabro-server/src/server/tests.rs\n+++ b/lib/crates/fabro-server/src/server/tests.rs\n@@ -9,10 +9,12 @@ use std::sync::{Arc as StdArc, Mutex as StdMutex};\n use axum::body::Body;\n use axum::http::{Method, Request, header};\n use chrono::{Duration as ChronoDuration, Utc};\n+use fabro_automation::{AutomationId, AutomationTarget};\n use fabro_config::ServerSettingsBuilder;\n use fabro_config::bind::Bind;\n use fabro_interview::{\n- AnswerValue, ControlInterviewer, Interviewer, Question, WorkerControlMessage,\n+ AnswerValue, ControlInterviewer, Interviewer, Question, WorkerControlDeliveryFrame,\n+ WorkerControlEnvelope, WorkerControlMessage,\n };\n use fabro_llm::types::{Message as LlmMessage, Request as LlmRequest, TokenCounts};\n use fabro_model::catalog::LlmCatalogSettings;\n@@ -32,6 +34,8 @@ use httpmock::Method::{GET, POST};\n use httpmock::MockServer;\n use serde_json::json;\n use tokio_stream::StreamExt as _;\n+use tokio_tungstenite::tungstenite::client::IntoClientRequest;\n+use tokio_tungstenite::tungstenite::protocol::Message as WebSocketMessage;\n use tower::ServiceExt;\n use tracing::field::{Field, Visit};\n use tracing::{Event as TracingEvent, Subscriber, subscriber};\n@@ -40,9 +44,13 @@ use tracing_subscriber::prelude::*;\n use tracing_subscriber::{Layer, Registry};\n \n use super::*;\n+use crate::automation_materializer::AutomationRunMaterializeInput;\n use crate::github_webhooks::compute_signature;\n use crate::jwt_auth::{AuthMode, ConfiguredAuth};\n use crate::test_support::*;\n+use crate::worker_control::{\n+ LocalWorkerControlBus, WorkerControlBus, WorkerControlCursor, WorkerControlReceiver,\n+};\n \n const MINIMAL_DOT: &str = r#\"digraph Test {\n graph [goal=\"Test\"]\n@@ -658,6 +666,292 @@ fn bearer_request(method: Method, path: &str, bearer: &str, body: Body) -> Reque\n .unwrap()\n }\n \n+struct WorkerControlWsTestServer {\n+ base_url: String,\n+ task: tokio::task::JoinHandle<()>,\n+}\n+\n+impl WorkerControlWsTestServer {\n+ async fn spawn(app: Router) -> Self {\n+ let listener = tokio::net::TcpListener::bind(\"127.0.0.1:0\")\n+ .await\n+ .expect(\"test WebSocket listener should bind\");\n+ let addr = listener\n+ .local_addr()\n+ .expect(\"test WebSocket listener should have a local address\");\n+ let task = tokio::spawn(async move {\n+ let result = axum::serve(\n+ listener,\n+ app.into_make_service_with_connect_info::(),\n+ )\n+ .await;\n+ if let Err(err) = result {\n+ tracing::debug!(error = %err, \"test WebSocket server stopped\");\n+ }\n+ });\n+ Self {\n+ base_url: format!(\"ws://{addr}\"),\n+ task,\n+ }\n+ }\n+\n+ fn worker_control_url(&self, run_id: RunId, after: Option<&str>) -> String {\n+ let mut url = format!(\n+ \"{}/api/v1/runs/{run_id}/worker/control-stream\",\n+ self.base_url\n+ );\n+ if let Some(after) = after {\n+ url.push_str(\"?after=\");\n+ url.push_str(after);\n+ }\n+ url\n+ }\n+}\n+\n+impl Drop for WorkerControlWsTestServer {\n+ fn drop(&mut self) {\n+ self.task.abort();\n+ }\n+}\n+\n+fn worker_control_ws_request(\n+ server: &WorkerControlWsTestServer,\n+ run_id: RunId,\n+ bearer: Option<&str>,\n+ after: Option<&str>,\n+) -> Request<()> {\n+ let mut request = server\n+ .worker_control_url(run_id, after)\n+ .into_client_request()\n+ .expect(\"test worker-control WebSocket request should build\");\n+ if let Some(bearer) = bearer {\n+ request.headers_mut().insert(\n+ header::AUTHORIZATION,\n+ format!(\"Bearer {bearer}\")\n+ .parse()\n+ .expect(\"test bearer header should parse\"),\n+ );\n+ }\n+ request\n+}\n+\n+async fn connect_worker_control_ws(\n+ server: &WorkerControlWsTestServer,\n+ run_id: RunId,\n+ bearer: &str,\n+ after: Option<&str>,\n+) -> tokio_tungstenite::WebSocketStream> {\n+ let request = worker_control_ws_request(server, run_id, Some(bearer), after);\n+ let (socket, _) = tokio_tungstenite::connect_async(request)\n+ .await\n+ .expect(\"worker-control WebSocket should connect\");\n+ socket\n+}\n+\n+async fn assert_worker_control_ws_rejected(\n+ server: &WorkerControlWsTestServer,\n+ run_id: RunId,\n+ bearer: Option<&str>,\n+ after: Option<&str>,\n+ expected: StatusCode,\n+) {\n+ let request = worker_control_ws_request(server, run_id, bearer, after);\n+ let error = tokio_tungstenite::connect_async(request)\n+ .await\n+ .expect_err(\"worker-control WebSocket should be rejected\");\n+ match error {\n+ tokio_tungstenite::tungstenite::Error::Http(response) => {\n+ assert_eq!(response.status(), expected);\n+ }\n+ other => panic!(\"expected HTTP rejection {expected}, got {other:#}\"),\n+ }\n+}\n+\n+async fn next_worker_control_frame(\n+ socket: &mut tokio_tungstenite::WebSocketStream<\n+ tokio_tungstenite::MaybeTlsStream,\n+ >,\n+) -> WorkerControlDeliveryFrame {\n+ let message = tokio::time::timeout(std::time::Duration::from_secs(2), async {\n+ loop {\n+ let message = futures_util::StreamExt::next(socket)\n+ .await\n+ .expect(\"worker-control WebSocket should remain open\")\n+ .expect(\"worker-control WebSocket frame should be ok\");\n+ match message {\n+ WebSocketMessage::Text(text) => return text,\n+ WebSocketMessage::Ping(payload) => {\n+ futures_util::SinkExt::send(socket, WebSocketMessage::Pong(payload))\n+ .await\n+ .expect(\"test worker-control pong should send\");\n+ }\n+ WebSocketMessage::Pong(_)\n+ | WebSocketMessage::Binary(_)\n+ | WebSocketMessage::Frame(_) => {}\n+ WebSocketMessage::Close(frame) => {\n+ panic!(\"worker-control WebSocket closed before text frame: {frame:?}\");\n+ }\n+ }\n+ }\n+ })\n+ .await\n+ .expect(\"worker-control frame should arrive\");\n+ serde_json::from_str(message.as_str()).expect(\"worker-control delivery frame should parse\")\n+}\n+\n+#[tokio::test(flavor = \"current_thread\")]\n+async fn worker_control_stream_rejects_missing_user_and_cross_run_auth() {\n+ let (_state, app) = jwt_auth_app();\n+ let user_bearer = issue_test_user_jwt();\n+ let run_id = create_run_with_bearer(&app, &user_bearer).await;\n+ let worker_bearer = issue_test_worker_token(&run_id);\n+ let other_run_id = create_run_with_bearer(&app, &user_bearer).await;\n+ let other_worker_bearer = issue_test_worker_token(&other_run_id);\n+ let server = WorkerControlWsTestServer::spawn(app).await;\n+\n+ assert_worker_control_ws_rejected(&server, run_id, None, None, StatusCode::UNAUTHORIZED).await;\n+ assert_worker_control_ws_rejected(\n+ &server,\n+ run_id,\n+ Some(&user_bearer),\n+ None,\n+ StatusCode::FORBIDDEN,\n+ )\n+ .await;\n+ assert_worker_control_ws_rejected(\n+ &server,\n+ run_id,\n+ Some(&other_worker_bearer),\n+ None,\n+ StatusCode::FORBIDDEN,\n+ )\n+ .await;\n+\n+ let mut socket = connect_worker_control_ws(&server, run_id, &worker_bearer, None).await;\n+ futures_util::SinkExt::send(&mut socket, WebSocketMessage::Close(None))\n+ .await\n+ .unwrap();\n+}\n+\n+#[tokio::test(flavor = \"current_thread\")]\n+async fn worker_control_stream_start_subscription_delivers_frames() {\n+ let (state, app) = jwt_auth_app();\n+ let user_bearer = issue_test_user_jwt();\n+ let run_id = create_run_with_bearer(&app, &user_bearer).await;\n+ let worker_bearer = issue_test_worker_token(&run_id);\n+ let server = WorkerControlWsTestServer::spawn(app).await;\n+ let mut socket = connect_worker_control_ws(&server, run_id, &worker_bearer, None).await;\n+\n+ let expected = WorkerControlEnvelope::cancel_run();\n+ let id = state\n+ .worker_control_bus\n+ .publish(run_id, expected.clone())\n+ .await\n+ .unwrap();\n+ let frame = next_worker_control_frame(&mut socket).await;\n+\n+ assert_eq!(frame.id, id.to_string());\n+ assert_eq!(frame.envelope, expected);\n+}\n+\n+#[tokio::test(flavor = \"current_thread\")]\n+async fn worker_control_stream_after_subscription_delivers_only_later_frames() {\n+ let (state, app) = jwt_auth_app();\n+ let user_bearer = issue_test_user_jwt();\n+ let run_id = create_run_with_bearer(&app, &user_bearer).await;\n+ let worker_bearer = issue_test_worker_token(&run_id);\n+ let first = state\n+ .worker_control_bus\n+ .publish(run_id, WorkerControlEnvelope::cancel_run())\n+ .await\n+ .unwrap();\n+ let expected = WorkerControlEnvelope::pause_run();\n+ let second = state\n+ .worker_control_bus\n+ .publish(run_id, expected.clone())\n+ .await\n+ .unwrap();\n+ let server = WorkerControlWsTestServer::spawn(app).await;\n+\n+ let mut socket =\n+ connect_worker_control_ws(&server, run_id, &worker_bearer, Some(first.as_str())).await;\n+ let frame = next_worker_control_frame(&mut socket).await;\n+\n+ assert_eq!(frame.id, second.to_string());\n+ assert_eq!(frame.envelope, expected);\n+}\n+\n+#[tokio::test(flavor = \"current_thread\")]\n+async fn worker_control_stream_invalid_cursor_is_http_gone_before_upgrade() {\n+ let (_state, app) = jwt_auth_app();\n+ let user_bearer = issue_test_user_jwt();\n+ let run_id = create_run_with_bearer(&app, &user_bearer).await;\n+ let worker_bearer = issue_test_worker_token(&run_id);\n+ let server = WorkerControlWsTestServer::spawn(app).await;\n+\n+ assert_worker_control_ws_rejected(\n+ &server,\n+ run_id,\n+ Some(&worker_bearer),\n+ Some(\"local:999\"),\n+ StatusCode::GONE,\n+ )\n+ .await;\n+}\n+\n+#[tokio::test(flavor = \"current_thread\")]\n+async fn worker_control_stream_rejects_missing_terminal_and_archived_runs() {\n+ let (state, app) = jwt_auth_app();\n+ let user_bearer = issue_test_user_jwt();\n+ let missing_run_id = RunId::new();\n+ let missing_worker_bearer = issue_test_worker_token(&missing_run_id);\n+ let terminal_run_id = RunId::new();\n+ create_succeeded_run(&state, terminal_run_id).await;\n+ let terminal_worker_bearer = issue_test_worker_token(&terminal_run_id);\n+ let archived_run_id = RunId::new();\n+ create_succeeded_run(&state, archived_run_id).await;\n+ let archived_worker_bearer = issue_test_worker_token(&archived_run_id);\n+ let archive_response = app\n+ .clone()\n+ .oneshot(\n+ Request::builder()\n+ .method(\"POST\")\n+ .uri(api(&format!(\"/runs/{archived_run_id}/archive\")))\n+ .header(header::AUTHORIZATION, format!(\"Bearer {user_bearer}\"))\n+ .body(Body::empty())\n+ .unwrap(),\n+ )\n+ .await\n+ .unwrap();\n+ response_json!(archive_response, StatusCode::OK).await;\n+ let server = WorkerControlWsTestServer::spawn(app).await;\n+\n+ assert_worker_control_ws_rejected(\n+ &server,\n+ missing_run_id,\n+ Some(&missing_worker_bearer),\n+ None,\n+ StatusCode::NOT_FOUND,\n+ )\n+ .await;\n+ assert_worker_control_ws_rejected(\n+ &server,\n+ terminal_run_id,\n+ Some(&terminal_worker_bearer),\n+ None,\n+ StatusCode::CONFLICT,\n+ )\n+ .await;\n+ assert_worker_control_ws_rejected(\n+ &server,\n+ archived_run_id,\n+ Some(&archived_worker_bearer),\n+ None,\n+ StatusCode::CONFLICT,\n+ )\n+ .await;\n+}\n+\n fn json_bearer_request(\n method: Method,\n path: &str,\n@@ -1649,6 +1943,8 @@ fn slack_app_state_with_secret_sources(\n http_client: Some(fabro_http::test_http_client().expect(\"test HTTP client should build\")),\n sandbox_provider_registry: None,\n shutdown: tokio_util::sync::CancellationToken::new(),\n+ worker_control_bus: None,\n+ automation_materializer_override: None,\n })\n .expect(\"slack test app state should build\")\n }\n@@ -1748,12 +2044,31 @@ fn slack_service_respects_disabled_server_config_even_with_vault_tokens() {\n http_client: Some(fabro_http::test_http_client().expect(\"test HTTP client should build\")),\n sandbox_provider_registry: None,\n shutdown: tokio_util::sync::CancellationToken::new(),\n+ worker_control_bus: None,\n+ automation_materializer_override: None,\n })\n .expect(\"slack disabled test app state should build\");\n \n assert!(state.slack_service.is_none());\n }\n \n+#[cfg(unix)]\n+#[test]\n+fn worker_command_uses_null_stdin_and_token_env() {\n+ let storage_dir = tempfile::tempdir().unwrap();\n+ let state = worker_command_test_state(storage_dir.path(), &[\"dev-token\"], Some(TEST_DEV_TOKEN));\n+ let cmd = worker_command(\n+ state.as_ref(),\n+ RunId::new(),\n+ RunExecutionMode::Start,\n+ storage_dir.path(),\n+ false,\n+ )\n+ .unwrap();\n+\n+ assert_worker_command_passes_token_only_by_env(&cmd);\n+}\n+\n #[cfg(unix)]\n #[test]\n fn worker_command_default_token_omits_agent_run_tools_scope() {\n@@ -2052,6 +2367,8 @@ methods = [\"dev-token\"]\n http_client: Some(fabro_http::test_http_client().expect(\"test HTTP client should build\")),\n sandbox_provider_registry: None,\n shutdown: tokio_util::sync::CancellationToken::new(),\n+ worker_control_bus: None,\n+ automation_materializer_override: None,\n }) else {\n panic!(\"build_app_state should require SESSION_SECRET\")\n };\n@@ -2179,6 +2496,8 @@ fn build_test_app_state_with_vault_path(vault_path: &Path) -> anyhow::Result crate::worker_token::\n .claims\n }\n \n+async fn worker_transport_with_receiver(\n+ run_id: RunId,\n+) -> (RunAnswerTransport, WorkerControlReceiver) {\n+ let bus = StdArc::new(LocalWorkerControlBus::new());\n+ let receiver = bus\n+ .subscribe(run_id, WorkerControlCursor::Start)\n+ .await\n+ .expect(\"test worker bus should subscribe\");\n+ // Ensure the subscription task is waiting before the test publishes.\n+ tokio::task::yield_now().await;\n+ let bus: StdArc = bus;\n+ let transport = RunAnswerTransport::Worker { run_id, bus };\n+ (transport, receiver)\n+}\n+\n+async fn recv_worker_control_envelope(\n+ receiver: &mut WorkerControlReceiver,\n+) -> WorkerControlEnvelope {\n+ receiver\n+ .recv()\n+ .await\n+ .expect(\"test worker control receiver should stay open\")\n+ .expect(\"test worker control delivery should succeed\")\n+ .envelope\n+}\n+\n #[tokio::test]\n-async fn subprocess_answer_transport_cancel_run_enqueues_cancel_message() {\n- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1);\n- let transport = RunAnswerTransport::Subprocess { control_tx };\n+async fn worker_answer_transport_cancel_run_publishes_cancel_message() {\n+ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await;\n \n transport.cancel_run().await.unwrap();\n \n assert_eq!(\n- control_rx.recv().await,\n- Some(WorkerControlEnvelope::cancel_run())\n+ recv_worker_control_envelope(&mut control_rx).await,\n+ WorkerControlEnvelope::cancel_run()\n );\n }\n \n #[tokio::test]\n-async fn subprocess_answer_transport_steer_enqueues_plain_steer_message() {\n- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1);\n- let transport = RunAnswerTransport::Subprocess { control_tx };\n+async fn worker_answer_transport_steer_publishes_plain_steer_message() {\n+ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await;\n let actor = Principal::System {\n system_kind: SystemActorKind::Engine,\n };\n@@ -2385,15 +2728,14 @@ async fn subprocess_answer_transport_steer_enqueues_plain_steer_message() {\n .unwrap();\n \n assert_eq!(\n- control_rx.recv().await,\n- Some(WorkerControlEnvelope::steer(\"try again\", actor))\n+ recv_worker_control_envelope(&mut control_rx).await,\n+ WorkerControlEnvelope::steer(\"try again\", actor)\n );\n }\n \n #[tokio::test]\n-async fn subprocess_answer_transport_interrupt_enqueues_interrupt_message() {\n- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1);\n- let transport = RunAnswerTransport::Subprocess { control_tx };\n+async fn worker_answer_transport_interrupt_publishes_interrupt_message() {\n+ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await;\n let actor = Principal::System {\n system_kind: SystemActorKind::Engine,\n };\n@@ -2401,15 +2743,14 @@ async fn subprocess_answer_transport_interrupt_enqueues_interrupt_message() {\n transport.interrupt(actor.clone()).await.unwrap();\n \n assert_eq!(\n- control_rx.recv().await,\n- Some(WorkerControlEnvelope::interrupt(actor))\n+ recv_worker_control_envelope(&mut control_rx).await,\n+ WorkerControlEnvelope::interrupt(actor)\n );\n }\n \n #[tokio::test]\n-async fn subprocess_answer_transport_interrupt_then_steer_enqueues_single_combined_message() {\n- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1);\n- let transport = RunAnswerTransport::Subprocess { control_tx };\n+async fn worker_answer_transport_interrupt_then_steer_publishes_single_combined_message() {\n+ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await;\n let actor = Principal::System {\n system_kind: SystemActorKind::Engine,\n };\n@@ -2420,19 +2761,15 @@ async fn subprocess_answer_transport_interrupt_then_steer_enqueues_single_combin\n .unwrap();\n \n assert_eq!(\n- control_rx.recv().await,\n- Some(WorkerControlEnvelope::interrupt_then_steer(\n- \"try again\",\n- actor\n- ))\n+ recv_worker_control_envelope(&mut control_rx).await,\n+ WorkerControlEnvelope::interrupt_then_steer(\"try again\", actor)\n );\n }\n \n #[tokio::test]\n-async fn subprocess_answer_transport_pair_commands_enqueue_control_messages() {\n- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(3);\n- let transport = RunAnswerTransport::Subprocess { control_tx };\n+async fn worker_answer_transport_pair_commands_publish_control_messages() {\n let run_id = fixtures::RUN_1;\n+ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await;\n let pair_id = \"01HZX6M29F1CD5YYMHT1F5D7WQ\".parse().unwrap();\n let message_id = \"01HZX6M4D7Y1QW0Q0P6V8Z4DR5\".parse().unwrap();\n let actor = Principal::System {\n@@ -2457,27 +2794,39 @@ async fn subprocess_answer_transport_pair_commands_enqueue_control_messages() {\n transport.end_pair(pair_id, actor.clone()).await.unwrap();\n \n assert_eq!(\n- control_rx.recv().await,\n- Some(WorkerControlEnvelope::start_pair(\n- run_id,\n- pair_id,\n- target,\n- actor.clone()\n- ))\n+ recv_worker_control_envelope(&mut control_rx).await,\n+ WorkerControlEnvelope::start_pair(run_id, pair_id, target, actor.clone())\n );\n assert_eq!(\n- control_rx.recv().await,\n- Some(WorkerControlEnvelope::pair_message(\n+ recv_worker_control_envelope(&mut control_rx).await,\n+ WorkerControlEnvelope::pair_message(\n pair_id,\n message_id,\n \"inspect this\",\n Some(\"client-1\".to_string()),\n actor.clone()\n- ))\n+ )\n );\n assert_eq!(\n- control_rx.recv().await,\n- Some(WorkerControlEnvelope::end_pair(pair_id, actor))\n+ recv_worker_control_envelope(&mut control_rx).await,\n+ WorkerControlEnvelope::end_pair(pair_id, actor)\n+ );\n+}\n+\n+#[tokio::test]\n+async fn worker_answer_transport_pause_and_unpause_publish_control_messages() {\n+ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await;\n+\n+ transport.pause_run().await.unwrap();\n+ transport.unpause_run().await.unwrap();\n+\n+ assert_eq!(\n+ recv_worker_control_envelope(&mut control_rx).await,\n+ WorkerControlEnvelope::pause_run()\n+ );\n+ assert_eq!(\n+ recv_worker_control_envelope(&mut control_rx).await,\n+ WorkerControlEnvelope::unpause_run()\n );\n }\n \n@@ -2854,6 +3203,155 @@ async fn post_run_manifest(app: &Router, manifest: serde_json::Value) -> serde_j\n response_json!(response, StatusCode::CREATED).await\n }\n \n+#[tokio::test]\n+async fn post_runs_create_regression_keeps_api_behavior_without_automation_metadata() {\n+ let state = TestAppStateBuilder::new().env_lookup(|_| None).build();\n+ let app = crate::test_support::build_test_router(Arc::clone(&state));\n+ let mut manifest = minimal_manifest_json(MINIMAL_DOT);\n+ manifest[\"title\"] = json!(\"API title\");\n+\n+ let body = post_run_manifest(&app, manifest).await;\n+ let run_id: RunId = body[\"id\"].as_str().unwrap().parse().unwrap();\n+\n+ assert_eq!(body[\"title\"], \"API title\");\n+ assert!(body[\"automation\"].is_null());\n+ assert_eq!(body[\"lifecycle\"][\"status\"][\"kind\"], \"submitted\");\n+ let summary = state\n+ .store\n+ .get_cached_summary(&run_id, Utc::now())\n+ .await\n+ .unwrap()\n+ .unwrap();\n+ assert!(summary.automation.is_none());\n+}\n+\n+#[tokio::test]\n+async fn create_run_from_manifest_helper_persists_without_automation_metadata() {\n+ let state = TestAppStateBuilder::new().env_lookup(|_| None).build();\n+ let manifest: RunManifest = serde_json::from_value(minimal_manifest_json(MINIMAL_DOT)).unwrap();\n+ let submitted_manifest_bytes = serde_json::to_vec(&manifest).unwrap();\n+ let run_id = RunId::new();\n+\n+ let response = Box::pin(handler::runs::create_run_from_manifest(\n+ Arc::clone(&state),\n+ handler::runs::CreateRunFromManifestRequest {\n+ manifest,\n+ submitted_manifest_bytes,\n+ explicit_run_id: Some(run_id),\n+ explicit_title_supplied: false,\n+ actor: Principal::System {\n+ system_kind: SystemActorKind::Engine,\n+ },\n+ headers: HeaderMap::new(),\n+ automation: None,\n+ },\n+ ))\n+ .await;\n+\n+ let body = response_json!(response, StatusCode::CREATED).await;\n+ assert_eq!(body[\"id\"], run_id.to_string());\n+ assert!(body[\"automation\"].is_null());\n+ let summary = state\n+ .store\n+ .get_cached_summary(&run_id, Utc::now())\n+ .await\n+ .unwrap()\n+ .unwrap();\n+ assert!(summary.automation.is_none());\n+}\n+\n+#[tokio::test]\n+async fn create_run_from_manifest_helper_persists_automation_metadata() {\n+ let state = TestAppStateBuilder::new().env_lookup(|_| None).build();\n+ let manifest: RunManifest = serde_json::from_value(minimal_manifest_json(MINIMAL_DOT)).unwrap();\n+ let submitted_manifest_bytes = serde_json::to_vec(&manifest).unwrap();\n+ let run_id = RunId::new();\n+ let automation = fabro_types::AutomationRef {\n+ id: \"nightly\".to_string(),\n+ name: Some(\"Nightly\".to_string()),\n+ trigger_id: Some(\"schedule\".to_string()),\n+ };\n+\n+ let response = Box::pin(handler::runs::create_run_from_manifest(\n+ Arc::clone(&state),\n+ handler::runs::CreateRunFromManifestRequest {\n+ manifest,\n+ submitted_manifest_bytes,\n+ explicit_run_id: Some(run_id),\n+ explicit_title_supplied: false,\n+ actor: Principal::System {\n+ system_kind: SystemActorKind::Engine,\n+ },\n+ headers: HeaderMap::new(),\n+ automation: Some(automation.clone()),\n+ },\n+ ))\n+ .await;\n+\n+ let body = response_json!(response, StatusCode::CREATED).await;\n+ assert_eq!(body[\"automation\"][\"id\"], automation.id);\n+ assert_eq!(\n+ body[\"automation\"][\"name\"],\n+ automation.name.as_deref().unwrap()\n+ );\n+ assert_eq!(\n+ body[\"automation\"][\"trigger_id\"],\n+ automation.trigger_id.as_deref().unwrap()\n+ );\n+ let summary = state\n+ .store\n+ .get_cached_summary(&run_id, Utc::now())\n+ .await\n+ .unwrap()\n+ .unwrap();\n+ assert_eq!(summary.automation, Some(automation));\n+}\n+\n+#[tokio::test]\n+async fn fake_automation_materializer_injection_captures_input_and_returns_manifest() {\n+ let materialized_manifest: RunManifest =\n+ serde_json::from_value(minimal_manifest_json(MINIMAL_DOT)).unwrap();\n+ let fake = TestAutomationRunMaterializer::succeed(\n+ materialized_manifest.clone(),\n+ b\"{\\\"fake\\\":true}\".to_vec(),\n+ );\n+ let state = TestAppStateBuilder::new()\n+ .automation_materializer(fake.clone())\n+ .build();\n+ let run_id = RunId::new();\n+ let user_settings_path = PathBuf::from(\"/tmp/fabro/settings.toml\");\n+ let temp_root = PathBuf::from(\"/tmp/fabro/automation\");\n+ let target = AutomationTarget {\n+ repository: \"fabro-sh/fabro\".to_string(),\n+ ref_selector: \"main\".to_string(),\n+ workflow: \"demo\".to_string(),\n+ };\n+\n+ let output = state\n+ .materialize_automation_run(AutomationRunMaterializeInput {\n+ automation_id: AutomationId::new(\"nightly\").unwrap(),\n+ target: target.clone(),\n+ run_id,\n+ user_settings_path: user_settings_path.clone(),\n+ temp_root: temp_root.clone(),\n+ })\n+ .await\n+ .expect(\"fake materializer should succeed\");\n+\n+ assert_eq!(\n+ serde_json::to_value(&output.manifest).unwrap(),\n+ serde_json::to_value(&materialized_manifest).unwrap()\n+ );\n+ assert_eq!(output.submitted_manifest_bytes, b\"{\\\"fake\\\":true}\".to_vec());\n+ let captured = fake.captured_inputs();\n+ assert_eq!(captured.len(), 1);\n+ assert_eq!(captured[0].automation_id.as_str(), \"nightly\");\n+ assert_eq!(captured[0].target, target);\n+ assert_eq!(captured[0].run_id, run_id);\n+ assert_eq!(captured[0].user_settings_path, user_settings_path);\n+ assert_eq!(captured[0].temp_root, temp_root);\n+}\n+\n async fn mock_openai_title_response<'a>(\n server: &'a MockServer,\n title: &str,\n@@ -5276,6 +5774,8 @@ fn create_github_token_app_state_with_env_lookup_and_llm_catalog_settings(\n http_client: Some(fabro_http::test_http_client().expect(\"test HTTP client should build\")),\n sandbox_provider_registry: None,\n shutdown: tokio_util::sync::CancellationToken::new(),\n+ worker_control_bus: None,\n+ automation_materializer_override: None,\n };\n let state = build_app_state(config).expect(\"test app state should build\");\n if let Some(token) = token {\n@@ -10678,12 +11178,8 @@ async fn steer_without_active_steerable_session_forwards_plain_steer_for_bufferi\n let state = test_app_state();\n let app = crate::test_support::build_test_router(Arc::clone(&state));\n let run_id = fixtures::RUN_1;\n- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1);\n- let _temp_dir = insert_running_control_run(\n- &state,\n- run_id,\n- Some(RunAnswerTransport::Subprocess { control_tx }),\n- );\n+ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await;\n+ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport));\n \n let req = Request::builder()\n .method(\"POST\")\n@@ -10694,7 +11190,7 @@ async fn steer_without_active_steerable_session_forwards_plain_steer_for_bufferi\n \n let response = app.oneshot(req).await.unwrap();\n assert_status!(response, StatusCode::ACCEPTED).await;\n- let envelope = control_rx.recv().await.unwrap();\n+ let envelope = recv_worker_control_envelope(&mut control_rx).await;\n assert!(matches!(\n envelope.message,\n WorkerControlMessage::Steer { ref text, .. } if text == \"try again\"\n@@ -10707,12 +11203,8 @@ async fn steer_with_active_non_steerable_session_returns_conflict() {\n let app = crate::test_support::build_test_router(Arc::clone(&state));\n let run_id = fixtures::RUN_1;\n let stage_id = StageId::new(\"agent\", 1);\n- let (control_tx, _control_rx) = tokio::sync::mpsc::channel(1);\n- let _temp_dir = insert_running_control_run(\n- &state,\n- run_id,\n- Some(RunAnswerTransport::Subprocess { control_tx }),\n- );\n+ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await;\n+ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport));\n {\n let mut runs = state.runs.lock().expect(\"runs lock poisoned\");\n runs.get_mut(&run_id)\n@@ -10739,12 +11231,8 @@ async fn steer_interrupt_without_active_steerable_session_returns_conflict() {\n let state = test_app_state();\n let app = crate::test_support::build_test_router(Arc::clone(&state));\n let run_id = fixtures::RUN_1;\n- let (control_tx, _control_rx) = tokio::sync::mpsc::channel(1);\n- let _temp_dir = insert_running_control_run(\n- &state,\n- run_id,\n- Some(RunAnswerTransport::Subprocess { control_tx }),\n- );\n+ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await;\n+ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport));\n \n let req = Request::builder()\n .method(\"POST\")\n@@ -10765,12 +11253,8 @@ async fn interrupt_with_active_steerable_session_forwards_interrupt() {\n let app = crate::test_support::build_test_router(Arc::clone(&state));\n let run_id = fixtures::RUN_1;\n let stage_id = StageId::new(\"agent\", 1);\n- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1);\n- let _temp_dir = insert_running_control_run(\n- &state,\n- run_id,\n- Some(RunAnswerTransport::Subprocess { control_tx }),\n- );\n+ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await;\n+ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport));\n {\n let mut runs = state.runs.lock().expect(\"runs lock poisoned\");\n runs.get_mut(&run_id)\n@@ -10787,7 +11271,7 @@ async fn interrupt_with_active_steerable_session_forwards_interrupt() {\n \n let response = app.oneshot(req).await.unwrap();\n assert_status!(response, StatusCode::ACCEPTED).await;\n- let envelope = control_rx.recv().await.unwrap();\n+ let envelope = recv_worker_control_envelope(&mut control_rx).await;\n assert!(matches!(\n envelope.message,\n WorkerControlMessage::Interrupt {\n@@ -10802,12 +11286,8 @@ async fn steer_interrupt_with_active_steerable_session_forwards_combined_control\n let app = crate::test_support::build_test_router(Arc::clone(&state));\n let run_id = fixtures::RUN_1;\n let stage_id = StageId::new(\"agent\", 1);\n- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1);\n- let _temp_dir = insert_running_control_run(\n- &state,\n- run_id,\n- Some(RunAnswerTransport::Subprocess { control_tx }),\n- );\n+ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await;\n+ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport));\n {\n let mut runs = state.runs.lock().expect(\"runs lock poisoned\");\n runs.get_mut(&run_id)\n@@ -10825,7 +11305,7 @@ async fn steer_interrupt_with_active_steerable_session_forwards_combined_control\n \n let response = app.oneshot(req).await.unwrap();\n assert_status!(response, StatusCode::ACCEPTED).await;\n- let envelope = control_rx.recv().await.unwrap();\n+ let envelope = recv_worker_control_envelope(&mut control_rx).await;\n assert!(matches!(\n envelope.message,\n WorkerControlMessage::InterruptThenSteer { ref text, .. } if text == \"try again\"\n@@ -10991,12 +11471,8 @@ async fn steer_with_active_acp_session_forwards_to_worker() {\n let state = test_app_state();\n let app = crate::test_support::build_test_router(Arc::clone(&state));\n let run_id = fixtures::RUN_1;\n- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1);\n- let _temp_dir = insert_running_control_run(\n- &state,\n- run_id,\n- Some(RunAnswerTransport::Subprocess { control_tx }),\n- );\n+ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await;\n+ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport));\n \n let started = acp_event_for_stage(&run_id, &workflow_event::Event::AgentAcpStarted {\n node_id: \"agent\".to_string(),\n@@ -11029,7 +11505,7 @@ async fn steer_with_active_acp_session_forwards_to_worker() {\n \n let response = app.oneshot(req).await.unwrap();\n assert_status!(response, StatusCode::ACCEPTED).await;\n- let envelope = control_rx.recv().await.unwrap();\n+ let envelope = recv_worker_control_envelope(&mut control_rx).await;\n assert!(matches!(\n envelope.message,\n WorkerControlMessage::Steer { ref text, .. } if text == \"try again\"\n@@ -11096,12 +11572,8 @@ async fn active_acp_steerable_marker_clears_on_terminal_paths() {\n let state = test_app_state();\n let app = crate::test_support::build_test_router(Arc::clone(&state));\n let run_id = fixtures::RUN_1;\n- let (control_tx, _control_rx) = tokio::sync::mpsc::channel(1);\n- let _temp_dir = insert_running_control_run(\n- &state,\n- run_id,\n- Some(RunAnswerTransport::Subprocess { control_tx }),\n- );\n+ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await;\n+ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport));\n let started = acp_event_for_stage(&run_id, &workflow_event::Event::AgentAcpStarted {\n node_id: \"agent\".to_string(),\n visit: 1,\n@@ -13031,11 +13503,13 @@ async fn pause_run_sets_pending_control_on_board_response() {\n let run_id_str = create_and_start_run(&app, MINIMAL_DOT).await;\n let run_id = run_id_str.parse::().unwrap();\n \n+ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await;\n {\n let mut runs = state.runs.lock().expect(\"runs lock poisoned\");\n let managed_run = runs.get_mut(&run_id).expect(\"run should exist\");\n managed_run.status = RunStatus::Running;\n managed_run.worker_pid = Some(u32::MAX);\n+ managed_run.answer_transport = Some(transport);\n }\n \n let req = Request::builder()\n@@ -13151,11 +13625,13 @@ async fn unpause_run_sets_pending_control() {\n let run_id_str = create_and_start_run(&app, MINIMAL_DOT).await;\n let run_id = run_id_str.parse::().unwrap();\n \n+ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await;\n {\n let mut runs = state.runs.lock().expect(\"runs lock poisoned\");\n let managed_run = runs.get_mut(&run_id).expect(\"run should exist\");\n managed_run.status = RunStatus::Paused { prior_block: None };\n managed_run.worker_pid = Some(u32::MAX);\n+ managed_run.answer_transport = Some(transport);\n }\n \n let req = Request::builder()\ndiff --git a/lib/crates/fabro-server/src/test_support.rs b/lib/crates/fabro-server/src/test_support.rs\nindex afd26f624..8e6bb3c20 100644\n--- a/lib/crates/fabro-server/src/test_support.rs\n+++ b/lib/crates/fabro-server/src/test_support.rs\n@@ -29,6 +29,12 @@ use tokio_util::sync::CancellationToken;\n use ulid::Ulid;\n \n use crate::auth;\n+<<<<<<< HEAD\n+=======\n+use crate::automation_materializer::AutomationRunMaterializer;\n+pub use crate::automation_materializer::TestAutomationRunMaterializer;\n+use crate::ip_allowlist::IpAllowlistConfig;\n+>>>>>>> origin/main\n use crate::jwt_auth::{AuthMode, ConfiguredAuth};\n #[cfg(test)]\n use crate::principal_middleware::{AuthContextSlot, RequestAuthContext};\n@@ -70,6 +76,7 @@ pub struct TestAppStateBuilder {\n server_secret_env: HashMap,\n env_lookup: EnvLookup,\n llm_catalog_settings: LlmCatalogSettings,\n+ automation_materializer: Option>,\n }\n \n impl Default for TestAppStateBuilder {\n@@ -88,6 +95,7 @@ impl Default for TestAppStateBuilder {\n server_secret_env: HashMap::new(),\n env_lookup: default_env_lookup(),\n llm_catalog_settings: LlmCatalogSettings::default(),\n+ automation_materializer: None,\n }\n }\n }\n@@ -144,6 +152,11 @@ impl TestAppStateBuilder {\n self\n }\n \n+ pub fn automation_materializer(mut self, materializer: TestAutomationRunMaterializer) -> Self {\n+ self.automation_materializer = Some(materializer.into_materializer());\n+ self\n+ }\n+\n pub fn provider_base_url(\n mut self,\n provider: impl Into,\n@@ -239,6 +252,9 @@ impl TestAppStateBuilder {\n ),\n sandbox_provider_registry: self.sandbox_provider_registry,\n shutdown: CancellationToken::new(),\n+ #[cfg(test)]\n+ worker_control_bus: None,\n+ automation_materializer_override: self.automation_materializer,\n })\n }\n }\ndiff --git a/lib/crates/fabro-server/src/worker_control/bus.rs b/lib/crates/fabro-server/src/worker_control/bus.rs\nnew file mode 100644\nindex 000000000..b6fadedb5\n--- /dev/null\n+++ b/lib/crates/fabro-server/src/worker_control/bus.rs\n@@ -0,0 +1,155 @@\n+use std::fmt;\n+\n+use fabro_interview::WorkerControlEnvelope;\n+use fabro_types::RunId;\n+use futures_util::future::BoxFuture;\n+use tokio::sync::mpsc;\n+\n+#[derive(Clone, PartialEq, Eq)]\n+pub(crate) struct WorkerControlMessageId(String);\n+\n+impl WorkerControlMessageId {\n+ #[must_use]\n+ pub(crate) fn new(value: impl Into) -> Self {\n+ Self(value.into())\n+ }\n+\n+ #[must_use]\n+ pub(crate) fn as_str(&self) -> &str {\n+ &self.0\n+ }\n+}\n+\n+impl fmt::Debug for WorkerControlMessageId {\n+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n+ f.debug_tuple(\"WorkerControlMessageId\")\n+ .field(&self.0)\n+ .finish()\n+ }\n+}\n+\n+impl fmt::Display for WorkerControlMessageId {\n+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n+ f.write_str(&self.0)\n+ }\n+}\n+\n+impl From for WorkerControlMessageId {\n+ fn from(value: String) -> Self {\n+ Self::new(value)\n+ }\n+}\n+\n+impl From<&str> for WorkerControlMessageId {\n+ fn from(value: &str) -> Self {\n+ Self::new(value)\n+ }\n+}\n+\n+/// Cursor for replaying a run's worker-control stream.\n+///\n+/// Future Redis Streams mapping:\n+/// - `Start` maps to `XREAD ... STREAMS fabro:run:{run_id}:control 0-0`.\n+/// - `After(id)` maps to `XREAD ... STREAMS fabro:run:{run_id}:control {id}`.\n+#[derive(Debug, Clone, PartialEq, Eq)]\n+pub(crate) enum WorkerControlCursor {\n+ Start,\n+ After(WorkerControlMessageId),\n+}\n+\n+impl WorkerControlCursor {\n+ pub(crate) fn from_after_query(after: Option<&str>) -> Result {\n+ match after {\n+ None => Ok(Self::Start),\n+ Some(value) if value.trim().is_empty() => Err(WorkerControlBusError::InvalidCursor {\n+ cursor: value.to_string(),\n+ reason: \"cursor id must not be empty\".to_string(),\n+ }),\n+ Some(value) => Ok(Self::After(WorkerControlMessageId::from(value))),\n+ }\n+ }\n+}\n+\n+#[derive(Debug, Clone, PartialEq, Eq)]\n+pub(crate) struct WorkerControlDelivery {\n+ pub(crate) id: WorkerControlMessageId,\n+ pub(crate) envelope: WorkerControlEnvelope,\n+}\n+\n+#[allow(\n+ dead_code,\n+ reason = \"The local backend does not construct every cross-backend error variant.\"\n+)]\n+#[derive(Debug, Clone, thiserror::Error, PartialEq, Eq)]\n+pub(crate) enum WorkerControlBusError {\n+ #[error(\"worker control backend is closed\")]\n+ Closed,\n+ #[error(\"worker control backend is unavailable\")]\n+ Unavailable,\n+ #[error(\"worker control cursor `{cursor}` is invalid: {reason}\")]\n+ InvalidCursor { cursor: String, reason: String },\n+ #[error(\"timed out publishing worker control message\")]\n+ PublishTimeout,\n+}\n+\n+impl WorkerControlBusError {\n+ #[must_use]\n+ pub(crate) fn invalid_cursor(cursor: impl Into, reason: impl Into) -> Self {\n+ Self::InvalidCursor {\n+ cursor: cursor.into(),\n+ reason: reason.into(),\n+ }\n+ }\n+}\n+\n+pub(crate) type WorkerControlReceiver =\n+ mpsc::Receiver>;\n+\n+pub(crate) trait WorkerControlBus: Send + Sync {\n+ fn publish(\n+ &self,\n+ run_id: RunId,\n+ envelope: WorkerControlEnvelope,\n+ ) -> BoxFuture<'_, Result>;\n+\n+ fn subscribe(\n+ &self,\n+ run_id: RunId,\n+ cursor: WorkerControlCursor,\n+ ) -> BoxFuture<'_, Result>;\n+\n+ fn cleanup_run(&self, run_id: RunId) -> BoxFuture<'_, ()>;\n+}\n+\n+#[cfg(test)]\n+mod tests {\n+ use super::*;\n+\n+ #[test]\n+ fn message_id_equality_and_debug_are_opaque() {\n+ let first = WorkerControlMessageId::from(\"local:42\");\n+ let same = WorkerControlMessageId::from(\"local:42\");\n+ let different = WorkerControlMessageId::from(\"local:43\");\n+\n+ assert_eq!(first, same);\n+ assert_ne!(first, different);\n+ assert_eq!(format!(\"{first}\"), \"local:42\");\n+ assert_eq!(format!(\"{first:?}\"), \"WorkerControlMessageId(\\\"local:42\\\")\");\n+ }\n+\n+ #[test]\n+ fn absent_after_query_parses_as_start_cursor() {\n+ assert_eq!(\n+ WorkerControlCursor::from_after_query(None).unwrap(),\n+ WorkerControlCursor::Start\n+ );\n+ }\n+\n+ #[test]\n+ fn present_after_query_parses_as_after_cursor() {\n+ assert_eq!(\n+ WorkerControlCursor::from_after_query(Some(\"local:42\")).unwrap(),\n+ WorkerControlCursor::After(WorkerControlMessageId::from(\"local:42\"))\n+ );\n+ }\n+}\ndiff --git a/lib/crates/fabro-server/src/worker_control/local.rs b/lib/crates/fabro-server/src/worker_control/local.rs\nnew file mode 100644\nindex 000000000..219ff18ec\n--- /dev/null\n+++ b/lib/crates/fabro-server/src/worker_control/local.rs\n@@ -0,0 +1,458 @@\n+use std::collections::{HashMap, VecDeque};\n+use std::sync::atomic::{AtomicU64, Ordering};\n+use std::sync::{Arc, Mutex};\n+\n+use fabro_interview::WorkerControlEnvelope;\n+use fabro_types::RunId;\n+use futures_util::FutureExt;\n+use futures_util::future::BoxFuture;\n+use tokio::sync::{Notify, mpsc};\n+\n+use super::{\n+ WorkerControlBus, WorkerControlBusError, WorkerControlCursor, WorkerControlDelivery,\n+ WorkerControlMessageId, WorkerControlReceiver,\n+};\n+\n+pub(crate) const LOCAL_WORKER_CONTROL_RETAINED_MESSAGES_PER_RUN: usize = 1024;\n+const LOCAL_WORKER_CONTROL_SUBSCRIBER_BUFFER: usize = 64;\n+\n+#[derive(Default)]\n+pub(crate) struct LocalWorkerControlBus {\n+ streams: Arc>>,\n+ next_sequence: Arc,\n+}\n+\n+struct LocalRunControlStream {\n+ messages: VecDeque,\n+ notify: Arc,\n+ has_trimmed: bool,\n+}\n+\n+#[derive(Clone)]\n+struct LocalMessage {\n+ sequence: u64,\n+ delivery: WorkerControlDelivery,\n+}\n+\n+impl LocalWorkerControlBus {\n+ #[must_use]\n+ pub(crate) fn new() -> Self {\n+ Self {\n+ streams: Arc::new(Mutex::new(HashMap::new())),\n+ next_sequence: Arc::new(AtomicU64::new(1)),\n+ }\n+ }\n+\n+ fn next_message_id(&self) -> (u64, WorkerControlMessageId) {\n+ let sequence = self.next_sequence.fetch_add(1, Ordering::Relaxed);\n+ (\n+ sequence,\n+ WorkerControlMessageId::new(format!(\"local:{sequence}\")),\n+ )\n+ }\n+\n+ fn subscribe_inner(\n+ &self,\n+ run_id: RunId,\n+ cursor: &WorkerControlCursor,\n+ ) -> Result {\n+ let (next_sequence, notify) = {\n+ let mut streams = self\n+ .streams\n+ .lock()\n+ .expect(\"worker control streams poisoned\");\n+ let stream = match cursor {\n+ WorkerControlCursor::Start => streams\n+ .entry(run_id)\n+ .or_insert_with(LocalRunControlStream::new),\n+ WorkerControlCursor::After(id) => streams.get_mut(&run_id).ok_or_else(|| {\n+ WorkerControlBusError::invalid_cursor(\n+ id.as_str(),\n+ \"message id is not retained for this run\",\n+ )\n+ })?,\n+ };\n+ let next_sequence = stream.next_sequence_for_cursor(cursor)?;\n+ (next_sequence, Arc::clone(&stream.notify))\n+ };\n+\n+ let (tx, rx) = mpsc::channel(LOCAL_WORKER_CONTROL_SUBSCRIBER_BUFFER);\n+ let streams = Arc::clone(&self.streams);\n+ tokio::spawn(async move {\n+ local_subscription_task(streams, run_id, notify, next_sequence, tx).await;\n+ });\n+ Ok(rx)\n+ }\n+\n+ #[cfg(test)]\n+ pub(crate) fn retained_len(&self, run_id: &RunId) -> usize {\n+ let streams = self\n+ .streams\n+ .lock()\n+ .expect(\"worker control streams poisoned\");\n+ streams\n+ .get(run_id)\n+ .map_or(0, |stream| stream.messages.len())\n+ }\n+}\n+\n+impl LocalRunControlStream {\n+ fn new() -> Self {\n+ Self {\n+ messages: VecDeque::new(),\n+ notify: Arc::new(Notify::new()),\n+ has_trimmed: false,\n+ }\n+ }\n+\n+ fn next_sequence_for_cursor(\n+ &self,\n+ cursor: &WorkerControlCursor,\n+ ) -> Result, WorkerControlBusError> {\n+ match cursor {\n+ WorkerControlCursor::Start if self.has_trimmed => Err(\n+ WorkerControlBusError::invalid_cursor(\"start\", \"retained local stream is trimmed\"),\n+ ),\n+ WorkerControlCursor::Start => Ok(self.messages.front().map(|message| message.sequence)),\n+ WorkerControlCursor::After(id) => {\n+ let requested_sequence = parse_local_sequence(id)?;\n+ let Some(position) = self\n+ .messages\n+ .iter()\n+ .position(|message| message.delivery.id == *id)\n+ else {\n+ return Err(WorkerControlBusError::invalid_cursor(\n+ id.as_str(),\n+ \"message id is not retained for this run\",\n+ ));\n+ };\n+ Ok(self\n+ .messages\n+ .get(position + 1)\n+ .map(|message| message.sequence)\n+ .or(Some(requested_sequence.saturating_add(1))))\n+ }\n+ }\n+ }\n+\n+ fn trim_retained(&mut self) {\n+ while self.messages.len() > LOCAL_WORKER_CONTROL_RETAINED_MESSAGES_PER_RUN {\n+ self.messages.pop_front();\n+ self.has_trimmed = true;\n+ }\n+ }\n+}\n+\n+fn parse_local_sequence(id: &WorkerControlMessageId) -> Result {\n+ let Some(raw) = id.as_str().strip_prefix(\"local:\") else {\n+ return Err(WorkerControlBusError::invalid_cursor(\n+ id.as_str(),\n+ \"local backend only understands local message ids\",\n+ ));\n+ };\n+ raw.parse::().map_err(|_| {\n+ WorkerControlBusError::invalid_cursor(id.as_str(), \"local message id sequence is invalid\")\n+ })\n+}\n+\n+async fn local_subscription_task(\n+ streams: Arc>>,\n+ run_id: RunId,\n+ notify: Arc,\n+ mut next_sequence: Option,\n+ tx: mpsc::Sender>,\n+) {\n+ loop {\n+ // Register interest *before* inspecting the stream so that a publish\n+ // racing with this read does not cause a lost wakeup. `notify_waiters`\n+ // does not leave a permit for future `notified()` calls.\n+ let notified = notify.notified();\n+ tokio::pin!(notified);\n+ notified.as_mut().enable();\n+\n+ let collected = {\n+ let streams_guard = streams.lock().expect(\"worker control streams poisoned\");\n+ match streams_guard.get(&run_id) {\n+ None => None,\n+ Some(stream) => {\n+ // A `Start` subscriber that joined before any publish lazily\n+ // adopts the first retained message as its cursor.\n+ if next_sequence.is_none() && stream.has_trimmed {\n+ Some(Err(WorkerControlBusError::invalid_cursor(\n+ \"start\",\n+ \"retained local stream is trimmed\",\n+ )))\n+ } else {\n+ if next_sequence.is_none() {\n+ next_sequence = stream.messages.front().map(|message| message.sequence);\n+ }\n+ match next_sequence {\n+ None => Some(Ok(Vec::new())),\n+ Some(next) => Some(collect_messages_from(&stream.messages, next)),\n+ }\n+ }\n+ }\n+ }\n+ };\n+\n+ let messages = match collected {\n+ None => return,\n+ Some(Err(err)) => {\n+ let _ = tx.send(Err(err)).await;\n+ return;\n+ }\n+ Some(Ok(messages)) => messages,\n+ };\n+\n+ if messages.is_empty() {\n+ notified.await;\n+ continue;\n+ }\n+\n+ for message in messages {\n+ next_sequence = Some(message.sequence.saturating_add(1));\n+ if tx.send(Ok(message.delivery)).await.is_err() {\n+ return;\n+ }\n+ }\n+ }\n+}\n+\n+/// Returns the retained messages with `sequence >= next`. Cheaper than scanning\n+/// the whole deque: `partition_point` is O(log N) and we only clone the tail.\n+fn collect_messages_from(\n+ messages: &VecDeque,\n+ next: u64,\n+) -> Result, WorkerControlBusError> {\n+ let Some(first_sequence) = messages.front().map(|message| message.sequence) else {\n+ return Ok(Vec::new());\n+ };\n+ if next < first_sequence {\n+ return Err(WorkerControlBusError::invalid_cursor(\n+ format!(\"local:{next}\"),\n+ \"subscriber fell behind retained local messages\",\n+ ));\n+ }\n+ let start = messages.partition_point(|message| message.sequence < next);\n+ Ok(messages.iter().skip(start).cloned().collect())\n+}\n+\n+impl WorkerControlBus for LocalWorkerControlBus {\n+ fn publish(\n+ &self,\n+ run_id: RunId,\n+ envelope: WorkerControlEnvelope,\n+ ) -> BoxFuture<'_, Result> {\n+ async move {\n+ let (sequence, id) = self.next_message_id();\n+ let delivery = WorkerControlDelivery {\n+ id: id.clone(),\n+ envelope,\n+ };\n+ let notify = {\n+ let mut streams = self\n+ .streams\n+ .lock()\n+ .expect(\"worker control streams poisoned\");\n+ let stream = streams\n+ .entry(run_id)\n+ .or_insert_with(LocalRunControlStream::new);\n+ stream\n+ .messages\n+ .push_back(LocalMessage { sequence, delivery });\n+ stream.trim_retained();\n+ Arc::clone(&stream.notify)\n+ };\n+ notify.notify_waiters();\n+ Ok(id)\n+ }\n+ .boxed()\n+ }\n+\n+ fn subscribe(\n+ &self,\n+ run_id: RunId,\n+ cursor: WorkerControlCursor,\n+ ) -> BoxFuture<'_, Result> {\n+ async move { self.subscribe_inner(run_id, &cursor) }.boxed()\n+ }\n+\n+ fn cleanup_run(&self, run_id: RunId) -> BoxFuture<'_, ()> {\n+ async move {\n+ let notify = {\n+ let mut streams = self\n+ .streams\n+ .lock()\n+ .expect(\"worker control streams poisoned\");\n+ streams.remove(&run_id).map(|stream| stream.notify)\n+ };\n+ if let Some(notify) = notify {\n+ notify.notify_waiters();\n+ }\n+ }\n+ .boxed()\n+ }\n+}\n+\n+#[cfg(test)]\n+mod tests {\n+ use std::time::Duration;\n+\n+ use fabro_interview::WorkerControlMessage;\n+ use fabro_types::fixtures;\n+\n+ use super::*;\n+\n+ async fn recv_delivery(receiver: &mut WorkerControlReceiver) -> WorkerControlDelivery {\n+ tokio::time::timeout(Duration::from_secs(1), receiver.recv())\n+ .await\n+ .expect(\"delivery should arrive\")\n+ .expect(\"subscription should remain open\")\n+ .expect(\"delivery should be ok\")\n+ }\n+\n+ #[tokio::test]\n+ async fn messages_publish_in_order() {\n+ let bus = LocalWorkerControlBus::new();\n+ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run())\n+ .await\n+ .unwrap();\n+ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::pause_run())\n+ .await\n+ .unwrap();\n+\n+ let mut receiver = bus\n+ .subscribe(fixtures::RUN_1, WorkerControlCursor::Start)\n+ .await\n+ .unwrap();\n+\n+ assert!(matches!(\n+ recv_delivery(&mut receiver).await.envelope.message,\n+ WorkerControlMessage::RunCancel\n+ ));\n+ assert!(matches!(\n+ recv_delivery(&mut receiver).await.envelope.message,\n+ WorkerControlMessage::RunPause\n+ ));\n+ }\n+\n+ #[tokio::test]\n+ async fn active_subscriber_receives_message_published_after_subscription() {\n+ let bus = LocalWorkerControlBus::new();\n+ let mut receiver = bus\n+ .subscribe(fixtures::RUN_1, WorkerControlCursor::Start)\n+ .await\n+ .unwrap();\n+\n+ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run())\n+ .await\n+ .unwrap();\n+\n+ assert!(matches!(\n+ recv_delivery(&mut receiver).await.envelope.message,\n+ WorkerControlMessage::RunCancel\n+ ));\n+ }\n+\n+ #[tokio::test]\n+ async fn messages_published_before_subscription_replay_from_start() {\n+ let bus = LocalWorkerControlBus::new();\n+ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run())\n+ .await\n+ .unwrap();\n+\n+ let mut receiver = bus\n+ .subscribe(fixtures::RUN_1, WorkerControlCursor::Start)\n+ .await\n+ .unwrap();\n+\n+ assert!(matches!(\n+ recv_delivery(&mut receiver).await.envelope.message,\n+ WorkerControlMessage::RunCancel\n+ ));\n+ }\n+\n+ #[tokio::test]\n+ async fn after_cursor_receives_only_later_messages() {\n+ let bus = LocalWorkerControlBus::new();\n+ let first_id = bus\n+ .publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run())\n+ .await\n+ .unwrap();\n+ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::pause_run())\n+ .await\n+ .unwrap();\n+\n+ let mut receiver = bus\n+ .subscribe(\n+ fixtures::RUN_1,\n+ WorkerControlCursor::After(first_id.clone()),\n+ )\n+ .await\n+ .unwrap();\n+\n+ assert!(matches!(\n+ recv_delivery(&mut receiver).await.envelope.message,\n+ WorkerControlMessage::RunPause\n+ ));\n+ assert!(receiver.try_recv().is_err());\n+ }\n+\n+ #[tokio::test]\n+ async fn trimming_bounds_retained_messages_and_invalidates_old_cursor() {\n+ let bus = LocalWorkerControlBus::new();\n+ let old_id = bus\n+ .publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run())\n+ .await\n+ .unwrap();\n+ for _ in 0..LOCAL_WORKER_CONTROL_RETAINED_MESSAGES_PER_RUN {\n+ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::pause_run())\n+ .await\n+ .unwrap();\n+ }\n+\n+ assert_eq!(\n+ bus.retained_len(&fixtures::RUN_1),\n+ LOCAL_WORKER_CONTROL_RETAINED_MESSAGES_PER_RUN\n+ );\n+ let err = bus\n+ .subscribe(fixtures::RUN_1, WorkerControlCursor::After(old_id))\n+ .await\n+ .unwrap_err();\n+ assert!(matches!(err, WorkerControlBusError::InvalidCursor { .. }));\n+\n+ let err = bus\n+ .subscribe(fixtures::RUN_1, WorkerControlCursor::Start)\n+ .await\n+ .unwrap_err();\n+ assert!(matches!(err, WorkerControlBusError::InvalidCursor { .. }));\n+ }\n+\n+ #[tokio::test]\n+ async fn after_cursor_for_unknown_run_does_not_create_stream() {\n+ let bus = LocalWorkerControlBus::new();\n+ let err = bus\n+ .subscribe(\n+ fixtures::RUN_1,\n+ WorkerControlCursor::After(WorkerControlMessageId::new(\"local:1\")),\n+ )\n+ .await\n+ .unwrap_err();\n+\n+ assert!(matches!(err, WorkerControlBusError::InvalidCursor { .. }));\n+ assert_eq!(bus.retained_len(&fixtures::RUN_1), 0);\n+ }\n+\n+ #[tokio::test]\n+ async fn cleanup_removes_retained_messages() {\n+ let bus = LocalWorkerControlBus::new();\n+ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run())\n+ .await\n+ .unwrap();\n+ assert_eq!(bus.retained_len(&fixtures::RUN_1), 1);\n+\n+ bus.cleanup_run(fixtures::RUN_1).await;\n+\n+ assert_eq!(bus.retained_len(&fixtures::RUN_1), 0);\n+ }\n+}\ndiff --git a/lib/crates/fabro-server/src/worker_control/mod.rs b/lib/crates/fabro-server/src/worker_control/mod.rs\nnew file mode 100644\nindex 000000000..6b6f70896\n--- /dev/null\n+++ b/lib/crates/fabro-server/src/worker_control/mod.rs\n@@ -0,0 +1,8 @@\n+mod bus;\n+mod local;\n+\n+pub(crate) use bus::{\n+ WorkerControlBus, WorkerControlBusError, WorkerControlCursor, WorkerControlDelivery,\n+ WorkerControlMessageId, WorkerControlReceiver,\n+};\n+pub(crate) use local::LocalWorkerControlBus;\ndiff --git a/lib/crates/fabro-workflow/src/operations/start.rs b/lib/crates/fabro-workflow/src/operations/start.rs\nindex 9d3f08d1c..23c90f3ff 100644\n--- a/lib/crates/fabro-workflow/src/operations/start.rs\n+++ b/lib/crates/fabro-workflow/src/operations/start.rs\n@@ -26,6 +26,7 @@ use fabro_types::{ManifestPath, RunId, RunRunnableSource, SandboxProviderKind};\n use fabro_vault::Vault;\n use tokio::runtime::Handle;\n use tokio::sync::RwLock as AsyncRwLock;\n+use tokio::time;\n use tokio_util::sync::CancellationToken;\n \n use crate::artifact_upload::ArtifactSink;\n@@ -987,6 +988,24 @@ impl Drop for DetachedRunBootstrapGuard {\n \n const POSTRUN_INTERRUPTED_MESSAGE: &str = \"Run interrupted before post-run finalization completed.\";\n const POSTRUN_CANCELLED_MESSAGE: &str = \"Run cancelled before post-run finalization completed.\";\n+const DETACHED_COMPLETION_GUARD_TERMINAL_GRACE: Duration = Duration::from_millis(25);\n+\n+async fn run_store_reaches_terminal(run_store: &RunStoreHandle, timeout: Duration) -> bool {\n+ let start = Instant::now();\n+ loop {\n+ if run_store\n+ .state()\n+ .await\n+ .is_ok_and(|state| state.status.is_terminal())\n+ {\n+ return true;\n+ }\n+ if start.elapsed() >= timeout {\n+ return false;\n+ }\n+ time::sleep(Duration::from_millis(10)).await;\n+ }\n+}\n \n struct DetachedRunCompletionGuard {\n event_sink: RunEventSink,\n@@ -1044,6 +1063,11 @@ impl Drop for DetachedRunCompletionGuard {\n let run_store = self.run_store.clone();\n if let Ok(handle) = Handle::try_current() {\n handle.spawn(async move {\n+ if run_store_reaches_terminal(&run_store, DETACHED_COMPLETION_GUARD_TERMINAL_GRACE)\n+ .await\n+ {\n+ return;\n+ }\n emit_workflow_run_failed(\n run_id,\n &run_store,\n", + "summary": { + "files_changed": 59, + "additions": 3602, + "deletions": 2011 + } + } + }, + { + "seq": 0, + "checkpoint": { + "timestamp": "2026-05-28T01:05:10.917390Z", + "current_node": "fixup", + "completed_nodes": [ + "start", + "toolchain", + "preflight_compile", + "preflight_lint", + "implement", + "simplify_opus", + "simplify_gpt", + "verify", + "fixup" + ], + "node_retries": {}, "context_values": { "internal.retry_count.verify": 0, "thread.start.current_node": "toolchain", - "failure_class": "deterministic", + "failure_class": "", "internal.work_dir": "/home/daytona/workspace/fabro", "graph.goal": "---\ntitle: \"refactor: Remove IP allowlisting\"\ntype: refactor\nstatus: active\ndate: 2026-05-27\n---\n\n# refactor: Remove IP allowlisting\n\n## Overview\n\nRemove Fabro's inbound server IP allowlisting feature completely. This removes the\nserver-wide `[server.ip_allowlist]` setting, the GitHub webhook-specific\n`[server.integrations.github.webhooks.ip_allowlist]` overlay, request middleware,\nclient-IP extraction, GitHub `/meta` hook-range expansion, settings API exposure,\ngenerated API client types, and the Settings > Security UI display.\n\nExisting configs that still contain removed IP allowlist keys should fail as\nunknown fields. This is an intentional hard removal, not a compatibility\ndeprecation.\n\n## Problem Frame\n\nThe feature is being removed from Fabro rather than maintained as an in-process\nnetwork access-control layer. Network source restrictions should be handled\noutside Fabro by reverse proxies, firewalls, VPNs, Tailscale, platform ingress,\nor other deployment-layer controls.\n\n## Requirements Trace\n\n- R1. Remove all runtime enforcement of inbound source-IP allowlisting from web,\n API, static asset, and GitHub webhook routes.\n- R2. Remove the server settings schema for global and GitHub webhook IP\n allowlists.\n- R3. Remove `IpAllowEntry` and related API schema/client types from public\n settings payloads.\n- R4. Keep unrelated allowlists intact: GitHub username allowlists, sandbox\n egress CIDR allow lists, and internal policy/test allowlists.\n- R5. Preserve existing authentication, GitHub webhook HMAC verification,\n routing, health checks, request logging, and settings hot-reload behavior.\n- R6. Treat old IP allowlist config as invalid after removal.\n- R7. Update operator-facing docs/changelog so users know to move source-IP\n restrictions upstream.\n\n## Scope Boundaries\n\n- Do not remove `[server.auth.github].allowed_usernames`.\n- Do not remove Daytona/sandbox `cidr_allow_list` network policy support.\n- Do not remove generic uses of \"allowlist\" in policy tests, markdown rendering,\n model controls, or other unrelated domains.\n- Do not add a migration, warning-only parser, or fallback compatibility path for\n old IP allowlist settings.\n- Do not change GitHub webhook signature verification or webhook route strategy.\n\n## Context & Research\n\n### Relevant Code and Patterns\n\n- Settings use sparse config layers in `lib/crates/fabro-config/src/layers/`\n and dense resolved types in `lib/crates/fabro-types/src/settings/`.\n- The OpenAPI spec at `docs/public/api-reference/fabro-api.yaml` is the source\n of truth for API wire shape; `cargo build -p fabro-api` regenerates Rust API\n code, and `cd lib/packages/fabro-api-client && bun run generate` regenerates\n the TypeScript client.\n- `lib/crates/fabro-server/src/server.rs` builds the router and currently wires\n IP allowlist middleware before other route dispatch behavior.\n- `lib/crates/fabro-server/src/serve.rs` currently resolves server and webhook\n allowlist configs at startup and creates a GitHub `/meta` resolver.\n- `apps/fabro-web/app/routes/settings-security.tsx` displays the settings API's\n `server.ip_allowlist` state.\n\n### Main Removal Surfaces\n\n- Config/types: `fabro-types`, `fabro-config`, config tests.\n- Server runtime: `fabro-server/src/ip_allowlist.rs`, router wiring, startup\n wiring, test helpers, routing/TCP tests.\n- API/contracts: OpenAPI schema, `fabro-api` type replacements/exports, generated\n TypeScript API client models.\n- Frontend/docs: Settings > Security copy, settings navigation copy, changelog.\n\n## Key Technical Decisions\n\n- **Hard removal:** Removed config keys should fail through existing\n `deny_unknown_fields` behavior. This keeps the change simple and makes stale\n deployment config visible immediately.\n- **Delete, do not stub:** Remove `IpAllowlistConfig` and middleware parameters\n instead of passing default empty configs through the router. Empty stubs would\n keep the feature shape alive and make future cleanup harder.\n- **Keep webhook auth unchanged:** GitHub webhook source-IP filtering goes away,\n but HMAC signature verification remains the security boundary for webhook\n payload authenticity.\n- **Generated clients follow OpenAPI:** Update the OpenAPI schemas first, then\n regenerate Rust and TypeScript API outputs rather than hand-editing generated\n client files except as a short-term cleanup if generation leaves stale exports.\n- **Docs mention upstream controls:** The changelog and security guidance should\n direct operators to network-layer controls, not a replacement Fabro setting.\n\n## Implementation Units\n\n- [ ] **Unit 1: Remove config schema and resolved types**\n\n**Goal:** Delete the IP allowlist settings contract from config parsing and dense\nserver settings.\n\n**Requirements:** R2, R3, R4, R6\n\n**Dependencies:** None\n\n**Files:**\n- Modify: `lib/crates/fabro-types/src/settings/server.rs`\n- Modify: `lib/crates/fabro-types/src/settings/mod.rs`\n- Modify: `lib/crates/fabro-types/Cargo.toml`\n- Modify: `lib/crates/fabro-config/src/layers/server.rs`\n- Modify: `lib/crates/fabro-config/src/layers/mod.rs`\n- Modify: `lib/crates/fabro-config/src/lib.rs`\n- Modify: `lib/crates/fabro-config/src/resolve/server.rs`\n- Modify: `lib/crates/fabro-config/src/tests/resolve_server.rs`\n\n**Approach:**\n- Remove `ServerNamespace.ip_allowlist` and its `test_default()` population.\n- Delete `ServerIpAllowlistSettings`, `ServerIpAllowlistOverrideSettings`, and\n `IpAllowEntry`.\n- Remove `ServerLayer.ip_allowlist`, `ServerIpAllowlistLayer`,\n `ServerIpAllowlistOverrideLayer`, and `IntegrationWebhooksLayer.ip_allowlist`.\n- Remove resolver functions and validation for global and webhook IP allowlists,\n including `github_meta_hooks` parsing and Unix socket trusted-proxy checks.\n- Remove `ipnet` from `fabro-types`. Keep `ipnet` in `fabro-config` because\n `resolve/environment.rs` still validates sandbox CIDR policy.\n- Replace old positive/negative IP allowlist config tests with unknown-field\n tests for `[server.ip_allowlist]` and\n `[server.integrations.github.webhooks.ip_allowlist]`.\n\n**Patterns to follow:**\n- Existing unknown-field tests in `lib/crates/fabro-config/src/tests/resolve_server.rs`\n for retired server settings.\n- Existing `ServerLayer`/`ServerNamespace` layer-to-resolved pattern for removing\n a server subdomain cleanly.\n\n**Test scenarios:**\n- Error path: TOML with `[server.ip_allowlist]` fails parsing/resolution with an\n unknown-field diagnostic mentioning `ip_allowlist`.\n- Error path: TOML with `[server.integrations.github.webhooks.ip_allowlist]`\n fails with an unknown-field diagnostic mentioning `ip_allowlist`.\n- Happy path: minimal valid server settings still resolve without any\n `ip_allowlist` field.\n- Integration: serialized `ServerSettings` JSON no longer includes\n `server.ip_allowlist`.\n\n**Verification:**\n- `fabro-config` and `fabro-types` compile without removed IP allowlist symbols.\n- Config tests prove stale allowlist settings are rejected.\n\n- [ ] **Unit 2: Remove server middleware and startup resolution**\n\n**Goal:** Remove all runtime source-IP filtering and GitHub `/meta` range\nresolution from `fabro-server`.\n\n**Requirements:** R1, R4, R5\n\n**Dependencies:** Unit 1\n\n**Files:**\n- Delete: `lib/crates/fabro-server/src/ip_allowlist.rs`\n- Modify: `lib/crates/fabro-server/src/lib.rs`\n- Modify: `lib/crates/fabro-server/src/server.rs`\n- Modify: `lib/crates/fabro-server/src/serve.rs`\n- Modify: `lib/crates/fabro-server/src/test_support.rs`\n- Modify: `lib/crates/fabro-server/Cargo.toml`\n- Modify: `lib/crates/fabro-server/src/auth/translate.rs`\n- Modify: `lib/crates/fabro-server/src/web_auth.rs`\n- Modify: `lib/crates/fabro-cli/tests/it/support/auth_harness.rs`\n\n**Approach:**\n- Remove the public `ip_allowlist` module export.\n- Delete `IpAllowlistConfig`, `IpAllowlist`, `GitHubMetaResolver`, client-IP\n extraction, middleware, and GitHub meta cache helpers.\n- Simplify `build_router_with_options` by removing its `Arc`\n parameter and removing `RouterOptions.github_webhook_ip_allowlist`.\n- Remove the global allowlist middleware layer from the main app router.\n- Simplify `github_webhook_routes` so it only receives the webhook secret and no\n route-specific allowlist config.\n- Remove startup creation of `GitHubMetaResolver`, `default_ip_allowlist`,\n `resolve_github_webhook_ip_allowlist`, and\n `resolve_startup_github_webhook_ip_allowlist`.\n- Replace all test/helper call sites that pass `IpAllowlistConfig::default()`\n with the simplified router signature.\n- Remove `ipnet` and any now-unused HTTP mocking/test-only dependencies from\n `fabro-server` if they are only used by the deleted module.\n- Consider whether `serve.rs` still needs `SocketAddr` for TCP\n `ConnectInfo`; if it was only present for allowlisting tests, remove the\n connect-info service wrapper and imports.\n\n**Patterns to follow:**\n- Keep router layers ordered as they are after the allowlist layer is removed:\n auth translation, demo routing, auth extension, canonical host, security\n headers, HTTP logging, request ID.\n- Existing webhook route HMAC tests and auth tests should remain the source of\n truth for webhook behavior.\n\n**Test scenarios:**\n- Happy path: API requests from any TCP remote address route normally when auth\n requirements are satisfied.\n- Happy path: `/health` remains accessible.\n- Integration: GitHub webhook route still rejects missing/invalid signatures and\n accepts valid signatures exactly as before.\n- Cleanup: there are no references to `IpAllowlistConfig`, `ip_allowlist_middleware`,\n `GitHubMetaResolver`, `github_meta_hooks`, or `github-meta-hooks.json`.\n\n**Verification:**\n- `fabro-server` and CLI auth harness tests compile against the simplified router\n API.\n- Runtime startup no longer performs any GitHub `/meta` fetch for webhook IP\n ranges.\n\n- [ ] **Unit 3: Update OpenAPI and generated API clients**\n\n**Goal:** Remove IP allowlist fields and schemas from public settings API\ncontracts and regenerated clients.\n\n**Requirements:** R3, R4\n\n**Dependencies:** Units 1 and 2\n\n**Files:**\n- Modify: `docs/public/api-reference/fabro-api.yaml`\n- Modify: `lib/crates/fabro-api/build.rs`\n- Modify: `lib/crates/fabro-api/src/lib.rs`\n- Modify: `lib/crates/fabro-api/tests/server_settings_round_trip.rs`\n- Modify/generated: `lib/packages/fabro-api-client/src/models/*`\n\n**Approach:**\n- Remove `ip_allowlist` from `ServerNamespace.required` and\n `ServerNamespace.properties`.\n- Remove `ServerIpAllowlistSettings`,\n `ServerIpAllowlistOverrideSettings`, `IpAllowEntry`,\n `LiteralIpAllowEntry`, and `GitHubMetaHooksEntry` schemas.\n- Remove `IntegrationWebhooksSettings.ip_allowlist` from required/properties.\n- Remove `with_replacement` entries and public re-exports for removed types in\n `fabro-api`.\n- Regenerate Rust API code by building `fabro-api`.\n- Regenerate TypeScript Axios client and ensure stale allowlist model files and\n index exports are gone.\n- Strengthen round-trip tests to assert settings JSON omits\n `server.ip_allowlist` and webhook `ip_allowlist`.\n\n**Patterns to follow:**\n- Existing OpenAPI-first workflow in `AGENTS.md`.\n- Existing `server_settings_family_reuses_domain_types` assertions for shared\n domain type identity.\n\n**Test scenarios:**\n- Contract: `ServerSettings` round-trips through API types without any IP\n allowlist field.\n- Contract: OpenAPI-generated TypeScript `ServerNamespace` has no\n `ip_allowlist` property.\n- Contract: `IntegrationWebhooksSettings` has only webhook strategy fields after\n removal.\n\n**Verification:**\n- Generated Rust and TypeScript clients match the updated OpenAPI spec.\n- `rg \"ServerIpAllowlist|IpAllowEntry|server-ip-allowlist|literal-ip-allow-entry\"`\n finds no remaining generated/public API references.\n\n- [ ] **Unit 4: Update web settings UI**\n\n**Goal:** Remove IP allowlist display from Settings > Security and align copy\nwith the new settings shape.\n\n**Requirements:** R3, R7\n\n**Dependencies:** Unit 3\n\n**Files:**\n- Modify: `apps/fabro-web/app/routes/settings-security.tsx`\n- Modify: `apps/fabro-web/app/routes/settings.tsx`\n\n**Approach:**\n- Change the Security page description from \"Authentication methods and network\n allowlist\" to authentication-only wording.\n- Remove destructuring and rendering of `settings.server.ip_allowlist`.\n- Remove the unused `Count` and `plural` imports if no longer used.\n- Update the Settings nav description for Security from \"Authentication and\n network allowlist\" to authentication-focused copy.\n\n**Patterns to follow:**\n- Existing `settings-panel` row layout for Auth methods and Allowed usernames.\n\n**Test scenarios:**\n- Typecheck: the route compiles against the regenerated API client with no\n `server.ip_allowlist` property.\n- UI behavior: Security page still renders auth methods and allowed usernames.\n- Cleanup: no frontend references to `ip_allowlist`, `IP allowlist`, or\n `trusted_proxy_count` remain.\n\n**Verification:**\n- `apps/fabro-web` typecheck passes against the new generated client.\n\n- [ ] **Unit 5: Update tests, docs, changelog, and dependency lockfile**\n\n**Goal:** Remove stale references and document the operator-facing behavior\nchange.\n\n**Requirements:** R4, R6, R7\n\n**Dependencies:** Units 1 through 4\n\n**Files:**\n- Modify: `lib/crates/fabro-server/tests/it/api/routing.rs`\n- Modify: `lib/crates/fabro-server/tests/it/api/tcp.rs`\n- Modify: `lib/crates/fabro-server/tests/it/api/settings.rs`\n- Modify: `docs/public/administration/security.mdx`\n- Create or modify: `docs/public/changelog/2026-05-27.mdx`\n- Modify: `Cargo.lock` if dependency graph changes\n\n**Approach:**\n- Delete route/TCP tests that only prove IP allowlist enforcement or\n `ConnectInfo` behavior.\n- Adjust any router setup helpers after the signature simplification.\n- Add a settings API assertion that `server.ip_allowlist` is absent from\n `/api/v1/settings`.\n- Update security docs to explicitly state that Fabro does not provide inbound\n source-IP allowlisting and operators should use upstream network controls.\n- Add a changelog entry dated 2026-05-27 describing the hard removal and the\n expected replacement at the deployment layer.\n- Run dependency resolution after removing direct `ipnet`/test dependencies;\n keep transitive or unrelated `ipnet` entries needed by sandbox CIDR validation.\n\n**Patterns to follow:**\n- Existing changelog style in `docs/public/changelog/2026-05-26.mdx`.\n- Existing settings API integration test style in\n `lib/crates/fabro-server/tests/it/api/settings.rs`.\n\n**Test scenarios:**\n- Settings API: response contains server auth, listen, storage, scheduler, and\n integrations fields, but not `server.ip_allowlist`.\n- Config compatibility: old IP allowlist TOML fails before startup rather than\n being silently ignored.\n- Docs validation: security docs no longer imply Fabro can restrict inbound\n source IPs internally.\n\n**Verification:**\n- `rg -n \"ip_allowlist|trusted_proxy_count|github_meta_hooks|IP allowlist|ip allowlist\"`\n returns only historical archived plan/brainstorm/spec references or unrelated\n non-server allowlist text.\n\n## System-Wide Impact\n\n- **Public API:** `GET /api/v1/settings` response shape changes by removing\n `server.ip_allowlist` and webhook `ip_allowlist`.\n- **Config compatibility:** Existing `settings.toml` files containing the removed\n keys become invalid. This is intentional.\n- **Runtime security posture:** Fabro no longer blocks requests based on source\n IP. Operators must enforce network source restrictions upstream.\n- **Webhook handling:** GitHub webhook HMAC verification remains unchanged; only\n optional source-IP filtering is removed.\n- **Generated clients:** Downstream TypeScript/Rust consumers that read\n `server.ip_allowlist` must update.\n\n## Risks & Mitigations\n\n| Risk | Mitigation |\n|------|------------|\n| Operators accidentally expose a server that previously relied on Fabro IP allowlisting | Changelog and security docs explicitly call out the removal and direct operators to upstream controls. |\n| Generated API clients retain stale types | Update OpenAPI first, regenerate both Rust and TypeScript clients, and run search checks for stale symbols. |\n| Unrelated allowlist functionality is removed by broad search/replace | Scope searches to `IpAllow`, `ip_allowlist`, `trusted_proxy_count`, and `github_meta_hooks`; preserve sandbox CIDR and GitHub username allowlists. |\n| Router signature cleanup breaks many tests | Update shared test helpers first, then compile-driven cleanup of remaining call sites. |\n| `ipnet` is removed where still needed | Keep `fabro-config` dependency if environment CIDR validation still imports `ipnet::IpNet`. |\n\n## Test Plan\n\n- `cargo build -p fabro-api`\n- `cd lib/packages/fabro-api-client && bun run generate`\n- `cargo nextest run -p fabro-config -p fabro-types -p fabro-api -p fabro-server`\n- `cd apps/fabro-web && bun run typecheck`\n- `cd apps/fabro-web && bun test`\n- `cargo +nightly-2026-04-14 fmt --check --all`\n- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings`\n\n## Assumptions\n\n- The accepted compatibility policy is hard removal: no warning-only parser, no\n migration, and no custom compatibility error path.\n- Public API consumers can tolerate a breaking settings shape change for this\n removed feature.\n- Historical documents in `docs/plans/`, `docs/brainstorms/`, and\n `docs/superpowers/` are archival and do not need rewriting.\n\n## Sources & References\n\n- Previous feature requirements:\n `docs/brainstorms/2026-04-15-ip-whitelist-requirements.md`\n- Previous implementation plan:\n `docs/plans/2026-04-15-002-feat-ip-allowlist-plan.md`\n- API workflow guidance: `AGENTS.md`\n- OpenAPI source of truth: `docs/public/api-reference/fabro-api.yaml`\n", - "failure_signature": "verify|deterministic|script failed with exit code: ## output from https://github.com/fabro-sh/fabro * branch main -> fetch_head .. main -> origin/main auto-merging cargo.lock auto-merging lib/crates/fabro-api/build.rs auto-merging lib/crates/fabro", + "failure_signature": "", "graph.model_stylesheet": "\n * { model: claude-opus-4-7; }\n ", "internal.node_visit_count": 1, "internal.retry_count.preflight_compile": 0, "internal.retry_count.preflight_lint": 0, + "response.fixup": "Clippy is clean. The merge conflict was the only issue. My fix resolves it by accepting the upstream changes (automation_materializer imports) and dropping the no-longer-needed `IpAllowlistConfig` import that this branch already removed.\n\nSummary:\n- The verify step failed because the auto-merge from origin/main left conflict markers in `lib/crates/fabro-server/src/test_support.rs`.\n- The conflict was between this branch's removal of `crate::ip_allowlist` and origin/main's addition of `crate::automation_materializer` imports.\n- Resolved by keeping the new `automation_materializer` imports and dropping the obsolete `IpAllowlistConfig` import.\n- `cargo check`, `cargo fmt`, and `cargo clippy --workspace --all-targets -- -D warnings` all pass.", "command.output": "blob://sha256/ba184126dad761158723a25e15818b2cb0ac7327ad06fa7513f7a7ce9377eee6", + "internal.retry_count.fixup": 0, + "thread.verify.current_node": "fixup", "response.simplify_opus": "## Review Summary\n\nAll three review agents came back essentially clean.\n\n- **Code Reuse Review** (e7156f0c): Clean. All new code follows existing local patterns (parse + `expect_err`, `body.get(x).is_none()`, `RouterOptions` destructure). No missed utilities.\n- **Code Quality Review** (59feb42b): Clean. Symbol audit confirms full removal of `IpAllowlistConfig`, `IpAllowlist`, `IpAllowEntry`, `LiteralIpAllowEntry`, `ip_allowlist_middleware`, `GitHubMetaResolver`, `github_meta_hooks`, `trusted_proxy_count`, `ServerIpAllowlistSettings`, `ServerIpAllowlistOverrideSettings`, `extract_client_ip`, `GitHubMetaCache`, and frontend `ip_allowlist`/`Count`/`plural` wiring. `deny_unknown_fields` enforces hard removal at the TOML boundary. No stubs, dead imports, or vestigial parameters from the refactor.\n- **Efficiency Review** (51c928ad): Clean. Per-request IP allowlist middleware (two layers — main app and webhook) gone. Startup GitHub `/meta` fetch + on-disk cache gone. `ConnectInfo` plumbing removed from `serve.rs` (both Tcp and Unix variants) and from `tests/it/api/tcp.rs`. No new hot-path work introduced anywhere.\n\n**One out-of-scope finding skipped:** the efficiency agent flagged `install.rs:710` as still using `into_make_service_with_connect_info::()` despite nothing in `install.rs` extracting `ConnectInfo` (the `detect_remote_ip` helper only reads the `x-forwarded-for` header). I verified this is true — it's the last `ConnectInfo` caller in the crate — but it's pre-existing code in the install flow, not introduced by this refactor, and unrelated to IP allowlisting. The agent rated it low-severity cleanup. Per the scope discipline of this refactor (hard removal of IP allowlisting only, no scope creep) and the system prompt's \"only make changes that are directly requested or clearly necessary,\" I'm leaving it alone.\n\nNo fixes applied — the diff was already clean.", - "outcome": "failed", - "last_response": "Review complete.\n\nWhat I checked:\n- Diff against merge base `a992a7d76...`\n- Remaining references to removed IP allowlist symbols/settings outside archival/build output\n- Key config/server/API/web/doc", - "current_node": "verify", + "outcome": "succeeded", + "last_response": "Clippy is clean. The merge conflict was the only issue. My fix resolves it by accepting the upstream changes (automation_materializer imports) and dropping the no-longer-needed `IpAllowlistConfig` imp", + "current_node": "fixup", "internal.retry_count.simplify_opus": 0, - "last_stage": "simplify_gpt", + "last_stage": "fixup", "internal.retry_count.implement": 0, "thread.implement.current_node": "simplify_opus", "internal.run_id": "01KSNYVQCKXPCSCEGWMZ1Q574Z", @@ -1397,7 +1655,7 @@ "thread.toolchain.current_node": "preflight_compile", "response.simplify_gpt": "Review complete.\n\nWhat I checked:\n- Diff against merge base `a992a7d76...`\n- Remaining references to removed IP allowlist symbols/settings outside archival/build output\n- Key config/server/API/web/docs changes for cleanup issues\n- Parallel review agents were launched for reuse, quality, and efficiency review; no actionable cleanup changes were needed.\n\nNo code changes were made during this review.\n\nVerification passed:\n- `cargo nextest run -p fabro-config -p fabro-types -p fabro-api -p fabro-server`\n - 1372 passed\n- `cd apps/fabro-web && bun run typecheck`\n - passed\n- Cleanup search:\n - `rg` found no remaining non-archival references to `ip_allowlist`, `trusted_proxy_count`, `github_meta_hooks`, `IpAllow`, `GitHubMetaResolver`, etc. excluding `target/` and archival docs.\n\nWorking tree is clean.", "internal.retry_count.simplify_gpt": 0, - "internal.thread_id": "simplify_gpt", + "internal.thread_id": "verify", "thread.simplify_opus.current_node": "simplify_gpt" }, "node_outcomes": { @@ -1559,6 +1817,47 @@ "active_time_ms": 1833544 } }, + "fixup": { + "status": "succeeded", + "context_updates": { + "last_response": "Clippy is clean. The merge conflict was the only issue. My fix resolves it by accepting the upstream changes (automation_materializer imports) and dropping the no-longer-needed `IpAllowlistConfig` imp", + "response.fixup": "Clippy is clean. The merge conflict was the only issue. My fix resolves it by accepting the upstream changes (automation_materializer imports) and dropping the no-longer-needed `IpAllowlistConfig` import that this branch already removed.\n\nSummary:\n- The verify step failed because the auto-merge from origin/main left conflict markers in `lib/crates/fabro-server/src/test_support.rs`.\n- The conflict was between this branch's removal of `crate::ip_allowlist` and origin/main's addition of `crate::automation_materializer` imports.\n- Resolved by keeping the new `automation_materializer` imports and dropping the obsolete `IpAllowlistConfig` import.\n- `cargo check`, `cargo fmt`, and `cargo clippy --workspace --all-targets -- -D warnings` all pass.", + "last_stage": "fixup" + }, + "notes": "Stage completed: fixup", + "usage": { + "input": { + "usage": { + "model": { + "provider": "anthropic", + "model_id": "claude-opus-4-7" + }, + "tokens": { + "input_tokens": 26910, + "output_tokens": 3681, + "reasoning_tokens": 0, + "cache_read_tokens": 397541, + "cache_write_tokens": 71523 + } + }, + "facts": { + "algorithm": "anthropic", + "cache_write_5m_tokens": 71523, + "cache_write_1h_tokens": 0 + } + }, + "total_usd_micros": 872363 + }, + "files_touched": [ + "/home/daytona/workspace/fabro/lib/crates/fabro-server/src/test_support.rs" + ], + "timing": { + "wall_time_ms": 0, + "inference_time_ms": 76530, + "tool_time_ms": 67058, + "active_time_ms": 143588 + } + }, "start": { "status": "succeeded", "usage": null @@ -1578,12 +1877,13 @@ } } }, - "next_node_id": "fixup", + "next_node_id": "verify", "node_visits": { "start": 1, "implement": 1, "toolchain": 1, "preflight_compile": 1, + "fixup": 1, "simplify_gpt": 1, "verify": 1, "preflight_lint": 1, @@ -1908,6 +2208,233 @@ }, "state": "succeeded" }, + "fixup@1": { + "first_event_seq": 1301, + "prompt": null, + "response": null, + "completion": null, + "provider_used": { + "mode": "agent", + "provider": "anthropic", + "model": "claude-opus-4-7" + }, + "diff": null, + "script_invocation": null, + "script_timing": null, + "parallel_results": null, + "output": null, + "started_at": "2026-05-28T01:02:46.632314Z", + "handler": "agent", + "usage": { + "input_tokens": 26910, + "output_tokens": 3681, + "total_tokens": 499655, + "reasoning_tokens": 0, + "cache_read_tokens": 397541, + "cache_write_tokens": 71523, + "total_usd_micros": 872363 + }, + "model": { + "provider": "anthropic", + "model_id": "claude-opus-4-7" + }, + "permission_level": "full", + "agent_tools": [ + { + "name": "AskUserQuestion", + "description": "Ask the human one or more questions and wait for their answers before continuing this stage.", + "source": { + "kind": "native" + }, + "category": "other", + "invoked": false + }, + { + "name": "TaskCreate", + "description": "Create pending tasks in the current session. Use concise subjects, descriptions, optional activeForm text, and metadata. Check TaskList first to avoid duplicate tasks.", + "source": { + "kind": "native" + }, + "category": "other", + "invoked": false + }, + { + "name": "TaskGet", + "description": "Get one task by taskId, including subject, status, description, owner, blockedBy, and blocks.", + "source": { + "kind": "native" + }, + "category": "other", + "invoked": false + }, + { + "name": "TaskList", + "description": "List tasks for the current session, including status, owner, and blocking dependencies. Use TaskGet with a taskId for full description and dependency details.", + "source": { + "kind": "native" + }, + "category": "other", + "invoked": false + }, + { + "name": "TaskUpdate", + "description": "Update an existing task's status, text, owner, metadata, or dependencies. Valid statuses are pending, in_progress, completed, and deleted. After completing a task, call TaskList to find newly unblocked work.", + "source": { + "kind": "native" + }, + "category": "other", + "invoked": false + }, + { + "name": "close_agent", + "description": "Close a running subagent that is no longer needed.", + "source": { + "kind": "native" + }, + "category": "subagent", + "invoked": false + }, + { + "name": "edit_file", + "description": "Edit a file by replacing an exact string. The old_string must be an exact match and unique unless replace_all is true; include surrounding context when needed. Read the file first and preserve existing indentation.", + "source": { + "kind": "native" + }, + "category": "write", + "invoked": true + }, + { + "name": "glob", + "description": "Find files by file names using a glob pattern. Use path to choose the search root. Prefer this over shell find or ls when locating repository files.", + "source": { + "kind": "native" + }, + "category": "read", + "invoked": false + }, + { + "name": "grep", + "description": "Search file contents with a regex pattern. Use path to choose the search root, glob_filter to limit matching files, case_insensitive for case folding, and max_results to cap output.", + "source": { + "kind": "native" + }, + "category": "read", + "invoked": true + }, + { + "name": "read_file", + "description": "Read files before editing them. Returns line-numbered text and supports offset/limit for large files. Use this instead of shell cat, head, tail, or sed when inspecting repository files.", + "source": { + "kind": "native" + }, + "category": "read", + "invoked": true + }, + { + "name": "send_input", + "description": "Send a follow-up message to a running subagent when new information or corrected instructions are needed.", + "source": { + "kind": "native" + }, + "category": "subagent", + "invoked": false + }, + { + "name": "shell", + "description": "Execute shell commands for terminal operations, package managers, tests and builds. Use dedicated tools for file reads, file edits, filename searches, and content searches. Provide timeout_ms for long-running commands.", + "source": { + "kind": "native" + }, + "category": "shell", + "invoked": true + }, + { + "name": "spawn_agent", + "description": "Spawn a subagent for independent work or context isolation. Use it for tasks that can proceed separately, and avoid duplicating the same work in the parent session.", + "source": { + "kind": "native" + }, + "category": "subagent", + "invoked": false + }, + { + "name": "wait", + "description": "Wait for a subagent to complete, then use the result to synthesize the outcome for the user.", + "source": { + "kind": "native" + }, + "category": "subagent", + "invoked": false + }, + { + "name": "web_fetch", + "description": "Fetch content from a URL that starts with http:// or https://. Pass a prompt to extract specific information or summarize the page; omit prompt to return the page content.", + "source": { + "kind": "native" + }, + "category": "other", + "invoked": false + }, + { + "name": "web_search", + "description": "Search the web using Brave Search when current external information is needed. Returns result titles, URLs, and descriptions; use web_fetch for a specific URL.", + "source": { + "kind": "native" + }, + "category": "other", + "invoked": false + }, + { + "name": "write_file", + "description": "Create new files, or overwrite an existing file only when replacement is explicitly intended. Prefer edit_file for targeted changes to existing files because write_file overwrites the full file content.", + "source": { + "kind": "native" + }, + "category": "write", + "invoked": false + } + ], + "context_window": { + "provider": "anthropic", + "model": "claude-opus-4-7", + "context_window_tokens": 1000000, + "input_tokens": 39179, + "usage_percent": 3.9179, + "count_method": "response_usage_scaled_breakdown", + "staleness": "live", + "generated_at": "2026-05-28T01:05:10.797693Z", + "event_seq": 1355, + "breakdown": [ + { + "category": "system_prompt", + "tokens": 2762, + "usage_percent": 0.2762 + }, + { + "category": "tools", + "tokens": 3138, + "usage_percent": 0.3138 + }, + { + "category": "memory", + "tokens": 6672, + "usage_percent": 0.6672 + }, + { + "category": "conversation", + "tokens": 26599, + "usage_percent": 2.6599 + }, + { + "category": "other", + "tokens": 8, + "usage_percent": 0.0008 + } + ], + "warnings": [] + }, + "state": "running" + }, "simplify_opus@1": { "first_event_seq": 710, "prompt": null, @@ -2182,7 +2709,12 @@ "first_event_seq": 1291, "prompt": null, "response": null, - "completion": null, + "completion": { + "outcome": "failed", + "notes": null, + "failure_reason": "Script failed with exit code: 1\n\n## output\nFrom https://github.com/fabro-sh/fabro\n * branch main -> FETCH_HEAD\n a992a7d76..475b4ab65 main -> origin/main\nAuto-merging Cargo.lock\nAuto-merging lib/crates/fabro-api/build.rs\nAuto-merging lib/crates/fabro-api/src/lib.rs\nAuto-merging lib/crates/fabro-server/Cargo.toml\nAuto-merging lib/crates/fabro-server/src/lib.rs\nAuto-merging lib/crates/fabro-server/src/serve.rs\nAuto-merging lib/crates/fabro-server/src/server.rs\nAuto-merging lib/crates/fabro-server/src/server/tests.rs\nAuto-merging lib/crates/fabro-server/src/test_support.rs\nCONFLICT (content): Merge conflict in lib/crates/fabro-server/src/test_support.rs\nAutomatic merge failed; fix conflicts and then commit the result.\n", + "timestamp": "2026-05-28T01:02:42.285529Z" + }, "provider_used": null, "diff": null, "script_invocation": { @@ -2190,11 +2722,27 @@ "command": "exec 2>&1\ngit fetch origin main 2>&1 && git merge --no-edit --no-stat origin/main 2>&1 && cargo +nightly-2026-04-14 fmt --all 2>&1 && cargo dev docs refresh 2>&1 && cargo +nightly-2026-04-14 fmt --check --all 2>&1 && { command -v rg >/dev/null 2>&1 || { echo 'rg is required for verify'; exit 127; }; } && ! rg -n 'AuthMode::Disabled|RunAuthMethod|RunSubjectProvenance|\\bActorRef\\b|\\bActorKind\\b|AuthenticatedSubject|AuthenticatedService|AuthorizeRunScoped|AuthorizeRunBlob|AuthorizeStageArtifact|AuthorizeCommandLog|auth_method\\s*==\\s*\"disabled\"' lib/crates apps lib/packages docs/public/api-reference/fabro-api.yaml 2>&1 && cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --workspace --status-level slow --profile ci 2>&1 && cargo dev docs check 2>&1 && bun install --frozen-lockfile 2>&1 && (cd apps/fabro-web && bun run typecheck) 2>&1 && (cd apps/fabro-web && bun run test) 2>&1 && (cd lib/packages/fabro-api-client && bun run typecheck) 2>&1 && cargo dev build -- -p fabro-cli --release 2>&1", "language": "shell" }, - "script_timing": null, + "script_timing": { + "output": "blob://sha256/ba184126dad761158723a25e15818b2cb0ac7327ad06fa7513f7a7ce9377eee6", + "exit_code": 1, + "duration_ms": 1604, + "termination": "exited", + "output_bytes": 711, + "live_streaming": true + }, "parallel_results": null, "output": null, + "output_bytes": 711, + "live_streaming": true, + "termination": "exited", "started_at": "2026-05-28T01:02:40.662746Z", "handler": "command", + "timing": { + "wall_time_ms": 1621, + "inference_time_ms": 0, + "tool_time_ms": 1604, + "active_time_ms": 1604 + }, "usage": { "input_tokens": 0, "output_tokens": 0, @@ -2203,7 +2751,7 @@ "cache_read_tokens": 0, "cache_write_tokens": 0 }, - "state": "running" + "state": "failed" }, "simplify_gpt@1": { "first_event_seq": 1075, diff --git a/stages/008-verify@1/diff.patch b/stages/008-verify@1/diff.patch new file mode 100644 index 000000000..1b2fc4bb3 --- /dev/null +++ b/stages/008-verify@1/diff.patch @@ -0,0 +1,4955 @@ +diff --git a/Cargo.lock b/Cargo.lock +index 7365db53d..2ef51ead4 100644 +--- a/Cargo.lock ++++ b/Cargo.lock +@@ -1889,6 +1889,7 @@ dependencies = [ + "tempfile", + "thiserror 2.0.18", + "tokio", ++ "tokio-tungstenite 0.26.2", + "tokio-util", + "toml 0.8.23", + "toml_edit", +@@ -2407,6 +2408,7 @@ dependencies = [ + "thiserror 2.0.18", + "tokio", + "tokio-stream", ++ "tokio-tungstenite 0.26.2", + "tokio-util", + "toml 0.8.23", + "toml_edit", +diff --git a/lib/crates/fabro-api/build.rs b/lib/crates/fabro-api/build.rs +index 88d465128..3ef344399 100644 +--- a/lib/crates/fabro-api/build.rs ++++ b/lib/crates/fabro-api/build.rs +@@ -617,6 +617,7 @@ fn main() { + ("SandboxTimestamps", "fabro_types::SandboxTimestamps", &[]), + ("AskFabro", "fabro_types::AskFabro", &[]), + ("Automation", "fabro_automation::Automation", &[]), ++ ("AutomationRef", "fabro_types::AutomationRef", &[]), + ("AutomationTarget", "fabro_automation::AutomationTarget", &[ + ]), + ( +diff --git a/lib/crates/fabro-api/src/lib.rs b/lib/crates/fabro-api/src/lib.rs +index 3837275f0..28903a67d 100644 +--- a/lib/crates/fabro-api/src/lib.rs ++++ b/lib/crates/fabro-api/src/lib.rs +@@ -39,32 +39,33 @@ pub mod types { + pub use fabro_types::{ + ActivatedSkill, AgentMcpToolSummary, AgentSkillActivationSource, AgentSkillSummary, + AgentToolCategory, AgentToolSource, AgentToolSummary, AgentToolsAvailableProps, AskFabro, +- AuthMethod, BilledTokenCounts, CommandTermination, Conclusion, CreateVariableRequest, +- DiffStats, DiffSummary, DirtyStatus, EventEnvelope, ExecOutputTail, FailureCategory, +- FailureDetail, FailureSignature, GitContext, IdpIdentity, IntegrationConnectionKind, +- IntegrationConnectionState, IntegrationConnectionStatus, IntegrationProvider, +- IntegrationStatus, InterviewOption, InterviewQuestionRecord, McpServerProjection, +- McpServerStatus, PairId, PairMessageId, PairMessageRecord, PairMessageRequest, PairRecord, +- PairStartRequest, PairStatus, PairTarget, PairTranscriptEntry, PairTranscriptResponse, +- PendingInterviewRecord, PermissionLevel, PreRunPushOutcome, Principal, PullRequest, +- PullRequestDetails, PullRequestDetailsStatus, PullRequestDetailsUnavailableReason, +- PullRequestLink, PullRequestMeta, PullRequestResponse, QuestionType, RepositoryRef, Run, +- RunApproval, RunApprovalState, RunClientProvenance, RunEvent, RunEventDetailContentKind, +- RunEventDetailResponse, RunFailure, RunPairStatusResponse, RunProjection, RunProvenance, +- RunRunnableSource, RunSandbox, RunSandboxFailure, RunSandboxInstance, RunSandboxKind, +- RunSandboxPlan, RunSandboxRuntime, RunServerProvenance, RunSize, SandboxDetails, +- SandboxInfo, SandboxListMeta, SandboxListResponse, SandboxNetwork, SandboxNetworkPolicy, +- SandboxNetworkPolicyMode, SandboxProviderKind, SandboxProviderLookupError, +- SandboxResources, SandboxService, SandboxServiceListResponse, SandboxState, +- SandboxTimestamps, SecretMetadata, SecretType, ServerSettings, SessionDetail, SessionId, +- SessionMessage, SessionRecord, SessionStatus, SessionSummary, SessionTurn, +- SkillsProjection, StageCompletion, StageContextWindow, StageContextWindowBreakdownItem, +- StageContextWindowCategory, StageContextWindowCountMethod, StageContextWindowProjection, +- StageContextWindowStaleness, StageContextWindowUnavailableReason, +- StageContextWindowWarning, StageHandler, StageModelUsage, StageOutcome, StageProjection, +- StageState, SubAgentProjection, SubAgentStatus, SystemActorKind, SystemIntegrationStatus, +- SystemIntegrationsResponse, TodoListProjection, TurnId, UpdateVariableRequest, +- UserPrincipal, Variable, VariableListResponse, WorkflowSettings, ++ AuthMethod, AutomationRef, BilledTokenCounts, CommandTermination, Conclusion, ++ CreateVariableRequest, DiffStats, DiffSummary, DirtyStatus, EventEnvelope, ExecOutputTail, ++ FailureCategory, FailureDetail, FailureSignature, GitContext, IdpIdentity, ++ IntegrationConnectionKind, IntegrationConnectionState, IntegrationConnectionStatus, ++ IntegrationProvider, IntegrationStatus, InterviewOption, InterviewQuestionRecord, ++ McpServerProjection, McpServerStatus, PairId, PairMessageId, PairMessageRecord, ++ PairMessageRequest, PairRecord, PairStartRequest, PairStatus, PairTarget, ++ PairTranscriptEntry, PairTranscriptResponse, PendingInterviewRecord, PermissionLevel, ++ PreRunPushOutcome, Principal, PullRequest, PullRequestDetails, PullRequestDetailsStatus, ++ PullRequestDetailsUnavailableReason, PullRequestLink, PullRequestMeta, PullRequestResponse, ++ QuestionType, RepositoryRef, Run, RunApproval, RunApprovalState, RunClientProvenance, ++ RunEvent, RunEventDetailContentKind, RunEventDetailResponse, RunFailure, ++ RunPairStatusResponse, RunProjection, RunProvenance, RunRunnableSource, RunSandbox, ++ RunSandboxFailure, RunSandboxInstance, RunSandboxKind, RunSandboxPlan, RunSandboxRuntime, ++ RunServerProvenance, RunSize, SandboxDetails, SandboxInfo, SandboxListMeta, ++ SandboxListResponse, SandboxNetwork, SandboxNetworkPolicy, SandboxNetworkPolicyMode, ++ SandboxProviderKind, SandboxProviderLookupError, SandboxResources, SandboxService, ++ SandboxServiceListResponse, SandboxState, SandboxTimestamps, SecretMetadata, SecretType, ++ ServerSettings, SessionDetail, SessionId, SessionMessage, SessionRecord, SessionStatus, ++ SessionSummary, SessionTurn, SkillsProjection, StageCompletion, StageContextWindow, ++ StageContextWindowBreakdownItem, StageContextWindowCategory, StageContextWindowCountMethod, ++ StageContextWindowProjection, StageContextWindowStaleness, ++ StageContextWindowUnavailableReason, StageContextWindowWarning, StageHandler, ++ StageModelUsage, StageOutcome, StageProjection, StageState, SubAgentProjection, ++ SubAgentStatus, SystemActorKind, SystemIntegrationStatus, SystemIntegrationsResponse, ++ TodoListProjection, TurnId, UpdateVariableRequest, UserPrincipal, Variable, ++ VariableListResponse, WorkflowSettings, + }; + + pub use crate::generated::types::*; +diff --git a/lib/crates/fabro-api/tests/run_summary_round_trip.rs b/lib/crates/fabro-api/tests/run_summary_round_trip.rs +index 0c5a70fb5..798448941 100644 +--- a/lib/crates/fabro-api/tests/run_summary_round_trip.rs ++++ b/lib/crates/fabro-api/tests/run_summary_round_trip.rs +@@ -3,9 +3,9 @@ use std::collections::HashMap; + + use chrono::{TimeZone, Utc}; + use fabro_api::types::{ +- RepositoryRef as ApiRepositoryRef, Run as ApiRun, RunApproval as ApiRunApproval, +- RunApprovalState as ApiRunApprovalState, RunRunnableSource as ApiRunRunnableSource, +- RunSize as ApiRunSize, ++ AutomationRef as ApiAutomationRef, RepositoryRef as ApiRepositoryRef, Run as ApiRun, ++ RunApproval as ApiRunApproval, RunApprovalState as ApiRunApprovalState, ++ RunRunnableSource as ApiRunRunnableSource, RunSize as ApiRunSize, + }; + use fabro_types::status::{RunStatus, SuccessReason}; + use fabro_types::{ +@@ -24,6 +24,7 @@ fn run_summary_reuses_domain_types() { + assert_same_type::(); + assert_same_type::(); + assert_same_type::(); ++ assert_same_type::(); + } + + #[test] +diff --git a/lib/crates/fabro-cli/Cargo.toml b/lib/crates/fabro-cli/Cargo.toml +index a4d15b5b5..d0b8b7373 100644 +--- a/lib/crates/fabro-cli/Cargo.toml ++++ b/lib/crates/fabro-cli/Cargo.toml +@@ -61,6 +61,7 @@ anyhow.workspace = true + miette.workspace = true + dotenvy.workspace = true + tokio.workspace = true ++tokio-tungstenite.workspace = true + tracing.workspace = true + tracing-subscriber.workspace = true + tracing-appender.workspace = true +@@ -128,3 +129,4 @@ fabro-test = { workspace = true } + fabro-macros = { path = "../fabro-macros" } + hkdf.workspace = true + reqwest = { workspace = true, features = ["cookies"] } ++tokio = { workspace = true, features = ["test-util", "macros"] } +diff --git a/lib/crates/fabro-cli/src/commands/run/runner.rs b/lib/crates/fabro-cli/src/commands/run/runner.rs +index fb3b31d43..ac1a2bbd0 100644 +--- a/lib/crates/fabro-cli/src/commands/run/runner.rs ++++ b/lib/crates/fabro-cli/src/commands/run/runner.rs +@@ -1,10 +1,4 @@ +-#![expect( +- clippy::disallowed_types, +- reason = "sync CLI `run` subprocess wrapper: reads server subprocess stdout line-by-line via \ +- std::io::BufReader; not on a Tokio path" +-)] +- +-use std::io::{BufRead as StdBufRead, BufReader as StdBufReader}; ++use std::collections::{HashSet, VecDeque}; + use std::path::{Path, PathBuf}; + use std::sync::Arc; + use std::time::Duration; +@@ -12,10 +6,14 @@ use std::time::Duration; + use anyhow::{Context, Result, anyhow}; + use async_trait::async_trait; + use fabro_api::types::RunManifest; ++use fabro_client::ServerTarget; + use fabro_config::user::active_settings_path; + use fabro_config::{ServerSettingsBuilder, Storage, load_llm_catalog_settings}; + use fabro_interview::{ +- AnswerSubmission, ControlInterviewer, WorkerControlEnvelope, WorkerControlMessage, ++ AnswerSubmission, ControlInterviewer, WORKER_CONTROL_INVALID_CURSOR_REASON, ++ WORKER_CONTROL_PONG_TIMEOUT_REASON, WORKER_CONTROL_WS_LIVENESS_TIMEOUT, ++ WORKER_CONTROL_WS_PING_INTERVAL, WorkerControlDeliveryFrame, WorkerControlEnvelope, ++ WorkerControlMessage, + }; + use fabro_model::Catalog; + use fabro_server::run_tool_manifest; +@@ -34,11 +32,23 @@ use fabro_workflow::operations::{self, StartServices}; + use fabro_workflow::run_control::RunControlState; + use fabro_workflow::runtime_store::{RunStoreBackend, RunStoreHandle}; + use fabro_workflow::services::FabroRunToolServices; ++use futures::{SinkExt, StreamExt}; + use jsonwebtoken::dangerous::insecure_decode; ++#[cfg(test)] ++use tokio::io::DuplexStream; ++use tokio::net::TcpStream; ++#[cfg(unix)] ++use tokio::net::UnixStream; + #[cfg(unix)] + use tokio::signal::unix::{SignalKind, signal}; +-use tokio::sync::{Mutex, RwLock as AsyncRwLock, mpsc}; +-use tokio::time::sleep; ++use tokio::sync::{Mutex, RwLock as AsyncRwLock, oneshot}; ++use tokio::task::JoinHandle; ++use tokio::time::{self, Instant, MissedTickBehavior}; ++use tokio_tungstenite::tungstenite::client::IntoClientRequest; ++use tokio_tungstenite::tungstenite::http::{HeaderValue, Request as WebSocketRequest, header}; ++use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode; ++use tokio_tungstenite::tungstenite::protocol::{self, Message as WebSocketMessage}; ++use tokio_tungstenite::{MaybeTlsStream, WebSocketStream, connect_async, tungstenite}; + use tokio_util::sync::CancellationToken; + + use crate::args::RunWorkerMode; +@@ -75,7 +85,7 @@ pub(crate) async fn execute( + let _ = fabro_proc::title_init(); + set_worker_title(&run_id, initial_worker_title_phase(mode)); + +- let target = server.parse::()?; ++ let target = server.parse::()?; + let client = server_client::connect_server_target_with_bearer(&target, worker_token).await?; + let run_store = HttpRunStore::connect(run_id, client.clone_for_reuse()).await?; + let run_state = run_store +@@ -110,13 +120,24 @@ pub(crate) async fn execute( + let cancel_token = CancellationToken::new(); + let emitter = Arc::new(Emitter::new(run_id)); + let steering_hub = Arc::new(fabro_workflow::SteeringHub::new(Arc::clone(&emitter))); +- spawn_worker_control_stream( +- Arc::clone(&interviewer), +- cancel_token.clone(), +- Arc::clone(&steering_hub), +- )?; + let run_control = RunControlState::new(); + install_signal_handlers(Arc::clone(&run_control), cancel_token.clone())?; ++ let mut control_manager = if run_state.status.is_terminal() { ++ None ++ } else { ++ Some(spawn_worker_control_manager( ++ target.clone(), ++ run_id, ++ worker_token.to_owned(), ++ Arc::clone(&interviewer), ++ cancel_token.clone(), ++ Arc::clone(&steering_hub), ++ Arc::clone(&run_control), ++ )) ++ }; ++ if let Some(control_manager) = &mut control_manager { ++ control_manager.wait_for_first_connection().await?; ++ } + let vault = load_worker_vault(storage_dir.as_deref())?; + let github_app = { + let vault_guard = match &vault { +@@ -158,13 +179,26 @@ pub(crate) async fn execute( + fabro_run_tools, + }; + +- match mode { +- RunWorkerMode::Start => { +- operations::start(&run_dir, services).await?; ++ let execution = async { ++ match mode { ++ RunWorkerMode::Start => operations::start(&run_dir, services).await, ++ RunWorkerMode::Resume => operations::resume(&run_dir, services).await, + } +- RunWorkerMode::Resume => { +- operations::resume(&run_dir, services).await?; ++ }; ++ ++ if let Some(mut control_manager) = control_manager { ++ tokio::select! { ++ result = execution => { ++ control_manager.finish(); ++ result?; ++ } ++ fatal = control_manager.fatal_control_loss() => { ++ control_manager.finish(); ++ return Err(fatal); ++ } + } ++ } else { ++ execution.await?; + } + + Ok(()) +@@ -254,96 +288,510 @@ fn load_worker_vault(storage_dir: Option<&Path>) -> Result, +- cancel_token: CancellationToken, +- steering_hub: Arc, +-) -> Result<()> { +- let (event_tx, event_rx) = mpsc::unbounded_channel(); +- tokio::spawn(handle_worker_control_stream_events( +- interviewer, +- cancel_token, +- steering_hub, +- event_rx, +- )); +- std::thread::Builder::new() +- .name("fabro-worker-control".to_string()) +- .spawn(move || { +- read_worker_control_stream_blocking(StdBufReader::new(std::io::stdin()), &event_tx); +- }) +- .context("failed to spawn worker control reader thread")?; +- Ok(()) ++#[derive(Default)] ++struct AppliedWorkerControlDeliveryIds { ++ last: Option, ++ order: VecDeque, ++ recent: HashSet, + } + +-fn read_worker_control_stream_blocking( +- mut reader: R, +- event_tx: &mpsc::UnboundedSender, +-) where +- R: StdBufRead, +-{ +- let mut line = String::new(); +- loop { +- line.clear(); +- match reader.read_line(&mut line) { +- Ok(0) | Err(_) => { +- let _ = event_tx.send(WorkerControlStreamEvent::Eof); +- break; +- } +- Ok(_) => { +- let line = line.trim_end_matches(['\r', '\n']).to_string(); +- if event_tx.send(WorkerControlStreamEvent::Line(line)).is_err() { +- break; +- } ++impl AppliedWorkerControlDeliveryIds { ++ fn last_applied_id(&self) -> Option<&str> { ++ self.last.as_deref() ++ } ++ ++ fn contains(&self, id: &str) -> bool { ++ self.recent.contains(id) ++ } ++ ++ fn record(&mut self, id: String) { ++ if !self.recent.insert(id.clone()) { ++ self.last = Some(id); ++ return; ++ } ++ self.order.push_back(id.clone()); ++ self.last = Some(id); ++ while self.order.len() > WORKER_CONTROL_APPLIED_ID_DEDUPE_CAPACITY { ++ if let Some(evicted) = self.order.pop_front() { ++ self.recent.remove(&evicted); + } + } + } + } + +-async fn handle_worker_control_stream_events( ++struct WorkerControlManagerHandle { ++ first_connection: Option>>, ++ fatal: Option>, ++ done: CancellationToken, ++ task: JoinHandle<()>, ++} ++ ++impl WorkerControlManagerHandle { ++ async fn wait_for_first_connection(&mut self) -> Result<()> { ++ let receiver = self ++ .first_connection ++ .take() ++ .context("worker control first-connection receiver missing")?; ++ receiver ++ .await ++ .context("worker control manager stopped before first connection")? ++ } ++ ++ async fn fatal_control_loss(&mut self) -> anyhow::Error { ++ let Some(receiver) = self.fatal.take() else { ++ return anyhow!("worker control fatal receiver missing"); ++ }; ++ receiver ++ .await ++ .unwrap_or_else(|_| anyhow!("worker control manager stopped before workflow completed")) ++ } ++ ++ fn finish(&self) { ++ self.done.cancel(); ++ self.task.abort(); ++ } ++} ++ ++#[derive(Debug)] ++struct WorkerControlStreamConnectRequest { ++ request: WebSocketRequest<()>, ++ unix_socket_path: Option, ++} ++ ++impl WorkerControlStreamConnectRequest { ++ fn request_for_tungstenite(&self) -> WebSocketRequest<()> { ++ self.request.clone() ++ } ++ ++ #[cfg(test)] ++ fn uri(&self) -> String { ++ self.request.uri().to_string() ++ } ++ ++ #[cfg(test)] ++ fn authorization(&self) -> Option<&str> { ++ self.request ++ .headers() ++ .get(header::AUTHORIZATION) ++ .and_then(|value| value.to_str().ok()) ++ } ++} ++ ++enum WorkerControlSocket { ++ Tcp(Box>>), ++ #[cfg(unix)] ++ Unix(Box>), ++ #[cfg(test)] ++ Test(Box>), ++} ++ ++impl WorkerControlSocket { ++ async fn send(&mut self, message: WebSocketMessage) -> Result<(), tungstenite::Error> { ++ match self { ++ Self::Tcp(socket) => socket.send(message).await, ++ #[cfg(unix)] ++ Self::Unix(socket) => socket.send(message).await, ++ #[cfg(test)] ++ Self::Test(socket) => socket.send(message).await, ++ } ++ } ++ ++ async fn next(&mut self) -> Option> { ++ match self { ++ Self::Tcp(socket) => socket.next().await, ++ #[cfg(unix)] ++ Self::Unix(socket) => socket.next().await, ++ #[cfg(test)] ++ Self::Test(socket) => socket.next().await, ++ } ++ } ++} ++ ++#[derive(Debug)] ++enum WorkerControlConnectError { ++ InvalidCursor, ++ Other(anyhow::Error), ++} ++ ++fn spawn_worker_control_manager( ++ target: ServerTarget, ++ run_id: RunId, ++ worker_token: String, ++ interviewer: Arc, ++ cancel_token: CancellationToken, ++ steering_hub: Arc, ++ run_control: Arc, ++) -> WorkerControlManagerHandle { ++ let (first_tx, first_rx) = oneshot::channel(); ++ let (fatal_tx, fatal_rx) = oneshot::channel(); ++ let done = CancellationToken::new(); ++ let task_done = done.clone(); ++ let task = tokio::spawn(async move { ++ run_worker_control_manager( ++ target, ++ run_id, ++ worker_token, ++ interviewer, ++ cancel_token, ++ steering_hub, ++ run_control, ++ task_done, ++ first_tx, ++ fatal_tx, ++ ) ++ .await; ++ }); ++ WorkerControlManagerHandle { ++ first_connection: Some(first_rx), ++ fatal: Some(fatal_rx), ++ done, ++ task, ++ } ++} ++ ++#[allow( ++ clippy::too_many_arguments, ++ reason = "Worker control manager owns the worker-side control dependencies." ++)] ++async fn run_worker_control_manager( ++ target: ServerTarget, ++ run_id: RunId, ++ worker_token: String, + interviewer: Arc, + cancel_token: CancellationToken, + steering_hub: Arc, +- mut event_rx: mpsc::UnboundedReceiver, ++ run_control: Arc, ++ done: CancellationToken, ++ first_tx: oneshot::Sender>, ++ fatal_tx: oneshot::Sender, + ) { +- while let Some(event) = event_rx.recv().await { +- match event { +- WorkerControlStreamEvent::Line(line) => { +- apply_worker_control_line(&interviewer, &cancel_token, &steering_hub, &line).await; ++ let mut first_tx = Some(first_tx); ++ let mut fatal_tx = Some(fatal_tx); ++ let mut backoff = WORKER_CONTROL_RECONNECT_INITIAL_BACKOFF; ++ let mut applied_ids = AppliedWorkerControlDeliveryIds::default(); ++ ++ while !done.is_cancelled() { ++ let request = match build_worker_control_stream_request( ++ &target, ++ &run_id, ++ &worker_token, ++ applied_ids.last_applied_id(), ++ ) { ++ Ok(request) => request, ++ Err(err) => { ++ report_fatal_control_loss( ++ &interviewer, ++ &cancel_token, ++ &mut first_tx, ++ &mut fatal_tx, ++ format!("failed to build worker control stream request: {err:#}"), ++ ) ++ .await; ++ return; ++ } ++ }; ++ ++ match connect_worker_control_stream(request).await { ++ Ok(mut socket) => { ++ if let Some(first_tx) = first_tx.take() { ++ let _ = first_tx.send(Ok(())); ++ } ++ backoff = WORKER_CONTROL_RECONNECT_INITIAL_BACKOFF; ++ match handle_worker_control_socket( ++ &mut socket, ++ &interviewer, ++ &cancel_token, ++ &steering_hub, ++ &run_control, ++ &mut applied_ids, ++ &done, ++ ) ++ .await ++ { ++ Ok(()) => {} ++ Err(WorkerControlConnectError::InvalidCursor) => { ++ report_fatal_control_loss( ++ &interviewer, ++ &cancel_token, ++ &mut first_tx, ++ &mut fatal_tx, ++ "worker control stream replay cursor is invalid".to_string(), ++ ) ++ .await; ++ return; ++ } ++ Err(WorkerControlConnectError::Other(err)) => { ++ tracing::debug!(error = %err, "Worker control stream disconnected"); ++ } ++ } + } +- WorkerControlStreamEvent::Eof => { +- interviewer.interrupt_all().await; ++ Err(WorkerControlConnectError::InvalidCursor) => { ++ report_fatal_control_loss( ++ &interviewer, ++ &cancel_token, ++ &mut first_tx, ++ &mut fatal_tx, ++ "worker control stream replay cursor is invalid".to_string(), ++ ) ++ .await; + return; + } ++ Err(WorkerControlConnectError::Other(err)) => { ++ tracing::debug!(error = %err, "Worker control stream connection failed"); ++ } + } ++ ++ sleep_or_done(&done, backoff).await; ++ backoff = next_worker_control_reconnect_backoff(backoff); + } ++} + ++async fn report_fatal_control_loss( ++ interviewer: &ControlInterviewer, ++ cancel_token: &CancellationToken, ++ first_tx: &mut Option>>, ++ fatal_tx: &mut Option>, ++ detail: String, ++) { ++ let message = format!("worker control channel lost: {detail}"); + interviewer.interrupt_all().await; ++ if let Some(first_tx) = first_tx.take() { ++ let _ = first_tx.send(Err(anyhow!(message.clone()))); ++ } ++ if let Some(fatal_tx) = fatal_tx.take() { ++ let _ = fatal_tx.send(anyhow!(message)); ++ } ++ cancel_token.cancel(); ++} ++ ++async fn sleep_or_done(done: &CancellationToken, delay: Duration) { ++ tokio::select! { ++ () = done.cancelled() => {} ++ () = time::sleep(delay) => {} ++ } ++} ++ ++fn next_worker_control_reconnect_backoff(current: Duration) -> Duration { ++ current ++ .saturating_mul(2) ++ .min(WORKER_CONTROL_RECONNECT_MAX_BACKOFF) ++} ++ ++fn build_worker_control_stream_request( ++ target: &ServerTarget, ++ run_id: &RunId, ++ worker_token: &str, ++ after: Option<&str>, ++) -> Result { ++ let (url, unix_socket_path) = match target { ++ ServerTarget::HttpUrl(_) => { ++ let base = target ++ .as_http_url() ++ .context("HTTP server target missing URL")?; ++ let websocket_base = if let Some(rest) = base.strip_prefix("http://") { ++ format!("ws://{rest}") ++ } else if let Some(rest) = base.strip_prefix("https://") { ++ format!("wss://{rest}") ++ } else { ++ anyhow::bail!("unsupported server URL scheme"); ++ }; ++ ( ++ worker_control_stream_url(&websocket_base, run_id, after), ++ None, ++ ) ++ } ++ ServerTarget::UnixSocket(path) => { ++ let url = worker_control_stream_url("ws://fabro", run_id, after); ++ (url, Some(path.as_path().to_path_buf())) ++ } ++ }; ++ let mut request = url ++ .as_str() ++ .into_client_request() ++ .context("failed to build worker control stream request")?; ++ request.headers_mut().insert( ++ header::AUTHORIZATION, ++ HeaderValue::from_str(&format!("Bearer {worker_token}")) ++ .context("failed to build worker control stream authorization header")?, ++ ); ++ Ok(WorkerControlStreamConnectRequest { ++ request, ++ unix_socket_path, ++ }) + } + +-async fn apply_worker_control_line( ++fn worker_control_stream_url(base: &str, run_id: &RunId, after: Option<&str>) -> String { ++ let mut url = format!("{base}/api/v1/runs/{run_id}/worker/control-stream"); ++ if let Some(after) = after { ++ url.push_str("?after="); ++ url.push_str(after); ++ } ++ url ++} ++ ++async fn connect_worker_control_stream( ++ request: WorkerControlStreamConnectRequest, ++) -> Result { ++ if let Some(path) = request.unix_socket_path.as_ref() { ++ #[cfg(unix)] ++ { ++ let ws_request = request.request_for_tungstenite(); ++ let stream = UnixStream::connect(path) ++ .await ++ .map_err(|err| WorkerControlConnectError::Other(anyhow::Error::new(err)))?; ++ let (socket, _) = tokio_tungstenite::client_async(ws_request, stream) ++ .await ++ .map_err(classify_tungstenite_error)?; ++ Ok(WorkerControlSocket::Unix(Box::new(socket))) ++ } ++ #[cfg(not(unix))] ++ { ++ let _ = path; ++ Err(WorkerControlConnectError::Other(anyhow!( ++ "Unix-socket worker control stream is not supported on this platform" ++ ))) ++ } ++ } else { ++ let (socket, _) = connect_async(request.request) ++ .await ++ .map_err(classify_tungstenite_error)?; ++ Ok(WorkerControlSocket::Tcp(Box::new(socket))) ++ } ++} ++ ++fn classify_tungstenite_error(error: tungstenite::Error) -> WorkerControlConnectError { ++ if let tungstenite::Error::Http(response) = &error { ++ if response.status().as_u16() == 410 { ++ return WorkerControlConnectError::InvalidCursor; ++ } ++ } ++ WorkerControlConnectError::Other(anyhow::Error::new(error)) ++} ++ ++async fn handle_worker_control_socket( ++ socket: &mut WorkerControlSocket, + interviewer: &ControlInterviewer, + cancel_token: &CancellationToken, + steering_hub: &fabro_workflow::SteeringHub, +- line: &str, +-) { +- if line.trim().is_empty() { +- return; ++ run_control: &RunControlState, ++ applied_ids: &mut AppliedWorkerControlDeliveryIds, ++ done: &CancellationToken, ++) -> Result<(), WorkerControlConnectError> { ++ let mut ping_interval = time::interval(WORKER_CONTROL_WS_PING_INTERVAL); ++ ping_interval.set_missed_tick_behavior(MissedTickBehavior::Delay); ++ let mut last_liveness = Instant::now(); ++ let liveness_timeout = time::sleep_until(last_liveness + WORKER_CONTROL_WS_LIVENESS_TIMEOUT); ++ tokio::pin!(liveness_timeout); ++ ++ loop { ++ liveness_timeout ++ .as_mut() ++ .reset(last_liveness + WORKER_CONTROL_WS_LIVENESS_TIMEOUT); ++ ++ tokio::select! { ++ () = done.cancelled() => return Ok(()), ++ _ = ping_interval.tick() => { ++ socket ++ .send(WebSocketMessage::Ping(Vec::new().into())) ++ .await ++ .map_err(|err| WorkerControlConnectError::Other(anyhow::Error::new(err)))?; ++ } ++ () = &mut liveness_timeout => { ++ let _ = socket ++ .send(WebSocketMessage::Close(Some(protocol::CloseFrame { ++ code: CloseCode::Away, ++ reason: WORKER_CONTROL_PONG_TIMEOUT_REASON.into(), ++ }))) ++ .await; ++ return Err(WorkerControlConnectError::Other(anyhow!( ++ "worker control WebSocket liveness timed out" ++ ))); ++ } ++ message = socket.next() => { ++ let Some(message) = message else { ++ return Ok(()); ++ }; ++ match message { ++ Ok(WebSocketMessage::Text(text)) => { ++ last_liveness = Instant::now(); ++ let frame = serde_json::from_str::(text.as_str()) ++ .map_err(|err| WorkerControlConnectError::Other(anyhow::Error::new(err)))?; ++ apply_worker_control_delivery_frame( ++ interviewer, ++ cancel_token, ++ steering_hub, ++ run_control, ++ applied_ids, ++ frame, ++ ) ++ .await; ++ } ++ Ok(WebSocketMessage::Ping(payload)) => { ++ last_liveness = Instant::now(); ++ socket ++ .send(WebSocketMessage::Pong(payload)) ++ .await ++ .map_err(|err| WorkerControlConnectError::Other(anyhow::Error::new(err)))?; ++ } ++ Ok(WebSocketMessage::Pong(_) | WebSocketMessage::Binary(_)) => { ++ last_liveness = Instant::now(); ++ } ++ Ok(WebSocketMessage::Close(frame)) => { ++ if frame.as_ref().is_some_and(|frame| { ++ frame.reason.as_str() == WORKER_CONTROL_INVALID_CURSOR_REASON ++ }) { ++ return Err(WorkerControlConnectError::InvalidCursor); ++ } ++ return Ok(()); ++ } ++ Ok(WebSocketMessage::Frame(_)) => {} ++ Err(err) => { ++ return Err(WorkerControlConnectError::Other(anyhow::Error::new(err))); ++ } ++ } ++ } ++ } + } ++} + +- let Ok(message) = serde_json::from_str::(line) else { +- return; +- }; ++async fn apply_worker_control_delivery_frame( ++ interviewer: &ControlInterviewer, ++ cancel_token: &CancellationToken, ++ steering_hub: &fabro_workflow::SteeringHub, ++ run_control: &RunControlState, ++ applied_ids: &mut AppliedWorkerControlDeliveryIds, ++ frame: WorkerControlDeliveryFrame, ++) -> bool { ++ // Duplicate ids cannot reach us under normal operation: the server replays ++ // strictly after the last applied id. Guard against a server-side bug or ++ // reconnect race by ignoring recently-applied delivery ids. ++ if applied_ids.contains(&frame.id) { ++ return false; ++ } ++ let frame_id = frame.id; ++ apply_worker_control_message( ++ interviewer, ++ cancel_token, ++ steering_hub, ++ run_control, ++ frame.envelope, ++ ) ++ .await; ++ applied_ids.record(frame_id); ++ true ++} + ++async fn apply_worker_control_message( ++ interviewer: &ControlInterviewer, ++ cancel_token: &CancellationToken, ++ steering_hub: &fabro_workflow::SteeringHub, ++ run_control: &RunControlState, ++ message: WorkerControlEnvelope, ++) { + match message.message { + WorkerControlMessage::InterviewAnswer { qid, answer, actor } => { + let _ = interviewer +@@ -354,6 +802,12 @@ async fn apply_worker_control_line( + cancel_token.cancel(); + interviewer.interrupt_all().await; + } ++ WorkerControlMessage::RunPause => { ++ run_control.request_pause(); ++ } ++ WorkerControlMessage::RunUnpause => { ++ run_control.request_unpause(); ++ } + WorkerControlMessage::Steer { text, actor } => { + steering_hub.deliver_steer(text, Some(actor)); + } +@@ -485,7 +939,7 @@ impl HttpRunStore { + Err(err) => last_error = Some(err), + } + if let Some(delay) = RUN_STORE_RETRY_DELAYS.get(attempt) { +- sleep(*delay).await; ++ time::sleep(*delay).await; + } + } + Err(last_error +@@ -757,10 +1211,14 @@ fn install_signal_handlers( + )] + mod tests { + use std::sync::Arc; ++ use std::time::Duration; + + use chrono::Utc; ++ use fabro_client::ServerTarget; + use fabro_config::Storage; +- use fabro_interview::{AnswerValue, ControlInterviewer, Interviewer, Question}; ++ use fabro_interview::{ ++ AnswerValue, ControlInterviewer, Interviewer, Question, WorkerControlEnvelope, ++ }; + use fabro_types::run_event::{ + InterviewCompletedProps, InterviewStartedProps, RunCompletedProps, RunControlEffectProps, + RunFailedProps, RunStatusTransitionProps, +@@ -771,12 +1229,17 @@ mod tests { + }; + use fabro_vault::{SecretType, Vault}; + use fabro_workflow::event::RunEventSink; ++ use fabro_workflow::run_control::RunControlState; ++ use tokio::time; ++ use tokio_tungstenite::tungstenite::protocol::{Message as TestWebSocketMessage, Role}; + use tokio_util::sync::CancellationToken; + + use super::{ +- WorkerControlStreamEvent, WorkerTitlePhase, apply_worker_control_line, +- handle_worker_control_stream_events, initial_worker_title_phase, load_worker_vault, +- read_worker_control_stream_blocking, stamp_system_worker, worker_title, ++ AppliedWorkerControlDeliveryIds, WorkerControlConnectError, WorkerControlSocket, ++ WorkerTitlePhase, apply_worker_control_delivery_frame, apply_worker_control_message, ++ build_worker_control_stream_request, connect_worker_control_stream, ++ handle_worker_control_socket, initial_worker_title_phase, load_worker_vault, ++ next_worker_control_reconnect_backoff, stamp_system_worker, worker_title, + worker_title_phase_for_event, + }; + use crate::args::RunWorkerMode; +@@ -1018,20 +1481,28 @@ mod tests { + } + + #[tokio::test] +- async fn worker_control_line_routes_answer_by_question_id() { ++ async fn worker_control_routes_answer_by_question_id() { + let interviewer = Arc::new(ControlInterviewer::new()); + let cancel_token = CancellationToken::new(); ++ let run_control = RunControlState::new(); + let mut question = Question::new("Approve?", QuestionType::YesNo); + question.id = "q-1".to_string(); + let ask_interviewer = Arc::clone(&interviewer); + let answer_task = tokio::spawn(async move { ask_interviewer.ask(question).await }); + + let hub = test_steering_hub(); +- apply_worker_control_line( ++ apply_worker_control_message( + &interviewer, + &cancel_token, + &hub, +- r#"{"v":1,"type":"interview.answer","qid":"q-1","answer":{"kind":"yes"},"actor":{"kind":"system","system_kind":"engine"}}"#, ++ &run_control, ++ WorkerControlEnvelope::interview_answer( ++ "q-1", ++ fabro_interview::AnswerSubmission::system( ++ fabro_interview::Answer::yes(), ++ fabro_types::SystemActorKind::Engine, ++ ), ++ ), + ) + .await; + +@@ -1041,9 +1512,10 @@ mod tests { + } + + #[tokio::test] +- async fn worker_control_line_cancel_sets_cancel_token_and_interrupts_pending_interviews() { ++ async fn worker_control_cancel_sets_cancel_token_and_interrupts_pending_interviews() { + let interviewer = Arc::new(ControlInterviewer::new()); + let cancel_token = CancellationToken::new(); ++ let run_control = RunControlState::new(); + let mut question = Question::new("Approve?", QuestionType::YesNo); + question.id = "q-1".to_string(); + let ask_interviewer = Arc::clone(&interviewer); +@@ -1051,11 +1523,12 @@ mod tests { + tokio::task::yield_now().await; + + let hub = test_steering_hub(); +- apply_worker_control_line( ++ apply_worker_control_message( + &interviewer, + &cancel_token, + &hub, +- r#"{"v":1,"type":"run.cancel"}"#, ++ &run_control, ++ WorkerControlEnvelope::cancel_run(), + ) + .await; + +@@ -1065,57 +1538,206 @@ mod tests { + } + + #[tokio::test] +- async fn blocking_worker_control_stream_emits_lines_and_eof() { +- let (event_tx, mut event_rx) = tokio::sync::mpsc::unbounded_channel(); ++ async fn worker_control_pause_and_unpause_route_to_run_control() { ++ let interviewer = Arc::new(ControlInterviewer::new()); ++ let cancel_token = CancellationToken::new(); ++ let run_control = RunControlState::new(); ++ let hub = test_steering_hub(); + +- read_worker_control_stream_blocking( +- std::io::Cursor::new( +- b"{\"v\":1,\"type\":\"run.cancel\"}\n{\"v\":1,\"type\":\"interview.answer\",\"qid\":\"q-1\",\"answer\":{\"kind\":\"yes\"},\"actor\":{\"kind\":\"system\",\"system_kind\":\"engine\"}}\n", +- ), +- &event_tx, ++ apply_worker_control_message( ++ &interviewer, ++ &cancel_token, ++ &hub, ++ &run_control, ++ WorkerControlEnvelope::pause_run(), ++ ) ++ .await; ++ assert!(run_control.pause_requested()); ++ ++ apply_worker_control_message( ++ &interviewer, ++ &cancel_token, ++ &hub, ++ &run_control, ++ WorkerControlEnvelope::unpause_run(), ++ ) ++ .await; ++ assert!(!run_control.pause_requested()); ++ } ++ ++ #[tokio::test] ++ async fn duplicate_delivery_ids_are_not_applied_twice() { ++ let interviewer = Arc::new(ControlInterviewer::new()); ++ let cancel_token = CancellationToken::new(); ++ let run_control = RunControlState::new(); ++ let hub = test_steering_hub(); ++ let mut applied_ids = AppliedWorkerControlDeliveryIds::default(); ++ let frame = fabro_interview::WorkerControlDeliveryFrame { ++ id: "local:1".to_string(), ++ envelope: WorkerControlEnvelope::pause_run(), ++ }; ++ ++ assert!( ++ apply_worker_control_delivery_frame( ++ &interviewer, ++ &cancel_token, ++ &hub, ++ &run_control, ++ &mut applied_ids, ++ frame.clone(), ++ ) ++ .await ++ ); ++ assert!( ++ !apply_worker_control_delivery_frame( ++ &interviewer, ++ &cancel_token, ++ &hub, ++ &run_control, ++ &mut applied_ids, ++ frame, ++ ) ++ .await ++ ); ++ ++ assert_eq!(applied_ids.last_applied_id(), Some("local:1")); ++ } ++ ++ #[test] ++ fn worker_control_request_construction_for_http_targets() { ++ let run_id = fixtures::RUN_1; ++ let request = build_worker_control_stream_request( ++ &ServerTarget::http_url("http://example.com:3000").unwrap(), ++ &run_id, ++ "worker-token", ++ None, ++ ) ++ .unwrap(); ++ assert_eq!( ++ request.uri(), ++ format!("ws://example.com:3000/api/v1/runs/{run_id}/worker/control-stream") ++ ); ++ assert_eq!(request.authorization(), Some("Bearer worker-token")); ++ ++ let reconnect = build_worker_control_stream_request( ++ &ServerTarget::http_url("https://example.com").unwrap(), ++ &run_id, ++ "worker-token", ++ Some("local:42"), ++ ) ++ .unwrap(); ++ assert_eq!( ++ reconnect.uri(), ++ format!("wss://example.com/api/v1/runs/{run_id}/worker/control-stream?after=local:42") + ); ++ } ++ ++ #[cfg(unix)] ++ #[test] ++ fn worker_control_request_construction_for_unix_socket_targets() { ++ let run_id = fixtures::RUN_1; ++ let request = build_worker_control_stream_request( ++ &ServerTarget::unix_socket_path("/tmp/fabro.sock").unwrap(), ++ &run_id, ++ "worker-token", ++ Some("local:42"), ++ ) ++ .unwrap(); + + assert_eq!( +- event_rx.try_recv(), +- Ok(WorkerControlStreamEvent::Line( +- r#"{"v":1,"type":"run.cancel"}"#.to_string() +- )) ++ request.uri(), ++ format!("ws://fabro/api/v1/runs/{run_id}/worker/control-stream?after=local:42") + ); + assert_eq!( +- event_rx.try_recv(), +- Ok(WorkerControlStreamEvent::Line( +- r#"{"v":1,"type":"interview.answer","qid":"q-1","answer":{"kind":"yes"},"actor":{"kind":"system","system_kind":"engine"}}"# +- .to_string() +- )) ++ request.unix_socket_path.as_deref(), ++ Some(std::path::Path::new("/tmp/fabro.sock")) + ); +- assert_eq!(event_rx.try_recv(), Ok(WorkerControlStreamEvent::Eof)); + } + +- #[tokio::test] +- async fn worker_control_event_loop_eof_interrupts_pending_interviews() { ++ #[test] ++ fn worker_control_reconnect_backoff_is_bounded() { ++ assert_eq!( ++ next_worker_control_reconnect_backoff(Duration::from_millis(100)), ++ Duration::from_millis(200) ++ ); ++ assert_eq!( ++ next_worker_control_reconnect_backoff(Duration::from_secs(4)), ++ Duration::from_secs(5) ++ ); ++ assert_eq!( ++ next_worker_control_reconnect_backoff(Duration::from_secs(5)), ++ Duration::from_secs(5) ++ ); ++ } ++ ++ #[tokio::test(start_paused = true)] ++ async fn worker_control_socket_times_out_without_liveness() { ++ let (worker_io, _server_io) = tokio::io::duplex(1024); ++ let worker_ws = ++ tokio_tungstenite::WebSocketStream::from_raw_socket(worker_io, Role::Client, None) ++ .await; ++ let mut socket = WorkerControlSocket::Test(Box::new(worker_ws)); + let interviewer = Arc::new(ControlInterviewer::new()); + let cancel_token = CancellationToken::new(); +- let mut question = Question::new("Approve?", QuestionType::YesNo); +- question.id = "q-1".to_string(); +- let ask_interviewer = Arc::clone(&interviewer); +- let answer_task = tokio::spawn(async move { ask_interviewer.ask(question).await }); +- let (event_tx, event_rx) = tokio::sync::mpsc::unbounded_channel(); ++ let hub = test_steering_hub(); ++ let run_control = RunControlState::new(); ++ let mut applied_ids = AppliedWorkerControlDeliveryIds::default(); ++ let done = CancellationToken::new(); ++ ++ let task = tokio::spawn(async move { ++ handle_worker_control_socket( ++ &mut socket, ++ &interviewer, ++ &cancel_token, ++ &hub, ++ &run_control, ++ &mut applied_ids, ++ &done, ++ ) ++ .await ++ }); + +- event_tx.send(WorkerControlStreamEvent::Eof).unwrap(); +- drop(event_tx); ++ tokio::task::yield_now().await; ++ time::advance(Duration::from_secs(46)).await; ++ let result = task.await.unwrap(); ++ assert!(matches!(result, Err(WorkerControlConnectError::Other(_)))); ++ } + +- let hub = test_steering_hub(); +- handle_worker_control_stream_events( +- Arc::clone(&interviewer), +- cancel_token.clone(), +- hub, +- event_rx, ++ #[cfg(unix)] ++ #[tokio::test] ++ async fn worker_control_unix_socket_handshake_completes() { ++ let temp = tempfile::tempdir().unwrap(); ++ let socket_path = temp.path().join("fabro.sock"); ++ let listener = tokio::net::UnixListener::bind(&socket_path).unwrap(); ++ let server = tokio::spawn(async move { ++ let (stream, _) = listener.accept().await.unwrap(); ++ let mut socket = tokio_tungstenite::accept_async(stream).await.unwrap(); ++ while let Some(message) = futures::StreamExt::next(&mut socket).await { ++ match message.unwrap() { ++ TestWebSocketMessage::Close(_) => break, ++ TestWebSocketMessage::Ping(payload) => { ++ futures::SinkExt::send(&mut socket, TestWebSocketMessage::Pong(payload)) ++ .await ++ .unwrap(); ++ } ++ _ => {} ++ } ++ } ++ }); ++ let request = build_worker_control_stream_request( ++ &ServerTarget::unix_socket_path(&socket_path).unwrap(), ++ &fixtures::RUN_1, ++ "worker-token", ++ None, + ) +- .await; ++ .unwrap(); + +- let answer = answer_task.await.unwrap().answer; +- assert_eq!(answer.value, AnswerValue::Interrupted); +- assert!(!cancel_token.is_cancelled()); ++ let mut socket = connect_worker_control_stream(request).await.unwrap(); ++ socket ++ .send(TestWebSocketMessage::Close(None)) ++ .await ++ .unwrap(); ++ server.await.unwrap(); + } + + #[tokio::test] +diff --git a/lib/crates/fabro-cli/tests/it/cmd/runner.rs b/lib/crates/fabro-cli/tests/it/cmd/runner.rs +index d7d87fb93..d5b307d35 100644 +--- a/lib/crates/fabro-cli/tests/it/cmd/runner.rs ++++ b/lib/crates/fabro-cli/tests/it/cmd/runner.rs +@@ -794,6 +794,66 @@ fn detached_run_answers_pending_question_without_interview_scratch_files() { + ))); + } + ++#[test] ++fn detached_run_cancel_reaches_worker_over_control_websocket() { ++ let context = auth_context(); ++ let run_id = unique_run_id(); ++ let workflow_path = context.temp_dir.join("cancel-over-control-websocket.fabro"); ++ let _gate = write_gated_workflow( ++ &workflow_path, ++ "cancel_over_control_websocket", ++ "Wait for cancellation", ++ ); ++ ++ let output = context ++ .command() ++ .args([ ++ "run", ++ "--detach", ++ "--run-id", ++ run_id.as_str(), ++ "--environment", ++ "local", ++ workflow_path.to_str().unwrap(), ++ ]) ++ .timeout(SHARED_DAEMON_TIMEOUT) ++ .output() ++ .expect("detached run should execute"); ++ assert!( ++ output.status.success(), ++ "detached run failed:\nstdout:\n{}\nstderr:\n{}", ++ String::from_utf8_lossy(&output.stdout), ++ String::from_utf8_lossy(&output.stderr) ++ ); ++ ++ let run_dir = context.find_run_dir(&run_id); ++ wait_for_event_names(&run_dir, &["run.running"]); ++ tokio::runtime::Runtime::new() ++ .expect("test runtime should build") ++ .block_on(async { ++ let (client, base_url) = ++ server_endpoint(&context.storage_dir).expect("server endpoint should exist"); ++ let response = client ++ .post(format!("{base_url}/api/v1/runs/{run_id}/cancel")) ++ .send() ++ .await ++ .expect("cancel request should succeed"); ++ assert_reqwest_status( ++ response, ++ fabro_http::StatusCode::OK, ++ format!("POST /api/v1/runs/{run_id}/cancel"), ++ ) ++ .await; ++ }); ++ ++ wait_for_status(&run_dir, &["failed"]); ++ let events = stored_worker_events(&run_dir); ++ assert!(events.iter().any(|event| matches!( ++ &event.body, ++ EventBody::RunFailed(props) if props.failure.reason == FailureReason::Cancelled ++ ))); ++} ++ + #[cfg(unix)] + #[test] + fn worker_exits_after_sigterm_cancel_even_when_stdin_stays_open() { +diff --git a/lib/crates/fabro-interview/src/control_protocol.rs b/lib/crates/fabro-interview/src/control_protocol.rs +index cdc820216..f76b61121 100644 +--- a/lib/crates/fabro-interview/src/control_protocol.rs ++++ b/lib/crates/fabro-interview/src/control_protocol.rs +@@ -1,3 +1,5 @@ ++use std::time::Duration; ++ + use fabro_types::{PairId, PairMessageId, PairTarget, Principal, RunId}; + use serde::{Deserialize, Serialize}; + +@@ -5,6 +7,25 @@ use crate::{Answer, AnswerSubmission, AnswerValue}; + + pub const WORKER_CONTROL_PROTOCOL_VERSION: u8 = 1; + ++/// Interval between worker-control WebSocket ping frames. ++/// ++/// Server and worker both initiate pings at this cadence; either side that ++/// fails to observe inbound traffic for [`WORKER_CONTROL_WS_LIVENESS_TIMEOUT`] ++/// closes the WebSocket. ++pub const WORKER_CONTROL_WS_PING_INTERVAL: Duration = Duration::from_secs(15); ++ ++/// Maximum quiet time allowed on a worker-control WebSocket before either side ++/// declares the connection dead. ++pub const WORKER_CONTROL_WS_LIVENESS_TIMEOUT: Duration = Duration::from_secs(45); ++ ++/// WebSocket close-frame reason used when the server can no longer prove ++/// replay correctness for the requested cursor. Workers must treat this as ++/// fatal control-channel loss. ++pub const WORKER_CONTROL_INVALID_CURSOR_REASON: &str = "invalid_cursor"; ++ ++/// WebSocket close-frame reason used when the ping/pong watchdog fires. ++pub const WORKER_CONTROL_PONG_TIMEOUT_REASON: &str = "pong_timeout"; ++ + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] + pub struct WorkerControlEnvelope { + pub v: u8, +@@ -33,6 +54,22 @@ impl WorkerControlEnvelope { + } + } + ++ #[must_use] ++ pub fn pause_run() -> Self { ++ Self { ++ v: WORKER_CONTROL_PROTOCOL_VERSION, ++ message: WorkerControlMessage::RunPause, ++ } ++ } ++ ++ #[must_use] ++ pub fn unpause_run() -> Self { ++ Self { ++ v: WORKER_CONTROL_PROTOCOL_VERSION, ++ message: WorkerControlMessage::RunUnpause, ++ } ++ } ++ + #[must_use] + pub fn steer(text: impl Into, actor: Principal) -> Self { + Self { +@@ -121,6 +158,10 @@ pub enum WorkerControlMessage { + }, + #[serde(rename = "run.cancel")] + RunCancel, ++ #[serde(rename = "run.pause")] ++ RunPause, ++ #[serde(rename = "run.unpause")] ++ RunUnpause, + #[serde(rename = "run.steer")] + Steer { text: String, actor: Principal }, + #[serde(rename = "run.interrupt")] +@@ -147,6 +188,12 @@ pub enum WorkerControlMessage { + PairEnd { pair_id: PairId, actor: Principal }, + } + ++#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] ++pub struct WorkerControlDeliveryFrame { ++ pub id: String, ++ pub envelope: WorkerControlEnvelope, ++} ++ + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] + #[serde(tag = "kind", rename_all = "snake_case")] + pub enum WorkerControlAnswer { +@@ -229,6 +276,26 @@ mod tests { + assert_eq!(parsed, envelope); + } + ++ #[test] ++ fn pause_run_round_trips_through_json() { ++ let envelope = WorkerControlEnvelope::pause_run(); ++ let json = serde_json::to_string(&envelope).unwrap(); ++ assert_eq!(json, r#"{"v":1,"type":"run.pause"}"#); ++ ++ let parsed: WorkerControlEnvelope = serde_json::from_str(&json).unwrap(); ++ assert_eq!(parsed, envelope); ++ } ++ ++ #[test] ++ fn unpause_run_round_trips_through_json() { ++ let envelope = WorkerControlEnvelope::unpause_run(); ++ let json = serde_json::to_string(&envelope).unwrap(); ++ assert_eq!(json, r#"{"v":1,"type":"run.unpause"}"#); ++ ++ let parsed: WorkerControlEnvelope = serde_json::from_str(&json).unwrap(); ++ assert_eq!(parsed, envelope); ++ } ++ + #[test] + fn steer_append_round_trips_through_json() { + let envelope = WorkerControlEnvelope::steer("try again", Principal::System { +@@ -326,4 +393,20 @@ mod tests { + serde_json::from_str(&serde_json::to_string(&end).unwrap()).unwrap(); + assert_eq!(parsed, end); + } ++ ++ #[test] ++ fn delivery_frame_round_trips_through_json() { ++ let frame = WorkerControlDeliveryFrame { ++ id: "local:42".to_string(), ++ envelope: WorkerControlEnvelope::cancel_run(), ++ }; ++ let json = serde_json::to_string(&frame).unwrap(); ++ assert_eq!( ++ json, ++ r#"{"id":"local:42","envelope":{"v":1,"type":"run.cancel"}}"# ++ ); ++ ++ let parsed: WorkerControlDeliveryFrame = serde_json::from_str(&json).unwrap(); ++ assert_eq!(parsed, frame); ++ } + } +diff --git a/lib/crates/fabro-interview/src/lib.rs b/lib/crates/fabro-interview/src/lib.rs +index 4c050c4f7..6d8c414a1 100644 +--- a/lib/crates/fabro-interview/src/lib.rs ++++ b/lib/crates/fabro-interview/src/lib.rs +@@ -222,8 +222,10 @@ pub use callback::CallbackInterviewer; + pub use console::ConsoleInterviewer; + pub use control::{ControlInterviewer, SubmitError}; + pub use control_protocol::{ +- WORKER_CONTROL_PROTOCOL_VERSION, WorkerControlAnswer, WorkerControlEnvelope, +- WorkerControlMessage, ++ WORKER_CONTROL_INVALID_CURSOR_REASON, WORKER_CONTROL_PONG_TIMEOUT_REASON, ++ WORKER_CONTROL_PROTOCOL_VERSION, WORKER_CONTROL_WS_LIVENESS_TIMEOUT, ++ WORKER_CONTROL_WS_PING_INTERVAL, WorkerControlAnswer, WorkerControlDeliveryFrame, ++ WorkerControlEnvelope, WorkerControlMessage, + }; + pub use queue::QueueInterviewer; + pub use recording::RecordingInterviewer; +diff --git a/lib/crates/fabro-server/Cargo.toml b/lib/crates/fabro-server/Cargo.toml +index c36eee94a..dfde73f66 100644 +--- a/lib/crates/fabro-server/Cargo.toml ++++ b/lib/crates/fabro-server/Cargo.toml +@@ -69,6 +69,7 @@ serde.workspace = true + serde_json.workspace = true + serde_yaml = "0.9" + anyhow.workspace = true ++async-trait.workspace = true + clap.workspace = true + toml.workspace = true + toml_edit.workspace = true +@@ -109,8 +110,8 @@ http-body-util = "0.1" + httpmock = "0.8" + serde_yaml = "0.9" + tracing-subscriber.workspace = true +-async-trait.workspace = true + tokio-util.workspace = true ++tokio-tungstenite.workspace = true + fabro-macros = { path = "../fabro-macros" } + fabro-sandbox = { path = "../fabro-sandbox", features = ["test-support"] } + fabro-test = { workspace = true } +diff --git a/lib/crates/fabro-server/src/automation_materializer.rs b/lib/crates/fabro-server/src/automation_materializer.rs +new file mode 100644 +index 000000000..74c9c2677 +--- /dev/null ++++ b/lib/crates/fabro-server/src/automation_materializer.rs +@@ -0,0 +1,797 @@ ++use std::path::{Path, PathBuf}; ++use std::time::Duration; ++ ++use anyhow::Context as _; ++use async_trait::async_trait; ++use base64::Engine as _; ++use base64::engine::general_purpose::STANDARD as BASE64_STANDARD; ++use fabro_api::types::RunManifest; ++use fabro_automation::{AutomationId, AutomationTarget}; ++use fabro_manifest::ManifestBuildInput; ++use fabro_types::{DirtyStatus, GitContext, PreRunPushOutcome, RunId}; ++use fabro_util::error::collect_chain; ++use tokio::process::Command; ++use tokio::{fs, task, time}; ++ ++const GIT_CLONE_TIMEOUT: Duration = Duration::from_mins(2); ++const GIT_FETCH_TIMEOUT: Duration = Duration::from_mins(1); ++const GIT_CHECKOUT_TIMEOUT: Duration = Duration::from_secs(30); ++const GIT_REV_PARSE_TIMEOUT: Duration = Duration::from_secs(10); ++ ++#[derive(Debug, Clone, PartialEq, Eq)] ++pub(crate) struct AutomationRunMaterializeInput { ++ pub automation_id: AutomationId, ++ pub target: AutomationTarget, ++ pub run_id: RunId, ++ pub user_settings_path: PathBuf, ++ pub temp_root: PathBuf, ++} ++ ++#[derive(Debug, Clone)] ++pub(crate) struct AutomationRunMaterialized { ++ pub manifest: RunManifest, ++ pub submitted_manifest_bytes: Vec, ++} ++ ++#[derive(thiserror::Error, Debug, Clone, PartialEq, Eq)] ++pub(crate) enum AutomationRunMaterializeError { ++ #[error("invalid automation target: {0}")] ++ InvalidTarget(String), ++ #[error("failed to clone automation repository: {0}")] ++ CloneFailed(String), ++ #[error("failed to resolve automation workflow: {0}")] ++ WorkflowNotFound(String), ++ #[error("failed to build run manifest: {0}")] ++ Manifest(String), ++} ++ ++#[async_trait] ++pub(crate) trait AutomationRunMaterializer: Send + Sync { ++ async fn materialize( ++ &self, ++ input: AutomationRunMaterializeInput, ++ ) -> Result; ++} ++ ++#[derive(Clone)] ++pub(crate) struct ProductionAutomationRunMaterializer { ++ github_credentials: Option, ++ github_api_base_url: String, ++ http_client: Option, ++} ++ ++impl ProductionAutomationRunMaterializer { ++ pub(crate) fn new( ++ github_credentials: Option, ++ github_api_base_url: String, ++ http_client: Option, ++ ) -> Self { ++ Self { ++ github_credentials, ++ github_api_base_url, ++ http_client, ++ } ++ } ++} ++ ++#[async_trait] ++impl AutomationRunMaterializer for ProductionAutomationRunMaterializer { ++ async fn materialize( ++ &self, ++ input: AutomationRunMaterializeInput, ++ ) -> Result { ++ let repo = parse_github_repository_slug(&input.target.repository)?; ++ fs::create_dir_all(&input.temp_root).await.map_err(|err| { ++ AutomationRunMaterializeError::CloneFailed(format!( ++ "failed to create temp root {}: {err}", ++ input.temp_root.display() ++ )) ++ })?; ++ let temp_dir = tempfile::Builder::new() ++ .prefix(&format!( ++ "automation-{}-{}-", ++ input.automation_id.as_str(), ++ input.run_id ++ )) ++ .tempdir_in(&input.temp_root) ++ .map_err(|err| { ++ AutomationRunMaterializeError::CloneFailed(format!( ++ "failed to create per-run temp directory under {}: {err}", ++ input.temp_root.display() ++ )) ++ })?; ++ let checkout_dir = temp_dir.path().join("repo"); ++ let clone_url = github_clone_url(&repo); ++ let auth = resolve_git_auth_config( ++ self.github_credentials.as_ref(), ++ &repo, ++ &self.github_api_base_url, ++ self.http_client.clone(), ++ ) ++ .await ++ .map_err(|err| { ++ AutomationRunMaterializeError::CloneFailed(render_error_chain(err.as_ref())) ++ })?; ++ ++ run_git_plan(build_clone_plan(&clone_url, &checkout_dir, auth.as_ref())).await?; ++ run_git_plan(build_fetch_ref_plan( ++ &clone_url, ++ &checkout_dir, ++ &input.target.ref_selector, ++ auth.as_ref(), ++ )) ++ .await?; ++ run_git_plan(build_checkout_ref_plan(&checkout_dir)).await?; ++ let checked_out_sha = run_git_plan(build_rev_parse_head_plan(&checkout_dir)) ++ .await ++ .map(|stdout| String::from_utf8_lossy(&stdout).trim().to_string())?; ++ ++ let manifest_input = ManifestFromCheckoutInput { ++ input, ++ checkout_dir, ++ repo, ++ checked_out_sha: Some(checked_out_sha), ++ }; ++ task::spawn_blocking(move || build_manifest_from_checkout(manifest_input)) ++ .await ++ .map_err(|err| { ++ AutomationRunMaterializeError::Manifest(format!( ++ "manifest build task failed: {err}" ++ )) ++ })? ++ } ++} ++ ++#[derive(Debug, Clone, PartialEq, Eq)] ++pub(crate) struct GithubRepository { ++ owner: String, ++ name: String, ++} ++ ++fn parse_github_repository_slug( ++ value: &str, ++) -> Result { ++ let Some((owner, repo)) = value.split_once('/') else { ++ return Err(AutomationRunMaterializeError::InvalidTarget(format!( ++ "repository must be a GitHub owner/repo slug: {value}" ++ ))); ++ }; ++ if repo.contains('/') || !valid_github_owner(owner) || !valid_github_repo(repo) { ++ return Err(AutomationRunMaterializeError::InvalidTarget(format!( ++ "repository must be a GitHub owner/repo slug: {value}" ++ ))); ++ } ++ Ok(GithubRepository { ++ owner: owner.to_string(), ++ name: repo.to_string(), ++ }) ++} ++ ++fn valid_github_owner(value: &str) -> bool { ++ if value.is_empty() || value.len() > 39 { ++ return false; ++ } ++ let bytes = value.as_bytes(); ++ let first = bytes[0]; ++ let last = bytes[bytes.len() - 1]; ++ (first.is_ascii_alphanumeric() && last.is_ascii_alphanumeric()) ++ && bytes ++ .iter() ++ .all(|byte| byte.is_ascii_alphanumeric() || *byte == b'-') ++} ++ ++fn valid_github_repo(value: &str) -> bool { ++ !value.is_empty() ++ && value.len() <= 100 ++ && value != "." ++ && value != ".." ++ && !value.starts_with('.') ++ && value ++ .bytes() ++ .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'_' | b'-')) ++} ++ ++fn github_clone_url(repo: &GithubRepository) -> String { ++ format!("https://github.com/{}/{}.git", repo.owner, repo.name) ++} ++ ++fn github_metadata_url(repo: &GithubRepository) -> String { ++ format!("https://github.com/{}/{}", repo.owner, repo.name) ++} ++ ++#[derive(Clone, Debug, PartialEq, Eq)] ++pub(crate) struct GitAuthConfig { ++ extraheader: Option, ++ sensitive_values: Vec, ++} ++ ++impl GitAuthConfig { ++ fn new(username: Option, password: Option) -> Self { ++ let Some(password) = password.filter(|value| !value.is_empty()) else { ++ return Self { ++ extraheader: None, ++ sensitive_values: Vec::new(), ++ }; ++ }; ++ let username = username ++ .filter(|value| !value.is_empty()) ++ .unwrap_or_else(|| "x-access-token".to_string()); ++ let encoded_credentials = BASE64_STANDARD.encode(format!("{username}:{password}")); ++ let extraheader = basic_auth_header_from_encoded(&encoded_credentials); ++ Self { ++ sensitive_values: vec![password, encoded_credentials, extraheader.clone()], ++ extraheader: Some(extraheader), ++ } ++ } ++ ++ fn git_env(&self, clone_url: &str) -> Vec<(String, String)> { ++ let Some(extraheader) = self.extraheader.as_ref() else { ++ return Vec::new(); ++ }; ++ vec![ ++ ("GIT_CONFIG_COUNT".to_string(), "1".to_string()), ++ ( ++ "GIT_CONFIG_KEY_0".to_string(), ++ format!("http.{clone_url}.extraheader"), ++ ), ++ ("GIT_CONFIG_VALUE_0".to_string(), extraheader.clone()), ++ ] ++ } ++ ++ fn sensitive_values(&self) -> &[String] { ++ &self.sensitive_values ++ } ++} ++ ++async fn resolve_git_auth_config( ++ credentials: Option<&fabro_github::GitHubCredentials>, ++ repo: &GithubRepository, ++ github_api_base_url: &str, ++ http_client: Option, ++) -> anyhow::Result> { ++ let Some(credentials) = credentials else { ++ return Ok(None); ++ }; ++ let context = match http_client { ++ Some(client) => { ++ fabro_github::GitHubContext::with_http_client(credentials, github_api_base_url, client) ++ } ++ None => fabro_github::GitHubContext::new(credentials, github_api_base_url), ++ }; ++ let (username, password) = ++ fabro_github::resolve_clone_credentials(&context, &repo.owner, &repo.name).await?; ++ Ok(Some(GitAuthConfig::new(username, password))) ++} ++ ++fn basic_auth_header(username: &str, password: &str) -> String { ++ basic_auth_header_from_encoded(&BASE64_STANDARD.encode(format!("{username}:{password}"))) ++} ++ ++fn basic_auth_header_from_encoded(encoded_credentials: &str) -> String { ++ format!("AUTHORIZATION: basic {encoded_credentials}") ++} ++ ++#[derive(Debug, Clone, PartialEq, Eq)] ++pub(crate) struct GitCommandPlan { ++ program: String, ++ args: Vec, ++ env: Vec<(String, String)>, ++ current_dir: Option, ++ timeout: Duration, ++ sensitive_values: Vec, ++} ++ ++impl GitCommandPlan { ++ fn new(args: impl IntoIterator>, timeout: Duration) -> Self { ++ Self { ++ program: "git".to_string(), ++ args: args.into_iter().map(Into::into).collect(), ++ env: vec![("GIT_TERMINAL_PROMPT".to_string(), "0".to_string())], ++ current_dir: None, ++ timeout, ++ sensitive_values: Vec::new(), ++ } ++ } ++ ++ fn current_dir(mut self, current_dir: impl Into) -> Self { ++ self.current_dir = Some(current_dir.into()); ++ self ++ } ++ ++ fn with_auth(mut self, clone_url: &str, auth: Option<&GitAuthConfig>) -> Self { ++ if let Some(auth) = auth { ++ self.env.extend(auth.git_env(clone_url)); ++ self.sensitive_values ++ .extend(auth.sensitive_values().iter().cloned()); ++ } ++ self ++ } ++ ++ fn env_value(&self, name: &str) -> Option<&str> { ++ self.env ++ .iter() ++ .find(|(key, _)| key == name) ++ .map(|(_, value)| value.as_str()) ++ } ++} ++ ++fn build_clone_plan( ++ clone_url: &str, ++ checkout_dir: &Path, ++ auth: Option<&GitAuthConfig>, ++) -> GitCommandPlan { ++ GitCommandPlan::new( ++ [ ++ "clone".to_string(), ++ "--depth".to_string(), ++ "1".to_string(), ++ "--no-checkout".to_string(), ++ clone_url.to_string(), ++ checkout_dir.display().to_string(), ++ ], ++ GIT_CLONE_TIMEOUT, ++ ) ++ .with_auth(clone_url, auth) ++} ++ ++fn build_fetch_ref_plan( ++ clone_url: &str, ++ checkout_dir: &Path, ++ ref_selector: &str, ++ auth: Option<&GitAuthConfig>, ++) -> GitCommandPlan { ++ GitCommandPlan::new( ++ [ ++ "fetch".to_string(), ++ "--depth".to_string(), ++ "1".to_string(), ++ "origin".to_string(), ++ "--".to_string(), ++ ref_selector.to_string(), ++ ], ++ GIT_FETCH_TIMEOUT, ++ ) ++ .current_dir(checkout_dir) ++ .with_auth(clone_url, auth) ++} ++ ++fn build_checkout_ref_plan(checkout_dir: &Path) -> GitCommandPlan { ++ GitCommandPlan::new( ++ ["checkout", "--force", "--detach", "FETCH_HEAD"], ++ GIT_CHECKOUT_TIMEOUT, ++ ) ++ .current_dir(checkout_dir) ++} ++ ++fn build_rev_parse_head_plan(checkout_dir: &Path) -> GitCommandPlan { ++ GitCommandPlan::new(["rev-parse", "HEAD"], GIT_REV_PARSE_TIMEOUT).current_dir(checkout_dir) ++} ++ ++async fn run_git_plan(plan: GitCommandPlan) -> Result, AutomationRunMaterializeError> { ++ let mut command = Command::new(&plan.program); ++ command.args(&plan.args); ++ command.envs(plan.env.iter().map(|(key, value)| (key, value))); ++ if let Some(current_dir) = plan.current_dir.as_ref() { ++ command.current_dir(current_dir); ++ } ++ command.kill_on_drop(true); ++ ++ let output = time::timeout(plan.timeout, command.output()) ++ .await ++ .map_err(|_| { ++ AutomationRunMaterializeError::CloneFailed(format!( ++ "{} timed out after {}s", ++ safe_command_label(&plan), ++ plan.timeout.as_secs() ++ )) ++ })? ++ .map_err(|err| { ++ AutomationRunMaterializeError::CloneFailed(format!( ++ "failed to run {}: {err}", ++ safe_command_label(&plan) ++ )) ++ })?; ++ ++ if output.status.success() { ++ return Ok(output.stdout); ++ } ++ ++ let stdout = String::from_utf8_lossy(&output.stdout); ++ let stderr = String::from_utf8_lossy(&output.stderr); ++ let mut message = format!( ++ "{} exited with status {}", ++ safe_command_label(&plan), ++ output.status ++ ); ++ if !stderr.trim().is_empty() { ++ message.push_str(": "); ++ message.push_str(stderr.trim()); ++ } else if !stdout.trim().is_empty() { ++ message.push_str(": "); ++ message.push_str(stdout.trim()); ++ } ++ Err(AutomationRunMaterializeError::CloneFailed( ++ redact_git_output(&message, &plan.sensitive_values), ++ )) ++} ++ ++fn safe_command_label(plan: &GitCommandPlan) -> String { ++ if plan.args.is_empty() { ++ plan.program.clone() ++ } else { ++ format!("{} {}", plan.program, plan.args.join(" ")) ++ } ++} ++ ++fn redact_git_output(text: &str, sensitive_values: &[String]) -> String { ++ let mut redacted = fabro_redact::redact_string(text); ++ for value in sensitive_values ++ .iter() ++ .map(String::as_str) ++ .filter(|value| !value.is_empty()) ++ { ++ redacted = redacted.replace(value, "REDACTED"); ++ } ++ redacted ++} ++ ++fn render_error_chain(error: &(dyn std::error::Error + 'static)) -> String { ++ collect_chain(error).join(": ") ++} ++ ++#[derive(Debug)] ++pub(crate) struct ManifestFromCheckoutInput { ++ input: AutomationRunMaterializeInput, ++ checkout_dir: PathBuf, ++ repo: GithubRepository, ++ checked_out_sha: Option, ++} ++ ++fn build_manifest_from_checkout( ++ args: ManifestFromCheckoutInput, ++) -> Result { ++ let ManifestFromCheckoutInput { ++ input, ++ checkout_dir, ++ repo, ++ checked_out_sha, ++ } = args; ++ let built = fabro_manifest::build_run_manifest(ManifestBuildInput { ++ workflow: input.target.workflow.as_str().into(), ++ cwd: checkout_dir, ++ run_id: Some(input.run_id), ++ user_settings_path: Some(input.user_settings_path), ++ ..ManifestBuildInput::default() ++ }) ++ .map_err(|err| manifest_build_error(&err))?; ++ ++ let mut manifest = built.manifest; ++ manifest.git = Some(GitContext { ++ origin_url: github_metadata_url(&repo), ++ branch: input.target.ref_selector, ++ sha: checked_out_sha, ++ dirty: DirtyStatus::Clean, ++ push_outcome: PreRunPushOutcome::NotAttempted, ++ }); ++ let submitted_manifest_bytes = serde_json::to_vec(&manifest) ++ .with_context(|| { ++ format!( ++ "failed to serialize materialized manifest for automation {}", ++ input.automation_id.as_str() ++ ) ++ }) ++ .map_err(|err| AutomationRunMaterializeError::Manifest(err.to_string()))?; ++ Ok(AutomationRunMaterialized { ++ manifest, ++ submitted_manifest_bytes, ++ }) ++} ++ ++fn manifest_build_error(error: &anyhow::Error) -> AutomationRunMaterializeError { ++ if error.chain().any(|source| { ++ source ++ .downcast_ref::() ++ .is_some_and(|err| matches!(err, fabro_config::Error::WorkflowNotFound(_))) ++ }) { ++ AutomationRunMaterializeError::WorkflowNotFound(render_error_chain(error.as_ref())) ++ } else { ++ AutomationRunMaterializeError::Manifest(render_error_chain(error.as_ref())) ++ } ++} ++ ++#[cfg(any(test, feature = "test-support"))] ++#[derive(Clone)] ++pub struct TestAutomationRunMaterializer { ++ inner: std::sync::Arc>, ++} ++ ++#[cfg(any(test, feature = "test-support"))] ++struct TestAutomationRunMaterializerState { ++ captured_inputs: Vec, ++ response: Result, ++} ++ ++#[cfg(any(test, feature = "test-support"))] ++impl TestAutomationRunMaterializer { ++ pub fn succeed(manifest: RunManifest, submitted_manifest_bytes: Vec) -> Self { ++ Self::new(Ok(AutomationRunMaterialized { ++ manifest, ++ submitted_manifest_bytes, ++ })) ++ } ++ ++ pub fn fail_invalid_target(message: impl Into) -> Self { ++ Self::new(Err(AutomationRunMaterializeError::InvalidTarget( ++ message.into(), ++ ))) ++ } ++ ++ fn new(response: Result) -> Self { ++ Self { ++ inner: std::sync::Arc::new(std::sync::Mutex::new(TestAutomationRunMaterializerState { ++ captured_inputs: Vec::new(), ++ response, ++ })), ++ } ++ } ++ ++ pub(crate) fn captured_inputs(&self) -> Vec { ++ self.inner ++ .lock() ++ .expect("test automation materializer lock poisoned") ++ .captured_inputs ++ .clone() ++ } ++ ++ pub(crate) fn into_materializer(self) -> std::sync::Arc { ++ std::sync::Arc::new(self) ++ } ++} ++ ++#[cfg(any(test, feature = "test-support"))] ++#[async_trait] ++impl AutomationRunMaterializer for TestAutomationRunMaterializer { ++ async fn materialize( ++ &self, ++ input: AutomationRunMaterializeInput, ++ ) -> Result { ++ let mut guard = self ++ .inner ++ .lock() ++ .expect("test automation materializer lock poisoned"); ++ guard.captured_inputs.push(input); ++ guard.response.clone() ++ } ++} ++ ++#[cfg(test)] ++mod tests { ++ #![expect( ++ clippy::disallowed_methods, ++ reason = "Materializer unit tests write small temporary workflow fixtures synchronously." ++ )] ++ ++ use std::fs; ++ use std::path::Path; ++ ++ use fabro_automation::{AutomationId, AutomationTarget}; ++ use fabro_types::{DirtyStatus, PreRunPushOutcome, RunId}; ++ use tempfile::TempDir; ++ ++ use super::*; ++ ++ fn target(repository: &str, ref_selector: &str, workflow: &str) -> AutomationTarget { ++ AutomationTarget { ++ repository: repository.to_string(), ++ ref_selector: ref_selector.to_string(), ++ workflow: workflow.to_string(), ++ } ++ } ++ ++ #[test] ++ fn target_repository_urls_are_github_metadata_urls_without_credentials() { ++ let repo = parse_github_repository_slug("fabro-sh/fabro").expect("slug should parse"); ++ ++ assert_eq!(repo.owner, "fabro-sh"); ++ assert_eq!(repo.name, "fabro"); ++ assert_eq!( ++ github_clone_url(&repo), ++ "https://github.com/fabro-sh/fabro.git" ++ ); ++ assert_eq!( ++ github_metadata_url(&repo), ++ "https://github.com/fabro-sh/fabro" ++ ); ++ assert!(!github_clone_url(&repo).contains('@')); ++ } ++ ++ #[test] ++ fn target_repository_validation_rejects_non_github_owner_repo_shapes() { ++ for value in [ ++ "fabro-sh", ++ "https://github.com/fabro-sh/fabro", ++ "fabro-sh/fabro/extra", ++ "-owner/repo", ++ "owner/.git", ++ ] { ++ let error = parse_github_repository_slug(value).expect_err("invalid slug should fail"); ++ assert!( ++ error.to_string().contains("invalid automation target"), ++ "unexpected error for {value}: {error}" ++ ); ++ } ++ } ++ ++ #[test] ++ fn ref_checkout_command_plans_use_argv_prompt_disable_and_timeouts() { ++ let repo = parse_github_repository_slug("fabro-sh/fabro").unwrap(); ++ let clone_url = github_clone_url(&repo); ++ let temp = TempDir::new().unwrap(); ++ let checkout_dir = temp.path().join("repo"); ++ ++ let clone = build_clone_plan(&clone_url, &checkout_dir, None); ++ assert_eq!(clone.program, "git"); ++ assert_eq!(clone.args, vec![ ++ "clone", ++ "--depth", ++ "1", ++ "--no-checkout", ++ "https://github.com/fabro-sh/fabro.git", ++ checkout_dir.to_str().unwrap(), ++ ]); ++ assert_eq!(clone.timeout, Duration::from_mins(2)); ++ assert_eq!(clone.env_value("GIT_TERMINAL_PROMPT"), Some("0")); ++ ++ let fetch = build_fetch_ref_plan(&clone_url, &checkout_dir, "feature/materialize", None); ++ assert_eq!(fetch.args, vec![ ++ "fetch", ++ "--depth", ++ "1", ++ "origin", ++ "--", ++ "feature/materialize", ++ ]); ++ assert_eq!(fetch.current_dir.as_deref(), Some(checkout_dir.as_path())); ++ assert_eq!(fetch.timeout, Duration::from_mins(1)); ++ assert_eq!(fetch.env_value("GIT_TERMINAL_PROMPT"), Some("0")); ++ ++ let checkout = build_checkout_ref_plan(&checkout_dir); ++ assert_eq!(checkout.args, vec![ ++ "checkout", ++ "--force", ++ "--detach", ++ "FETCH_HEAD" ++ ]); ++ assert_eq!( ++ checkout.current_dir.as_deref(), ++ Some(checkout_dir.as_path()) ++ ); ++ assert_eq!(checkout.timeout, Duration::from_secs(30)); ++ assert_eq!(checkout.env_value("GIT_TERMINAL_PROMPT"), Some("0")); ++ } ++ ++ #[test] ++ fn credential_redaction_removes_tokens_and_basic_auth_headers() { ++ let secret = "ghu_materializer_secret"; ++ let basic = basic_auth_header("x-access-token", secret); ++ let message = format!( ++ "fatal: could not read Username for https://github.com/fabro-sh/fabro.git; token={secret}; header={basic}" ++ ); ++ ++ let redacted = redact_git_output(&message, &[secret.to_string(), basic.clone()]); ++ ++ assert!(!redacted.contains(secret), "token leaked: {redacted}"); ++ assert!( ++ !redacted.contains(&basic), ++ "basic header leaked: {redacted}" ++ ); ++ let encoded_secret = BASE64_STANDARD.encode(format!("x-access-token:{secret}")); ++ assert!( ++ !redacted.contains(&encoded_secret), ++ "encoded credential leaked: {redacted}" ++ ); ++ assert!( ++ redacted.contains("REDACTED"), ++ "expected redaction marker: {redacted}" ++ ); ++ } ++ ++ #[test] ++ fn credential_config_env_keeps_clone_url_uncredentialed() { ++ let repo = parse_github_repository_slug("fabro-sh/fabro").unwrap(); ++ let clone_url = github_clone_url(&repo); ++ let auth = GitAuthConfig::new( ++ Some("x-access-token".to_string()), ++ Some("ghu_secret".to_string()), ++ ); ++ let plan = build_clone_plan(&clone_url, Path::new("/tmp/fabro-checkout"), Some(&auth)); ++ ++ assert!( ++ plan.args ++ .iter() ++ .any(|arg| arg == "https://github.com/fabro-sh/fabro.git") ++ ); ++ assert!(plan.args.iter().all(|arg| !arg.contains("ghu_secret"))); ++ assert_eq!(plan.env_value("GIT_CONFIG_COUNT"), Some("1")); ++ assert_eq!( ++ plan.env_value("GIT_CONFIG_KEY_0"), ++ Some("http.https://github.com/fabro-sh/fabro.git.extraheader") ++ ); ++ assert!( ++ plan.env_value("GIT_CONFIG_VALUE_0") ++ .is_some_and(|value| value.starts_with("AUTHORIZATION: basic ")) ++ ); ++ } ++ ++ #[test] ++ fn workflow_path_resolution_builds_manifest_from_checkout_directory() { ++ let temp = TempDir::new().unwrap(); ++ let checkout = temp.path().join("checkout"); ++ let workflow_dir = checkout.join(".fabro/workflows/demo"); ++ fs::create_dir_all(&workflow_dir).unwrap(); ++ fs::write(checkout.join(".fabro/project.toml"), "_version = 1\n").unwrap(); ++ fs::write( ++ workflow_dir.join("workflow.fabro"), ++ r#"digraph Demo { graph [goal="Ship automation"] start [shape=Mdiamond] exit [shape=Msquare] start -> exit }"#, ++ ) ++ .unwrap(); ++ fs::write( ++ workflow_dir.join("workflow.toml"), ++ "_version = 1\n[workflow]\ngraph = \"workflow.fabro\"\n", ++ ) ++ .unwrap(); ++ let user_settings_path = temp.path().join("settings.toml"); ++ fs::write(&user_settings_path, "_version = 1\n").unwrap(); ++ let run_id = RunId::new(); ++ let repo = parse_github_repository_slug("fabro-sh/fabro").unwrap(); ++ let sha = "0123456789abcdef0123456789abcdef01234567".to_string(); ++ ++ let materialized = build_manifest_from_checkout(ManifestFromCheckoutInput { ++ input: AutomationRunMaterializeInput { ++ automation_id: AutomationId::new("nightly").unwrap(), ++ target: target("fabro-sh/fabro", "main", "demo"), ++ run_id, ++ user_settings_path: user_settings_path.clone(), ++ temp_root: temp.path().to_path_buf(), ++ }, ++ checkout_dir: checkout.clone(), ++ repo, ++ checked_out_sha: Some(sha.clone()), ++ }) ++ .expect("manifest should build from checkout"); ++ ++ assert_eq!( ++ materialized.manifest.run_id.as_deref(), ++ Some(run_id.to_string().as_str()) ++ ); ++ assert_eq!(materialized.manifest.cwd, checkout.display().to_string()); ++ assert_eq!( ++ materialized.manifest.target.path, ++ ".fabro/workflows/demo/workflow.fabro" ++ ); ++ assert!( ++ materialized ++ .manifest ++ .configs ++ .iter() ++ .any(|config| config.path.as_deref() == Some(user_settings_path.to_str().unwrap())) ++ ); ++ let git = materialized ++ .manifest ++ .git ++ .as_ref() ++ .expect("git context should be set"); ++ assert_eq!(git.origin_url, "https://github.com/fabro-sh/fabro"); ++ assert_eq!(git.branch, "main"); ++ assert_eq!(git.sha.as_deref(), Some(sha.as_str())); ++ assert_eq!(git.dirty, DirtyStatus::Clean); ++ assert_eq!(git.push_outcome, PreRunPushOutcome::NotAttempted); ++ let submitted_manifest: serde_json::Value = ++ serde_json::from_slice(&materialized.submitted_manifest_bytes) ++ .expect("submitted bytes should be a manifest"); ++ assert_eq!( ++ submitted_manifest, ++ serde_json::to_value(&materialized.manifest).unwrap() ++ ); ++ } ++} +diff --git a/lib/crates/fabro-server/src/lib.rs b/lib/crates/fabro-server/src/lib.rs +index 443300c09..76805781c 100644 +--- a/lib/crates/fabro-server/src/lib.rs ++++ b/lib/crates/fabro-server/src/lib.rs +@@ -9,6 +9,11 @@ + )] + + pub mod auth; ++#[allow( ++ dead_code, ++ reason = "Automation scheduler wiring will call the materializer; issue #398 adds the shared core first." ++)] ++mod automation_materializer; + mod canonical_host; + mod canonical_origin; + pub mod csp; +@@ -43,6 +48,7 @@ pub mod static_files; + #[cfg(any(test, feature = "test-support"))] + pub mod test_support; + pub mod web_auth; ++mod worker_control; + mod worker_token; + + pub use error::{ApiError, Error, Result}; +diff --git a/lib/crates/fabro-server/src/principal_middleware.rs b/lib/crates/fabro-server/src/principal_middleware.rs +index acc9eb250..1b037a6ab 100644 +--- a/lib/crates/fabro-server/src/principal_middleware.rs ++++ b/lib/crates/fabro-server/src/principal_middleware.rs +@@ -58,6 +58,7 @@ pub(crate) struct RequestAuth(pub(crate) AuthContextSlot); + pub(crate) struct RequiredUser(pub(crate) UserPrincipal); + pub(crate) struct RequiredRunManagementActor(pub(crate) Principal); + pub(crate) struct RequireRunScoped(pub(crate) RunId); ++pub(crate) struct RequireWorkerRunScoped(pub(crate) RunId); + pub(crate) struct RequireRunManagementTarget(pub(crate) RunId, pub(crate) Principal); + pub(crate) struct RequireRunBlob(pub(crate) RunId, pub(crate) RunBlobId); + pub(crate) struct RequireRunStageScoped(pub(crate) RunId, pub(crate) String); +@@ -245,6 +246,23 @@ impl FromRequestParts> for RequireRunScoped { + } + } + ++impl FromRequestParts> for RequireWorkerRunScoped { ++ type Rejection = Response; ++ ++ async fn from_request_parts( ++ parts: &mut Parts, ++ state: &Arc, ++ ) -> Result { ++ let Path(id): Path = Path::from_request_parts(parts, state) ++ .await ++ .map_err(IntoResponse::into_response)?; ++ let run_id = parse_run_id_path(&id)?; ++ require_worker_for_run(&auth_slot_from_parts(parts), &run_id) ++ .map_err(IntoResponse::into_response)?; ++ Ok(Self(run_id)) ++ } ++} ++ + impl FromRequestParts> for RequireRunManagementTarget { + type Rejection = Response; + +@@ -418,6 +436,15 @@ fn require_worker_or_user_for_run( + } + } + ++fn require_worker_for_run(slot: &AuthContextSlot, route_run_id: &RunId) -> Result<(), ApiError> { ++ let context = slot.0.lock().expect("auth context lock poisoned"); ++ match &context.principal { ++ Principal::Worker { run_id } if run_id == route_run_id => Ok(()), ++ Principal::Worker { .. } | Principal::User(_) => Err(ApiError::forbidden()), ++ _ => Err(auth_rejection(context.auth_status, context.auth_error_code)), ++ } ++} ++ + fn require_run_management_target( + slot: &AuthContextSlot, + route_run_id: &RunId, +diff --git a/lib/crates/fabro-server/src/serve.rs b/lib/crates/fabro-server/src/serve.rs +index f4a3c5089..3042cbae1 100644 +--- a/lib/crates/fabro-server/src/serve.rs ++++ b/lib/crates/fabro-server/src/serve.rs +@@ -776,6 +776,10 @@ where + http_client: None, + sandbox_provider_registry: None, + shutdown: shutdown.clone(), ++ #[cfg(test)] ++ worker_control_bus: None, ++ #[cfg(any(test, feature = "test-support"))] ++ automation_materializer_override: None, + })?; + let reconciled = reconcile_incomplete_runs_on_startup(&state).await?; + if reconciled > 0 { +diff --git a/lib/crates/fabro-server/src/server.rs b/lib/crates/fabro-server/src/server.rs +index 529a374dd..88260152c 100644 +--- a/lib/crates/fabro-server/src/server.rs ++++ b/lib/crates/fabro-server/src/server.rs +@@ -124,7 +124,7 @@ use sha2::{Digest, Sha256}; + use tempfile::NamedTempFile; + use tokio::fs; + use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader}; +-use tokio::process::{ChildStderr, ChildStdin, Command}; ++use tokio::process::{ChildStderr, Command}; + use tokio::sync::broadcast::error::RecvError; + use tokio::sync::{ + Mutex as AsyncMutex, Notify, OwnedMutexGuard, RwLock as AsyncRwLock, Semaphore, broadcast, +@@ -140,6 +140,10 @@ use tracing::{Instrument, debug, error, info, warn}; + use ulid::Ulid; + + use crate::auth::{self, GithubEndpoints, auth_translation_middleware, demo_routing_middleware}; ++use crate::automation_materializer::{ ++ AutomationRunMaterializeError, AutomationRunMaterializeInput, AutomationRunMaterialized, ++ AutomationRunMaterializer, ProductionAutomationRunMaterializer, ++}; + use crate::canonical_origin::resolve_canonical_origin; + use crate::error::ApiError; + use crate::github_webhooks::{ +@@ -148,14 +152,15 @@ use crate::github_webhooks::{ + use crate::jwt_auth::{self, AuthMode}; + use crate::principal_middleware::{ + AuthContextSlot, RequestAuth, RequestAuthContext, RequireRunBlob, RequireRunManagementTarget, +- RequireRunScoped, RequireRunStageScoped, RequireStageArtifact, RequiredUser, +- principal_middleware, ++ RequireRunScoped, RequireRunStageScoped, RequireStageArtifact, RequireWorkerRunScoped, ++ RequiredUser, principal_middleware, + }; + use crate::request_id::{self, RequestId}; + use crate::run_files::{FilesInFlight, new_files_in_flight}; + use crate::server_secrets::{LlmClientResult, ServerSecrets}; + use crate::spawn_env::{apply_render_graph_env, apply_worker_env}; + use crate::startup::load_startup_vault; ++use crate::worker_control::{LocalWorkerControlBus, WorkerControlBus, WorkerControlBusError}; + use crate::worker_token::{WorkerScopeSet, WorkerTokenKeys, issue_worker_token_with_scopes}; + use crate::{ + canonical_host, demo, diagnostics, run_manifest, security_headers, static_files, web_auth, +@@ -267,7 +272,6 @@ enum ExecutionResult { + + const WORKER_CANCEL_GRACE: Duration = Duration::from_secs(5); + const TERMINAL_DELETE_WORKER_GRACE: Duration = Duration::from_millis(50); +-const WORKER_CONTROL_QUEUE_CAPACITY: usize = 8; + const WORKER_CONTROL_ENQUEUE_TIMEOUT: Duration = Duration::from_secs(1); + /// Per-model billing totals. + #[derive(Default)] +@@ -289,8 +293,9 @@ pub(crate) type RegistryFactoryOverride = + + #[derive(Clone)] + enum RunAnswerTransport { +- Subprocess { +- control_tx: mpsc::Sender, ++ Worker { ++ run_id: RunId, ++ bus: Arc, + }, + InProcess { + interviewer: Arc, +@@ -312,18 +317,46 @@ enum PairTransportError { + } + + impl RunAnswerTransport { ++ async fn publish_worker_control( ++ run_id: RunId, ++ bus: &Arc, ++ message: WorkerControlEnvelope, ++ ) -> Result<(), WorkerControlBusError> { ++ timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, bus.publish(run_id, message)) ++ .await ++ .map_err(|_| WorkerControlBusError::PublishTimeout)? ++ .map(|_| ()) ++ } ++ ++ fn answer_error_from_bus(error: &WorkerControlBusError) -> AnswerTransportError { ++ match error { ++ WorkerControlBusError::PublishTimeout => AnswerTransportError::Timeout, ++ WorkerControlBusError::Closed ++ | WorkerControlBusError::Unavailable ++ | WorkerControlBusError::InvalidCursor { .. } => AnswerTransportError::Closed, ++ } ++ } ++ ++ fn pair_error_from_bus(error: &WorkerControlBusError) -> PairTransportError { ++ match error { ++ WorkerControlBusError::PublishTimeout => PairTransportError::Timeout, ++ WorkerControlBusError::Closed ++ | WorkerControlBusError::Unavailable ++ | WorkerControlBusError::InvalidCursor { .. } => PairTransportError::Closed, ++ } ++ } ++ + async fn submit( + &self, + qid: &str, + submission: AnswerSubmission, + ) -> Result<(), AnswerTransportError> { + match self { +- Self::Subprocess { control_tx } => { ++ Self::Worker { run_id, bus } => { + let message = WorkerControlEnvelope::interview_answer(qid.to_string(), submission); +- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message)) ++ Self::publish_worker_control(*run_id, bus, message) + .await +- .map_err(|_| AnswerTransportError::Timeout)? +- .map_err(|_| AnswerTransportError::Closed) ++ .map_err(|err| Self::answer_error_from_bus(&err)) + } + Self::InProcess { interviewer, .. } => interviewer + .submit(qid, submission) +@@ -334,12 +367,11 @@ impl RunAnswerTransport { + + async fn cancel_run(&self) -> Result<(), AnswerTransportError> { + match self { +- Self::Subprocess { control_tx } => { ++ Self::Worker { run_id, bus } => { + let message = WorkerControlEnvelope::cancel_run(); +- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message)) ++ Self::publish_worker_control(*run_id, bus, message) + .await +- .map_err(|_| AnswerTransportError::Timeout)? +- .map_err(|_| AnswerTransportError::Closed) ++ .map_err(|err| Self::answer_error_from_bus(&err)) + } + Self::InProcess { interviewer, .. } => { + interviewer.cancel_all().await; +@@ -352,12 +384,11 @@ impl RunAnswerTransport { + /// in-process steering hub. + async fn steer(&self, text: String, actor: Principal) -> Result<(), AnswerTransportError> { + match self { +- Self::Subprocess { control_tx } => { ++ Self::Worker { run_id, bus } => { + let message = WorkerControlEnvelope::steer(text, actor); +- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message)) ++ Self::publish_worker_control(*run_id, bus, message) + .await +- .map_err(|_| AnswerTransportError::Timeout)? +- .map_err(|_| AnswerTransportError::Closed) ++ .map_err(|err| Self::answer_error_from_bus(&err)) + } + Self::InProcess { steering_hub, .. } => { + steering_hub.deliver_steer(text, Some(actor)); +@@ -368,12 +399,11 @@ impl RunAnswerTransport { + + async fn interrupt(&self, actor: Principal) -> Result<(), AnswerTransportError> { + match self { +- Self::Subprocess { control_tx } => { ++ Self::Worker { run_id, bus } => { + let message = WorkerControlEnvelope::interrupt(actor); +- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message)) ++ Self::publish_worker_control(*run_id, bus, message) + .await +- .map_err(|_| AnswerTransportError::Timeout)? +- .map_err(|_| AnswerTransportError::Closed) ++ .map_err(|err| Self::answer_error_from_bus(&err)) + } + Self::InProcess { steering_hub, .. } => { + steering_hub.interrupt(Some(&actor)); +@@ -388,12 +418,11 @@ impl RunAnswerTransport { + actor: Principal, + ) -> Result<(), AnswerTransportError> { + match self { +- Self::Subprocess { control_tx } => { ++ Self::Worker { run_id, bus } => { + let message = WorkerControlEnvelope::interrupt_then_steer(text, actor); +- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message)) ++ Self::publish_worker_control(*run_id, bus, message) + .await +- .map_err(|_| AnswerTransportError::Timeout)? +- .map_err(|_| AnswerTransportError::Closed) ++ .map_err(|err| Self::answer_error_from_bus(&err)) + } + Self::InProcess { steering_hub, .. } => { + steering_hub.interrupt_then_steer(&text, Some(&actor)); +@@ -410,12 +439,14 @@ impl RunAnswerTransport { + actor: Principal, + ) -> Result<(), PairTransportError> { + match self { +- Self::Subprocess { control_tx } => { ++ Self::Worker { ++ run_id: worker_run_id, ++ bus, ++ } => { + let message = WorkerControlEnvelope::start_pair(run_id, pair_id, target, actor); +- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message)) ++ Self::publish_worker_control(*worker_run_id, bus, message) + .await +- .map_err(|_| PairTransportError::Timeout)? +- .map_err(|_| PairTransportError::Closed) ++ .map_err(|err| Self::pair_error_from_bus(&err)) + } + Self::InProcess { steering_hub, .. } => steering_hub + .start_pair(run_id, pair_id, target, Some(actor)) +@@ -433,7 +464,7 @@ impl RunAnswerTransport { + actor: Principal, + ) -> Result<(), PairTransportError> { + match self { +- Self::Subprocess { control_tx } => { ++ Self::Worker { run_id, bus } => { + let message = WorkerControlEnvelope::pair_message( + pair_id, + message_id, +@@ -441,10 +472,9 @@ impl RunAnswerTransport { + client_message_id.clone(), + actor, + ); +- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message)) ++ Self::publish_worker_control(*run_id, bus, message) + .await +- .map_err(|_| PairTransportError::Timeout)? +- .map_err(|_| PairTransportError::Closed) ++ .map_err(|err| Self::pair_error_from_bus(&err)) + } + Self::InProcess { steering_hub, .. } => steering_hub + .send_pair_message(pair_id, message_id, text, client_message_id, Some(actor)) +@@ -455,12 +485,11 @@ impl RunAnswerTransport { + + async fn end_pair(&self, pair_id: PairId, actor: Principal) -> Result<(), PairTransportError> { + match self { +- Self::Subprocess { control_tx } => { ++ Self::Worker { run_id, bus } => { + let message = WorkerControlEnvelope::end_pair(pair_id, actor); +- timeout(WORKER_CONTROL_ENQUEUE_TIMEOUT, control_tx.send(message)) ++ Self::publish_worker_control(*run_id, bus, message) + .await +- .map_err(|_| PairTransportError::Timeout)? +- .map_err(|_| PairTransportError::Closed) ++ .map_err(|err| Self::pair_error_from_bus(&err)) + } + Self::InProcess { steering_hub, .. } => steering_hub + .end_pair(pair_id, Some(actor)) +@@ -468,6 +497,30 @@ impl RunAnswerTransport { + .map_err(PairTransportError::Control), + } + } ++ ++ async fn pause_run(&self) -> Result<(), AnswerTransportError> { ++ match self { ++ Self::Worker { run_id, bus } => { ++ let message = WorkerControlEnvelope::pause_run(); ++ Self::publish_worker_control(*run_id, bus, message) ++ .await ++ .map_err(|err| Self::answer_error_from_bus(&err)) ++ } ++ Self::InProcess { .. } => Err(AnswerTransportError::Closed), ++ } ++ } ++ ++ async fn unpause_run(&self) -> Result<(), AnswerTransportError> { ++ match self { ++ Self::Worker { run_id, bus } => { ++ let message = WorkerControlEnvelope::unpause_run(); ++ Self::publish_worker_control(*run_id, bus, message) ++ .await ++ .map_err(|err| Self::answer_error_from_bus(&err)) ++ } ++ Self::InProcess { .. } => Err(AnswerTransportError::Closed), ++ } ++ } + } + + #[derive(Debug, Clone)] +@@ -1008,10 +1061,13 @@ pub struct AppState { + session_runtimes: SessionRuntimeManager, + artifact_store: ArtifactStore, + automation_store: Arc, ++ #[cfg(any(test, feature = "test-support"))] ++ automation_materializer_override: Option>, + worker_tokens: WorkerTokenKeys, + started_at: Instant, + resource_sampler: resource_sampler::ResourceSampler, + max_concurrent_runs: usize, ++ pub(crate) worker_control_bus: Arc, + scheduler_notify: Notify, + global_event_tx: broadcast::Sender, + /// Per-run coalescing registry for `GET /runs/{id}/files`. Concurrent +@@ -1048,6 +1104,33 @@ impl AppState { + pub(crate) fn automation_store(&self) -> &AutomationStore { + &self.automation_store + } ++ ++ #[allow( ++ dead_code, ++ reason = "Automation scheduler wiring will call this after issue #398's materialization core." ++ )] ++ pub(crate) async fn materialize_automation_run( ++ &self, ++ input: AutomationRunMaterializeInput, ++ ) -> Result { ++ #[cfg(any(test, feature = "test-support"))] ++ if let Some(materializer) = self.automation_materializer_override.as_ref() { ++ return materializer.materialize(input).await; ++ } ++ ++ let settings = self.server_settings(); ++ let credentials = self ++ .github_credentials(&settings.server.integrations.github) ++ .ok() ++ .flatten(); ++ ProductionAutomationRunMaterializer::new( ++ credentials, ++ self.github_api_base_url.clone(), ++ self.http_client.clone(), ++ ) ++ .materialize(input) ++ .await ++ } + } + + pub(crate) struct AskFabroReadiness { +@@ -1129,21 +1212,25 @@ async fn lock_pull_request_create( + } + + pub(crate) struct AppStateConfig { +- pub(crate) resolved_settings: ResolvedAppStateSettings, ++ pub(crate) resolved_settings: ResolvedAppStateSettings, + pub(crate) registry_factory_override: Option>, +- pub(crate) max_concurrent_runs: usize, +- pub(crate) store: Arc, +- pub(crate) artifact_store: ArtifactStore, +- pub(crate) vault_path: PathBuf, +- pub(crate) variables_path: PathBuf, +- pub(crate) preloaded_vault: Option, +- pub(crate) server_secrets: ServerSecrets, +- pub(crate) env_lookup: EnvLookup, +- pub(crate) github_api_base_url: Option, +- pub(crate) active_config_path: PathBuf, +- pub(crate) http_client: Option, ++ pub(crate) max_concurrent_runs: usize, ++ pub(crate) store: Arc, ++ pub(crate) artifact_store: ArtifactStore, ++ pub(crate) vault_path: PathBuf, ++ pub(crate) variables_path: PathBuf, ++ pub(crate) preloaded_vault: Option, ++ pub(crate) server_secrets: ServerSecrets, ++ pub(crate) env_lookup: EnvLookup, ++ pub(crate) github_api_base_url: Option, ++ pub(crate) active_config_path: PathBuf, ++ pub(crate) http_client: Option, + pub(crate) sandbox_provider_registry: Option, +- pub(crate) shutdown: CancellationToken, ++ pub(crate) shutdown: CancellationToken, ++ #[cfg(test)] ++ pub(crate) worker_control_bus: Option>, ++ #[cfg(any(test, feature = "test-support"))] ++ pub(crate) automation_materializer_override: Option>, + } + + #[derive(Clone)] +@@ -2177,6 +2264,10 @@ pub(crate) fn build_app_state(config: AppStateConfig) -> anyhow::Result anyhow::Result = { ++ #[cfg(test)] ++ { ++ worker_control_bus.unwrap_or_else(|| Arc::new(LocalWorkerControlBus::new())) ++ } ++ #[cfg(not(test))] ++ { ++ Arc::new(LocalWorkerControlBus::new()) ++ } ++ }; + Ok(Arc::new(AppState { + runs: Mutex::new(HashMap::new()), + aggregate_billing: Mutex::new(BillingAccumulator::default()), +@@ -2267,10 +2368,13 @@ pub(crate) fn build_app_state(config: AppStateConfig) -> anyhow::Result { +@@ -2966,6 +3077,7 @@ async fn finish_cancelled_run_before_execution(state: &Arc, run_id: Ru + clear_live_run_state(managed_run); + } + drop(runs); ++ cleanup_worker_control_bus_for_run(state.as_ref(), run_id); + state.scheduler_notify.notify_one(); + } + +@@ -3101,6 +3213,7 @@ fn fail_managed_run(state: &Arc, run_id: RunId, reason: FailureReason, + managed_run.error = Some(message); + clear_live_run_state(managed_run); + } ++ cleanup_worker_control_bus_for_run(state.as_ref(), run_id); + } + + fn update_live_run_from_event(state: &AppState, run_id: RunId, event: &RunEvent) { +@@ -3176,6 +3289,7 @@ fn update_live_run_from_event(state: &AppState, run_id: RunId, event: &RunEvent) + managed_run.active_api_targets.clear(); + managed_run.active_steerable_stages.clear(); + managed_run.active_non_steerable_stages.clear(); ++ cleanup_worker_control_bus_for_run(state, run_id); + } + EventBody::RunFailed(props) => { + managed_run.status = RunStatus::Failed { +@@ -3188,6 +3302,7 @@ fn update_live_run_from_event(state: &AppState, run_id: RunId, event: &RunEvent) + managed_run.active_api_targets.clear(); + managed_run.active_steerable_stages.clear(); + managed_run.active_non_steerable_stages.clear(); ++ cleanup_worker_control_bus_for_run(state, run_id); + } + // Track active agent sessions by steerability. Activated/deactivated + // are leased by session id so stale deactivations cannot clear a newer +@@ -3272,20 +3387,6 @@ async fn drain_worker_stderr(run_id: RunId, stderr: ChildStderr) -> anyhow::Resu + Ok(()) + } + +-async fn pump_worker_control_jsonl( +- mut stdin: ChildStdin, +- mut control_rx: mpsc::Receiver, +-) -> anyhow::Result<()> { +- while let Some(message) = control_rx.recv().await { +- let mut line = serde_json::to_vec(&message)?; +- line.push(b'\n'); +- stdin.write_all(&line).await?; +- stdin.flush().await?; +- } +- +- Ok(()) +-} +- + async fn append_worker_exit_failure( + run_store: &fabro_store::RunDatabase, + run_id: RunId, +@@ -3369,7 +3470,7 @@ fn worker_command( + .arg(run_id.to_string()) + .arg("--mode") + .arg(worker_mode_arg(mode)) +- .stdin(Stdio::piped()) ++ .stdin(Stdio::null()) + .stdout(worker_stdout) + .stderr(Stdio::piped()); + +@@ -4048,25 +4149,6 @@ async fn execute_run_subprocess(state: Arc, run_id: RunId) { + } + } + +- let Some(stdin) = child.stdin.take() else { +- let message = "Worker stdin pipe was unavailable".to_string(); +- tracing::error!(run_id = %run_id, "{message}"); +- let _ = child.start_kill(); +- let failure_event = workflow_event::Event::workflow_run_failed_from_error( +- &WorkflowError::engine(message.clone()), +- fabro_types::RunTiming::default(), +- FailureReason::LaunchFailed, +- None, +- None, +- None, +- None, +- ); +- let _ = workflow_event::append_event(&run_store, &run_id, &failure_event).await; +- fail_managed_run(&state, run_id, FailureReason::LaunchFailed, message); +- state.scheduler_notify.notify_one(); +- return; +- }; +- + let Some(stderr) = child.stderr.take() else { + let message = "Worker stderr pipe was unavailable".to_string(); + tracing::error!(run_id = %run_id, "{message}"); +@@ -4086,15 +4168,16 @@ async fn execute_run_subprocess(state: Arc, run_id: RunId) { + return; + }; + +- let (control_tx, control_rx) = mpsc::channel(WORKER_CONTROL_QUEUE_CAPACITY); + { + let mut runs = state.runs.lock().expect("runs lock poisoned"); + if let Some(managed_run) = runs.get_mut(&run_id) { +- managed_run.answer_transport = Some(RunAnswerTransport::Subprocess { control_tx }); ++ managed_run.answer_transport = Some(RunAnswerTransport::Worker { ++ run_id, ++ bus: Arc::clone(&state.worker_control_bus), ++ }); + } + } + +- let control_task = tokio::spawn(pump_worker_control_jsonl(stdin, control_rx)); + let stderr_task = tokio::spawn(drain_worker_stderr(run_id, stderr)); + + let wait_status = match child.wait().await { +@@ -4119,9 +4202,6 @@ async fn execute_run_subprocess(state: Arc, run_id: RunId) { + } + }; + +- control_task.abort(); +- let _ = control_task.await; +- + match stderr_task.await { + Ok(Ok(())) => {} + Ok(Err(err)) => { +diff --git a/lib/crates/fabro-server/src/server/handler/lifecycle.rs b/lib/crates/fabro-server/src/server/handler/lifecycle.rs +index 0c2697133..d8a2589ac 100644 +--- a/lib/crates/fabro-server/src/server/handler/lifecycle.rs ++++ b/lib/crates/fabro-server/src/server/handler/lifecycle.rs +@@ -387,10 +387,6 @@ async fn cancel_run( + | RunStatus::Running + | RunStatus::Blocked { .. } + | RunStatus::Paused { .. } => { +- let use_cancel_signal = !matches!( +- managed_run.answer_transport, +- Some(RunAnswerTransport::InProcess { .. }) +- ); + let persist_cancelled_status = matches!( + managed_run.status, + RunStatus::Submitted | RunStatus::Pending { .. } | RunStatus::Runnable +@@ -404,9 +400,7 @@ async fn cancel_run( + persist_cancelled_status, + managed_run.answer_transport.clone(), + managed_run.cancel_token.clone(), +- use_cancel_signal +- .then(|| managed_run.cancel_tx.take()) +- .flatten(), ++ managed_run.cancel_tx.take(), + managed_run.worker_pid, + )) + } +@@ -436,22 +430,29 @@ async fn cancel_run( + if let Some(token) = &cancel_token { + token.cancel(); + } +- let sent_cancel_signal = if let Some(cancel_tx) = cancel_tx { ++ let sent_in_process_cancel = if let Some(cancel_tx) = cancel_tx { + let _ = cancel_tx.send(()); + true + } else { + false + }; +- if let Some(answer_transport) = answer_transport { +- if !(sent_cancel_signal && matches!(answer_transport, RunAnswerTransport::InProcess { .. })) ++ let delivered_control = if let Some(answer_transport) = answer_transport { ++ if sent_in_process_cancel ++ && matches!(answer_transport, RunAnswerTransport::InProcess { .. }) + { +- let _ = answer_transport.cancel_run().await; ++ true ++ } else { ++ answer_transport.cancel_run().await.is_ok() ++ } ++ } else { ++ false ++ }; ++ if !delivered_control { ++ if let Some(worker_pid) = worker_pid { ++ #[cfg(unix)] ++ fabro_proc::sigterm(worker_pid); ++ schedule_worker_kill(Arc::clone(&state), id, worker_pid); + } +- } +- if let Some(worker_pid) = worker_pid { +- #[cfg(unix)] +- fabro_proc::sigterm(worker_pid); +- schedule_worker_kill(Arc::clone(&state), id, worker_pid); + } + + if persist_cancelled_status { +@@ -504,9 +505,9 @@ async fn unmanaged_cancel_response( + /// How `pause_run` should enact the transition, chosen from the current run + /// status. + enum PauseMode { +- /// Worker is running; ask it to pause via SIGUSR1. Status flips to +- /// `Paused` once the worker acknowledges. +- Signal { worker_pid: u32 }, ++ /// Worker is running; ask it to pause via the worker control bus. Status ++ /// flips to `Paused` once the worker acknowledges. ++ Transport { transport: RunAnswerTransport }, + /// Worker is blocked on a human gate; flip to `Paused` directly by + /// appending `RunPaused` ourselves. + AppendEvent, +@@ -514,8 +515,9 @@ enum PauseMode { + + /// How `unpause_run` should enact the transition. + enum UnpauseMode { +- /// No outstanding block; ask the worker to resume via SIGUSR2. +- Signal { worker_pid: u32 }, ++ /// No outstanding block; ask the worker to resume via the worker control ++ /// bus. ++ Transport { transport: RunAnswerTransport }, + /// Was paused while blocked; append `RunUnpaused` and let the reducer + /// restore the underlying blocked state from `Paused { prior_block }`. + AppendEvent, +@@ -544,11 +546,11 @@ async fn pause_run( + let runs = state.runs.lock().expect("runs lock poisoned"); + match runs.get(&id) { + Some(managed_run) if managed_run.status == RunStatus::Running => { +- let Some(worker_pid) = managed_run.worker_pid else { ++ let Some(transport) = managed_run.answer_transport.clone() else { + return ApiError::new(StatusCode::CONFLICT, "Run worker is not available.") + .into_response(); + }; +- PauseMode::Signal { worker_pid } ++ PauseMode::Transport { transport } + } + Some(managed_run) if matches!(managed_run.status, RunStatus::Blocked { .. }) => { + PauseMode::AppendEvent +@@ -578,11 +580,14 @@ async fn pause_run( + return ApiError::new(StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response(); + } + match mode { +- PauseMode::Signal { worker_pid } => { +- #[cfg(unix)] +- fabro_proc::sigusr1(worker_pid); +- #[cfg(not(unix))] +- let _ = worker_pid; ++ PauseMode::Transport { transport } => { ++ if transport.pause_run().await.is_err() { ++ return ApiError::new( ++ StatusCode::SERVICE_UNAVAILABLE, ++ "Failed to deliver pause request to the active run.", ++ ) ++ .into_response(); ++ } + } + PauseMode::AppendEvent => { + if let Some(response) = synchronous_transition(state.as_ref(), id, |events| { +@@ -625,11 +630,11 @@ async fn unpause_run( + prior_block: Some(_), + } => UnpauseMode::AppendEvent, + RunStatus::Paused { prior_block: None } => { +- let Some(worker_pid) = managed_run.worker_pid else { ++ let Some(transport) = managed_run.answer_transport.clone() else { + return ApiError::new(StatusCode::CONFLICT, "Run worker is not available.") + .into_response(); + }; +- UnpauseMode::Signal { worker_pid } ++ UnpauseMode::Transport { transport } + } + _ => { + return ApiError::new(StatusCode::CONFLICT, "Run is not paused.") +@@ -658,11 +663,14 @@ async fn unpause_run( + return ApiError::new(StatusCode::INTERNAL_SERVER_ERROR, err.to_string()).into_response(); + } + match mode { +- UnpauseMode::Signal { worker_pid } => { +- #[cfg(unix)] +- fabro_proc::sigusr2(worker_pid); +- #[cfg(not(unix))] +- let _ = worker_pid; ++ UnpauseMode::Transport { transport } => { ++ if transport.unpause_run().await.is_err() { ++ return ApiError::new( ++ StatusCode::SERVICE_UNAVAILABLE, ++ "Failed to deliver unpause request to the active run.", ++ ) ++ .into_response(); ++ } + } + UnpauseMode::AppendEvent => { + if let Some(response) = synchronous_transition(state.as_ref(), id, |events| { +diff --git a/lib/crates/fabro-server/src/server/handler/mod.rs b/lib/crates/fabro-server/src/server/handler/mod.rs +index 0804cccb3..33c6aa8eb 100644 +--- a/lib/crates/fabro-server/src/server/handler/mod.rs ++++ b/lib/crates/fabro-server/src/server/handler/mod.rs +@@ -15,7 +15,7 @@ mod lifecycle; + mod models; + mod pair; + mod pull_requests; +-mod runs; ++pub(in crate::server) mod runs; + mod sandbox; + mod sandboxes; + mod secrets; +@@ -23,6 +23,7 @@ mod sessions; + mod steer; + pub(in crate::server) mod system; + mod variables; ++mod worker_control; + + pub(super) use system::{health, openapi_spec}; + +@@ -167,6 +168,7 @@ pub(super) fn real_routes() -> Router> { + .merge(models::routes()) + .merge(secrets::routes()) + .merge(variables::routes()) ++ .merge(worker_control::routes()) + .merge(sessions::routes()) + .merge(system::routes()) + .merge(completions::routes()) +diff --git a/lib/crates/fabro-server/src/server/handler/runs.rs b/lib/crates/fabro-server/src/server/handler/runs.rs +index c4928d261..3e83bddbb 100644 +--- a/lib/crates/fabro-server/src/server/handler/runs.rs ++++ b/lib/crates/fabro-server/src/server/handler/runs.rs +@@ -21,9 +21,10 @@ use fabro_interview::AnswerSubmission; + use fabro_llm::client::Client as LlmClient; + use fabro_types::settings::ResolveEnvError; + use fabro_types::{ +- Principal, RunClientProvenance, RunId, RunProvenance, RunServerProvenance, StageContextWindow, +- StageContextWindowStaleness, StageContextWindowUnavailableReason, StageHandler, +- StageModelUsage, StageProjection, SystemActorKind, WorkflowSettings, parse_blob_ref, ++ AutomationRef, Principal, RunClientProvenance, RunId, RunProvenance, RunServerProvenance, ++ StageContextWindow, StageContextWindowStaleness, StageContextWindowUnavailableReason, ++ StageHandler, StageModelUsage, StageProjection, SystemActorKind, WorkflowSettings, ++ parse_blob_ref, + }; + use fabro_util::version::FABRO_VERSION; + use fabro_workflow::command_log::{command_log_path, read_json_string_blob, read_log_slice}; +@@ -598,12 +599,50 @@ async fn create_run( + Err(err) => return ApiError::bad_request(err.to_string()).into_response(), + }; + let explicit_title_supplied = req.title.is_some(); ++ Box::pin(create_run_from_manifest( ++ state, ++ CreateRunFromManifestRequest { ++ manifest: req, ++ submitted_manifest_bytes: body.to_vec(), ++ explicit_run_id: None, ++ explicit_title_supplied, ++ actor, ++ headers, ++ automation: None, ++ }, ++ )) ++ .await ++} ++ ++pub(crate) struct CreateRunFromManifestRequest { ++ pub(crate) manifest: RunManifest, ++ pub(crate) submitted_manifest_bytes: Vec, ++ pub(crate) explicit_run_id: Option, ++ pub(crate) explicit_title_supplied: bool, ++ pub(crate) actor: Principal, ++ pub(crate) headers: HeaderMap, ++ pub(crate) automation: Option, ++} ++ ++pub(crate) async fn create_run_from_manifest( ++ state: Arc, ++ request: CreateRunFromManifestRequest, ++) -> Response { ++ let CreateRunFromManifestRequest { ++ manifest, ++ submitted_manifest_bytes, ++ explicit_run_id, ++ explicit_title_supplied, ++ actor, ++ headers, ++ automation, ++ } = request; + let manifest_run_defaults = state.manifest_run_defaults(); + let manifest_environment_defaults = state.manifest_environment_defaults(); + let mut prepared = match run_manifest::prepare_manifest_with_environment_defaults( + manifest_run_defaults.as_ref(), + manifest_environment_defaults.as_ref(), +- &req, ++ &manifest, + ) { + Ok(prepared) => prepared, + Err(err) => return ApiError::bad_request(err.to_string()).into_response(), +@@ -612,7 +651,9 @@ async fn create_run( + return ApiError::bad_request(format!("Run config variable interpolation failed: {err}")) + .into_response(); + } +- let run_id = prepared.run_id.unwrap_or_else(RunId::new); ++ let run_id = explicit_run_id ++ .or(prepared.run_id) ++ .unwrap_or_else(RunId::new); + let provider = run_manifest::effective_sandbox_provider(&prepared.settings.run); + if let Some(error) = + run_manifest::sandbox_provider_policy_error(&state.server_settings(), provider) +@@ -653,7 +694,8 @@ async fn create_run( + ); + create_input.run_id = Some(run_id); + create_input.provenance = Some(run_provenance(&headers, &actor)); +- create_input.submitted_manifest_bytes = Some(body.to_vec()); ++ create_input.submitted_manifest_bytes = Some(submitted_manifest_bytes); ++ create_input.automation = automation; + + let storage_root = match resolve_interp_string(&state.server_settings().server.storage.root) { + Ok(path) => PathBuf::from(path), +diff --git a/lib/crates/fabro-server/src/server/handler/worker_control.rs b/lib/crates/fabro-server/src/server/handler/worker_control.rs +new file mode 100644 +index 000000000..d643a904e +--- /dev/null ++++ b/lib/crates/fabro-server/src/server/handler/worker_control.rs +@@ -0,0 +1,166 @@ ++use std::sync::Arc; ++ ++use axum::extract::ws::{ ++ CloseFrame, Message as WsMessage, WebSocket, WebSocketUpgrade, close_code, ++}; ++use fabro_interview::{ ++ WORKER_CONTROL_INVALID_CURSOR_REASON, WORKER_CONTROL_PONG_TIMEOUT_REASON, ++ WORKER_CONTROL_WS_LIVENESS_TIMEOUT, WORKER_CONTROL_WS_PING_INTERVAL, ++ WorkerControlDeliveryFrame, ++}; ++use futures_util::{SinkExt, StreamExt}; ++use tokio::time::{self, Instant, MissedTickBehavior}; ++ ++use super::super::{ ++ ApiError, AppState, IntoResponse, Query, RequireWorkerRunScoped, Response, Router, State, ++ StatusCode, get, ++}; ++use crate::worker_control::{WorkerControlBusError, WorkerControlCursor, WorkerControlReceiver}; ++ ++#[derive(Debug, serde::Deserialize)] ++struct WorkerControlStreamQuery { ++ after: Option, ++} ++ ++pub(super) fn routes() -> Router> { ++ Router::new().route( ++ "/runs/{id}/worker/control-stream", ++ get(worker_control_stream), ++ ) ++} ++ ++async fn worker_control_stream( ++ RequireWorkerRunScoped(id): RequireWorkerRunScoped, ++ State(state): State>, ++ Query(query): Query, ++ ws: WebSocketUpgrade, ++) -> Response { ++ let cached = match state.store.get_cached_run(&id).await { ++ Ok(Some(cached)) => cached, ++ Ok(None) => return ApiError::not_found("Run not found.").into_response(), ++ Err(err) => { ++ return ApiError::new(StatusCode::INTERNAL_SERVER_ERROR, err.to_string()) ++ .into_response(); ++ } ++ }; ++ if cached.projection.archived_at.is_some() { ++ return ApiError::new(StatusCode::CONFLICT, "Run is archived.").into_response(); ++ } ++ if cached.projection.status.is_terminal() { ++ return ApiError::new( ++ StatusCode::CONFLICT, ++ "Run is terminal and cannot accept worker control streams.", ++ ) ++ .into_response(); ++ } ++ ++ let cursor = match WorkerControlCursor::from_after_query(query.after.as_deref()) { ++ Ok(cursor) => cursor, ++ Err(err @ WorkerControlBusError::InvalidCursor { .. }) => { ++ return ApiError::new(StatusCode::GONE, err.to_string()).into_response(); ++ } ++ Err(err) => return worker_control_bus_error_response(&err), ++ }; ++ let receiver = match state.worker_control_bus.subscribe(id, cursor).await { ++ Ok(receiver) => receiver, ++ Err(err @ WorkerControlBusError::InvalidCursor { .. }) => { ++ return ApiError::new(StatusCode::GONE, err.to_string()).into_response(); ++ } ++ Err(err) => return worker_control_bus_error_response(&err), ++ }; ++ ++ ws.on_upgrade(move |socket| worker_control_websocket(socket, receiver)) ++} ++ ++fn worker_control_bus_error_response(err: &WorkerControlBusError) -> Response { ++ let status = match err { ++ WorkerControlBusError::Closed | WorkerControlBusError::Unavailable => { ++ StatusCode::SERVICE_UNAVAILABLE ++ } ++ WorkerControlBusError::PublishTimeout => StatusCode::SERVICE_UNAVAILABLE, ++ WorkerControlBusError::InvalidCursor { .. } => StatusCode::GONE, ++ }; ++ ApiError::new(status, err.to_string()).into_response() ++} ++ ++async fn worker_control_websocket(socket: WebSocket, mut receiver: WorkerControlReceiver) { ++ let (mut sender, mut receiver_ws) = socket.split(); ++ let mut ping_interval = time::interval(WORKER_CONTROL_WS_PING_INTERVAL); ++ ping_interval.set_missed_tick_behavior(MissedTickBehavior::Delay); ++ let mut last_liveness = Instant::now(); ++ let liveness_timeout = time::sleep_until(last_liveness + WORKER_CONTROL_WS_LIVENESS_TIMEOUT); ++ tokio::pin!(liveness_timeout); ++ ++ loop { ++ liveness_timeout ++ .as_mut() ++ .reset(last_liveness + WORKER_CONTROL_WS_LIVENESS_TIMEOUT); ++ ++ tokio::select! { ++ delivery = receiver.recv() => { ++ let Some(delivery) = delivery else { ++ let _ = sender.send(WsMessage::Close(None)).await; ++ return; ++ }; ++ let delivery = match delivery { ++ Ok(delivery) => delivery, ++ Err(WorkerControlBusError::InvalidCursor { .. }) => { ++ let _ = sender.send(invalid_cursor_close_message()).await; ++ return; ++ } ++ Err(_) => { ++ let _ = sender.send(WsMessage::Close(None)).await; ++ return; ++ } ++ }; ++ let frame = WorkerControlDeliveryFrame { ++ id: delivery.id.to_string(), ++ envelope: delivery.envelope, ++ }; ++ let Ok(text) = serde_json::to_string(&frame) else { ++ let _ = sender.send(WsMessage::Close(None)).await; ++ return; ++ }; ++ if sender.send(WsMessage::Text(text.into())).await.is_err() { ++ return; ++ } ++ } ++ message = receiver_ws.next() => { ++ let Some(message) = message else { ++ return; ++ }; ++ match message { ++ Ok(WsMessage::Ping(payload)) => { ++ last_liveness = Instant::now(); ++ if sender.send(WsMessage::Pong(payload)).await.is_err() { ++ return; ++ } ++ } ++ Ok(WsMessage::Pong(_) | WsMessage::Text(_) | WsMessage::Binary(_)) => { ++ last_liveness = Instant::now(); ++ } ++ Ok(WsMessage::Close(_)) | Err(_) => return, ++ } ++ } ++ _ = ping_interval.tick() => { ++ if sender.send(WsMessage::Ping(Vec::new().into())).await.is_err() { ++ return; ++ } ++ } ++ () = &mut liveness_timeout => { ++ let _ = sender.send(WsMessage::Close(Some(CloseFrame { ++ code: close_code::AWAY, ++ reason: WORKER_CONTROL_PONG_TIMEOUT_REASON.into(), ++ }))).await; ++ return; ++ } ++ } ++ } ++} ++ ++fn invalid_cursor_close_message() -> WsMessage { ++ WsMessage::Close(Some(CloseFrame { ++ code: close_code::POLICY, ++ reason: WORKER_CONTROL_INVALID_CURSOR_REASON.into(), ++ })) ++} +diff --git a/lib/crates/fabro-server/src/server/tests.rs b/lib/crates/fabro-server/src/server/tests.rs +index df5d26152..866db6757 100644 +--- a/lib/crates/fabro-server/src/server/tests.rs ++++ b/lib/crates/fabro-server/src/server/tests.rs +@@ -9,10 +9,12 @@ use std::sync::{Arc as StdArc, Mutex as StdMutex}; + use axum::body::Body; + use axum::http::{Method, Request, header}; + use chrono::{Duration as ChronoDuration, Utc}; ++use fabro_automation::{AutomationId, AutomationTarget}; + use fabro_config::ServerSettingsBuilder; + use fabro_config::bind::Bind; + use fabro_interview::{ +- AnswerValue, ControlInterviewer, Interviewer, Question, WorkerControlMessage, ++ AnswerValue, ControlInterviewer, Interviewer, Question, WorkerControlDeliveryFrame, ++ WorkerControlEnvelope, WorkerControlMessage, + }; + use fabro_llm::types::{Message as LlmMessage, Request as LlmRequest, TokenCounts}; + use fabro_model::catalog::LlmCatalogSettings; +@@ -32,6 +34,8 @@ use httpmock::Method::{GET, POST}; + use httpmock::MockServer; + use serde_json::json; + use tokio_stream::StreamExt as _; ++use tokio_tungstenite::tungstenite::client::IntoClientRequest; ++use tokio_tungstenite::tungstenite::protocol::Message as WebSocketMessage; + use tower::ServiceExt; + use tracing::field::{Field, Visit}; + use tracing::{Event as TracingEvent, Subscriber, subscriber}; +@@ -40,9 +44,13 @@ use tracing_subscriber::prelude::*; + use tracing_subscriber::{Layer, Registry}; + + use super::*; ++use crate::automation_materializer::AutomationRunMaterializeInput; + use crate::github_webhooks::compute_signature; + use crate::jwt_auth::{AuthMode, ConfiguredAuth}; + use crate::test_support::*; ++use crate::worker_control::{ ++ LocalWorkerControlBus, WorkerControlBus, WorkerControlCursor, WorkerControlReceiver, ++}; + + const MINIMAL_DOT: &str = r#"digraph Test { + graph [goal="Test"] +@@ -658,6 +666,292 @@ fn bearer_request(method: Method, path: &str, bearer: &str, body: Body) -> Reque + .unwrap() + } + ++struct WorkerControlWsTestServer { ++ base_url: String, ++ task: tokio::task::JoinHandle<()>, ++} ++ ++impl WorkerControlWsTestServer { ++ async fn spawn(app: Router) -> Self { ++ let listener = tokio::net::TcpListener::bind("127.0.0.1:0") ++ .await ++ .expect("test WebSocket listener should bind"); ++ let addr = listener ++ .local_addr() ++ .expect("test WebSocket listener should have a local address"); ++ let task = tokio::spawn(async move { ++ let result = axum::serve( ++ listener, ++ app.into_make_service_with_connect_info::(), ++ ) ++ .await; ++ if let Err(err) = result { ++ tracing::debug!(error = %err, "test WebSocket server stopped"); ++ } ++ }); ++ Self { ++ base_url: format!("ws://{addr}"), ++ task, ++ } ++ } ++ ++ fn worker_control_url(&self, run_id: RunId, after: Option<&str>) -> String { ++ let mut url = format!( ++ "{}/api/v1/runs/{run_id}/worker/control-stream", ++ self.base_url ++ ); ++ if let Some(after) = after { ++ url.push_str("?after="); ++ url.push_str(after); ++ } ++ url ++ } ++} ++ ++impl Drop for WorkerControlWsTestServer { ++ fn drop(&mut self) { ++ self.task.abort(); ++ } ++} ++ ++fn worker_control_ws_request( ++ server: &WorkerControlWsTestServer, ++ run_id: RunId, ++ bearer: Option<&str>, ++ after: Option<&str>, ++) -> Request<()> { ++ let mut request = server ++ .worker_control_url(run_id, after) ++ .into_client_request() ++ .expect("test worker-control WebSocket request should build"); ++ if let Some(bearer) = bearer { ++ request.headers_mut().insert( ++ header::AUTHORIZATION, ++ format!("Bearer {bearer}") ++ .parse() ++ .expect("test bearer header should parse"), ++ ); ++ } ++ request ++} ++ ++async fn connect_worker_control_ws( ++ server: &WorkerControlWsTestServer, ++ run_id: RunId, ++ bearer: &str, ++ after: Option<&str>, ++) -> tokio_tungstenite::WebSocketStream> { ++ let request = worker_control_ws_request(server, run_id, Some(bearer), after); ++ let (socket, _) = tokio_tungstenite::connect_async(request) ++ .await ++ .expect("worker-control WebSocket should connect"); ++ socket ++} ++ ++async fn assert_worker_control_ws_rejected( ++ server: &WorkerControlWsTestServer, ++ run_id: RunId, ++ bearer: Option<&str>, ++ after: Option<&str>, ++ expected: StatusCode, ++) { ++ let request = worker_control_ws_request(server, run_id, bearer, after); ++ let error = tokio_tungstenite::connect_async(request) ++ .await ++ .expect_err("worker-control WebSocket should be rejected"); ++ match error { ++ tokio_tungstenite::tungstenite::Error::Http(response) => { ++ assert_eq!(response.status(), expected); ++ } ++ other => panic!("expected HTTP rejection {expected}, got {other:#}"), ++ } ++} ++ ++async fn next_worker_control_frame( ++ socket: &mut tokio_tungstenite::WebSocketStream< ++ tokio_tungstenite::MaybeTlsStream, ++ >, ++) -> WorkerControlDeliveryFrame { ++ let message = tokio::time::timeout(std::time::Duration::from_secs(2), async { ++ loop { ++ let message = futures_util::StreamExt::next(socket) ++ .await ++ .expect("worker-control WebSocket should remain open") ++ .expect("worker-control WebSocket frame should be ok"); ++ match message { ++ WebSocketMessage::Text(text) => return text, ++ WebSocketMessage::Ping(payload) => { ++ futures_util::SinkExt::send(socket, WebSocketMessage::Pong(payload)) ++ .await ++ .expect("test worker-control pong should send"); ++ } ++ WebSocketMessage::Pong(_) ++ | WebSocketMessage::Binary(_) ++ | WebSocketMessage::Frame(_) => {} ++ WebSocketMessage::Close(frame) => { ++ panic!("worker-control WebSocket closed before text frame: {frame:?}"); ++ } ++ } ++ } ++ }) ++ .await ++ .expect("worker-control frame should arrive"); ++ serde_json::from_str(message.as_str()).expect("worker-control delivery frame should parse") ++} ++ ++#[tokio::test(flavor = "current_thread")] ++async fn worker_control_stream_rejects_missing_user_and_cross_run_auth() { ++ let (_state, app) = jwt_auth_app(); ++ let user_bearer = issue_test_user_jwt(); ++ let run_id = create_run_with_bearer(&app, &user_bearer).await; ++ let worker_bearer = issue_test_worker_token(&run_id); ++ let other_run_id = create_run_with_bearer(&app, &user_bearer).await; ++ let other_worker_bearer = issue_test_worker_token(&other_run_id); ++ let server = WorkerControlWsTestServer::spawn(app).await; ++ ++ assert_worker_control_ws_rejected(&server, run_id, None, None, StatusCode::UNAUTHORIZED).await; ++ assert_worker_control_ws_rejected( ++ &server, ++ run_id, ++ Some(&user_bearer), ++ None, ++ StatusCode::FORBIDDEN, ++ ) ++ .await; ++ assert_worker_control_ws_rejected( ++ &server, ++ run_id, ++ Some(&other_worker_bearer), ++ None, ++ StatusCode::FORBIDDEN, ++ ) ++ .await; ++ ++ let mut socket = connect_worker_control_ws(&server, run_id, &worker_bearer, None).await; ++ futures_util::SinkExt::send(&mut socket, WebSocketMessage::Close(None)) ++ .await ++ .unwrap(); ++} ++ ++#[tokio::test(flavor = "current_thread")] ++async fn worker_control_stream_start_subscription_delivers_frames() { ++ let (state, app) = jwt_auth_app(); ++ let user_bearer = issue_test_user_jwt(); ++ let run_id = create_run_with_bearer(&app, &user_bearer).await; ++ let worker_bearer = issue_test_worker_token(&run_id); ++ let server = WorkerControlWsTestServer::spawn(app).await; ++ let mut socket = connect_worker_control_ws(&server, run_id, &worker_bearer, None).await; ++ ++ let expected = WorkerControlEnvelope::cancel_run(); ++ let id = state ++ .worker_control_bus ++ .publish(run_id, expected.clone()) ++ .await ++ .unwrap(); ++ let frame = next_worker_control_frame(&mut socket).await; ++ ++ assert_eq!(frame.id, id.to_string()); ++ assert_eq!(frame.envelope, expected); ++} ++ ++#[tokio::test(flavor = "current_thread")] ++async fn worker_control_stream_after_subscription_delivers_only_later_frames() { ++ let (state, app) = jwt_auth_app(); ++ let user_bearer = issue_test_user_jwt(); ++ let run_id = create_run_with_bearer(&app, &user_bearer).await; ++ let worker_bearer = issue_test_worker_token(&run_id); ++ let first = state ++ .worker_control_bus ++ .publish(run_id, WorkerControlEnvelope::cancel_run()) ++ .await ++ .unwrap(); ++ let expected = WorkerControlEnvelope::pause_run(); ++ let second = state ++ .worker_control_bus ++ .publish(run_id, expected.clone()) ++ .await ++ .unwrap(); ++ let server = WorkerControlWsTestServer::spawn(app).await; ++ ++ let mut socket = ++ connect_worker_control_ws(&server, run_id, &worker_bearer, Some(first.as_str())).await; ++ let frame = next_worker_control_frame(&mut socket).await; ++ ++ assert_eq!(frame.id, second.to_string()); ++ assert_eq!(frame.envelope, expected); ++} ++ ++#[tokio::test(flavor = "current_thread")] ++async fn worker_control_stream_invalid_cursor_is_http_gone_before_upgrade() { ++ let (_state, app) = jwt_auth_app(); ++ let user_bearer = issue_test_user_jwt(); ++ let run_id = create_run_with_bearer(&app, &user_bearer).await; ++ let worker_bearer = issue_test_worker_token(&run_id); ++ let server = WorkerControlWsTestServer::spawn(app).await; ++ ++ assert_worker_control_ws_rejected( ++ &server, ++ run_id, ++ Some(&worker_bearer), ++ Some("local:999"), ++ StatusCode::GONE, ++ ) ++ .await; ++} ++ ++#[tokio::test(flavor = "current_thread")] ++async fn worker_control_stream_rejects_missing_terminal_and_archived_runs() { ++ let (state, app) = jwt_auth_app(); ++ let user_bearer = issue_test_user_jwt(); ++ let missing_run_id = RunId::new(); ++ let missing_worker_bearer = issue_test_worker_token(&missing_run_id); ++ let terminal_run_id = RunId::new(); ++ create_succeeded_run(&state, terminal_run_id).await; ++ let terminal_worker_bearer = issue_test_worker_token(&terminal_run_id); ++ let archived_run_id = RunId::new(); ++ create_succeeded_run(&state, archived_run_id).await; ++ let archived_worker_bearer = issue_test_worker_token(&archived_run_id); ++ let archive_response = app ++ .clone() ++ .oneshot( ++ Request::builder() ++ .method("POST") ++ .uri(api(&format!("/runs/{archived_run_id}/archive"))) ++ .header(header::AUTHORIZATION, format!("Bearer {user_bearer}")) ++ .body(Body::empty()) ++ .unwrap(), ++ ) ++ .await ++ .unwrap(); ++ response_json!(archive_response, StatusCode::OK).await; ++ let server = WorkerControlWsTestServer::spawn(app).await; ++ ++ assert_worker_control_ws_rejected( ++ &server, ++ missing_run_id, ++ Some(&missing_worker_bearer), ++ None, ++ StatusCode::NOT_FOUND, ++ ) ++ .await; ++ assert_worker_control_ws_rejected( ++ &server, ++ terminal_run_id, ++ Some(&terminal_worker_bearer), ++ None, ++ StatusCode::CONFLICT, ++ ) ++ .await; ++ assert_worker_control_ws_rejected( ++ &server, ++ archived_run_id, ++ Some(&archived_worker_bearer), ++ None, ++ StatusCode::CONFLICT, ++ ) ++ .await; ++} ++ + fn json_bearer_request( + method: Method, + path: &str, +@@ -1649,6 +1943,8 @@ fn slack_app_state_with_secret_sources( + http_client: Some(fabro_http::test_http_client().expect("test HTTP client should build")), + sandbox_provider_registry: None, + shutdown: tokio_util::sync::CancellationToken::new(), ++ worker_control_bus: None, ++ automation_materializer_override: None, + }) + .expect("slack test app state should build") + } +@@ -1748,12 +2044,31 @@ fn slack_service_respects_disabled_server_config_even_with_vault_tokens() { + http_client: Some(fabro_http::test_http_client().expect("test HTTP client should build")), + sandbox_provider_registry: None, + shutdown: tokio_util::sync::CancellationToken::new(), ++ worker_control_bus: None, ++ automation_materializer_override: None, + }) + .expect("slack disabled test app state should build"); + + assert!(state.slack_service.is_none()); + } + ++#[cfg(unix)] ++#[test] ++fn worker_command_uses_null_stdin_and_token_env() { ++ let storage_dir = tempfile::tempdir().unwrap(); ++ let state = worker_command_test_state(storage_dir.path(), &["dev-token"], Some(TEST_DEV_TOKEN)); ++ let cmd = worker_command( ++ state.as_ref(), ++ RunId::new(), ++ RunExecutionMode::Start, ++ storage_dir.path(), ++ false, ++ ) ++ .unwrap(); ++ ++ assert_worker_command_passes_token_only_by_env(&cmd); ++} ++ + #[cfg(unix)] + #[test] + fn worker_command_default_token_omits_agent_run_tools_scope() { +@@ -2052,6 +2367,8 @@ methods = ["dev-token"] + http_client: Some(fabro_http::test_http_client().expect("test HTTP client should build")), + sandbox_provider_registry: None, + shutdown: tokio_util::sync::CancellationToken::new(), ++ worker_control_bus: None, ++ automation_materializer_override: None, + }) else { + panic!("build_app_state should require SESSION_SECRET") + }; +@@ -2179,6 +2496,8 @@ fn build_test_app_state_with_vault_path(vault_path: &Path) -> anyhow::Result crate::worker_token:: + .claims + } + ++async fn worker_transport_with_receiver( ++ run_id: RunId, ++) -> (RunAnswerTransport, WorkerControlReceiver) { ++ let bus = StdArc::new(LocalWorkerControlBus::new()); ++ let receiver = bus ++ .subscribe(run_id, WorkerControlCursor::Start) ++ .await ++ .expect("test worker bus should subscribe"); ++ // Ensure the subscription task is waiting before the test publishes. ++ tokio::task::yield_now().await; ++ let bus: StdArc = bus; ++ let transport = RunAnswerTransport::Worker { run_id, bus }; ++ (transport, receiver) ++} ++ ++async fn recv_worker_control_envelope( ++ receiver: &mut WorkerControlReceiver, ++) -> WorkerControlEnvelope { ++ receiver ++ .recv() ++ .await ++ .expect("test worker control receiver should stay open") ++ .expect("test worker control delivery should succeed") ++ .envelope ++} ++ + #[tokio::test] +-async fn subprocess_answer_transport_cancel_run_enqueues_cancel_message() { +- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1); +- let transport = RunAnswerTransport::Subprocess { control_tx }; ++async fn worker_answer_transport_cancel_run_publishes_cancel_message() { ++ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await; + + transport.cancel_run().await.unwrap(); + + assert_eq!( +- control_rx.recv().await, +- Some(WorkerControlEnvelope::cancel_run()) ++ recv_worker_control_envelope(&mut control_rx).await, ++ WorkerControlEnvelope::cancel_run() + ); + } + + #[tokio::test] +-async fn subprocess_answer_transport_steer_enqueues_plain_steer_message() { +- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1); +- let transport = RunAnswerTransport::Subprocess { control_tx }; ++async fn worker_answer_transport_steer_publishes_plain_steer_message() { ++ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await; + let actor = Principal::System { + system_kind: SystemActorKind::Engine, + }; +@@ -2385,15 +2728,14 @@ async fn subprocess_answer_transport_steer_enqueues_plain_steer_message() { + .unwrap(); + + assert_eq!( +- control_rx.recv().await, +- Some(WorkerControlEnvelope::steer("try again", actor)) ++ recv_worker_control_envelope(&mut control_rx).await, ++ WorkerControlEnvelope::steer("try again", actor) + ); + } + + #[tokio::test] +-async fn subprocess_answer_transport_interrupt_enqueues_interrupt_message() { +- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1); +- let transport = RunAnswerTransport::Subprocess { control_tx }; ++async fn worker_answer_transport_interrupt_publishes_interrupt_message() { ++ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await; + let actor = Principal::System { + system_kind: SystemActorKind::Engine, + }; +@@ -2401,15 +2743,14 @@ async fn subprocess_answer_transport_interrupt_enqueues_interrupt_message() { + transport.interrupt(actor.clone()).await.unwrap(); + + assert_eq!( +- control_rx.recv().await, +- Some(WorkerControlEnvelope::interrupt(actor)) ++ recv_worker_control_envelope(&mut control_rx).await, ++ WorkerControlEnvelope::interrupt(actor) + ); + } + + #[tokio::test] +-async fn subprocess_answer_transport_interrupt_then_steer_enqueues_single_combined_message() { +- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1); +- let transport = RunAnswerTransport::Subprocess { control_tx }; ++async fn worker_answer_transport_interrupt_then_steer_publishes_single_combined_message() { ++ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await; + let actor = Principal::System { + system_kind: SystemActorKind::Engine, + }; +@@ -2420,19 +2761,15 @@ async fn subprocess_answer_transport_interrupt_then_steer_enqueues_single_combin + .unwrap(); + + assert_eq!( +- control_rx.recv().await, +- Some(WorkerControlEnvelope::interrupt_then_steer( +- "try again", +- actor +- )) ++ recv_worker_control_envelope(&mut control_rx).await, ++ WorkerControlEnvelope::interrupt_then_steer("try again", actor) + ); + } + + #[tokio::test] +-async fn subprocess_answer_transport_pair_commands_enqueue_control_messages() { +- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(3); +- let transport = RunAnswerTransport::Subprocess { control_tx }; ++async fn worker_answer_transport_pair_commands_publish_control_messages() { + let run_id = fixtures::RUN_1; ++ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await; + let pair_id = "01HZX6M29F1CD5YYMHT1F5D7WQ".parse().unwrap(); + let message_id = "01HZX6M4D7Y1QW0Q0P6V8Z4DR5".parse().unwrap(); + let actor = Principal::System { +@@ -2457,27 +2794,39 @@ async fn subprocess_answer_transport_pair_commands_enqueue_control_messages() { + transport.end_pair(pair_id, actor.clone()).await.unwrap(); + + assert_eq!( +- control_rx.recv().await, +- Some(WorkerControlEnvelope::start_pair( +- run_id, +- pair_id, +- target, +- actor.clone() +- )) ++ recv_worker_control_envelope(&mut control_rx).await, ++ WorkerControlEnvelope::start_pair(run_id, pair_id, target, actor.clone()) + ); + assert_eq!( +- control_rx.recv().await, +- Some(WorkerControlEnvelope::pair_message( ++ recv_worker_control_envelope(&mut control_rx).await, ++ WorkerControlEnvelope::pair_message( + pair_id, + message_id, + "inspect this", + Some("client-1".to_string()), + actor.clone() +- )) ++ ) + ); + assert_eq!( +- control_rx.recv().await, +- Some(WorkerControlEnvelope::end_pair(pair_id, actor)) ++ recv_worker_control_envelope(&mut control_rx).await, ++ WorkerControlEnvelope::end_pair(pair_id, actor) ++ ); ++} ++ ++#[tokio::test] ++async fn worker_answer_transport_pause_and_unpause_publish_control_messages() { ++ let (transport, mut control_rx) = worker_transport_with_receiver(fixtures::RUN_1).await; ++ ++ transport.pause_run().await.unwrap(); ++ transport.unpause_run().await.unwrap(); ++ ++ assert_eq!( ++ recv_worker_control_envelope(&mut control_rx).await, ++ WorkerControlEnvelope::pause_run() ++ ); ++ assert_eq!( ++ recv_worker_control_envelope(&mut control_rx).await, ++ WorkerControlEnvelope::unpause_run() + ); + } + +@@ -2854,6 +3203,155 @@ async fn post_run_manifest(app: &Router, manifest: serde_json::Value) -> serde_j + response_json!(response, StatusCode::CREATED).await + } + ++#[tokio::test] ++async fn post_runs_create_regression_keeps_api_behavior_without_automation_metadata() { ++ let state = TestAppStateBuilder::new().env_lookup(|_| None).build(); ++ let app = crate::test_support::build_test_router(Arc::clone(&state)); ++ let mut manifest = minimal_manifest_json(MINIMAL_DOT); ++ manifest["title"] = json!("API title"); ++ ++ let body = post_run_manifest(&app, manifest).await; ++ let run_id: RunId = body["id"].as_str().unwrap().parse().unwrap(); ++ ++ assert_eq!(body["title"], "API title"); ++ assert!(body["automation"].is_null()); ++ assert_eq!(body["lifecycle"]["status"]["kind"], "submitted"); ++ let summary = state ++ .store ++ .get_cached_summary(&run_id, Utc::now()) ++ .await ++ .unwrap() ++ .unwrap(); ++ assert!(summary.automation.is_none()); ++} ++ ++#[tokio::test] ++async fn create_run_from_manifest_helper_persists_without_automation_metadata() { ++ let state = TestAppStateBuilder::new().env_lookup(|_| None).build(); ++ let manifest: RunManifest = serde_json::from_value(minimal_manifest_json(MINIMAL_DOT)).unwrap(); ++ let submitted_manifest_bytes = serde_json::to_vec(&manifest).unwrap(); ++ let run_id = RunId::new(); ++ ++ let response = Box::pin(handler::runs::create_run_from_manifest( ++ Arc::clone(&state), ++ handler::runs::CreateRunFromManifestRequest { ++ manifest, ++ submitted_manifest_bytes, ++ explicit_run_id: Some(run_id), ++ explicit_title_supplied: false, ++ actor: Principal::System { ++ system_kind: SystemActorKind::Engine, ++ }, ++ headers: HeaderMap::new(), ++ automation: None, ++ }, ++ )) ++ .await; ++ ++ let body = response_json!(response, StatusCode::CREATED).await; ++ assert_eq!(body["id"], run_id.to_string()); ++ assert!(body["automation"].is_null()); ++ let summary = state ++ .store ++ .get_cached_summary(&run_id, Utc::now()) ++ .await ++ .unwrap() ++ .unwrap(); ++ assert!(summary.automation.is_none()); ++} ++ ++#[tokio::test] ++async fn create_run_from_manifest_helper_persists_automation_metadata() { ++ let state = TestAppStateBuilder::new().env_lookup(|_| None).build(); ++ let manifest: RunManifest = serde_json::from_value(minimal_manifest_json(MINIMAL_DOT)).unwrap(); ++ let submitted_manifest_bytes = serde_json::to_vec(&manifest).unwrap(); ++ let run_id = RunId::new(); ++ let automation = fabro_types::AutomationRef { ++ id: "nightly".to_string(), ++ name: Some("Nightly".to_string()), ++ trigger_id: Some("schedule".to_string()), ++ }; ++ ++ let response = Box::pin(handler::runs::create_run_from_manifest( ++ Arc::clone(&state), ++ handler::runs::CreateRunFromManifestRequest { ++ manifest, ++ submitted_manifest_bytes, ++ explicit_run_id: Some(run_id), ++ explicit_title_supplied: false, ++ actor: Principal::System { ++ system_kind: SystemActorKind::Engine, ++ }, ++ headers: HeaderMap::new(), ++ automation: Some(automation.clone()), ++ }, ++ )) ++ .await; ++ ++ let body = response_json!(response, StatusCode::CREATED).await; ++ assert_eq!(body["automation"]["id"], automation.id); ++ assert_eq!( ++ body["automation"]["name"], ++ automation.name.as_deref().unwrap() ++ ); ++ assert_eq!( ++ body["automation"]["trigger_id"], ++ automation.trigger_id.as_deref().unwrap() ++ ); ++ let summary = state ++ .store ++ .get_cached_summary(&run_id, Utc::now()) ++ .await ++ .unwrap() ++ .unwrap(); ++ assert_eq!(summary.automation, Some(automation)); ++} ++ ++#[tokio::test] ++async fn fake_automation_materializer_injection_captures_input_and_returns_manifest() { ++ let materialized_manifest: RunManifest = ++ serde_json::from_value(minimal_manifest_json(MINIMAL_DOT)).unwrap(); ++ let fake = TestAutomationRunMaterializer::succeed( ++ materialized_manifest.clone(), ++ b"{\"fake\":true}".to_vec(), ++ ); ++ let state = TestAppStateBuilder::new() ++ .automation_materializer(fake.clone()) ++ .build(); ++ let run_id = RunId::new(); ++ let user_settings_path = PathBuf::from("/tmp/fabro/settings.toml"); ++ let temp_root = PathBuf::from("/tmp/fabro/automation"); ++ let target = AutomationTarget { ++ repository: "fabro-sh/fabro".to_string(), ++ ref_selector: "main".to_string(), ++ workflow: "demo".to_string(), ++ }; ++ ++ let output = state ++ .materialize_automation_run(AutomationRunMaterializeInput { ++ automation_id: AutomationId::new("nightly").unwrap(), ++ target: target.clone(), ++ run_id, ++ user_settings_path: user_settings_path.clone(), ++ temp_root: temp_root.clone(), ++ }) ++ .await ++ .expect("fake materializer should succeed"); ++ ++ assert_eq!( ++ serde_json::to_value(&output.manifest).unwrap(), ++ serde_json::to_value(&materialized_manifest).unwrap() ++ ); ++ assert_eq!(output.submitted_manifest_bytes, b"{\"fake\":true}".to_vec()); ++ let captured = fake.captured_inputs(); ++ assert_eq!(captured.len(), 1); ++ assert_eq!(captured[0].automation_id.as_str(), "nightly"); ++ assert_eq!(captured[0].target, target); ++ assert_eq!(captured[0].run_id, run_id); ++ assert_eq!(captured[0].user_settings_path, user_settings_path); ++ assert_eq!(captured[0].temp_root, temp_root); ++} ++ + async fn mock_openai_title_response<'a>( + server: &'a MockServer, + title: &str, +@@ -5276,6 +5774,8 @@ fn create_github_token_app_state_with_env_lookup_and_llm_catalog_settings( + http_client: Some(fabro_http::test_http_client().expect("test HTTP client should build")), + sandbox_provider_registry: None, + shutdown: tokio_util::sync::CancellationToken::new(), ++ worker_control_bus: None, ++ automation_materializer_override: None, + }; + let state = build_app_state(config).expect("test app state should build"); + if let Some(token) = token { +@@ -10678,12 +11178,8 @@ async fn steer_without_active_steerable_session_forwards_plain_steer_for_bufferi + let state = test_app_state(); + let app = crate::test_support::build_test_router(Arc::clone(&state)); + let run_id = fixtures::RUN_1; +- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1); +- let _temp_dir = insert_running_control_run( +- &state, +- run_id, +- Some(RunAnswerTransport::Subprocess { control_tx }), +- ); ++ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await; ++ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport)); + + let req = Request::builder() + .method("POST") +@@ -10694,7 +11190,7 @@ async fn steer_without_active_steerable_session_forwards_plain_steer_for_bufferi + + let response = app.oneshot(req).await.unwrap(); + assert_status!(response, StatusCode::ACCEPTED).await; +- let envelope = control_rx.recv().await.unwrap(); ++ let envelope = recv_worker_control_envelope(&mut control_rx).await; + assert!(matches!( + envelope.message, + WorkerControlMessage::Steer { ref text, .. } if text == "try again" +@@ -10707,12 +11203,8 @@ async fn steer_with_active_non_steerable_session_returns_conflict() { + let app = crate::test_support::build_test_router(Arc::clone(&state)); + let run_id = fixtures::RUN_1; + let stage_id = StageId::new("agent", 1); +- let (control_tx, _control_rx) = tokio::sync::mpsc::channel(1); +- let _temp_dir = insert_running_control_run( +- &state, +- run_id, +- Some(RunAnswerTransport::Subprocess { control_tx }), +- ); ++ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await; ++ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport)); + { + let mut runs = state.runs.lock().expect("runs lock poisoned"); + runs.get_mut(&run_id) +@@ -10739,12 +11231,8 @@ async fn steer_interrupt_without_active_steerable_session_returns_conflict() { + let state = test_app_state(); + let app = crate::test_support::build_test_router(Arc::clone(&state)); + let run_id = fixtures::RUN_1; +- let (control_tx, _control_rx) = tokio::sync::mpsc::channel(1); +- let _temp_dir = insert_running_control_run( +- &state, +- run_id, +- Some(RunAnswerTransport::Subprocess { control_tx }), +- ); ++ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await; ++ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport)); + + let req = Request::builder() + .method("POST") +@@ -10765,12 +11253,8 @@ async fn interrupt_with_active_steerable_session_forwards_interrupt() { + let app = crate::test_support::build_test_router(Arc::clone(&state)); + let run_id = fixtures::RUN_1; + let stage_id = StageId::new("agent", 1); +- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1); +- let _temp_dir = insert_running_control_run( +- &state, +- run_id, +- Some(RunAnswerTransport::Subprocess { control_tx }), +- ); ++ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await; ++ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport)); + { + let mut runs = state.runs.lock().expect("runs lock poisoned"); + runs.get_mut(&run_id) +@@ -10787,7 +11271,7 @@ async fn interrupt_with_active_steerable_session_forwards_interrupt() { + + let response = app.oneshot(req).await.unwrap(); + assert_status!(response, StatusCode::ACCEPTED).await; +- let envelope = control_rx.recv().await.unwrap(); ++ let envelope = recv_worker_control_envelope(&mut control_rx).await; + assert!(matches!( + envelope.message, + WorkerControlMessage::Interrupt { +@@ -10802,12 +11286,8 @@ async fn steer_interrupt_with_active_steerable_session_forwards_combined_control + let app = crate::test_support::build_test_router(Arc::clone(&state)); + let run_id = fixtures::RUN_1; + let stage_id = StageId::new("agent", 1); +- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1); +- let _temp_dir = insert_running_control_run( +- &state, +- run_id, +- Some(RunAnswerTransport::Subprocess { control_tx }), +- ); ++ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await; ++ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport)); + { + let mut runs = state.runs.lock().expect("runs lock poisoned"); + runs.get_mut(&run_id) +@@ -10825,7 +11305,7 @@ async fn steer_interrupt_with_active_steerable_session_forwards_combined_control + + let response = app.oneshot(req).await.unwrap(); + assert_status!(response, StatusCode::ACCEPTED).await; +- let envelope = control_rx.recv().await.unwrap(); ++ let envelope = recv_worker_control_envelope(&mut control_rx).await; + assert!(matches!( + envelope.message, + WorkerControlMessage::InterruptThenSteer { ref text, .. } if text == "try again" +@@ -10991,12 +11471,8 @@ async fn steer_with_active_acp_session_forwards_to_worker() { + let state = test_app_state(); + let app = crate::test_support::build_test_router(Arc::clone(&state)); + let run_id = fixtures::RUN_1; +- let (control_tx, mut control_rx) = tokio::sync::mpsc::channel(1); +- let _temp_dir = insert_running_control_run( +- &state, +- run_id, +- Some(RunAnswerTransport::Subprocess { control_tx }), +- ); ++ let (transport, mut control_rx) = worker_transport_with_receiver(run_id).await; ++ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport)); + + let started = acp_event_for_stage(&run_id, &workflow_event::Event::AgentAcpStarted { + node_id: "agent".to_string(), +@@ -11029,7 +11505,7 @@ async fn steer_with_active_acp_session_forwards_to_worker() { + + let response = app.oneshot(req).await.unwrap(); + assert_status!(response, StatusCode::ACCEPTED).await; +- let envelope = control_rx.recv().await.unwrap(); ++ let envelope = recv_worker_control_envelope(&mut control_rx).await; + assert!(matches!( + envelope.message, + WorkerControlMessage::Steer { ref text, .. } if text == "try again" +@@ -11096,12 +11572,8 @@ async fn active_acp_steerable_marker_clears_on_terminal_paths() { + let state = test_app_state(); + let app = crate::test_support::build_test_router(Arc::clone(&state)); + let run_id = fixtures::RUN_1; +- let (control_tx, _control_rx) = tokio::sync::mpsc::channel(1); +- let _temp_dir = insert_running_control_run( +- &state, +- run_id, +- Some(RunAnswerTransport::Subprocess { control_tx }), +- ); ++ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await; ++ let _temp_dir = insert_running_control_run(&state, run_id, Some(transport)); + let started = acp_event_for_stage(&run_id, &workflow_event::Event::AgentAcpStarted { + node_id: "agent".to_string(), + visit: 1, +@@ -13031,11 +13503,13 @@ async fn pause_run_sets_pending_control_on_board_response() { + let run_id_str = create_and_start_run(&app, MINIMAL_DOT).await; + let run_id = run_id_str.parse::().unwrap(); + ++ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await; + { + let mut runs = state.runs.lock().expect("runs lock poisoned"); + let managed_run = runs.get_mut(&run_id).expect("run should exist"); + managed_run.status = RunStatus::Running; + managed_run.worker_pid = Some(u32::MAX); ++ managed_run.answer_transport = Some(transport); + } + + let req = Request::builder() +@@ -13151,11 +13625,13 @@ async fn unpause_run_sets_pending_control() { + let run_id_str = create_and_start_run(&app, MINIMAL_DOT).await; + let run_id = run_id_str.parse::().unwrap(); + ++ let (transport, _control_rx) = worker_transport_with_receiver(run_id).await; + { + let mut runs = state.runs.lock().expect("runs lock poisoned"); + let managed_run = runs.get_mut(&run_id).expect("run should exist"); + managed_run.status = RunStatus::Paused { prior_block: None }; + managed_run.worker_pid = Some(u32::MAX); ++ managed_run.answer_transport = Some(transport); + } + + let req = Request::builder() +diff --git a/lib/crates/fabro-server/src/test_support.rs b/lib/crates/fabro-server/src/test_support.rs +index afd26f624..8e6bb3c20 100644 +--- a/lib/crates/fabro-server/src/test_support.rs ++++ b/lib/crates/fabro-server/src/test_support.rs +@@ -29,6 +29,12 @@ use tokio_util::sync::CancellationToken; + use ulid::Ulid; + + use crate::auth; ++<<<<<<< HEAD ++======= ++use crate::automation_materializer::AutomationRunMaterializer; ++pub use crate::automation_materializer::TestAutomationRunMaterializer; ++use crate::ip_allowlist::IpAllowlistConfig; ++>>>>>>> origin/main + use crate::jwt_auth::{AuthMode, ConfiguredAuth}; + #[cfg(test)] + use crate::principal_middleware::{AuthContextSlot, RequestAuthContext}; +@@ -70,6 +76,7 @@ pub struct TestAppStateBuilder { + server_secret_env: HashMap, + env_lookup: EnvLookup, + llm_catalog_settings: LlmCatalogSettings, ++ automation_materializer: Option>, + } + + impl Default for TestAppStateBuilder { +@@ -88,6 +95,7 @@ impl Default for TestAppStateBuilder { + server_secret_env: HashMap::new(), + env_lookup: default_env_lookup(), + llm_catalog_settings: LlmCatalogSettings::default(), ++ automation_materializer: None, + } + } + } +@@ -144,6 +152,11 @@ impl TestAppStateBuilder { + self + } + ++ pub fn automation_materializer(mut self, materializer: TestAutomationRunMaterializer) -> Self { ++ self.automation_materializer = Some(materializer.into_materializer()); ++ self ++ } ++ + pub fn provider_base_url( + mut self, + provider: impl Into, +@@ -239,6 +252,9 @@ impl TestAppStateBuilder { + ), + sandbox_provider_registry: self.sandbox_provider_registry, + shutdown: CancellationToken::new(), ++ #[cfg(test)] ++ worker_control_bus: None, ++ automation_materializer_override: self.automation_materializer, + }) + } + } +diff --git a/lib/crates/fabro-server/src/worker_control/bus.rs b/lib/crates/fabro-server/src/worker_control/bus.rs +new file mode 100644 +index 000000000..b6fadedb5 +--- /dev/null ++++ b/lib/crates/fabro-server/src/worker_control/bus.rs +@@ -0,0 +1,155 @@ ++use std::fmt; ++ ++use fabro_interview::WorkerControlEnvelope; ++use fabro_types::RunId; ++use futures_util::future::BoxFuture; ++use tokio::sync::mpsc; ++ ++#[derive(Clone, PartialEq, Eq)] ++pub(crate) struct WorkerControlMessageId(String); ++ ++impl WorkerControlMessageId { ++ #[must_use] ++ pub(crate) fn new(value: impl Into) -> Self { ++ Self(value.into()) ++ } ++ ++ #[must_use] ++ pub(crate) fn as_str(&self) -> &str { ++ &self.0 ++ } ++} ++ ++impl fmt::Debug for WorkerControlMessageId { ++ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { ++ f.debug_tuple("WorkerControlMessageId") ++ .field(&self.0) ++ .finish() ++ } ++} ++ ++impl fmt::Display for WorkerControlMessageId { ++ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { ++ f.write_str(&self.0) ++ } ++} ++ ++impl From for WorkerControlMessageId { ++ fn from(value: String) -> Self { ++ Self::new(value) ++ } ++} ++ ++impl From<&str> for WorkerControlMessageId { ++ fn from(value: &str) -> Self { ++ Self::new(value) ++ } ++} ++ ++/// Cursor for replaying a run's worker-control stream. ++/// ++/// Future Redis Streams mapping: ++/// - `Start` maps to `XREAD ... STREAMS fabro:run:{run_id}:control 0-0`. ++/// - `After(id)` maps to `XREAD ... STREAMS fabro:run:{run_id}:control {id}`. ++#[derive(Debug, Clone, PartialEq, Eq)] ++pub(crate) enum WorkerControlCursor { ++ Start, ++ After(WorkerControlMessageId), ++} ++ ++impl WorkerControlCursor { ++ pub(crate) fn from_after_query(after: Option<&str>) -> Result { ++ match after { ++ None => Ok(Self::Start), ++ Some(value) if value.trim().is_empty() => Err(WorkerControlBusError::InvalidCursor { ++ cursor: value.to_string(), ++ reason: "cursor id must not be empty".to_string(), ++ }), ++ Some(value) => Ok(Self::After(WorkerControlMessageId::from(value))), ++ } ++ } ++} ++ ++#[derive(Debug, Clone, PartialEq, Eq)] ++pub(crate) struct WorkerControlDelivery { ++ pub(crate) id: WorkerControlMessageId, ++ pub(crate) envelope: WorkerControlEnvelope, ++} ++ ++#[allow( ++ dead_code, ++ reason = "The local backend does not construct every cross-backend error variant." ++)] ++#[derive(Debug, Clone, thiserror::Error, PartialEq, Eq)] ++pub(crate) enum WorkerControlBusError { ++ #[error("worker control backend is closed")] ++ Closed, ++ #[error("worker control backend is unavailable")] ++ Unavailable, ++ #[error("worker control cursor `{cursor}` is invalid: {reason}")] ++ InvalidCursor { cursor: String, reason: String }, ++ #[error("timed out publishing worker control message")] ++ PublishTimeout, ++} ++ ++impl WorkerControlBusError { ++ #[must_use] ++ pub(crate) fn invalid_cursor(cursor: impl Into, reason: impl Into) -> Self { ++ Self::InvalidCursor { ++ cursor: cursor.into(), ++ reason: reason.into(), ++ } ++ } ++} ++ ++pub(crate) type WorkerControlReceiver = ++ mpsc::Receiver>; ++ ++pub(crate) trait WorkerControlBus: Send + Sync { ++ fn publish( ++ &self, ++ run_id: RunId, ++ envelope: WorkerControlEnvelope, ++ ) -> BoxFuture<'_, Result>; ++ ++ fn subscribe( ++ &self, ++ run_id: RunId, ++ cursor: WorkerControlCursor, ++ ) -> BoxFuture<'_, Result>; ++ ++ fn cleanup_run(&self, run_id: RunId) -> BoxFuture<'_, ()>; ++} ++ ++#[cfg(test)] ++mod tests { ++ use super::*; ++ ++ #[test] ++ fn message_id_equality_and_debug_are_opaque() { ++ let first = WorkerControlMessageId::from("local:42"); ++ let same = WorkerControlMessageId::from("local:42"); ++ let different = WorkerControlMessageId::from("local:43"); ++ ++ assert_eq!(first, same); ++ assert_ne!(first, different); ++ assert_eq!(format!("{first}"), "local:42"); ++ assert_eq!(format!("{first:?}"), "WorkerControlMessageId(\"local:42\")"); ++ } ++ ++ #[test] ++ fn absent_after_query_parses_as_start_cursor() { ++ assert_eq!( ++ WorkerControlCursor::from_after_query(None).unwrap(), ++ WorkerControlCursor::Start ++ ); ++ } ++ ++ #[test] ++ fn present_after_query_parses_as_after_cursor() { ++ assert_eq!( ++ WorkerControlCursor::from_after_query(Some("local:42")).unwrap(), ++ WorkerControlCursor::After(WorkerControlMessageId::from("local:42")) ++ ); ++ } ++} +diff --git a/lib/crates/fabro-server/src/worker_control/local.rs b/lib/crates/fabro-server/src/worker_control/local.rs +new file mode 100644 +index 000000000..219ff18ec +--- /dev/null ++++ b/lib/crates/fabro-server/src/worker_control/local.rs +@@ -0,0 +1,458 @@ ++use std::collections::{HashMap, VecDeque}; ++use std::sync::atomic::{AtomicU64, Ordering}; ++use std::sync::{Arc, Mutex}; ++ ++use fabro_interview::WorkerControlEnvelope; ++use fabro_types::RunId; ++use futures_util::FutureExt; ++use futures_util::future::BoxFuture; ++use tokio::sync::{Notify, mpsc}; ++ ++use super::{ ++ WorkerControlBus, WorkerControlBusError, WorkerControlCursor, WorkerControlDelivery, ++ WorkerControlMessageId, WorkerControlReceiver, ++}; ++ ++pub(crate) const LOCAL_WORKER_CONTROL_RETAINED_MESSAGES_PER_RUN: usize = 1024; ++const LOCAL_WORKER_CONTROL_SUBSCRIBER_BUFFER: usize = 64; ++ ++#[derive(Default)] ++pub(crate) struct LocalWorkerControlBus { ++ streams: Arc>>, ++ next_sequence: Arc, ++} ++ ++struct LocalRunControlStream { ++ messages: VecDeque, ++ notify: Arc, ++ has_trimmed: bool, ++} ++ ++#[derive(Clone)] ++struct LocalMessage { ++ sequence: u64, ++ delivery: WorkerControlDelivery, ++} ++ ++impl LocalWorkerControlBus { ++ #[must_use] ++ pub(crate) fn new() -> Self { ++ Self { ++ streams: Arc::new(Mutex::new(HashMap::new())), ++ next_sequence: Arc::new(AtomicU64::new(1)), ++ } ++ } ++ ++ fn next_message_id(&self) -> (u64, WorkerControlMessageId) { ++ let sequence = self.next_sequence.fetch_add(1, Ordering::Relaxed); ++ ( ++ sequence, ++ WorkerControlMessageId::new(format!("local:{sequence}")), ++ ) ++ } ++ ++ fn subscribe_inner( ++ &self, ++ run_id: RunId, ++ cursor: &WorkerControlCursor, ++ ) -> Result { ++ let (next_sequence, notify) = { ++ let mut streams = self ++ .streams ++ .lock() ++ .expect("worker control streams poisoned"); ++ let stream = match cursor { ++ WorkerControlCursor::Start => streams ++ .entry(run_id) ++ .or_insert_with(LocalRunControlStream::new), ++ WorkerControlCursor::After(id) => streams.get_mut(&run_id).ok_or_else(|| { ++ WorkerControlBusError::invalid_cursor( ++ id.as_str(), ++ "message id is not retained for this run", ++ ) ++ })?, ++ }; ++ let next_sequence = stream.next_sequence_for_cursor(cursor)?; ++ (next_sequence, Arc::clone(&stream.notify)) ++ }; ++ ++ let (tx, rx) = mpsc::channel(LOCAL_WORKER_CONTROL_SUBSCRIBER_BUFFER); ++ let streams = Arc::clone(&self.streams); ++ tokio::spawn(async move { ++ local_subscription_task(streams, run_id, notify, next_sequence, tx).await; ++ }); ++ Ok(rx) ++ } ++ ++ #[cfg(test)] ++ pub(crate) fn retained_len(&self, run_id: &RunId) -> usize { ++ let streams = self ++ .streams ++ .lock() ++ .expect("worker control streams poisoned"); ++ streams ++ .get(run_id) ++ .map_or(0, |stream| stream.messages.len()) ++ } ++} ++ ++impl LocalRunControlStream { ++ fn new() -> Self { ++ Self { ++ messages: VecDeque::new(), ++ notify: Arc::new(Notify::new()), ++ has_trimmed: false, ++ } ++ } ++ ++ fn next_sequence_for_cursor( ++ &self, ++ cursor: &WorkerControlCursor, ++ ) -> Result, WorkerControlBusError> { ++ match cursor { ++ WorkerControlCursor::Start if self.has_trimmed => Err( ++ WorkerControlBusError::invalid_cursor("start", "retained local stream is trimmed"), ++ ), ++ WorkerControlCursor::Start => Ok(self.messages.front().map(|message| message.sequence)), ++ WorkerControlCursor::After(id) => { ++ let requested_sequence = parse_local_sequence(id)?; ++ let Some(position) = self ++ .messages ++ .iter() ++ .position(|message| message.delivery.id == *id) ++ else { ++ return Err(WorkerControlBusError::invalid_cursor( ++ id.as_str(), ++ "message id is not retained for this run", ++ )); ++ }; ++ Ok(self ++ .messages ++ .get(position + 1) ++ .map(|message| message.sequence) ++ .or(Some(requested_sequence.saturating_add(1)))) ++ } ++ } ++ } ++ ++ fn trim_retained(&mut self) { ++ while self.messages.len() > LOCAL_WORKER_CONTROL_RETAINED_MESSAGES_PER_RUN { ++ self.messages.pop_front(); ++ self.has_trimmed = true; ++ } ++ } ++} ++ ++fn parse_local_sequence(id: &WorkerControlMessageId) -> Result { ++ let Some(raw) = id.as_str().strip_prefix("local:") else { ++ return Err(WorkerControlBusError::invalid_cursor( ++ id.as_str(), ++ "local backend only understands local message ids", ++ )); ++ }; ++ raw.parse::().map_err(|_| { ++ WorkerControlBusError::invalid_cursor(id.as_str(), "local message id sequence is invalid") ++ }) ++} ++ ++async fn local_subscription_task( ++ streams: Arc>>, ++ run_id: RunId, ++ notify: Arc, ++ mut next_sequence: Option, ++ tx: mpsc::Sender>, ++) { ++ loop { ++ // Register interest *before* inspecting the stream so that a publish ++ // racing with this read does not cause a lost wakeup. `notify_waiters` ++ // does not leave a permit for future `notified()` calls. ++ let notified = notify.notified(); ++ tokio::pin!(notified); ++ notified.as_mut().enable(); ++ ++ let collected = { ++ let streams_guard = streams.lock().expect("worker control streams poisoned"); ++ match streams_guard.get(&run_id) { ++ None => None, ++ Some(stream) => { ++ // A `Start` subscriber that joined before any publish lazily ++ // adopts the first retained message as its cursor. ++ if next_sequence.is_none() && stream.has_trimmed { ++ Some(Err(WorkerControlBusError::invalid_cursor( ++ "start", ++ "retained local stream is trimmed", ++ ))) ++ } else { ++ if next_sequence.is_none() { ++ next_sequence = stream.messages.front().map(|message| message.sequence); ++ } ++ match next_sequence { ++ None => Some(Ok(Vec::new())), ++ Some(next) => Some(collect_messages_from(&stream.messages, next)), ++ } ++ } ++ } ++ } ++ }; ++ ++ let messages = match collected { ++ None => return, ++ Some(Err(err)) => { ++ let _ = tx.send(Err(err)).await; ++ return; ++ } ++ Some(Ok(messages)) => messages, ++ }; ++ ++ if messages.is_empty() { ++ notified.await; ++ continue; ++ } ++ ++ for message in messages { ++ next_sequence = Some(message.sequence.saturating_add(1)); ++ if tx.send(Ok(message.delivery)).await.is_err() { ++ return; ++ } ++ } ++ } ++} ++ ++/// Returns the retained messages with `sequence >= next`. Cheaper than scanning ++/// the whole deque: `partition_point` is O(log N) and we only clone the tail. ++fn collect_messages_from( ++ messages: &VecDeque, ++ next: u64, ++) -> Result, WorkerControlBusError> { ++ let Some(first_sequence) = messages.front().map(|message| message.sequence) else { ++ return Ok(Vec::new()); ++ }; ++ if next < first_sequence { ++ return Err(WorkerControlBusError::invalid_cursor( ++ format!("local:{next}"), ++ "subscriber fell behind retained local messages", ++ )); ++ } ++ let start = messages.partition_point(|message| message.sequence < next); ++ Ok(messages.iter().skip(start).cloned().collect()) ++} ++ ++impl WorkerControlBus for LocalWorkerControlBus { ++ fn publish( ++ &self, ++ run_id: RunId, ++ envelope: WorkerControlEnvelope, ++ ) -> BoxFuture<'_, Result> { ++ async move { ++ let (sequence, id) = self.next_message_id(); ++ let delivery = WorkerControlDelivery { ++ id: id.clone(), ++ envelope, ++ }; ++ let notify = { ++ let mut streams = self ++ .streams ++ .lock() ++ .expect("worker control streams poisoned"); ++ let stream = streams ++ .entry(run_id) ++ .or_insert_with(LocalRunControlStream::new); ++ stream ++ .messages ++ .push_back(LocalMessage { sequence, delivery }); ++ stream.trim_retained(); ++ Arc::clone(&stream.notify) ++ }; ++ notify.notify_waiters(); ++ Ok(id) ++ } ++ .boxed() ++ } ++ ++ fn subscribe( ++ &self, ++ run_id: RunId, ++ cursor: WorkerControlCursor, ++ ) -> BoxFuture<'_, Result> { ++ async move { self.subscribe_inner(run_id, &cursor) }.boxed() ++ } ++ ++ fn cleanup_run(&self, run_id: RunId) -> BoxFuture<'_, ()> { ++ async move { ++ let notify = { ++ let mut streams = self ++ .streams ++ .lock() ++ .expect("worker control streams poisoned"); ++ streams.remove(&run_id).map(|stream| stream.notify) ++ }; ++ if let Some(notify) = notify { ++ notify.notify_waiters(); ++ } ++ } ++ .boxed() ++ } ++} ++ ++#[cfg(test)] ++mod tests { ++ use std::time::Duration; ++ ++ use fabro_interview::WorkerControlMessage; ++ use fabro_types::fixtures; ++ ++ use super::*; ++ ++ async fn recv_delivery(receiver: &mut WorkerControlReceiver) -> WorkerControlDelivery { ++ tokio::time::timeout(Duration::from_secs(1), receiver.recv()) ++ .await ++ .expect("delivery should arrive") ++ .expect("subscription should remain open") ++ .expect("delivery should be ok") ++ } ++ ++ #[tokio::test] ++ async fn messages_publish_in_order() { ++ let bus = LocalWorkerControlBus::new(); ++ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run()) ++ .await ++ .unwrap(); ++ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::pause_run()) ++ .await ++ .unwrap(); ++ ++ let mut receiver = bus ++ .subscribe(fixtures::RUN_1, WorkerControlCursor::Start) ++ .await ++ .unwrap(); ++ ++ assert!(matches!( ++ recv_delivery(&mut receiver).await.envelope.message, ++ WorkerControlMessage::RunCancel ++ )); ++ assert!(matches!( ++ recv_delivery(&mut receiver).await.envelope.message, ++ WorkerControlMessage::RunPause ++ )); ++ } ++ ++ #[tokio::test] ++ async fn active_subscriber_receives_message_published_after_subscription() { ++ let bus = LocalWorkerControlBus::new(); ++ let mut receiver = bus ++ .subscribe(fixtures::RUN_1, WorkerControlCursor::Start) ++ .await ++ .unwrap(); ++ ++ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run()) ++ .await ++ .unwrap(); ++ ++ assert!(matches!( ++ recv_delivery(&mut receiver).await.envelope.message, ++ WorkerControlMessage::RunCancel ++ )); ++ } ++ ++ #[tokio::test] ++ async fn messages_published_before_subscription_replay_from_start() { ++ let bus = LocalWorkerControlBus::new(); ++ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run()) ++ .await ++ .unwrap(); ++ ++ let mut receiver = bus ++ .subscribe(fixtures::RUN_1, WorkerControlCursor::Start) ++ .await ++ .unwrap(); ++ ++ assert!(matches!( ++ recv_delivery(&mut receiver).await.envelope.message, ++ WorkerControlMessage::RunCancel ++ )); ++ } ++ ++ #[tokio::test] ++ async fn after_cursor_receives_only_later_messages() { ++ let bus = LocalWorkerControlBus::new(); ++ let first_id = bus ++ .publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run()) ++ .await ++ .unwrap(); ++ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::pause_run()) ++ .await ++ .unwrap(); ++ ++ let mut receiver = bus ++ .subscribe( ++ fixtures::RUN_1, ++ WorkerControlCursor::After(first_id.clone()), ++ ) ++ .await ++ .unwrap(); ++ ++ assert!(matches!( ++ recv_delivery(&mut receiver).await.envelope.message, ++ WorkerControlMessage::RunPause ++ )); ++ assert!(receiver.try_recv().is_err()); ++ } ++ ++ #[tokio::test] ++ async fn trimming_bounds_retained_messages_and_invalidates_old_cursor() { ++ let bus = LocalWorkerControlBus::new(); ++ let old_id = bus ++ .publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run()) ++ .await ++ .unwrap(); ++ for _ in 0..LOCAL_WORKER_CONTROL_RETAINED_MESSAGES_PER_RUN { ++ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::pause_run()) ++ .await ++ .unwrap(); ++ } ++ ++ assert_eq!( ++ bus.retained_len(&fixtures::RUN_1), ++ LOCAL_WORKER_CONTROL_RETAINED_MESSAGES_PER_RUN ++ ); ++ let err = bus ++ .subscribe(fixtures::RUN_1, WorkerControlCursor::After(old_id)) ++ .await ++ .unwrap_err(); ++ assert!(matches!(err, WorkerControlBusError::InvalidCursor { .. })); ++ ++ let err = bus ++ .subscribe(fixtures::RUN_1, WorkerControlCursor::Start) ++ .await ++ .unwrap_err(); ++ assert!(matches!(err, WorkerControlBusError::InvalidCursor { .. })); ++ } ++ ++ #[tokio::test] ++ async fn after_cursor_for_unknown_run_does_not_create_stream() { ++ let bus = LocalWorkerControlBus::new(); ++ let err = bus ++ .subscribe( ++ fixtures::RUN_1, ++ WorkerControlCursor::After(WorkerControlMessageId::new("local:1")), ++ ) ++ .await ++ .unwrap_err(); ++ ++ assert!(matches!(err, WorkerControlBusError::InvalidCursor { .. })); ++ assert_eq!(bus.retained_len(&fixtures::RUN_1), 0); ++ } ++ ++ #[tokio::test] ++ async fn cleanup_removes_retained_messages() { ++ let bus = LocalWorkerControlBus::new(); ++ bus.publish(fixtures::RUN_1, WorkerControlEnvelope::cancel_run()) ++ .await ++ .unwrap(); ++ assert_eq!(bus.retained_len(&fixtures::RUN_1), 1); ++ ++ bus.cleanup_run(fixtures::RUN_1).await; ++ ++ assert_eq!(bus.retained_len(&fixtures::RUN_1), 0); ++ } ++} +diff --git a/lib/crates/fabro-server/src/worker_control/mod.rs b/lib/crates/fabro-server/src/worker_control/mod.rs +new file mode 100644 +index 000000000..6b6f70896 +--- /dev/null ++++ b/lib/crates/fabro-server/src/worker_control/mod.rs +@@ -0,0 +1,8 @@ ++mod bus; ++mod local; ++ ++pub(crate) use bus::{ ++ WorkerControlBus, WorkerControlBusError, WorkerControlCursor, WorkerControlDelivery, ++ WorkerControlMessageId, WorkerControlReceiver, ++}; ++pub(crate) use local::LocalWorkerControlBus; +diff --git a/lib/crates/fabro-workflow/src/operations/start.rs b/lib/crates/fabro-workflow/src/operations/start.rs +index 9d3f08d1c..23c90f3ff 100644 +--- a/lib/crates/fabro-workflow/src/operations/start.rs ++++ b/lib/crates/fabro-workflow/src/operations/start.rs +@@ -26,6 +26,7 @@ use fabro_types::{ManifestPath, RunId, RunRunnableSource, SandboxProviderKind}; + use fabro_vault::Vault; + use tokio::runtime::Handle; + use tokio::sync::RwLock as AsyncRwLock; ++use tokio::time; + use tokio_util::sync::CancellationToken; + + use crate::artifact_upload::ArtifactSink; +@@ -987,6 +988,24 @@ impl Drop for DetachedRunBootstrapGuard { + + const POSTRUN_INTERRUPTED_MESSAGE: &str = "Run interrupted before post-run finalization completed."; + const POSTRUN_CANCELLED_MESSAGE: &str = "Run cancelled before post-run finalization completed."; ++const DETACHED_COMPLETION_GUARD_TERMINAL_GRACE: Duration = Duration::from_millis(25); ++ ++async fn run_store_reaches_terminal(run_store: &RunStoreHandle, timeout: Duration) -> bool { ++ let start = Instant::now(); ++ loop { ++ if run_store ++ .state() ++ .await ++ .is_ok_and(|state| state.status.is_terminal()) ++ { ++ return true; ++ } ++ if start.elapsed() >= timeout { ++ return false; ++ } ++ time::sleep(Duration::from_millis(10)).await; ++ } ++} + + struct DetachedRunCompletionGuard { + event_sink: RunEventSink, +@@ -1044,6 +1063,11 @@ impl Drop for DetachedRunCompletionGuard { + let run_store = self.run_store.clone(); + if let Ok(handle) = Handle::try_current() { + handle.spawn(async move { ++ if run_store_reaches_terminal(&run_store, DETACHED_COMPLETION_GUARD_TERMINAL_GRACE) ++ .await ++ { ++ return; ++ } + emit_workflow_run_failed( + run_id, + &run_store, diff --git a/stages/008-verify@1/output.log b/stages/008-verify@1/output.log new file mode 100644 index 000000000..d8a75bb62 --- /dev/null +++ b/stages/008-verify@1/output.log @@ -0,0 +1 @@ +blob://sha256/ba184126dad761158723a25e15818b2cb0ac7327ad06fa7513f7a7ce9377eee6 \ No newline at end of file diff --git a/stages/008-verify@1/script_timing.json b/stages/008-verify@1/script_timing.json new file mode 100644 index 000000000..d9dbb957f --- /dev/null +++ b/stages/008-verify@1/script_timing.json @@ -0,0 +1,8 @@ +{ + "output": "blob://sha256/ba184126dad761158723a25e15818b2cb0ac7327ad06fa7513f7a7ce9377eee6", + "exit_code": 1, + "duration_ms": 1604, + "termination": "exited", + "output_bytes": 711, + "live_streaming": true +} \ No newline at end of file diff --git a/stages/008-verify@1/status.json b/stages/008-verify@1/status.json new file mode 100644 index 000000000..35b9e1dba --- /dev/null +++ b/stages/008-verify@1/status.json @@ -0,0 +1,6 @@ +{ + "outcome": "failed", + "notes": null, + "failure_reason": "Script failed with exit code: 1\n\n## output\nFrom https://github.com/fabro-sh/fabro\n * branch main -> FETCH_HEAD\n a992a7d76..475b4ab65 main -> origin/main\nAuto-merging Cargo.lock\nAuto-merging lib/crates/fabro-api/build.rs\nAuto-merging lib/crates/fabro-api/src/lib.rs\nAuto-merging lib/crates/fabro-server/Cargo.toml\nAuto-merging lib/crates/fabro-server/src/lib.rs\nAuto-merging lib/crates/fabro-server/src/serve.rs\nAuto-merging lib/crates/fabro-server/src/server.rs\nAuto-merging lib/crates/fabro-server/src/server/tests.rs\nAuto-merging lib/crates/fabro-server/src/test_support.rs\nCONFLICT (content): Merge conflict in lib/crates/fabro-server/src/test_support.rs\nAutomatic merge failed; fix conflicts and then commit the result.\n", + "timestamp": "2026-05-28T01:02:42.285529Z" +} \ No newline at end of file diff --git a/stages/009-fixup@1/prompt.md b/stages/009-fixup@1/prompt.md new file mode 100644 index 000000000..f4fdab6c6 --- /dev/null +++ b/stages/009-fixup@1/prompt.md @@ -0,0 +1,443 @@ +Goal: --- +title: "refactor: Remove IP allowlisting" +type: refactor +status: active +date: 2026-05-27 +--- + +# refactor: Remove IP allowlisting + +## Overview + +Remove Fabro's inbound server IP allowlisting feature completely. This removes the +server-wide `[server.ip_allowlist]` setting, the GitHub webhook-specific +`[server.integrations.github.webhooks.ip_allowlist]` overlay, request middleware, +client-IP extraction, GitHub `/meta` hook-range expansion, settings API exposure, +generated API client types, and the Settings > Security UI display. + +Existing configs that still contain removed IP allowlist keys should fail as +unknown fields. This is an intentional hard removal, not a compatibility +deprecation. + +## Problem Frame + +The feature is being removed from Fabro rather than maintained as an in-process +network access-control layer. Network source restrictions should be handled +outside Fabro by reverse proxies, firewalls, VPNs, Tailscale, platform ingress, +or other deployment-layer controls. + +## Requirements Trace + +- R1. Remove all runtime enforcement of inbound source-IP allowlisting from web, + API, static asset, and GitHub webhook routes. +- R2. Remove the server settings schema for global and GitHub webhook IP + allowlists. +- R3. Remove `IpAllowEntry` and related API schema/client types from public + settings payloads. +- R4. Keep unrelated allowlists intact: GitHub username allowlists, sandbox + egress CIDR allow lists, and internal policy/test allowlists. +- R5. Preserve existing authentication, GitHub webhook HMAC verification, + routing, health checks, request logging, and settings hot-reload behavior. +- R6. Treat old IP allowlist config as invalid after removal. +- R7. Update operator-facing docs/changelog so users know to move source-IP + restrictions upstream. + +## Scope Boundaries + +- Do not remove `[server.auth.github].allowed_usernames`. +- Do not remove Daytona/sandbox `cidr_allow_list` network policy support. +- Do not remove generic uses of "allowlist" in policy tests, markdown rendering, + model controls, or other unrelated domains. +- Do not add a migration, warning-only parser, or fallback compatibility path for + old IP allowlist settings. +- Do not change GitHub webhook signature verification or webhook route strategy. + +## Context & Research + +### Relevant Code and Patterns + +- Settings use sparse config layers in `lib/crates/fabro-config/src/layers/` + and dense resolved types in `lib/crates/fabro-types/src/settings/`. +- The OpenAPI spec at `docs/public/api-reference/fabro-api.yaml` is the source + of truth for API wire shape; `cargo build -p fabro-api` regenerates Rust API + code, and `cd lib/packages/fabro-api-client && bun run generate` regenerates + the TypeScript client. +- `lib/crates/fabro-server/src/server.rs` builds the router and currently wires + IP allowlist middleware before other route dispatch behavior. +- `lib/crates/fabro-server/src/serve.rs` currently resolves server and webhook + allowlist configs at startup and creates a GitHub `/meta` resolver. +- `apps/fabro-web/app/routes/settings-security.tsx` displays the settings API's + `server.ip_allowlist` state. + +### Main Removal Surfaces + +- Config/types: `fabro-types`, `fabro-config`, config tests. +- Server runtime: `fabro-server/src/ip_allowlist.rs`, router wiring, startup + wiring, test helpers, routing/TCP tests. +- API/contracts: OpenAPI schema, `fabro-api` type replacements/exports, generated + TypeScript API client models. +- Frontend/docs: Settings > Security copy, settings navigation copy, changelog. + +## Key Technical Decisions + +- **Hard removal:** Removed config keys should fail through existing + `deny_unknown_fields` behavior. This keeps the change simple and makes stale + deployment config visible immediately. +- **Delete, do not stub:** Remove `IpAllowlistConfig` and middleware parameters + instead of passing default empty configs through the router. Empty stubs would + keep the feature shape alive and make future cleanup harder. +- **Keep webhook auth unchanged:** GitHub webhook source-IP filtering goes away, + but HMAC signature verification remains the security boundary for webhook + payload authenticity. +- **Generated clients follow OpenAPI:** Update the OpenAPI schemas first, then + regenerate Rust and TypeScript API outputs rather than hand-editing generated + client files except as a short-term cleanup if generation leaves stale exports. +- **Docs mention upstream controls:** The changelog and security guidance should + direct operators to network-layer controls, not a replacement Fabro setting. + +## Implementation Units + +- [ ] **Unit 1: Remove config schema and resolved types** + +**Goal:** Delete the IP allowlist settings contract from config parsing and dense +server settings. + +**Requirements:** R2, R3, R4, R6 + +**Dependencies:** None + +**Files:** +- Modify: `lib/crates/fabro-types/src/settings/server.rs` +- Modify: `lib/crates/fabro-types/src/settings/mod.rs` +- Modify: `lib/crates/fabro-types/Cargo.toml` +- Modify: `lib/crates/fabro-config/src/layers/server.rs` +- Modify: `lib/crates/fabro-config/src/layers/mod.rs` +- Modify: `lib/crates/fabro-config/src/lib.rs` +- Modify: `lib/crates/fabro-config/src/resolve/server.rs` +- Modify: `lib/crates/fabro-config/src/tests/resolve_server.rs` + +**Approach:** +- Remove `ServerNamespace.ip_allowlist` and its `test_default()` population. +- Delete `ServerIpAllowlistSettings`, `ServerIpAllowlistOverrideSettings`, and + `IpAllowEntry`. +- Remove `ServerLayer.ip_allowlist`, `ServerIpAllowlistLayer`, + `ServerIpAllowlistOverrideLayer`, and `IntegrationWebhooksLayer.ip_allowlist`. +- Remove resolver functions and validation for global and webhook IP allowlists, + including `github_meta_hooks` parsing and Unix socket trusted-proxy checks. +- Remove `ipnet` from `fabro-types`. Keep `ipnet` in `fabro-config` because + `resolve/environment.rs` still validates sandbox CIDR policy. +- Replace old positive/negative IP allowlist config tests with unknown-field + tests for `[server.ip_allowlist]` and + `[server.integrations.github.webhooks.ip_allowlist]`. + +**Patterns to follow:** +- Existing unknown-field tests in `lib/crates/fabro-config/src/tests/resolve_server.rs` + for retired server settings. +- Existing `ServerLayer`/`ServerNamespace` layer-to-resolved pattern for removing + a server subdomain cleanly. + +**Test scenarios:** +- Error path: TOML with `[server.ip_allowlist]` fails parsing/resolution with an + unknown-field diagnostic mentioning `ip_allowlist`. +- Error path: TOML with `[server.integrations.github.webhooks.ip_allowlist]` + fails with an unknown-field diagnostic mentioning `ip_allowlist`. +- Happy path: minimal valid server settings still resolve without any + `ip_allowlist` field. +- Integration: serialized `ServerSettings` JSON no longer includes + `server.ip_allowlist`. + +**Verification:** +- `fabro-config` and `fabro-types` compile without removed IP allowlist symbols. +- Config tests prove stale allowlist settings are rejected. + +- [ ] **Unit 2: Remove server middleware and startup resolution** + +**Goal:** Remove all runtime source-IP filtering and GitHub `/meta` range +resolution from `fabro-server`. + +**Requirements:** R1, R4, R5 + +**Dependencies:** Unit 1 + +**Files:** +- Delete: `lib/crates/fabro-server/src/ip_allowlist.rs` +- Modify: `lib/crates/fabro-server/src/lib.rs` +- Modify: `lib/crates/fabro-server/src/server.rs` +- Modify: `lib/crates/fabro-server/src/serve.rs` +- Modify: `lib/crates/fabro-server/src/test_support.rs` +- Modify: `lib/crates/fabro-server/Cargo.toml` +- Modify: `lib/crates/fabro-server/src/auth/translate.rs` +- Modify: `lib/crates/fabro-server/src/web_auth.rs` +- Modify: `lib/crates/fabro-cli/tests/it/support/auth_harness.rs` + +**Approach:** +- Remove the public `ip_allowlist` module export. +- Delete `IpAllowlistConfig`, `IpAllowlist`, `GitHubMetaResolver`, client-IP + extraction, middleware, and GitHub meta cache helpers. +- Simplify `build_router_with_options` by removing its `Arc` + parameter and removing `RouterOptions.github_webhook_ip_allowlist`. +- Remove the global allowlist middleware layer from the main app router. +- Simplify `github_webhook_routes` so it only receives the webhook secret and no + route-specific allowlist config. +- Remove startup creation of `GitHubMetaResolver`, `default_ip_allowlist`, + `resolve_github_webhook_ip_allowlist`, and + `resolve_startup_github_webhook_ip_allowlist`. +- Replace all test/helper call sites that pass `IpAllowlistConfig::default()` + with the simplified router signature. +- Remove `ipnet` and any now-unused HTTP mocking/test-only dependencies from + `fabro-server` if they are only used by the deleted module. +- Consider whether `serve.rs` still needs `SocketAddr` for TCP + `ConnectInfo`; if it was only present for allowlisting tests, remove the + connect-info service wrapper and imports. + +**Patterns to follow:** +- Keep router layers ordered as they are after the allowlist layer is removed: + auth translation, demo routing, auth extension, canonical host, security + headers, HTTP logging, request ID. +- Existing webhook route HMAC tests and auth tests should remain the source of + truth for webhook behavior. + +**Test scenarios:** +- Happy path: API requests from any TCP remote address route normally when auth + requirements are satisfied. +- Happy path: `/health` remains accessible. +- Integration: GitHub webhook route still rejects missing/invalid signatures and + accepts valid signatures exactly as before. +- Cleanup: there are no references to `IpAllowlistConfig`, `ip_allowlist_middleware`, + `GitHubMetaResolver`, `github_meta_hooks`, or `github-meta-hooks.json`. + +**Verification:** +- `fabro-server` and CLI auth harness tests compile against the simplified router + API. +- Runtime startup no longer performs any GitHub `/meta` fetch for webhook IP + ranges. + +- [ ] **Unit 3: Update OpenAPI and generated API clients** + +**Goal:** Remove IP allowlist fields and schemas from public settings API +contracts and regenerated clients. + +**Requirements:** R3, R4 + +**Dependencies:** Units 1 and 2 + +**Files:** +- Modify: `docs/public/api-reference/fabro-api.yaml` +- Modify: `lib/crates/fabro-api/build.rs` +- Modify: `lib/crates/fabro-api/src/lib.rs` +- Modify: `lib/crates/fabro-api/tests/server_settings_round_trip.rs` +- Modify/generated: `lib/packages/fabro-api-client/src/models/*` + +**Approach:** +- Remove `ip_allowlist` from `ServerNamespace.required` and + `ServerNamespace.properties`. +- Remove `ServerIpAllowlistSettings`, + `ServerIpAllowlistOverrideSettings`, `IpAllowEntry`, + `LiteralIpAllowEntry`, and `GitHubMetaHooksEntry` schemas. +- Remove `IntegrationWebhooksSettings.ip_allowlist` from required/properties. +- Remove `with_replacement` entries and public re-exports for removed types in + `fabro-api`. +- Regenerate Rust API code by building `fabro-api`. +- Regenerate TypeScript Axios client and ensure stale allowlist model files and + index exports are gone. +- Strengthen round-trip tests to assert settings JSON omits + `server.ip_allowlist` and webhook `ip_allowlist`. + +**Patterns to follow:** +- Existing OpenAPI-first workflow in `AGENTS.md`. +- Existing `server_settings_family_reuses_domain_types` assertions for shared + domain type identity. + +**Test scenarios:** +- Contract: `ServerSettings` round-trips through API types without any IP + allowlist field. +- Contract: OpenAPI-generated TypeScript `ServerNamespace` has no + `ip_allowlist` property. +- Contract: `IntegrationWebhooksSettings` has only webhook strategy fields after + removal. + +**Verification:** +- Generated Rust and TypeScript clients match the updated OpenAPI spec. +- `rg "ServerIpAllowlist|IpAllowEntry|server-ip-allowlist|literal-ip-allow-entry"` + finds no remaining generated/public API references. + +- [ ] **Unit 4: Update web settings UI** + +**Goal:** Remove IP allowlist display from Settings > Security and align copy +with the new settings shape. + +**Requirements:** R3, R7 + +**Dependencies:** Unit 3 + +**Files:** +- Modify: `apps/fabro-web/app/routes/settings-security.tsx` +- Modify: `apps/fabro-web/app/routes/settings.tsx` + +**Approach:** +- Change the Security page description from "Authentication methods and network + allowlist" to authentication-only wording. +- Remove destructuring and rendering of `settings.server.ip_allowlist`. +- Remove the unused `Count` and `plural` imports if no longer used. +- Update the Settings nav description for Security from "Authentication and + network allowlist" to authentication-focused copy. + +**Patterns to follow:** +- Existing `settings-panel` row layout for Auth methods and Allowed usernames. + +**Test scenarios:** +- Typecheck: the route compiles against the regenerated API client with no + `server.ip_allowlist` property. +- UI behavior: Security page still renders auth methods and allowed usernames. +- Cleanup: no frontend references to `ip_allowlist`, `IP allowlist`, or + `trusted_proxy_count` remain. + +**Verification:** +- `apps/fabro-web` typecheck passes against the new generated client. + +- [ ] **Unit 5: Update tests, docs, changelog, and dependency lockfile** + +**Goal:** Remove stale references and document the operator-facing behavior +change. + +**Requirements:** R4, R6, R7 + +**Dependencies:** Units 1 through 4 + +**Files:** +- Modify: `lib/crates/fabro-server/tests/it/api/routing.rs` +- Modify: `lib/crates/fabro-server/tests/it/api/tcp.rs` +- Modify: `lib/crates/fabro-server/tests/it/api/settings.rs` +- Modify: `docs/public/administration/security.mdx` +- Create or modify: `docs/public/changelog/2026-05-27.mdx` +- Modify: `Cargo.lock` if dependency graph changes + +**Approach:** +- Delete route/TCP tests that only prove IP allowlist enforcement or + `ConnectInfo` behavior. +- Adjust any router setup helpers after the signature simplification. +- Add a settings API assertion that `server.ip_allowlist` is absent from + `/api/v1/settings`. +- Update security docs to explicitly state that Fabro does not provide inbound + source-IP allowlisting and operators should use upstream network controls. +- Add a changelog entry dated 2026-05-27 describing the hard removal and the + expected replacement at the deployment layer. +- Run dependency resolution after removing direct `ipnet`/test dependencies; + keep transitive or unrelated `ipnet` entries needed by sandbox CIDR validation. + +**Patterns to follow:** +- Existing changelog style in `docs/public/changelog/2026-05-26.mdx`. +- Existing settings API integration test style in + `lib/crates/fabro-server/tests/it/api/settings.rs`. + +**Test scenarios:** +- Settings API: response contains server auth, listen, storage, scheduler, and + integrations fields, but not `server.ip_allowlist`. +- Config compatibility: old IP allowlist TOML fails before startup rather than + being silently ignored. +- Docs validation: security docs no longer imply Fabro can restrict inbound + source IPs internally. + +**Verification:** +- `rg -n "ip_allowlist|trusted_proxy_count|github_meta_hooks|IP allowlist|ip allowlist"` + returns only historical archived plan/brainstorm/spec references or unrelated + non-server allowlist text. + +## System-Wide Impact + +- **Public API:** `GET /api/v1/settings` response shape changes by removing + `server.ip_allowlist` and webhook `ip_allowlist`. +- **Config compatibility:** Existing `settings.toml` files containing the removed + keys become invalid. This is intentional. +- **Runtime security posture:** Fabro no longer blocks requests based on source + IP. Operators must enforce network source restrictions upstream. +- **Webhook handling:** GitHub webhook HMAC verification remains unchanged; only + optional source-IP filtering is removed. +- **Generated clients:** Downstream TypeScript/Rust consumers that read + `server.ip_allowlist` must update. + +## Risks & Mitigations + +| Risk | Mitigation | +|------|------------| +| Operators accidentally expose a server that previously relied on Fabro IP allowlisting | Changelog and security docs explicitly call out the removal and direct operators to upstream controls. | +| Generated API clients retain stale types | Update OpenAPI first, regenerate both Rust and TypeScript clients, and run search checks for stale symbols. | +| Unrelated allowlist functionality is removed by broad search/replace | Scope searches to `IpAllow`, `ip_allowlist`, `trusted_proxy_count`, and `github_meta_hooks`; preserve sandbox CIDR and GitHub username allowlists. | +| Router signature cleanup breaks many tests | Update shared test helpers first, then compile-driven cleanup of remaining call sites. | +| `ipnet` is removed where still needed | Keep `fabro-config` dependency if environment CIDR validation still imports `ipnet::IpNet`. | + +## Test Plan + +- `cargo build -p fabro-api` +- `cd lib/packages/fabro-api-client && bun run generate` +- `cargo nextest run -p fabro-config -p fabro-types -p fabro-api -p fabro-server` +- `cd apps/fabro-web && bun run typecheck` +- `cd apps/fabro-web && bun test` +- `cargo +nightly-2026-04-14 fmt --check --all` +- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings` + +## Assumptions + +- The accepted compatibility policy is hard removal: no warning-only parser, no + migration, and no custom compatibility error path. +- Public API consumers can tolerate a breaking settings shape change for this + removed feature. +- Historical documents in `docs/plans/`, `docs/brainstorms/`, and + `docs/superpowers/` are archival and do not need rewriting. + +## Sources & References + +- Previous feature requirements: + `docs/brainstorms/2026-04-15-ip-whitelist-requirements.md` +- Previous implementation plan: + `docs/plans/2026-04-15-002-feat-ip-allowlist-plan.md` +- API workflow guidance: `AGENTS.md` +- OpenAPI source of truth: `docs/public/api-reference/fabro-api.yaml` + + +## Completed stages +- **toolchain**: succeeded + - Script: `command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1` + - Output: + ``` + cargo 1.95.0 (f2d3ce0bd 2026-03-21) + ``` +- **preflight_compile**: succeeded + - Script: `cargo check -q --workspace 2>&1` + - Output: (empty) +- **preflight_lint**: succeeded + - Script: `cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1` + - Output: (empty) +- **implement**: succeeded + - Model: gpt-5.5, 2.6m tokens in / 35.4k out + - Files: /home/daytona/workspace/fabro/docs/public/changelog/2026-05-27.mdx +- **simplify_opus**: succeeded + - Model: claude-opus-4-7, 21.1k tokens in / 5.5k out +- **simplify_gpt**: succeeded + - Model: gpt-5.5, 445.4k tokens in / 4.4k out +- **verify**: failed + - Script: `git fetch origin main 2>&1 && git merge --no-edit --no-stat origin/main 2>&1 && cargo +nightly-2026-04-14 fmt --all 2>&1 && cargo dev docs refresh 2>&1 && cargo +nightly-2026-04-14 fmt --check --all 2>&1 && { command -v rg >/dev/null 2>&1 || { echo 'rg is required for verify'; exit 127; }; } && ! rg -n 'AuthMode::Disabled|RunAuthMethod|RunSubjectProvenance|\bActorRef\b|\bActorKind\b|AuthenticatedSubject|AuthenticatedService|AuthorizeRunScoped|AuthorizeRunBlob|AuthorizeStageArtifact|AuthorizeCommandLog|auth_method\s*==\s*"disabled"' lib/crates apps lib/packages docs/public/api-reference/fabro-api.yaml 2>&1 && cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --workspace --status-level slow --profile ci 2>&1 && cargo dev docs check 2>&1 && bun install --frozen-lockfile 2>&1 && (cd apps/fabro-web && bun run typecheck) 2>&1 && (cd apps/fabro-web && bun run test) 2>&1 && (cd lib/packages/fabro-api-client && bun run typecheck) 2>&1 && cargo dev build -- -p fabro-cli --release 2>&1` + - Output: + ``` + From https://github.com/fabro-sh/fabro + * branch main -> FETCH_HEAD + a992a7d76..475b4ab65 main -> origin/main + Auto-merging Cargo.lock + Auto-merging lib/crates/fabro-api/build.rs + Auto-merging lib/crates/fabro-api/src/lib.rs + Auto-merging lib/crates/fabro-server/Cargo.toml + Auto-merging lib/crates/fabro-server/src/lib.rs + Auto-merging lib/crates/fabro-server/src/serve.rs + Auto-merging lib/crates/fabro-server/src/server.rs + Auto-merging lib/crates/fabro-server/src/server/tests.rs + Auto-merging lib/crates/fabro-server/src/test_support.rs + CONFLICT (content): Merge conflict in lib/crates/fabro-server/src/test_support.rs + Automatic merge failed; fix conflicts and then commit the result. + ``` + +## Context +- failure_class: deterministic +- failure_signature: verify|deterministic|script failed with exit code: ## output from https://github.com/fabro-sh/fabro * branch main -> fetch_head .. main -> origin/main auto-merging cargo.lock auto-merging lib/crates/fabro-api/build.rs auto-merging lib/crates/fabro + + +The verify step failed. Read the build output from context and fix all format, clippy, Rust test, docs, TypeScript typecheck/test, and build failures. \ No newline at end of file diff --git a/stages/009-fixup@1/provider_used.json b/stages/009-fixup@1/provider_used.json new file mode 100644 index 000000000..672cc50e5 --- /dev/null +++ b/stages/009-fixup@1/provider_used.json @@ -0,0 +1,5 @@ +{ + "mode": "agent", + "provider": "anthropic", + "model": "claude-opus-4-7" +} \ No newline at end of file diff --git a/stages/009-fixup@1/response.md b/stages/009-fixup@1/response.md new file mode 100644 index 000000000..44690e98d --- /dev/null +++ b/stages/009-fixup@1/response.md @@ -0,0 +1,7 @@ +Clippy is clean. The merge conflict was the only issue. My fix resolves it by accepting the upstream changes (automation_materializer imports) and dropping the no-longer-needed `IpAllowlistConfig` import that this branch already removed. + +Summary: +- The verify step failed because the auto-merge from origin/main left conflict markers in `lib/crates/fabro-server/src/test_support.rs`. +- The conflict was between this branch's removal of `crate::ip_allowlist` and origin/main's addition of `crate::automation_materializer` imports. +- Resolved by keeping the new `automation_materializer` imports and dropping the obsolete `IpAllowlistConfig` import. +- `cargo check`, `cargo fmt`, and `cargo clippy --workspace --all-targets -- -D warnings` all pass. \ No newline at end of file