fabro/docs/internal/parallel-strategy.md

Parallel fan-out / fan-in strategy

Status: proposed (not yet implemented). Line numbers reference the tree at the time of writing and will drift.

This document specifies the intended product behavior of parallel fan-out (shape=component) and fan-in (shape=tripleoctagon). Appendix A catalogs pre-existing bugs this design resolves. Appendix B lists behavior changes relative to today's implementation.

1. Introduction: goals and current weaknesses

The goal of this design is a parallel execution model that is simple, coherent, and correct:

• Simple — a user should be able to predict what a fan-out/fan-in does from the graph alone. One mental model ("branches produce candidates; fan-in picks a commit; downstream sees all responses"), no new syntax for synthesis, no incantations (special fidelity settings, escape-hatch attributes) to make the flagship patterns work.
• Coherent — the same rules apply regardless of node type or channel. What holds for a sequential command node's output should hold for a branch command node's output; the workspace (git) facet and the text (context) facet should follow parallel logic; selection should live in exactly one place.
• Correct — the engine must do what the graph, the docs, and the recorded run state say it does. Nodes drawn in the graph must run; documented outputs must exist; a judge asked to pick the best candidate must be shown the candidates.

Today's implementation misses all three. Issue #490 is the visible symptom: a synthesis node after fan-in never sees branch responses — only {id, status, head_sha} metadata — while two tutorials promise the opposite. Investigation showed a broader incoherence:

• Inherited spec gap. The Attractor spec (fabro's ancestor) deliberately isolates branch context and never defines a channel for branch output text. Its fan-in pseudocode judges candidates it cannot see (llm_evaluate receives only statuses) and sorts by a score field nothing sets. Fabro inherited this gap faithfully.
• Missing compensation. Kilroy (a sibling Attractor implementation) compensates with a file/git handoff: the post-merge node's prompt is injected with each branch's worktree_dir, logs_root, and head_sha plus instructions to read/merge them. Fabro has no equivalent, but its tutorials promise kilroy-like behavior ("the synth node receives all four perspectives in its preamble").
• Accidental semantics. Several adjacent behaviors are unprincipled accidents rather than decisions: branches execute a single node and silently skip chained nodes; two handlers fast-forward to two different definitions of "winner"; branch nodes run with a stale preamble built for the fan-out node; selection is vacuous in both modes; nested fan-outs silently lose git isolation. Appendix A catalogs these.

2. Core model: candidates

A parallel branch is an isolated unit of execution that produces a candidate. A candidate has exactly three facets:

Facet	Content	Carried by
Commit	Workspace state produced by the branch	Per-branch git commit (head_sha)
Response	Text output (LLM response, command output)	Branch stage records + context keys
Verdict	Terminal status plus optional numeric score	parallel.results entries

Fan-out produces N candidates in isolation. Fan-in selects one commit to continue on. Downstream nodes (synthesis) get all responses. Selection and synthesis are distinct concerns: selection needs a node (tripleoctagon); synthesis is any ordinary downstream node, because responses propagate.

The post-fan-in contract, in one sentence: after fan-in, the run looks as if every branch had run sequentially, and the winner ran last.

3. Topology: branches are subgraphs

Each outgoing edge of a fan-out node starts a branch. A branch executes as a subgraph walk: the engine traverses nodes and edges from the branch entry node until it reaches the join node (the fan-in). Multi-node chains (fork -> plan_a; plan_a -> impl_a; impl_a -> merge) run every node in the chain. This matches the Attractor spec (execute_subgraph) and kilroy (runSubgraphUntil); today fabro executes exactly one node per branch and silently skips the rest (Appendix A.2).

Structural validation (lint):

• Every path from a branch entry must converge on the run's join node. A branch path that escapes the join (reaches exit, or a node outside the fan-out region) is a validation error.
• A nested component node inside a branch is rejected until worktree-from-worktree isolation is implemented (today it silently runs with no git isolation; Appendix A.6).
• fidelity="full" on a fan-in's outgoing edge gets a lint warning: branches run on different threads, so full fidelity can never carry branch outputs (it drops the preamble entirely).

4. Node types in branches

No type-based restrictions. Restriction is structural (§3), not by allowlist:

• Agent / prompt nodes — the primary case.
• Command / script / tool nodes — first-class. Deterministic fan-out (test matrices, benchmark bake-offs across worktrees) is a supported pattern with no LLM anywhere: branches emit score via status fields, heuristic selection picks the winner by measurement.
• Conditionals — meaningful under subgraph branches: they route within the branch.
• Human gates — allowed; each branch may pause independently.
• Nested parallel — rejected by lint until isolation composes (§3).

5. Git isolation

Unchanged mechanics, with ownership fixed:

1. Before fan-out, checkpoint the sandbox to produce base_sha.
2. Each branch gets a worktree on a branch ref (fabro/run/parallel/<run>/<node>/pass<N>/<branch>), rooted at base_sha. The branch's internal.work_dir points at the worktree.
3. After a branch completes, git add -A + commit (--allow-empty) yields the candidate's head_sha.
4. Worktrees are removed after the join. Loser branch refs are kept so downstream nodes and humans can git show/git diff any candidate.
5. Fan-in exclusively owns the fast-forward. The parallel handler performs no merge. After selection, fan-in fast-forwards the primary workspace to the winner's head_sha. (Today both handlers fast-forward, to potentially different winners; Appendix A.3.)

Degradation without git (no repo, or git isolation disabled): branches share the primary sandbox with no workspace isolation, head_sha is absent from candidates, and fan-in performs no merge. Response and verdict facets work unchanged — prompt-only ensembles do not require git.

6. Execution and stage recording

Branch nodes execute as real stages, recorded through the normal ExecutionState::record path and namespaced under the fan-out (e.g. stage a@1 within fork@1). Consequences (all fixes to current behavior):

• Branch prompts/responses appear in events, fabro dump, and the web UI as ordinary stages.
• Each branch node gets a freshly built preamble for its own position, via the standard lifecycle, instead of reusing the fan-out node's stale preamble.
• Branch stages participate in the standard retry policy per node.

7. Context merge-back at fan-in

When fan-in completes, branch context updates are applied to the parent context with a collision rule:

• Per-node keys (response.<branch_node_id>, structured-output fields namespaced by node) apply for all branches. Branch node IDs are unique, so no collisions.
• Singleton keys (last_stage, last_response, command.output, un-namespaced status fields) are taken from the winner only.
• Failed branches' response.<id> values are still applied (a synthesis node analyzing disagreement wants to see the failure text). Their singletons are never applied.

Fan-in additionally writes (as today):

• parallel.results — one entry per candidate: {id, status, head_sha?, score?}.
• parallel.branch_count, parallel.fan_in.best_id, parallel.fan_in.best_outcome, parallel.fan_in.best_head_sha.

No file is materialized into the run workspace. parallel.results reaches LLM consumers through the preamble's context section, and agents can git show any candidate via its head_sha. (The current docs claim parallel_results.json is available to downstream nodes; that claim is false today and should be corrected rather than implemented — see open question 4 for the one consumer this leaves unserved.)

8. Selection

Fan-in selects the winning candidate. Two modes, as today, but with real signal:

• Heuristic (no prompt on the fan-in node): rank by status (succeeded < partially_succeeded < failed), then score descending, then lexical id. score becomes settable: branches emit it via structured-output / status fields, which now survive into parallel.results (§7).
• LLM judge (fan-in node has a prompt and a backend is configured): the judge prompt includes, per candidate: id, status, score, a bounded response excerpt, and git diff --stat vs. base_sha when git isolation is active. Today the judge sees only {id, status, head_sha} and cannot possibly discriminate (Appendix A.4).

Selection determines the commit facet only. It does not suppress loser responses (§7) or loser refs (§5).

9. Downstream visibility (preamble)

Prompt templates render once at manifest build time with {goal, inputs} only; the preamble is the sole channel for runtime context into a fresh-session node. Therefore:

• At compact (default) fidelity, branch stage summaries render their responses inline, bounded, with a See: <blob path> reference when truncated — the same treatment command.output already receives at compact. Rationale: post-fan-in branch responses are unrecoverable through any fidelity setting (different threads), exactly like command output.
• The per-branch response budget is larger than command output's 25-line tail (ensemble analyses front-load their substance; a small tail amputates it). Exact budget TBD at implementation; must remain bounded so N long branches cannot blow the downstream context. Agent-type synthesis nodes can read the full text from the blob/artifact reference.
• summary:high renders the same with its larger budget. truncate carries goal only (explicit opt-out). full remains the degenerate case and lints (§3).

10. Join policies

• wait_all (default): all branches run to completion; join proceeds when all are terminal. Succeeds if no branch failed, else partially succeeds (fan-in fails only when all candidates failed).
• first_success: join proceeds at the first successful branch. Remaining branches are cancelled; cancelled branches record a terminal cancelled stage (their partial responses are not merged back). The sole successful branch is the winner.
• k_of_n / quorum (kilroy has them): deliberately not added now. The surface stays minimal until a concrete need appears.

Open questions

1. Implementation phasing: subgraph branches (§3) are the largest lift. Response propagation (§6–§9) fixes #490 and both tutorials on its own and can ship first.
2. first_success cancellation semantics for in-flight agent sessions (graceful stop vs. abort; what the cancelled stage records).
3. Exact preamble budget per branch response (§9).
4. Context access for deterministic post-fan-in consumers. Command/script nodes have no channel to context (no preamble, no template rendering in script), so a deterministic aggregator after fan-in cannot learn candidate head_shas. Candidate mechanisms: a results file under a checkpoint-excluded workspace path, or an env var (e.g. FABRO_PARALLEL_RESULTS) pointing at a file outside the workspace. A bare workspace file is ruled out: checkpoint commits git add -A, so it would leak into run history and PRs. Design alongside the deterministic fan-out pattern (§4).

---

Appendix A: pre-existing bugs

Cataloged against the current tree; line numbers will drift.

1. Branch outputs dropped (#490). The fan-out task reads only outcome.status and head_sha from each branch; branch context_updates (including response.<id>) are discarded with the forked context (fabro-workflow/src/handler/parallel.rs:389-397,465-470). No channel carries branch text to downstream nodes. Confirmed by live repro on fabro-testing (run 01KX148ZAMMJRAADHK1HBF7PC3, server 0.287.0-nightly.0).
2. Chained branch nodes silently skipped. Branches execute exactly one node; the engine then jumps to the join (parallel.rs:388-397,612,628; fabro-core/src/executor.rs:421). In fork -> a; a -> a2; a2 -> merge, a2 never runs and nothing warns.
3. Double fast-forward with two different winner definitions. The parallel handler fast-forwards the lexically first successful branch (parallel.rs:511-538); fan-in then fast-forwards its selected winner (fan_in.rs:120-131). If selection ever picks a non-lexical-first branch, the second --ff-only merge cannot succeed (sibling commits diverge). Masked today only because selection is vacuous (A.4).
4. Selection is vacuous. Heuristic tie-breaks on a score field nothing can set (scores would arrive via branch context updates, which are dropped per A.1). The LLM judge prompt is serde_json::to_string_pretty of parallel.results — id/status/head_sha only (fan_in.rs:247-250). Both modes reduce to "first successful branch, alphabetically."
5. Branch preambles are stale. Branches run via dispatch_handler, bypassing the lifecycle's per-node preamble rebuild; each branch node inherits current.preamble as computed for the fan-out node itself.
6. Nested parallel silently loses git isolation. Branch EngineServices are built with git_state: RwLock::new(None) (parallel.rs:380), so a component node inside a branch runs its own branches with no worktrees and no warning.
7. Docs contradict the engine. tutorials/ensemble.mdx:85 and tutorials/parallel-review.mdx:81 claim the post-merge node receives all branch perspectives in its preamble (false, per A.1). workflows/stages-and-nodes.mdx:195 claims merged results are available to downstream nodes as parallel_results.json (the file exists only under stages/<fork>@1/ in dumps, not in any node's working directory).
8. fidelity="full" across a fan-in is a trap. It drops the preamble (metadata included) and cannot attach to any branch thread; raising fidelity strictly reduces what the downstream node sees. No lint warns.

Appendix B: behavior changes vs. today

Changes a user could observe if this spec is implemented as written.

1. Chained branch nodes execute. Graphs that (unknowingly) relied on single-node branch semantics will now run the full chain (fixes A.2; may lengthen existing runs).
2. New validation errors. Branch paths that don't converge on the join, and nested component nodes inside branches, become lint failures for graphs that previously ran (with wrong or silently degraded semantics).
3. Fan-in owns the fast-forward. The workspace after fan-in may land on a different commit than today whenever selection (scores, LLM judge) disagrees with lexical-first order. The parallel handler no longer merges.
4. Post-fan-in context is richer. response.<id> for every branch, winner-sourced singletons (last_stage, last_response, command.output), and score in parallel.results. Today those singletons retain their pre-fork values; workflows with edge conditions over them could route differently.
5. Preambles after fan-in grow. Branch responses render inline at compact fidelity (bounded). Downstream nodes see more tokens per run; snapshot tests over preambles will churn.
6. Branch executions become visible stages. Events, dumps, the web UI, and the stage list gain per-branch stages (a@1 under fork@1). Consumers of events.jsonl / the API will see new stage records.
7. parallel_results.json docs claim is corrected, not implemented. workflows/stages-and-nodes.mdx:195 is updated to describe the real channels (context key + preamble); no file appears in the workspace (open question 4 covers deterministic consumers).
8. Loser branch refs are documented as retained and become part of the product contract instead of an accident of not deleting them.
9. first_success cancels losers explicitly and records cancelled stages; today's exact cancellation behavior is unspecified.
10. LLM judge prompts change shape. Fan-in nodes with prompts now send candidate excerpts and diff stats to the judge — more tokens, different (better) selections than today's id-only prompt.