Conflicts were between this branch's parallel-branch identity work and
main's stage billing, review targets, and live stage timing.
- Stage fixtures: main added `billing` to each per-file `makeStage`; this
branch had hoisted one builder into `lib/test-utils`. Kept the hoisted
builder and gave it `billing: makeBilledTokenCounts()`, so both intents
hold and the field list stays in one place. `stage-sidebar.test.ts` also
builds raw `RunStage` wire payloads, so it keeps importing
`makeBilledTokenCounts` directly.
- Import lists (`run_projection.rs`, `fabro-api/src/lib.rs`,
`run_state.rs`, `stage_projection_round_trip.rs`): unioned both sides —
`ParallelBranchId` alongside `timing`, `ReviewTarget`,
`ReviewTargetKind`, `AttrValue`, `Node`, and
`StageToolBatchProjection`.
- `fabro-server` tests: git interleaved two unrelated new tests into one
body. Split them back into
`list_run_stages_exposes_parallel_branch_identity` and
`run_billing_includes_live_stage_timing_in_rows_and_totals`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Branches bypass the engine's stage.started/stage.completed lifecycle, so
no SWR key invalidated the stages list while a fork ran. The new live
branch rows stayed frozen at their first observed state until an
incidental refetch. Map parallel.* events to the stages list, run events,
and graph keys.
Also:
- Label branch rows with formatStageLabel so a re-entered branch renders
as `review_glm@2`, matching the sidebar and waterfall.
- Build branch rows in one pass and count live outcomes in one loop.
- Name ParallelBranchId in the OpenAPI spec and reuse fabro_types::
ParallelBranchId, replacing two copies of an inline string format.
- Hoist makeStage and textContent into lib/test-utils so widening Stage
cannot leave per-file fixtures stale (tests are excluded from
typecheck, so the two component-test copies had already gone stale).
- Query stat tiles by data-stat instead of an exact Tailwind class.
- Reuse append_scoped_stage_event's body via append_event_with_scope and
add test_branch_event instead of poking envelope fields.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Conflict in apps/fabro-web/app/components/interview-dock.tsx. Main moved
the dock onto the shared collapsible `RunDockShell` and replaced the
local button constants with shared ones.
Kept main's structure whole and re-applied the review target rendering
onto it: the question paragraph in the shell's `body` becomes the linked
`ReviewTargetQuestion` when the target passes `safeReviewTarget`, and
plain text otherwise. Both now share main's paragraph classes through
`QUESTION_TEXT`, so the two renderings stay visually identical.
`peek` keeps using `question.text`, which is the correct plain-text
collapsed summary for a review target question.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`active_time_ms` was only ever computed from terminal stage events, so a
stage still running contributed zero to the run rollup. A run parked in one
long agent stage reported 2m 8s of active time against 16m 53s of wall
clock — the two finished stages — while the running stage had been doing
continuous inference and tool work for over 14 minutes.
`live_run_timing` summed `filter_map(|stage| stage.timing)`, and
`stage.timing` is only written at finalization. Wall time ticked live off
`start_time`; active time did not tick at all.
Stage projections now accumulate brackets from the event log:
- Closing an inference bracket folds its span into `live_inference_ms`
instead of discarding it, including across retries, matching the
in-process stopwatch.
- Tool calls open a batch on the first outstanding call and close it when
the last one drains, so tools running concurrently within a turn count
once — the same span `execute_tool_calls` is bracketed by. Summing
per-call durations would over-count parallel tool use. Subagent tool
events are excluded; they run inside the root call's span already.
- `StageProjection::live_timing(now)` composes accumulators with any open
bracket, per handler: agent stages use the brackets, prompt and command
stages count elapsed time as inference and tool respectively, and
handlers that wait on a human, timer, condition, or child branches
report zero.
Active is clamped to wall per stage. A worker killed mid-turn leaves its
bracket open forever, and without the clamp it would tick up unbounded.
The clamp does not need to detect the dead worker: a stage cannot have been
active longer than it has existed. `watchdog.timeout` remains the authority
on whether a run is stuck. The clamp is deliberately not applied at run
level, where concurrent branches can legitimately sum past run wall time.
Timing is derived from events rather than emitted by the worker, so this
needs no event-schema change and applies to runs already stored.
`StageProjection.timing` keeps its terminal-only meaning, and the
authoritative breakdown still replaces the live estimate at terminal
events.
The billing endpoint had the same hole behind its `wall_only` fallback:
running stages reported zero inference/tool/active. Not visible in the
product, which renders only `wall_time_ms`, but wrong for any other
consumer of `GET /runs/{id}/billing`.
Parallel branch stages lose their breakdown permanently, even after
completion, because `parallel.branch.completed` carries only `duration_ms`.
That is a separate data-loss bug, tracked in #644.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
During a long LLM turn the durable event stream was silent: between
`agent.tool.completed` and the next `agent.message` nothing was emitted,
so "the model is generating" and "the worker is wedged" were
indistinguishable from the run store, SSE, or the UI.
The signal already existed. `AssistantTextStart` fired at exactly the
right point — after `build_request()`, after compaction, immediately
before the stream opens — then was classified as streaming noise and
thrown away. This promotes it rather than inventing a new one.
Two events, each asserting only what is provable when it is emitted:
- `agent.llm.started` carries the *requested* provider/model. No usage,
no cost, no context window: none of it exists yet, and failover can
re-target, so `agent.message` stays authoritative for what answered.
- `agent.llm.first_output` is edge-triggered on the first output of an
attempt and names what arrived. `ToolCall` is required, not optional:
a turn that opens with a tool call produces no text or reasoning
delta, so a latch keyed on those two would stay silent for exactly
the tool-heavy rounds where liveness matters most.
`agent.llm.retry` now also fires on the one previously invisible
mid-turn path — a stream that ends without a finish event, which
replays the turn and discards its output with nothing to show for it.
Its `attempt` field was already fed by two independent counters, so an
optional `phase` (open | consume) names which loop it counts.
`StageProjection.inference` projects the open bracket. `Some` means
"the event log contains an unclosed inference bracket", not "the model
is computing now" — a SIGKILLed worker leaves it open, which is the
truthful statement of what we know, and `watchdog.timeout` remains the
authority on actually-stuck.
The close is the subtle part. Terminal cancel and wall-clock timeout
tear the session down through `discard_session` without emitting a
message, error, or interrupt, so a session-lifecycle backstop is
required. It has to be `agent.session.ended`, not
`agent.session.deactivated`: deactivation is emitted by `lease.release()`
*before* the forwarder drains queued agent events, so a queued
`agent.llm.started` can arrive after it and re-open the bracket. But
`agent.session.ended` carries no stage identity, so the close takes
ordering from the event and identity from the projection, scanning for
brackets the ending session opened. A normal stage lookup there finds
no target and silently no-ops.
Presentation states what the log proves and nothing more: no progress
bar or ETA (no completion estimate exists), "reasoning" only when the
provider sent reasoning output, elapsed counted since the request
opened, and no live animation once the run is terminal.
Scope is session-backed agent stages. One-shot completions call
`client.complete` directly and never build a session; covering them
means moving the emit point into `fabro-llm`, filed as a follow-up.
`agent.output.start` was never persisted — it existed in a name map,
an `unreachable!` arm, and docs — so the rename carries no migration
risk. Corrects `events.md`, which documented it as a real emitted
event, and the v2 proposal, which mapped it to `message.part.started`
despite it firing before the request opens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Kimi profile was registering the Anthropic task tools. Both persist
through the same TodoRuntime, but they model opposite interactions: TaskCreate
and TaskUpdate mutate individual tasks against tracked ids, while Kimi Code's
TodoList replaces the whole list in one call. Of the two surfaces fabro already
had, Kimi was given the one furthest from what its models are trained on.
Add TodoListKind::KimiTodos and a TodoList tool matching Kimi Code's contract
exactly:
TodoList({ todos?: [{ title, status: pending | in_progress | done }] })
Omitting `todos` reads the list, an empty array clears it, and a list replaces
it. Reconciliation mirrors update_plan -- items are identified by their text,
so re-submitting a list preserves identity for unchanged entries -- and the
runtime, projections, and events are unchanged.
Two differences from the existing surfaces were behavioral rather than
cosmetic. Items carry only `title`, where TaskCreate requires both `subject`
and `description`, so a model with nothing to say for a description had to
invent one. And the terminal status is spelled `done`; `completed` is the
Anthropic and Codex spelling, and a model emitting `done` against the old
schema got a validation error rather than a todo. The internal representation
stays TodoStatus::Completed; only the wire vocabulary differs.
Kimi todo lists are session-scoped like OpenAI plans, so the root-agent
projection excludes subagent lists the same way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Normalize the readable reasoning providers already return into a
canonical `ReasoningOutput` and carry it through the `agent.message`
run event to storage, SSE, and JSONL.
The shape is derived from the final response's canonical message
content rather than stored a second time, so there is no duplicate
source of truth and retried or replaced streaming buffers never
become durable reasoning. OpenAI-compatible `reasoning_details` are
now preserved verbatim as an opaque content part; only known readable
members are normalized out of them, leaving encrypted entries for a
later provider-aware replay phase.
This phase is passive: no request parameters change, no capability
guessing, and no newly observed provider field is replayed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Guard EventScan seeks against sequences past MAX_EVENT_SEQ: a
seven-digit start prefix sorts below six-digit event keys, so an
unvalidated since_seq like 5000000 returned an incorrect slice of
history instead of an empty page. An end bound past MAX_EVENT_SEQ now
delegates to the unbounded scan, which is equivalent because no stored
sequence exceeds it.
Clamp the descending exclusive end to just past the newest stored
event, so an oversized before_seq cursor pages from the newest event
instead of probing empty key space and returning nothing.
Split RunEventListParams out of EventListParams so before_seq and
order are only accepted by /runs/{id}/events; the session, stage,
pair transcript, and demo endpoints go back to ignoring them instead
of accepting order=desc while returning ascending results.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolves conflicts with the shared-checkout parallel rewrite (#607) and the
cached-run/billing dedup (de60eb900):
- handler/parallel.rs: rebuilt on main's shared-checkout version. Branch
ordinals are still reserved inside the branch task right before
ParallelBranchStarted (with graph_visit/resumed_from_stage_id), and the
reserved StageScope is shared with post-await error paths via a OnceLock
slot instead of main's dispatch-time visit=1 scope, so completion events
are never emitted under a guessed ordinal.
- billing.rs: keep this branch's run_stage_from_projection (RunStage grew
graph_visit/resumed_from_stage_id and a typed id), adopt main's
state.cached_run() and drop the removed run_stage_from_stage_id import.
- run_projection.rs: adopt main's typed parallel_results
(Option<Vec<ParallelBranchResult>>).
- run_event/misc.rs: union of both sides' imports.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On a projection-cache miss, descending pagination recovered the latest
sequence by scanning the run's entire event prefix, making a cold-cache
order=desc request O(total_events). Binary-search the zero-padded
sequence key space with single-entry probes instead, bounding recovery
to O(log MAX_EVENT_SEQ) reads. The probe predicate (smallest stored
sequence at or above a bound) stays monotone across gaps left by
failed appends.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolved conflicts against main's shared-projection-cache rework:
- projection_cache.rs: kept main's projection_snapshot and dropped this
branch's last_seq accessor, which it subsumes; latest_event_seq now
reads the sequence from projection_snapshot.
- run_store.rs: kept main's EventScan cursor and added a seek_before
constructor so the backward-pagination range scan bounds its end key
through the same abstraction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Event keys zero-pad seq to six digits, so an exclusive end bound past
MAX_EVENT_SEQ formatted as a seven-digit prefix that sorts before real
event keys, producing an inverted scan range. This made the newest page
come back empty once a run reached MAX_EVENT_SEQ, and let a client
supplied before_seq beyond MAX_EVENT_SEQ garble the range. Clamp the
bound and treat anything past MAX_EVENT_SEQ as unbounded; no stored
sequence exceeds it, so the results are equivalent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolved conflict in run_store.rs tests: kept both the new
list_events_before_with_limit tests from this branch and the
append_event_rejects_sequences_beyond_key_order_limit test from main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add AppState::cached_run with the standard 500/404 mapping and use it
everywhere handlers read the shared run-projection cache. This also
normalizes two inconsistencies: graph-source cache errors now map to
500 (was 502), and a missing projection in PR create/unlink now maps
to the canonical 404 (was a bespoke 500).
- Extract an EventScan cursor shared by the four run-event scan loops,
delegate list_events_from to the paginated variant, and stop the
stage-event scan once its page is full instead of walking the rest of
the log.
- Hold Arc<RunProjection> in the local projection cache so opening a run
no longer deep-copies the projection (copy-on-write via Arc::make_mut),
and drop the now-unreachable shared-cache branch in last_event_seq.
- Trim hot-path clones: run_files serves the projection Arc directly,
run-state serializes by reference, artifacts only checks existence, and
the command-log handler opens a reader only for the CAS-blob branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A node cancelled (or lost to a crash) mid-flight and then resumed now
starts a new stage execution with the next StageId ordinal (work@2)
instead of reusing and clearing the cancelled execution's projection.
The old execution stays immutable with its own events, session, output,
timing, billing, and termination state.
Engine:
- Add a run-scoped StageExecutionTracker on RunServices with per-node
high-water marks. Ordinals are reserved after the StageStart hook
passes on the first attempt (retries reuse the reservation), ensured
at the composite checkpoint pre-step for hook-skips, and reserved in
on_terminal_reached for terminal nodes' synthetic events.
- Keep three concepts distinct: graph visit (max_visits/checkpoints,
unchanged), stage execution ordinal (the @N in StageId), and handler
attempt. The tracker is not checkpointed; the append-only stage event
history is its durable source of truth.
- resume() seeds the allocator from the run projection and computes a
node -> StageId provenance map of executions observed after the
selected checkpoint, threaded through execute_persisted_run,
RunSession, and InitOptions.
Events and projections:
- stage.started, parallel.branch.started, and checkpoint.completed
carry optional graph_visit and resumed_from_stage_id; StageProjection
stores both. Old events deserialize with None and legacy duplicate
stage.started replays keep last-attempt behavior.
- The CheckpointCompleted reducer is envelope-first: diffs and
skipped-stage synthesis attach to the exact execution StageId, an
existing Retrying projection finalizes as Skipped without losing
identity, and historical node_outcomes no longer create or collide
with newer ordinals (node_visits remains a legacy fallback).
Handlers:
- Parallel fan-out reserves child ordinals through the shared tracker,
derives worktree pass{N} from the parent's execution ordinal, and
seeds branch contexts with explicit child stage scopes so branch
lifecycle and nested handler events agree.
- Artifact capture and manager-loop child logs follow the ordinal.
API and UI:
- RunStage documents visit as the execution ordinal and adds optional
graph_visit and resumed_from_stage_id; Rust and TypeScript clients
regenerated.
- The web sidebar lists both executions chronologically; resumed stages
show a "Resumed from" link in the stage detail header and hover
popover, with the graph visit surfaced when it diverges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Extract a shared fetch_run_events_page helper so the three client
paging loops (full list, until, tail) no longer repeat the request/
convert/has_more skeleton; fold the tail loop's two descending-order
checks into one and drop its redundant had_events flag.
- Skip the latest-seq lookup in list_events_before_with_limit when the
caller supplies a before_seq cursor, so a cold projection cache costs
at most one full history scan per pagination session instead of one
per page.
- Remove the dead before_seq max(1) clamp and the passthrough order()
accessor from EventListParams.
- Document the CLI --tail 0 --follow seeding trick and the reader
event_seq placeholder invariant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Unify list_events_from with list_events_from_with_limit so projection
replay shares the seek path instead of duplicating the decode loop
- Bound the event scan with keys::run_events_range instead of an
unbounded range plus a manual prefix break, so slatedb never touches
SSTs belonging to other runs or namespaces
- Store reader event_seq as None instead of a valid-looking sentinel of
1, so appends through a reader-built inner fail as ReadOnly rather
than writing duplicate sequences
- Borrow keys during scans instead of allocating a String per entry,
drop a dead branch in cached_events_from, collapse recover_next_seq's
single-caller parameters, and document the zero-padded key ordering
invariant the seek depends on
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
