Splitting on the first colon in FromStr broke bare model IDs that
legitimately contain one. A reference like "llama3:8b" parsed as
provider "llama3" selector "8b", and since "llama3" is not a provider
the lookup failed instead of passing the ID through to the pinned
provider. Verified against origin/main: canonical_session_model with
"future-model:latest" pinned to openrouter returned the passthrough
before and a 400 after.
This is not fixable by choosing a different separator. Bedrock
inference-profile ARNs contain both colons and slashes, and
docs/public/integrations/bedrock.mdx tells users to put arbitrary
inference-profile IDs in api_id. Only the registry can tell a provider
prefix from a model ID that happens to contain the separator.
FromStr now leaves colon-bearing tokens bare, and ModelRef::qualify
promotes only those whose prefix names a known provider. resolve()
applies it, so the fallback path is covered; sessions.rs applies it
before its own match so it keeps its tailored ambiguity messages.
ModelRegistry is now implemented for Catalog in fabro-types, replacing
the CatalogModelRegistry wrapper that existed only in start.rs, so both
call sites share one registry view.
Covered by regression tests at both surfaces, plus qualify unit tests
for ollama tags and Bedrock ARNs. The pre-existing passthrough test
canonical_session_model_preserves_unknown_passthrough_on_selected_provider
passes again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Conflict in apps/fabro-web/app/components/interview-dock.tsx. Main moved
the dock onto the shared collapsible `RunDockShell` and replaced the
local button constants with shared ones.
Kept main's structure whole and re-applied the review target rendering
onto it: the question paragraph in the shell's `body` becomes the linked
`ReviewTargetQuestion` when the target passes `safeReviewTarget`, and
plain text otherwise. Both now share main's paragraph classes through
`QUESTION_TEXT`, so the two renderings stay visually identical.
`peek` keeps using `question.text`, which is the correct plain-text
collapsed summary for a review target question.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The process environment is no longer a configuration source. `{{ vars.NAME }}`
(non-sensitive, server-stored) and `{{ secrets.NAME }}` (vault-backed) cover
both cases, and reading the worker's ambient environment made a run's inputs
depend on how its process happened to be launched.
`Namespace::Env` is kept but wired to nothing, so `{{ env.NAME }}` still
parses and fails with a message naming its replacement rather than reaching
a consumer as literal text. `ResolveCtx::with_env` is gone, so no call site
can opt back in.
Two long-standing warts were env-only and go with it:
- `InterpString::resolve_or_source`, the "fall back to the raw template
source on failure" path, which let an unresolved token reach a sandbox or
the GitHub API as literal `{{ ... }}` text. Its own comment noted it was
slated for hard-error semantics.
- `RunEnvironmentSettings::resolve_env`'s matching source fallback for
env-only values.
Both carried `#[expect(clippy::disallowed_methods)]` escape hatches. Every
run-boundary resolver — sandbox env, prepare steps, MCP transports, GitHub
permissions, Slack channels, run goal files, provider extra_headers — now
fails closed instead.
Hooks lose their `allowed_env_vars` allowlist, `resolve_header`, and
`HeaderResolveError` along with the `E: Env` generic threaded through the
executor. They keep `{{ vars.* }}`, which `RunSettings::substitute_variables`
already substitutes server-side at run creation.
`allowed_env_vars` is removed from the OpenAPI spec and the generated
TypeScript client. The docs example showing `{{ env.* }}` in
`[server.slatedb.s3].bucket` was already wrong — that field is a plain
String and never interpolated — and is now a literal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The model indicator on a stage page hovered to provider, model, and
reasoning effort only. Seeing what a stage actually spent meant leaving
for the Billing tab, which reports per node rather than per visit.
The stage list had no token data to show, so add a per-visit `billing`
block to `GET /runs/{id}/stages`. The Billing tab's pricing rule (a
provider-reported cost wins, otherwise the server catalog prices the
tokens) was private to `billing_rollup`; move it to
`StageProjection::billed_usage` and drive both call sites from it so the
two views cannot drift.
The popover's buckets use the Billing tab's labels verbatim. It stays
scoped to one visit, so a looped node's row on the Billing tab is the sum
of what each of its visits shows here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`active_time_ms` was only ever computed from terminal stage events, so a
stage still running contributed zero to the run rollup. A run parked in one
long agent stage reported 2m 8s of active time against 16m 53s of wall
clock — the two finished stages — while the running stage had been doing
continuous inference and tool work for over 14 minutes.
`live_run_timing` summed `filter_map(|stage| stage.timing)`, and
`stage.timing` is only written at finalization. Wall time ticked live off
`start_time`; active time did not tick at all.
Stage projections now accumulate brackets from the event log:
- Closing an inference bracket folds its span into `live_inference_ms`
instead of discarding it, including across retries, matching the
in-process stopwatch.
- Tool calls open a batch on the first outstanding call and close it when
the last one drains, so tools running concurrently within a turn count
once — the same span `execute_tool_calls` is bracketed by. Summing
per-call durations would over-count parallel tool use. Subagent tool
events are excluded; they run inside the root call's span already.
- `StageProjection::live_timing(now)` composes accumulators with any open
bracket, per handler: agent stages use the brackets, prompt and command
stages count elapsed time as inference and tool respectively, and
handlers that wait on a human, timer, condition, or child branches
report zero.
Active is clamped to wall per stage. A worker killed mid-turn leaves its
bracket open forever, and without the clamp it would tick up unbounded.
The clamp does not need to detect the dead worker: a stage cannot have been
active longer than it has existed. `watchdog.timeout` remains the authority
on whether a run is stuck. The clamp is deliberately not applied at run
level, where concurrent branches can legitimately sum past run wall time.
Timing is derived from events rather than emitted by the worker, so this
needs no event-schema change and applies to runs already stored.
`StageProjection.timing` keeps its terminal-only meaning, and the
authoritative breakdown still replaces the live estimate at terminal
events.
The billing endpoint had the same hole behind its `wall_only` fallback:
running stages reported zero inference/tool/active. Not visible in the
product, which renders only `wall_time_ms`, but wrong for any other
consumer of `GET /runs/{id}/billing`.
Parallel branch stages lose their breakdown permanently, even after
completion, because `parallel.branch.completed` carries only `duration_ms`.
That is a separate data-loss bug, tracked in #644.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
During a long LLM turn the durable event stream was silent: between
`agent.tool.completed` and the next `agent.message` nothing was emitted,
so "the model is generating" and "the worker is wedged" were
indistinguishable from the run store, SSE, or the UI.
The signal already existed. `AssistantTextStart` fired at exactly the
right point — after `build_request()`, after compaction, immediately
before the stream opens — then was classified as streaming noise and
thrown away. This promotes it rather than inventing a new one.
Two events, each asserting only what is provable when it is emitted:
- `agent.llm.started` carries the *requested* provider/model. No usage,
no cost, no context window: none of it exists yet, and failover can
re-target, so `agent.message` stays authoritative for what answered.
- `agent.llm.first_output` is edge-triggered on the first output of an
attempt and names what arrived. `ToolCall` is required, not optional:
a turn that opens with a tool call produces no text or reasoning
delta, so a latch keyed on those two would stay silent for exactly
the tool-heavy rounds where liveness matters most.
`agent.llm.retry` now also fires on the one previously invisible
mid-turn path — a stream that ends without a finish event, which
replays the turn and discards its output with nothing to show for it.
Its `attempt` field was already fed by two independent counters, so an
optional `phase` (open | consume) names which loop it counts.
`StageProjection.inference` projects the open bracket. `Some` means
"the event log contains an unclosed inference bracket", not "the model
is computing now" — a SIGKILLed worker leaves it open, which is the
truthful statement of what we know, and `watchdog.timeout` remains the
authority on actually-stuck.
The close is the subtle part. Terminal cancel and wall-clock timeout
tear the session down through `discard_session` without emitting a
message, error, or interrupt, so a session-lifecycle backstop is
required. It has to be `agent.session.ended`, not
`agent.session.deactivated`: deactivation is emitted by `lease.release()`
*before* the forwarder drains queued agent events, so a queued
`agent.llm.started` can arrive after it and re-open the bracket. But
`agent.session.ended` carries no stage identity, so the close takes
ordering from the event and identity from the projection, scanning for
brackets the ending session opened. A normal stage lookup there finds
no target and silently no-ops.
Presentation states what the log proves and nothing more: no progress
bar or ETA (no completion estimate exists), "reasoning" only when the
provider sent reasoning output, elapsed counted since the request
opened, and no live animation once the run is terminal.
Scope is session-backed agent stages. One-shot completions call
`client.complete` directly and never build a session; covering them
means moving the emit point into `fabro-llm`, filed as a follow-up.
`agent.output.start` was never persisted — it existed in a name map,
an `unreachable!` arm, and docs — so the rename carries no migration
risk. Corrects `events.md`, which documented it as a real emitted
event, and the v2 proposal, which mapped it to `message.part.started`
despite it firing before the request opens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Normalize the readable reasoning providers already return into a
canonical `ReasoningOutput` and carry it through the `agent.message`
run event to storage, SSE, and JSONL.
The shape is derived from the final response's canonical message
content rather than stored a second time, so there is no duplicate
source of truth and retried or replaced streaming buffers never
become durable reasoning. OpenAI-compatible `reasoning_details` are
now preserved verbatim as an opaque content part; only known readable
members are normalized out of them, leaving encrypted entries for a
later provider-aware replay phase.
This phase is passive: no request parameters change, no capability
guessing, and no newly observed provider field is replayed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolves conflicts with the shared-checkout parallel rewrite (#607) and the
cached-run/billing dedup (de60eb900):
- handler/parallel.rs: rebuilt on main's shared-checkout version. Branch
ordinals are still reserved inside the branch task right before
ParallelBranchStarted (with graph_visit/resumed_from_stage_id), and the
reserved StageScope is shared with post-await error paths via a OnceLock
slot instead of main's dispatch-time visit=1 scope, so completion events
are never emitted under a guessed ordinal.
- billing.rs: keep this branch's run_stage_from_projection (RunStage grew
graph_visit/resumed_from_stage_id and a typed id), adopt main's
state.cached_run() and drop the removed run_stage_from_stage_id import.
- run_projection.rs: adopt main's typed parallel_results
(Option<Vec<ParallelBranchResult>>).
- run_event/misc.rs: union of both sides' imports.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolved conflicts against main's shared-projection-cache rework:
- projection_cache.rs: kept main's projection_snapshot and dropped this
branch's last_seq accessor, which it subsumes; latest_event_seq now
reads the sequence from projection_snapshot.
- run_store.rs: kept main's EventScan cursor and added a seek_before
constructor so the backward-pagination range scan bounds its end key
through the same abstraction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolved conflict in run_store.rs tests: kept both the new
list_events_before_with_limit tests from this branch and the
append_event_rejects_sequences_beyond_key_order_limit test from main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A node cancelled (or lost to a crash) mid-flight and then resumed now
starts a new stage execution with the next StageId ordinal (work@2)
instead of reusing and clearing the cancelled execution's projection.
The old execution stays immutable with its own events, session, output,
timing, billing, and termination state.
Engine:
- Add a run-scoped StageExecutionTracker on RunServices with per-node
high-water marks. Ordinals are reserved after the StageStart hook
passes on the first attempt (retries reuse the reservation), ensured
at the composite checkpoint pre-step for hook-skips, and reserved in
on_terminal_reached for terminal nodes' synthetic events.
- Keep three concepts distinct: graph visit (max_visits/checkpoints,
unchanged), stage execution ordinal (the @N in StageId), and handler
attempt. The tracker is not checkpointed; the append-only stage event
history is its durable source of truth.
- resume() seeds the allocator from the run projection and computes a
node -> StageId provenance map of executions observed after the
selected checkpoint, threaded through execute_persisted_run,
RunSession, and InitOptions.
Events and projections:
- stage.started, parallel.branch.started, and checkpoint.completed
carry optional graph_visit and resumed_from_stage_id; StageProjection
stores both. Old events deserialize with None and legacy duplicate
stage.started replays keep last-attempt behavior.
- The CheckpointCompleted reducer is envelope-first: diffs and
skipped-stage synthesis attach to the exact execution StageId, an
existing Retrying projection finalizes as Skipped without losing
identity, and historical node_outcomes no longer create or collide
with newer ordinals (node_visits remains a legacy fallback).
Handlers:
- Parallel fan-out reserves child ordinals through the shared tracker,
derives worktree pass{N} from the parent's execution ordinal, and
seeds branch contexts with explicit child stage scopes so branch
lifecycle and nested handler events agree.
- Artifact capture and manager-loop child logs follow the ordinal.
API and UI:
- RunStage documents visit as the execution ordinal and adds optional
graph_visit and resumed_from_stage_id; Rust and TypeScript clients
regenerated.
- The web sidebar lists both executions chronologically; resumed stages
show a "Resumed from" link in the stage detail header and hover
popover, with the graph visit surfaced when it diverges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cumulative implement + simplify_fable diff recovered from the run's meta
branch (fabro/meta/01KY7YH7RYCJ1BDVTTP96ZA4HV, stage 006 diff.patch).
The run validated this tree clean: cargo nextest (7,007 passed), clippy,
fmt, TS client regen + typecheck, web tests (679 passed), docs check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The crate reorganization renamed lib/crates/ to lib/apps|components|foundation/.
Git followed all modified files across the rename; the only conflict was the
newly added codec/cache.rs, now placed at lib/components/fabro-llm/src/codec/cache.rs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Anthropic prompt caching is opt-in per request: without explicit
ephemeral cache_control breakpoints in the body, no cache writes or
reads ever happen. The OpenAI-compatible codec never emitted them, so
every run on openrouter Claude models billed the full conversation at
the uncached input rate on every turn (0 cache tokens on the billing
page, confirmed by OpenRouter's activity portal).
- Add a `cache_control_breakpoints` model feature declaring that a
route only caches when the request marks the cacheable prefix; set it
on the builtin OpenRouter Claude rows. Catalog build rejects the flag
without `prompt_cache`.
- Teach the Chat Completions wire shape a parts-form content variant so
a message can carry the annotation; unmarked messages keep the
plain-string form for compatibility with strict servers.
- Mark the last system message (covers tools + system upstream) and the
second-to-last user turn, counting tool results as user turns —
mirroring the anthropic codec's placement so agent loops get
incremental cache hits.
- Extract the shared placement/opt-out policy into codec::cache and
refactor the anthropic codec onto it; anthropic wire snapshots are
unchanged.
- Honor `provider_options.<name>.auto_cache = false` as an opt-out and
consume the control key instead of merging it into the body.
- Mirror the new feature through settings (fabro-config), the OpenAPI
schema, and the generated TypeScript client.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
## What
Per-step environment in `run.prepare.steps[].env` was parsed and then
**dropped** before it reached the resolved run settings, so prepare
steps could never see their declared env. This PR carries that env all
the way through to the executor, resolves prepare-step interpolation at
the run boundary, and fixes an argv-quoting bug.
Three things:
1. **Per-step env is carried through.** `RunPrepareSettings` now holds
`steps: Vec<PreparedStep>` (command plus per-step `env`) instead of a
flat `commands: Vec<String>`. The per-step env reaches `exec_command`,
which already accepts per-command env vars, and is merged on top of the
base sandbox environment.
2. **Interpolation resolves at the run boundary.** Prepare-step
`script`/`command` and per-step `env` values are carried in source form
out of the portable config resolve layer (so `fabro validate` stays
portable and never requires env to be set). Their `{{ env.* }}` tokens
resolve in the process that actually runs the steps, via
`RunPrepareSettings::resolve_step_env` — mirroring the existing MCP
transport env resolution. A missing env var is a **hard error**
(fail-closed); there is no fallback to the unresolved literal.
3. **Argv is shell-quoted.** Argv-style prepare steps were assembled
with `join(" ")`, so an argument containing spaces or quotes was
re-split by the shell. They are now shell-quoted per element with the
shared `shell_quote()` helper. `script` steps stay verbatim because they
are raw shell snippets.
## How
- `RunPrepareSettings.commands: Vec<String>` becomes
`RunPrepareSettings.steps: Vec<PreparedStep>` where `PreparedStep {
command, env }`. The server-side `{{ vars.* }}` substitution pass now
walks each step's command and env.
- New `RunPrepareSettings::resolve_step_env(env_lookup)` resolves `{{
env.* }}` in each step's command and env values, returning a hard error
on a missing var (and a loud `Unavailable` error for reserved
`secrets`/`inputs` tokens).
- The run boundary (`fabro_workflow::operations::start`) gains
`runtime_setup_commands`, the prepare-step counterpart to
`runtime_mcp_server`. `LifecycleOptions` now carries `Vec<SetupCommand>`
(command + env), and the initialize phase passes each step's env to
`exec_command`.
- `resolve_prepare` shell-quotes each argv element and carries per-step
env in source form. The stale lint suppression on the resolved fields is
rewritten to describe the deliberate source preservation that now
resolves at the run boundary.
- The shell-quoting helper moves to a shared `fabro_util::shell` module
(backed by `shlex`); `fabro_sandbox::shell_quote` delegates to it so the
config resolve layer and sandbox code share one audited implementation.
- The OpenAPI `RunPrepareSettings` schema and the generated TypeScript
client are updated to the new `steps`/`PreparedStep` shape.
## Testing
- `cargo build --workspace`
- `cargo +nightly-2026-04-14 fmt --check --all`
- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D
warnings`
- `cargo nextest run` for `fabro-util`, `fabro-types`, `fabro-config`,
`fabro-sandbox`, `fabro-api`, `fabro-workflow`, `fabro-server`,
`fabro-cli` (provider keys stripped) — all green.
- `cd lib/packages/fabro-api-client && bun run typecheck` — clean.
New tests cover: per-step env carried through resolution; script/command
+ env resolved at the run boundary; a missing env var is a hard error
(in both the command and a per-step env value); reserved `secrets`
tokens surface as `Unavailable`; argv elements are shell-quoted (an arg
with spaces/quotes is correctly quoted) while a `script` stays verbatim;
and an end-to-end check that per-step env reaches the executed setup
command (with a negative control proving the success is attributable to
the per-step env).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Makes hook interpolation typed end-to-end and fail-closed, and removes
the bespoke template engine on HTTP-hook headers.
- **Typed end-to-end.** Hook `command`, `url`, header values, `prompt`,
and `model` are now carried as a typed `InterpString` from the config
resolve layer all the way to the executor. The executor resolves each
segment at hook fire time from the typed value instead of collapsing it
to a `String` and re-parsing it. This mirrors the MCP transport env
resolution boundary (`resolve_transport_env` / `runtime_mcp_server`).
- **Narrow header tokens.** HTTP-hook headers previously ran through
MiniJinja with an env allowlist
(`TemplateContext::with_env_lookup_allowed`). They now resolve through
the same narrow `{{ ns.NAME }}` token resolver as every other hook field
— no template engine, no allowlist.
- **Fail-closed everywhere.** A missing or out-of-scope `{{ env.* }}` /
`{{ secrets.* }}` token in a command, URL, header, prompt, or model is
now a hard error that blocks the hook rather than firing it with a
half-resolved or empty value. Previously command hooks failed closed but
http/prompt/agent hooks failed open (warned and proceeded), which could
dispatch an HTTP request with an empty credential header or run an LLM
call against a half-rendered prompt. Transport-level outcomes (non-2xx
responses, connection errors, unparseable bodies) stay fail-open.
A follow-up cleanup commit removes the template engine's `env` namespace
(`with_env_lookup` / `with_env_lookup_allowed` / the `EnvLookup`
object), which the header path was the last consumer of.
## How
- `fabro-types` and `fabro-hooks` `HookType` / `HookDefinition` now type
the interpolatable fields as `InterpString`. `InterpString` serializes
as its raw source, so persisted run specs and checkpoints round-trip
unchanged.
- The `fabro-config` resolve layer clones the typed `InterpString`
through instead of calling `as_source()`, so the fields no longer leak
unresolved template text — the old "source preservation" `#[expect]`
annotations on the hook resolvers are gone.
- The executor's single `resolve_interp` helper resolves a typed
`InterpString` and is shared by the command, http, prompt, and agent
paths; resolution failure maps to `HookDecision::Block`, which the
runner already reports loudly (error for blocking hooks, warn for
non-blocking).
## Testing
- New unit tests: fire-time resolution from the typed value (no
re-parse), narrow-token header resolution, and fail-closed behavior for
HTTP url, HTTP header, and prompt hooks on a missing variable (the hook
does not fire and the resolution error surfaces).
- Existing hook tests updated and kept green.
- Gates: `cargo build --workspace`, `cargo +nightly-2026-04-14 fmt
--check --all`, `cargo +nightly-2026-04-14 clippy --workspace
--all-targets -- -D warnings`, and `cargo nextest run` for the touched
crates (`fabro-hooks`, `fabro-types`, `fabro-config`, `fabro-template`,
`fabro-workflow`, `fabro-server`, and the `fabro-cli` hook/config
tests), all green.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Adds the **mcp-servers HTTP API**: `GET/POST /api/v1/mcp-servers` and
`GET/PUT/DELETE /api/v1/mcp-servers/{id}` on top of the merged
`fabro-mcp-store` foundation and OpenAPI spec.
This includes the AppState wiring needed for the catalog to work end to
end: `McpServerStore` construction from `{active-config-dir}/mcps/`, an
`AppState` accessor, the `fabro-server` dependency, and route
registration for list/create/get/replace/delete handlers.
The API mirrors the automations concurrency pattern with ETags on
read/write responses and required `If-Match` headers for replace/delete.
## Resolved before merge
- **Credential-omitting read model:** read responses now return
`McpServerView` / `McpTransportView`, so stored env/header values are
not exposed by GET/list/create/replace responses. Responses include only
`env_keys` / `header_keys`; persisted values remain available to runtime
execution.
- **Manifest catalog references:** run manifest validation, graph
rendering, preflight, and run creation now resolve server-managed MCP
catalog references such as `[run.agent.mcps.<name>] id = "..."`.
- **Schema strictness:** unknown MCP transport fields are rejected,
aligning the reused Rust domain type with the OpenAPI
`additionalProperties: false` contract.
- **Create response headers:** the `POST /mcp-servers` 201 response now
documents its `ETag` header in OpenAPI.
## Follow-up intentionally left out
Credential-literal validation remains structural only: create/replace
currently accept literal env/header values and persist them for runtime
use. The warn-vs-hard-reject UX is a separate follow-up for the settings
UI; it is not a response-omission issue.
## Testing
Current PR checks are green:
- Rust: format, clippy, generated docs, Linux tests
- TypeScript: build, test, typecheck
Local checks run during the simplify/CI-fix pass:
- `cargo +nightly-2026-04-14 fmt --check --all`
- `cargo +nightly-2026-04-14 clippy --locked --workspace --all-targets
-- -D warnings`
- `cargo nextest run -p fabro-config run_agent_mcps`
- `cargo nextest run -p fabro-mcp-store`
- `cargo nextest run -p fabro-api --test mcp_server_round_trip`
- `cargo build -p fabro-api`
- `cargo nextest run -p fabro-server --features test-support
system_sandbox_provider`
- `cargo nextest run -p fabro-server --features test-support --test it
mcp_servers`
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
