The model indicator on a stage page hovered to provider, model, and
reasoning effort only. Seeing what a stage actually spent meant leaving
for the Billing tab, which reports per node rather than per visit.
The stage list had no token data to show, so add a per-visit `billing`
block to `GET /runs/{id}/stages`. The Billing tab's pricing rule (a
provider-reported cost wins, otherwise the server catalog prices the
tokens) was private to `billing_rollup`; move it to
`StageProjection::billed_usage` and drive both call sites from it so the
two views cannot drift.
The popover's buckets use the Billing tab's labels verbatim. It stays
scoped to one visit, so a looped node's row on the Billing tab is the sum
of what each of its visits shows here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DOT parser created a node for every edge endpoint, and nothing
recorded whether a node came from a declaration or was synthesized from
an edge. The edge_target_exists rule only checked whether the node id
was present in the graph, which was always true by then, so a misspelled
endpoint became an attribute-free node that defaulted to shape=box — an
LLM stage. Validation emitted a prompt_on_llm_nodes warning and exited 0.
Node now carries `implicit`, set only when the parser synthesizes the
node from an edge endpoint. A declaration anywhere in the workflow
clears it, so order does not matter and subgraph declarations count.
Node::new leaves it false, so programmatic construction and graphs
deserialized from older checkpoints read as declared.
edge_target_exists treats an endpoint as valid only when it exists and
is declared, reporting each undeclared node once. The near-identical
missing-source and missing-target branches collapse into one path. The
import transform copies the flag onto spliced nodes so an edge-only node
inside an imported fragment is caught too.
parse_and_validate_human_gate had two edge-only nodes and now declares
them; it was an instance of the bug rather than a casualty of the fix.
No shipped workflow, docs example, or CLI fixture relied on the old
behavior.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`active_time_ms` was only ever computed from terminal stage events, so a
stage still running contributed zero to the run rollup. A run parked in one
long agent stage reported 2m 8s of active time against 16m 53s of wall
clock — the two finished stages — while the running stage had been doing
continuous inference and tool work for over 14 minutes.
`live_run_timing` summed `filter_map(|stage| stage.timing)`, and
`stage.timing` is only written at finalization. Wall time ticked live off
`start_time`; active time did not tick at all.
Stage projections now accumulate brackets from the event log:
- Closing an inference bracket folds its span into `live_inference_ms`
instead of discarding it, including across retries, matching the
in-process stopwatch.
- Tool calls open a batch on the first outstanding call and close it when
the last one drains, so tools running concurrently within a turn count
once — the same span `execute_tool_calls` is bracketed by. Summing
per-call durations would over-count parallel tool use. Subagent tool
events are excluded; they run inside the root call's span already.
- `StageProjection::live_timing(now)` composes accumulators with any open
bracket, per handler: agent stages use the brackets, prompt and command
stages count elapsed time as inference and tool respectively, and
handlers that wait on a human, timer, condition, or child branches
report zero.
Active is clamped to wall per stage. A worker killed mid-turn leaves its
bracket open forever, and without the clamp it would tick up unbounded.
The clamp does not need to detect the dead worker: a stage cannot have been
active longer than it has existed. `watchdog.timeout` remains the authority
on whether a run is stuck. The clamp is deliberately not applied at run
level, where concurrent branches can legitimately sum past run wall time.
Timing is derived from events rather than emitted by the worker, so this
needs no event-schema change and applies to runs already stored.
`StageProjection.timing` keeps its terminal-only meaning, and the
authoritative breakdown still replaces the live estimate at terminal
events.
The billing endpoint had the same hole behind its `wall_only` fallback:
running stages reported zero inference/tool/active. Not visible in the
product, which renders only `wall_time_ms`, but wrong for any other
consumer of `GET /runs/{id}/billing`.
Parallel branch stages lose their breakdown permanently, even after
completion, because `parallel.branch.completed` carries only `duration_ms`.
That is a separate data-loss bug, tracked in #644.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codex drives the GPT-5.6 models with a much narrower tool set than the
other OpenAI models: a shell, `apply_patch`, and `update_plan`. It has no
file-read, file-write, grep, glob, or fetch tool at all -- reading and
searching go through the shell, and every write goes through
`apply_patch`. Offering 5.6 fabro's extra tools advertises affordances its
instructions never mention, so this adds a profile that registers only
what Codex does.
The profile is selected per model via `agent_profile = "gpt56"` on the six
5.6 rows (three each on `openai` and `openrouter`), following the existing
Kimi-over-a-gateway pattern. Every other model on those providers keeps
its provider default, with no code branch and no version sniffing.
- `ToolVocabulary::Codex` renames `shell` to `shell_command`; a strum
alias keeps `from_any_name` resolving it to `NativeTool::Shell`, so
permissions, categories, and telemetry still key on the canonical name.
- `shell_command` gains `workdir`, passed to the `cwd` argument
`execute_shell_command` already accepted, with Codex's "always set
`workdir`, do not `cd`" guidance.
- `prompts/gpt56.md.j2` is adapted from Codex's 5.6 `base_instructions`,
which are byte-identical across Sol, Terra, and Luna. A header comment
records provenance and the departures fabro's harness forces.
This is an alignment-only pass: it matches Codex's tool contract while
keeping direct tool calls. Codex actually drives 5.6 in code mode, with a
single `exec` tool taking JavaScript and every other tool reached through
a `tools` object inside a V8 isolate. That is deliberately out of scope.
Luna's `multi_agent_version: v1` (vs v2 on Sol and Terra) is also out of
scope. It only changes the sub-agent tool set, which fabro registers from
the caller rather than the profile, and fabro's current set matches
neither version exactly.
Two server cancel-timing tests are adjusted. `gpt-5.6-sol` is the
`openai` provider's default model, so runs that name no model now build a
3-tool profile instead of an 8-tool one and reach their first stage
sooner. `full_http_lifecycle_cancel` asserted `status.kind == "blocked"`
at the instant of cancel, which the worker is free to change the moment it
is signaled; it now accepts either live state, matching the tolerance its
own comment already documents for `pending_control`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Model default reasoning explicitly at the provider-route level so always-reasoning endpoints without effort controls receive summary headroom. Cap all summary requests at model output limits and bound retained visible summaries to the original allowance. Reuse builtin catalog fixtures and named budget constants in tests, and document the new model setting.
During a long LLM turn the durable event stream was silent: between
`agent.tool.completed` and the next `agent.message` nothing was emitted,
so "the model is generating" and "the worker is wedged" were
indistinguishable from the run store, SSE, or the UI.
The signal already existed. `AssistantTextStart` fired at exactly the
right point — after `build_request()`, after compaction, immediately
before the stream opens — then was classified as streaming noise and
thrown away. This promotes it rather than inventing a new one.
Two events, each asserting only what is provable when it is emitted:
- `agent.llm.started` carries the *requested* provider/model. No usage,
no cost, no context window: none of it exists yet, and failover can
re-target, so `agent.message` stays authoritative for what answered.
- `agent.llm.first_output` is edge-triggered on the first output of an
attempt and names what arrived. `ToolCall` is required, not optional:
a turn that opens with a tool call produces no text or reasoning
delta, so a latch keyed on those two would stay silent for exactly
the tool-heavy rounds where liveness matters most.
`agent.llm.retry` now also fires on the one previously invisible
mid-turn path — a stream that ends without a finish event, which
replays the turn and discards its output with nothing to show for it.
Its `attempt` field was already fed by two independent counters, so an
optional `phase` (open | consume) names which loop it counts.
`StageProjection.inference` projects the open bracket. `Some` means
"the event log contains an unclosed inference bracket", not "the model
is computing now" — a SIGKILLed worker leaves it open, which is the
truthful statement of what we know, and `watchdog.timeout` remains the
authority on actually-stuck.
The close is the subtle part. Terminal cancel and wall-clock timeout
tear the session down through `discard_session` without emitting a
message, error, or interrupt, so a session-lifecycle backstop is
required. It has to be `agent.session.ended`, not
`agent.session.deactivated`: deactivation is emitted by `lease.release()`
*before* the forwarder drains queued agent events, so a queued
`agent.llm.started` can arrive after it and re-open the bracket. But
`agent.session.ended` carries no stage identity, so the close takes
ordering from the event and identity from the projection, scanning for
brackets the ending session opened. A normal stage lookup there finds
no target and silently no-ops.
Presentation states what the log proves and nothing more: no progress
bar or ETA (no completion estimate exists), "reasoning" only when the
provider sent reasoning output, elapsed counted since the request
opened, and no live animation once the run is terminal.
Scope is session-backed agent stages. One-shot completions call
`client.complete` directly and never build a session; covering them
means moving the emit point into `fabro-llm`, filed as a follow-up.
`agent.output.start` was never persisted — it existed in a name map,
an `unreachable!` arm, and docs — so the rename carries no migration
risk. Corrects `events.md`, which documented it as a real emitted
event, and the v2 proposal, which mapped it to `message.part.started`
despite it firing before the request opens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fabro advertised Bash while its three backends implemented three
different contracts: Daytona evaluated commands through `sh`, and
Docker's streaming, stdio, and setup paths used a login shell. Bash-only
syntax silently misbehaved depending on provider and code path, and
login profiles could change PATH and command behavior per image.
Make `bash -c` the enforced interpreter for every command string the
Unix sandbox API accepts, on every production backend and through both
buffered and streaming execution. This selects the interpreter only —
no `errexit`, no `pipefail`, no login mode — so `false | true` still
succeeds and a workflow that wants other semantics writes them into its
own command.
Local resolves `bash` through the worker's PATH (NixOS has no
/bin/bash) and reuses that one executable across all three command
paths. Docker and Daytona require /bin/bash with no `sh` fallback.
Fresh initialization and resume/start now verify Bash through a shared
marker-validating probe before reporting the sandbox usable, so a
missing or non-Bash interpreter fails at the lifecycle boundary with
provider-specific remediation instead of on the first command. The
probe also rejects Bash in POSIX mode, which an image whose `bash` is
really `sh` would otherwise pass.
Sandbox MCP scripts and the detached launch wrapper move under the same
contract; host-side stdio MCP scripts, hooks, and interactive terminals
are separate executors and keep their existing `sh` behavior.
The `shell` tool's name and JSON schema are unchanged across providers;
only its prose now identifies `command` as Bash source.
BREAKING CHANGE: sandbox commands no longer load login-shell profiles,
so environment set in /etc/profile.d/*.sh, ~/.bash_profile, or
nvm/rbenv/sdkman initializers is gone. Move those exports into the
Dockerfile's ENV or the Daytona snapshot image.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shell executor rendered every returned ExecResult and returned
Ok(output), so nonzero exits, timeouts, and cancellations reached
execute_one_tool() as successes. That false ToolResult propagated
consistently: agent.tool.completed recorded is_error: false, the success
post-tool hook ran, Anthropic saw is_error: false, OpenAI Responses saw a
completed function-call output, and CLI/web rendered a successful tool
call.
ExecResult::is_success() is now the authoritative predicate. The executor
runs through exec_command_streaming() with a sink callback, so it keeps
the production providers' stream provenance and partial-output capture,
and drops the exec 2>&1 prefix that merged stderr into stdout before
Fabro could report it. Model-facing text labels termination, exit code,
duration, and either separate stdout/stderr sections or one combined
section when the provider cannot separate streams.
Session-bound dispatch also emits a typed agent.tool.process.completed
event carrying the process metadata, streams_separated, and bounded
redacted output tails. It is subordinate diagnostic data: the following
agent.tool.completed remains the one tool-protocol completion and the
authoritative owner of is_error, so consumers need no new row.
Nonzero, timed-out, and cancelled commands intentionally change from
successful to failed tool results, and PostToolUseFailure replaces
PostToolUse for them. On Docker the agent shell tool now uses the
streaming path's bash -lc supervisor, which terminates the process group
on timeout instead of leaving container-side processes running.
The public shell schema is unchanged and pinned by an exact assertion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Kimi profile was registering the Anthropic task tools. Both persist
through the same TodoRuntime, but they model opposite interactions: TaskCreate
and TaskUpdate mutate individual tasks against tracked ids, while Kimi Code's
TodoList replaces the whole list in one call. Of the two surfaces fabro already
had, Kimi was given the one furthest from what its models are trained on.
Add TodoListKind::KimiTodos and a TodoList tool matching Kimi Code's contract
exactly:
TodoList({ todos?: [{ title, status: pending | in_progress | done }] })
Omitting `todos` reads the list, an empty array clears it, and a list replaces
it. Reconciliation mirrors update_plan -- items are identified by their text,
so re-submitting a list preserves identity for unchanged entries -- and the
runtime, projections, and events are unchanged.
Two differences from the existing surfaces were behavioral rather than
cosmetic. Items carry only `title`, where TaskCreate requires both `subject`
and `description`, so a model with nothing to say for a description had to
invent one. And the terminal status is spelled `done`; `completed` is the
Anthropic and Codex spelling, and a model emitting `done` against the old
schema got a validation error rather than a todo. The internal representation
stays TodoStatus::Completed; only the wire vocabulary differs.
Kimi todo lists are session-scoped like OpenAI plans, so the root-agent
projection excludes subagent lists the same way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Kimi models ran on the OpenAI profile, which exists to look like Codex. Give
them their own profile derived from Kimi Code's system prompt.
Routing is per model, not per provider, because Kimi models are served both
directly by Moonshot and through gateways. `kimi` sets agent_profile at the
provider level; the Kimi model rows on `openrouter` set it individually, so a
gateway route behaves like the direct one while other OpenRouter models keep
the provider's OpenAI profile.
The profile targets a measured failure. Across two observed K3 implementation
stages, 32 of 35 tool failures were the same thing: writes to files the model
had not read, rejected by the workspace read-before-write guard, or
`old_string` values reconstructed from memory rather than taken from a read.
Kimi Code drills this rule in its own tool descriptions, so the profile does
too -- `edit_file` and `write_file` carry Kimi-specific descriptions naming the
guard and the failure text the model will see, alongside a "Reading Before
Writing" section in the system prompt. Profiles own their tool registries, so
this re-describes the tools for Kimi only; every other profile is untouched and
the executors and JSON schemas are shared unchanged.
Tool names stay fabro's existing snake_case. Whether Kimi Code's PascalCase
vocabulary measurably helps is untested, and renaming would also mean updating
the name-keyed categories in tool_permissions.rs, where an unknown tool falls
back to Shell. That is a separate change to make on evidence.
The prompt is a subtractive port: capabilities fabro does not have -- plan
mode, background tasks, cron, subagent swarms, the cwd tree listing -- are
dropped rather than promised. The shell timeout default matches Kimi Code's 60s
and memory discovery reads AGENTS.md, which is the only instruction file Kimi
Code looks for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Normalize the readable reasoning providers already return into a
canonical `ReasoningOutput` and carry it through the `agent.message`
run event to storage, SSE, and JSONL.
The shape is derived from the final response's canonical message
content rather than stored a second time, so there is no duplicate
source of truth and retried or replaced streaming buffers never
become durable reasoning. OpenAI-compatible `reasoning_details` are
now preserved verbatim as an opaque content part; only known readable
members are normalized out of them, leaving encrypted entries for a
later provider-aware replay phase.
This phase is passive: no request parameters change, no capability
guessing, and no newly observed provider field is replayed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolves conflicts with the shared-checkout parallel rewrite (#607) and the
cached-run/billing dedup (de60eb900):
- handler/parallel.rs: rebuilt on main's shared-checkout version. Branch
ordinals are still reserved inside the branch task right before
ParallelBranchStarted (with graph_visit/resumed_from_stage_id), and the
reserved StageScope is shared with post-await error paths via a OnceLock
slot instead of main's dispatch-time visit=1 scope, so completion events
are never emitted under a guessed ordinal.
- billing.rs: keep this branch's run_stage_from_projection (RunStage grew
graph_visit/resumed_from_stage_id and a typed id), adopt main's
state.cached_run() and drop the removed run_stage_from_stage_id import.
- run_projection.rs: adopt main's typed parallel_results
(Option<Vec<ParallelBranchResult>>).
- run_event/misc.rs: union of both sides' imports.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolved conflicts against main's shared-projection-cache rework:
- projection_cache.rs: kept main's projection_snapshot and dropped this
branch's last_seq accessor, which it subsumes; latest_event_seq now
reads the sequence from projection_snapshot.
- run_store.rs: kept main's EventScan cursor and added a seek_before
constructor so the backward-pagination range scan bounds its end key
through the same abstraction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- The catalog comment showed `fabro provider login fireworks`, but
`--provider` is a required flag: `fabro provider login --provider fireworks`.
- The remote-server `fabro model test` example omitted `--provider fireworks`,
which could resolve the slug against a different provider.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolved conflict in run_store.rs tests: kept both the new
list_events_before_with_limit tests from this branch and the
append_event_rejects_sequences_beyond_key_order_limit test from main.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot review flagged that dropping #[serde(default)] makes newer
clients hard-fail against servers that predate the controls field.
The late-added Model fields (default, small_default, configured) set
the precedent: required in the OpenAPI spec, defaulted on
deserialization. An empty controls list already means "unsupported",
so the degraded value is semantically correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the missing with_replacement for ModelControls so progenitor reuses
fabro_model::ModelControls instead of generating a dead parallel DTO,
re-export it from fabro_api::types, and assert type identity in the
round-trip test.
Drop #[serde(default)] from Model.controls and
ModelControls.reasoning_effort: the OpenAPI spec marks both required,
matching the strict deserialization of the sibling features/costs
fields. Update CLI stub payloads to include the now-required field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Rewrite the Fireworks tool round-trip E2E test on the shared
run_model_test deep-test pattern used by the OpenRouter and Poolside
opt-in provider tests, instead of a fourth hand-rolled copy of the
multiply-tool scaffold.
- Drop the "(via Fireworks)" display-name suffix from slugs that have no
first-party provider (kimi-k2.6, deepseek-v4-*, minimax-m2.7),
matching the OpenRouter convention; rename "Qwen 3.7 Plus" to
"Qwen3.7 Plus" to match existing Qwen entries.
- Fix kimi-k2.6 vision flag to false, matching the OpenRouter entry for
the same slug (the portability test asserts they are the same model).
- Assert small_default_for_provider and per-model family/vision/
reasoning in the catalog tests, mirroring sibling provider tests.
- Add Troubleshooting and Further reading sections to the Fireworks
docs page, matching the other opt-in provider pages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
