A worker whose server is killed outright retried its control-stream
connect forever at a 5 s capped backoff, so it never finalized, never
stopped its sandbox, and lived until reboot. The worker now tracks the
start of each run of continuous connection failure and gives up after
60 s (10 s once its parent is pid 1), through the existing fatal
control-loss path that interrupts interviews and cancels the run. After
that fatal fires, the runner keeps driving the cancelled pipeline for a
bounded grace so `conclude` can stop the sandbox before the process
exits, instead of dropping the pipeline future mid-flight.
The test harness's stale-daemon reaper only matched `fabro server`
titles bound under `/tmp/.tmp*`, but `TestContext` roots are
`.ft-<label>-*` under `std::env::temp_dir()`, so daemons bound there
were never reaped. The regex now also matches sockets below a `.ft-`
directory component under any parent, and the unit test covers both
roots plus real-looking paths and `.ft-` outside a directory component.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Move the lithos-llm pin from 55add459 to 43a42ac28e9d9bcf40a91abc02be4f12ca274ebb,
and the three Pebble pins from a39f43e to 67c9f48, Pebble `main`, which pins
that same lithos-llm revision so Cargo holds one lithos-llm crate. lithos-llm
`main` (ca19fac) is one commit further; that commit touches only its nightly
workflow, so this pin stays on the revision Pebble unifies with.
The `openai`, `anthropic`, `gemini`, and `openai-compatible` features are
gone upstream; each expanded to `runtime`, which `bedrock` implies, so the
four names leave the fabro-llm feature list. Every other manifest already
names `runtime`.
The catalog schema now names one adapter and many codecs per provider.
`adapter` defaults to `http`, `codecs = [...]` replaces `codec` and defaults
to `["openai-chat"]`, and the loader rejects the old `codec` key and the four
protocol-named adapter ids. Every inline catalog in tests and docs moves to
the new shape: the `openai-compatible` + `openai-chat` pair is dropped as the
default, `adapter = "openai"` + `codec = "openai-responses"` becomes
`codecs = ["openai-responses"]`, and the one test that swaps in a custom
adapter id now adds the line instead of replacing one. The settings
reference, the API schema's `Provider.adapter` description, and the SDK page
describe the new fields; the three `docs/superpowers/plans/` files that show
the old shape are dated, unchecked historical plans and are left as they are.
The implied agent profile for an operator provider that declares none used
to read the removed protocol adapter ids; it now reads the provider's first
codec (Anthropic Messages and Gemini map to their harnesses, the `bedrock`
adapter to Anthropic, everything else to OpenAI), with a test for the codec
path.
Absorbing the rest of the range: OpenRouter and Fireworks now ship enabled,
so the two fabro-llm tests that used OpenRouter as the disabled fixture use
`bedrock-openai`, and the docs and comments that said the two ship disabled
are corrected. The built-in catalog grew past 100 enabled model rows
(Vercel, TypeSafe, and the enabled OpenRouter and Fireworks rosters), so the
pagination shape test walks `page[offset]` to the last page instead of
assuming one page fits.
`cargo update -p` on the four crates also re-resolved a few already-locked
edges to match the lithos-llm lockfile: `windows-sys` 0.61.2/0.60.2 ->
0.59.0 under dirs-sys, errno, nu-ansi-term, quinn-udp, rustix,
rustls-platform-verifier, tempfile, terminal_size, and winapi-util;
`windows-core` 0.61.2 -> 0.62.2 under iana-time-zone; `errno` 0.2.8 ->
0.3.14 under signal-hook-registry; and `indexmap` 2.13.0 as a new public
dependency of lithos-llm. No package version was added or removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lithos-llm attaches a cost to every response at the client: the codec
keeps a provider-reported cost when the provider supplies one, and the
resolver fills the catalog's price for the route when it does not.
Pebble records that priced usage on every assistant turn and sums it,
so each AssistantMessage on the stream, and the store fold's live stage
usage, already carries the cost. Fabro's catalog re-pricing of the same
tokens was redundant, and is gone.
model_usage_from_llm, with_reported_cost, and every estimate_cost call
in fabro are deleted. The pebble handler's stage_usage groups pebble's
accounts by route and sums them with Usage::saturating_add, keeping the
cost and source pebble carried, so the terminal stage.completed usage is
the live fold's sum; it no longer fails when the catalog does not know a
provider. A one-shot prompt stage records the response's own usage and
cost as lithos-llm returned it. The per-model price cards in fabro-llm's
API module stay.
Tests: the pebble handler sums Catalog and Provider costs per row and
leaves a row's and the total's cost unknown once an answer was unpriced;
the store fold shows the same tokens and cost live and at completion,
and None at both for an unpriced answer; the agent integration test
compares the whole completed Usage with the live fold, cost included;
a one-shot prompt stage on a mocked OpenAI-compatible provider reports a
Catalog cost that lithos-llm's resolver attached, with no fabro pricing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A row with no model usage keeps its dash; a usage with tokens but no
cost, such as a model the catalog cannot price or a total with an
unpriced part, reads "unknown" rather than looking like zero.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Pebble main a39f43e26effdf99635eaf343f095c17157c9c93 (pebble #22) carries
an assistant turn's usage as Usage in the session record and moves the
record format to version 5. CodingRuntime::from_record refuses a record
in another format with UnsupportedRecord { version, supported } before
it reads the route. Fabro persists those records in SQLite for Ask Fabro
resume, and old runs get no migration, so a record written by an older
build is read back as stored and refused on the next turn.
Two tests pin that down. The store reads pebble's own version 4 fixture
back through get without a parse error and reports it unsupported. A
resumed Ask Fabro session whose stored record declares the previous
format fails its next turn with the agent_error code and the message
"session record format version 4 is not supported (this build requires
5)", runs no turn, and leaves the stored record in place.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
run.completed, run.failed, stage.completed, stage.failed, and
agent.message carry lithos-llm's Usage; the API reference navigation
names the run usage endpoint.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The run detail tab, route, query hook, and query key say usage. Every
read of input_tokens, output_tokens, total_tokens, reasoning_tokens,
cache_read_tokens, cache_write_tokens, and total_usd_micros moves to
usage.tokens and usage.cost, with lib/usage.ts replacing lib/billing.ts:
totalTokens sums the five buckets, and costSourceTag names a cost that
the provider reported or that was summed from differently sourced parts.
The Usage tab and the stage popover show that tag next to such a cost.
Test fixtures build a Usage through makeUsage; the failing set of the
web tests is unchanged from main (the same 13 environment failures).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The generated models follow the spec: TokenCounts, Cost, Usage,
ModelUsage, UsageModelRef, UsageStageRef, Speed, RunUsage,
RunUsageStage, RunUsageTotals, UsageByModel, AggregateUsage, and
AggregateUsageTotals replace the billing models, and UsageApi replaces
BillingApi. The stale billing model files are deleted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Re-pin lithos-llm to 55add4596b861a0623d00c3a54aa5c147c8d504b and
pebble to c91810fe51aece80359b9cd8efea971af0c46925, where token usage
and cost travel together as Usage { tokens: TokenCounts, cost:
Option<Cost> }. Fabro now carries that one type everywhere it used to
carry BilledTokenCounts, BilledModelUsage, UsdMicros, or a token count
beside a cost_usd_micros.
fabro-types: billing.rs is usage.rs with ModelRef, ModelUsage { model,
usage }, sum_usage, and usage_is_empty; billing_rollup.rs is
usage_rollup.rs with ProjectionUsageStage, ProjectionUsageByModel,
ProjectionUsageRollup, and usage_rollup_from_projection. Every usage
field is named usage: StageProjection.usage and usage_by_model,
Outcome<Option<ModelUsage>>, stage.completed and stage.failed usage and
usage_by_model, prompt.completed usage, run.completed and run.failed
usage (total_usd_micros is gone), Conclusion.usage, StageSummary.usage,
Run.usage. RunSize buckets by Cost.
fabro-workflow: model_usage_from_llm prices tokens from the catalog with
a Catalog cost source, with_reported_cost keeps a provider cost, and the
pebble handler's stage_usage groups pebble's accounts by model and sums
rows with Usage::saturating_add, so a total has a cost only when every
priced part was priced. The store fold's live usage is the agent's
usage plus its descendants'.
API: the OpenAPI spec deletes BilledTokenCounts, BilledModelUsage,
CompletionUsage, CompletionCost, TokenUsage, and RunBillingSummary,
adds TokenCounts, Cost, Usage, and ModelUsage, and renames every
billing schema, property, tag, path, and operation to usage. fabro-api
reuses lithos-llm's and fabro-types' types through with_replacement,
with a round-trip test per replacement.
Old stored runs get no migration: their pebble events in the old shape
read back with zero usage, and their rebuilt projections lose agent
usage.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When the driver reports an output loss on a streaming exec, the pebble
Environment adapter now appends one line to the stderr it hands back:
[sandbox] N output frame(s), M bytes dropped by the provider
Pebble renders the result's stderr into the tool output, so the model
and the run log both see that the command's output is incomplete rather
than reading a silently shortened stream. The adapter also logs one
`warn!` with `dropped_frames`, `dropped_bytes`, and the command's first
word, bounded, so the operator can find the event without the log
carrying the command itself.
The loss is not folded into pebble's per-stream capture stats: the
dropped frames' stream is unknown and the counts are of encoded bytes,
so attributing them to stdout or stderr would be a guess. The driver's
`truncated` flags on both captures already say the counts undercount.
`ExecOutputTail` is pebble-owned and mirrored in the OpenAPI spec, so
the run events are left alone.
Tests cover the appended line with and without existing stderr, a
lossless command over the scripted double staying unchanged, the
bounded program name, and a lossy command end to end through
`Environment::exec` over a scripted sandbox whose exec facet reports a
loss (the driver's scripted double has no knob for it).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Moves the eight sandbox-driver crates from ddb32e1 to 64c14b8, which
brings sandbox-driver PR #22 (Daytona output resync): the Daytona
plugin's encoded-exec decoder no longer fails a command on a torn frame.
It discards through the next newline, counts the loss in the new
`sandbox_driver::OutputLoss { dropped_frames, dropped_bytes }`, and
carries it as `ExecStreamingResult::output_loss` (serde default) across
the plugin wire. A loss also sets `truncated` on both captures, so
`into_complete()` refuses while `run_streaming` completes. PR #21
(supervisor generations) was already on main under the previous pin.
Nothing in fabro needed a source change for the bump. The lockfile
changes only the eight driver source lines; the unrelated windows-sys,
windows-core, and errno flips `cargo update` proposed are not taken, and
`cargo metadata --locked` accepts the result.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The live Daytona tests create a provider sandbox and delete it on their
last line, so any panic or failed assertion before that line leaks a
running, billed sandbox. Two leaked that way on 2026-09-14 when a
sandbox-driver decoder flake panicked daytona_playwright_mcp_sandbox_transport.
Add fabro_sandbox::test_support::DeletedOnDrop, a guard that owns the
RunSandbox (Deref keeps the tests reading unchanged), offers an explicit
delete(self) for the happy path, and deletes from Drop otherwise. The
drop-time delete runs on its own thread and runtime because the test's
runtime may be unwinding. It reconnects the provider through the
ProviderAccess the test built the sandbox with, because the sandbox's
own handle pools HTTP connections whose tasks live on the test's runtime;
a live check of that path timed out after 10s.
Every Daytona test that creates a sandbox now holds it through the guard.
Unit tests over the scripted double prove delete-on-drop runs once, an
explicit delete runs once, and a panic inside catch_unwind still deletes
with and without a runtime.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Since #852, `fabro exec --verbose` only turned on the request/response
middleware on the LLM client and no longer printed tool calls, tool
results, or the transcript. Pebble #18 gives pebble-cli-core rendering
options, so `--verbose` now runs the prompt through
`run_prompt_with(..., RenderOptions::verbose())`: each tool call's
arguments and result in full under its `[tool]` and `[result]` lines,
plus the transcript. The middleware is enabled as before. Without the
flag the renderer gets the default options, so the output is unchanged.
The twin shell test now scripts the tool call and the final answer as
two turns, so the answer on stdout is the scripted one rather than the
twin's fallback echo, and it asserts that stderr carries no result,
reasoning, or verbose blocks. A new twin test runs the same prompt with
`--verbose` and asserts the tool and result blocks and the request dump.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Pebble #18 adds tool-result and transcript rendering options to
pebble-cli-core (RenderOptions, Renderer::options, and
session::run_prompt_with). Pebble #19 keeps a paired human's hold across
a route failover; no embedder change is needed for it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`ps --json` reports the digraph name as `workflow_graph_name` and reserves
`workflow_name` for an explicit `[workflow] name` (6a86ced77). That change
updated the ps tests but not this ignored e2e test, which still expected
the graph name under `workflow_name`. The test now asserts the contract
the ps tests assert: `workflow_name` is null for a bare graph file and
`workflow_graph_name` is the digraph name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The provider probe runs inside the isolated server, which never sees the
test process environment. The test used to store `OPENAI_BASE_URL` in the
vault, and cd74013d0 dropped that entry without replacing it, so the
server probed the real OpenAI API with the namespace as its key and the
doctor reported the provider as failed. The server settings now repoint
the `openai` provider at the twin through the operator `[llm]` overlay.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The four hook tests and arc_e2e_with_real_llm in workflow/hooks.rs failed
in twin mode for three reasons, all in the test fixtures.
The hooked workflows were written as `<name>.toml` beside `<name>.fabro`.
Version packaging accepts a config only as `workflow.toml` beside its
graph (44dccfa3d), so `fabro run` failed at collection. Each hooked
workflow now lives in its own `<name>/` directory as `workflow.toml`.
The isolated server never learned the twin's base URL. The CLI command
carried `OPENAI_BASE_URL`, but the run executes in the server, which does
not see the test process environment, so it called the real OpenAI API
with the namespace as its key. The twin-mode server settings now repoint
the `openai` provider at the twin through the operator `[llm]` overlay,
the same way `run_uses_vault_credentials_for_worker_execution` does.
With the server reaching the twin, the hook scenarios were consumed by
the wrong request: the server asks the model for a run title in the same
namespace before the hook fires, and the scenarios had no matcher. The
block test then saw the twin's default response and the hook failed open,
so the run succeeded. Hook scenarios now match on the `Hook prompt:`
prefix of the evaluator's user message.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The e2e nextest profile flags a test as slow after 10s and kills it after
three periods, 30s in all. A Daytona live test creates a remote sandbox,
installs tools in it, and waits for the provider; the Playwright MCP test
also fetches a 114 MiB browser and completes an MCP handshake through the
preview URL. A measured live run took 40.6s, so the documented
`--profile e2e --run-ignored only` command killed it before it could
report its own result.
Add an e2e override for every `daytona_` test that raises the slow period
to 60s and the kill to 20 periods, so a slow provider has room while a hung
test still ends.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The pin moves from the head of the section-4-driver-items branch (a92c0db6,
since merged as #20) to main, which adds #21: the protocol crate's
PluginSupervisor gains numbered generations, a health probe before a
generation serves, and a refusal of a replacement that reports another
resource namespace. Petri pins the same revision, so the two runners share
one sandbox-driver.
cargo build --workspace and cargo nextest run -p fabro-sandbox -p
fabro-workflow pass (1502 tests, 48 skipped).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DvCtm3CHs5TFbQqUWtkoMX
The stack this branch grew from was rebase-merged, so main carries the same
content under new commits; every conflict resolves to this branch's side.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
StageProjection.agent_control and the AgentControlState schema go with the
field; the agent's activity on AgentSessionProjection carries the fact.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The session_projection_parity module pinned fabro's stage fold to pebble's
SessionProjection while both existed; the stage view reads the fold now,
and the one usage rule has its own tests. StageProjection.agent_control
and AgentControlState go too: pebble's fold carries the interrupted and
steered facts as agent.activity, and the stage's state says whether the
stage still runs, which is what the reset on fabro's own stage events was
for. The run-detail banner reads activity plus state.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The demo agent stage's stored events now read as one pebble session: MCP
servers up and failed, skills, a subagent, a failover, a compaction, and a
written file, ending with ProcessingEnd. Demo mode serves the run state it
answered not_implemented to, with the agent stage carrying the coding
agent's fold of those events, so the stage sidebar renders them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Pebble's CodingAgentEvent stream is the agent event contract: every event
except streaming deltas is stored verbatim under its derived name and
folded into StageProjection.agent with pebble's SessionProjection. The
events doc, the events strategy, and the v2 shape doc say so, list the
agent events fabro still emits for facts pebble cannot know, and tell
consumers to read the fold rather than fold the events again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
