Reduce duplication and over-specification introduced with the Modal
provider, without changing shipped behavior.
- Extract enabled_provider_catalog and assert_deep_tool_round_trip in
the fabro-llm integration tests. The Poolside, Fireworks, OpenRouter,
and Modal deep round trips were four near-identical copies.
- Add ApiCredential::with_extra_headers for providers that authenticate
with request headers instead of an API key.
- Replace the unreachable require_env guards in the Modal e2e test with
the std::env::var form used by every sibling test, and register
MODAL_TOKEN_ID and MODAL_TOKEN_SECRET in EnvVars.
- Collapse modal_requires_both_vault_proxy_tokens to a single case. The
loop rebuilt the whole built-in catalog per iteration.
- Drop tautological and over-specified assertions: the api_key_url doc
URL, the forced default/probe lookups on a single-model provider, and
the get_on_provider loop that could not fail.
- Inline the single-use modal_env_catalog fixture and note why it
overrides the shipped secrets templates.
- Sort the MODAL_* keys in .env.example, and record in modal.toml why
api_id keeps the Hugging Face capitalization.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Take `Arc<Catalog>` by value again through the validation entry points.
`AppState::catalog()` returns an owned `Arc`, so `&state.catalog()` was
cloning, borrowing the temporary, then cloning again at the leaf. Every
consumer ends up owning the `Arc`, so by-value is the honest shape and it
drops one clone per call. The one caller holding the catalog in a field
now says `Arc::clone(&self.catalog)` explicitly.
- Correct the `RenderMode` doc comment. It claimed `Strict` is "used by
run-create", but run-create renders `Structural` and promotes the
resulting warnings to errors itself; `Strict` has no production caller.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ModelResolutionOptions` was a field-for-field duplicate of the existing
public `ModelResolutionTransform`, down to a verbatim copy of its `new()`.
`pipeline::transform` then unpacked one to rebuild the other, cloning the
catalog Arc and the eligible-provider set on the way.
- Delete `ModelResolutionOptions`. `TransformOptions.model_resolution` now
holds an `Option<ModelResolutionTransform>` directly, so the TRANSFORM
step is `resolution.apply(graph)?` with no rebuild and no clones. This
is consistent with `custom_transforms`, which already holds transforms.
- Add `ModelResolutionTransform::catalog()` so the VALIDATE step can reach
the same catalog for its lint rules. That is the only new code needed.
- Drop `CatalogScope` from `operations::validate`, which was a third copy
of the same fields. The three entry points now hand a partially built
transform to `validate_resolving_models`, which completes it with the
workflow's default provider once the workflow is resolved.
- Extract `validate_child_workflow` in `manager_loop`, collapsing two
near-identical validate-and-unwrap blocks.
- Point the transform tests at their own `transform_options()` helper via
struct-update syntax instead of respelling all seven fields, and drop a
HashSet -> Vec -> HashSet round trip from the create test helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up cleanup on the catalog-free validation split. Same behavior,
fewer parallel code paths.
- Make the catalog an explicit `Option<&Catalog>` on `pipeline::validate`
instead of a `validate` / `validate_with_catalog` pair, so each call
site states whether catalog rules run.
- Collapse `preprocess_and_validate`, `preprocess_and_validate_structural`,
and `preprocess` into one function that takes `TransformOptions`. Its
`model_resolution` field is now the single source of truth for catalog
awareness, which drops a 12-argument signature and the
`too_many_arguments` allow.
- Replace the duplicated resolve-and-preprocess block in
`operations::validate` with one `validate_in_scope` helper, and drop the
HashSet -> Vec -> HashSet round trip on the catalog path.
- Extract `configured_default_provider`, previously duplicated between
`operations::create` and `operations::validate`.
- Delete `validate_manifest_with_environment_defaults`, which had no
callers outside its own module.
- Share the `server-model.fabro` fixture between the two CLI tests instead
of inlining it twice. The validate test now asserts the rendered output
through the usual snapshot helper, which also removes a hand-rolled
`std::fs::write` and its clippy allow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The model indicator on a stage page hovered to provider, model, and
reasoning effort only. Seeing what a stage actually spent meant leaving
for the Billing tab, which reports per node rather than per visit.
The stage list had no token data to show, so add a per-visit `billing`
block to `GET /runs/{id}/stages`. The Billing tab's pricing rule (a
provider-reported cost wins, otherwise the server catalog prices the
tokens) was private to `billing_rollup`; move it to
`StageProjection::billed_usage` and drive both call sites from it so the
two views cannot drift.
The popover's buckets use the Billing tab's labels verbatim. It stays
scoped to one visit, so a looped node's row on the Billing tab is the sum
of what each of its visits shows here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DOT parser created a node for every edge endpoint, and nothing
recorded whether a node came from a declaration or was synthesized from
an edge. The edge_target_exists rule only checked whether the node id
was present in the graph, which was always true by then, so a misspelled
endpoint became an attribute-free node that defaulted to shape=box — an
LLM stage. Validation emitted a prompt_on_llm_nodes warning and exited 0.
Node now carries `implicit`, set only when the parser synthesizes the
node from an edge endpoint. A declaration anywhere in the workflow
clears it, so order does not matter and subgraph declarations count.
Node::new leaves it false, so programmatic construction and graphs
deserialized from older checkpoints read as declared.
edge_target_exists treats an endpoint as valid only when it exists and
is declared, reporting each undeclared node once. The near-identical
missing-source and missing-target branches collapse into one path. The
import transform copies the flag onto spliced nodes so an edge-only node
inside an imported fragment is caught too.
parse_and_validate_human_gate had two edge-only nodes and now declares
them; it was an instance of the bug rather than a casualty of the fix.
No shipped workflow, docs example, or CLI fixture relied on the old
behavior.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check tested for `never reconstruct it from memory`, a phrase the
Kimi edit description no longer contains after it was reworded to match
Kimi Code. A one-sided `!contains` against a literal cannot tell "the
phrase is absent because nothing leaked" from "the phrase is absent
everywhere", so it silently stopped protecting anything.
Assert the marker is present in Kimi's own description and absent from
the stock one. Removing the marker from the description now fails the
test instead of quietly disarming it, verified by doing exactly that.
Also correct the grep docs: all three sandbox implementations probe for
`rg` and fall back to POSIX `grep`, so the page should not imply a
single engine. Pre-existing, adjacent to the lines this branch touched.
Reported by Copilot review on #646.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither harness puts read-before-edit mechanics in the system prompt.
Kimi Code's `system.md` has no such section; the rules live in
`edit.md`, `write.md`, and `read.md`. Codex's prompts say nothing about
reading before an edit at all, and its editing guidance is attached to
`apply_patch`.
Drop the `# Reading Before Writing` section from the Kimi prompt and
carry its content in the Edit, Write, and Read descriptions, worded as
Kimi Code words it. Nothing is lost: every bullet in the removed
section was already covered by a tool description.
Two behaviors change to match upstream. Edit now says not to issue
consecutive edits against the same file, since the first invalidates
the second's `old_string` -- Kimi Code's stated reason. Read now says
not to re-read solely to confirm a write landed, which both harnesses
call out as waste; the previous prompt asked for exactly that re-read.
The gpt56 profile already followed the Codex split and is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`active_time_ms` was only ever computed from terminal stage events, so a
stage still running contributed zero to the run rollup. A run parked in one
long agent stage reported 2m 8s of active time against 16m 53s of wall
clock — the two finished stages — while the running stage had been doing
continuous inference and tool work for over 14 minutes.
`live_run_timing` summed `filter_map(|stage| stage.timing)`, and
`stage.timing` is only written at finalization. Wall time ticked live off
`start_time`; active time did not tick at all.
Stage projections now accumulate brackets from the event log:
- Closing an inference bracket folds its span into `live_inference_ms`
instead of discarding it, including across retries, matching the
in-process stopwatch.
- Tool calls open a batch on the first outstanding call and close it when
the last one drains, so tools running concurrently within a turn count
once — the same span `execute_tool_calls` is bracketed by. Summing
per-call durations would over-count parallel tool use. Subagent tool
events are excluded; they run inside the root call's span already.
- `StageProjection::live_timing(now)` composes accumulators with any open
bracket, per handler: agent stages use the brackets, prompt and command
stages count elapsed time as inference and tool respectively, and
handlers that wait on a human, timer, condition, or child branches
report zero.
Active is clamped to wall per stage. A worker killed mid-turn leaves its
bracket open forever, and without the clamp it would tick up unbounded.
The clamp does not need to detect the dead worker: a stage cannot have been
active longer than it has existed. `watchdog.timeout` remains the authority
on whether a run is stuck. The clamp is deliberately not applied at run
level, where concurrent branches can legitimately sum past run wall time.
Timing is derived from events rather than emitted by the worker, so this
needs no event-schema change and applies to runs already stored.
`StageProjection.timing` keeps its terminal-only meaning, and the
authoritative breakdown still replaces the live estimate at terminal
events.
The billing endpoint had the same hole behind its `wall_only` fallback:
running stages reported zero inference/tool/active. Not visible in the
product, which renders only `wall_time_ms`, but wrong for any other
consumer of `GET /runs/{id}/billing`.
Parallel branch stages lose their breakdown permanently, even after
completion, because `parallel.branch.completed` carries only `duration_ms`.
That is a separate data-loss bug, tracked in #644.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ReadBeforeWriteSandbox` blocked writes to any existing file the agent
had not read, tracked by a session read set populated only by
`read_file`, `grep`, `read_many_files`, and the Kimi `Read`.
The gpt56 profile has none of those. It mirrors Codex's tool contract --
`shell_command`, `apply_patch`/`edit_file`, `update_plan`, `web_search`
-- and reads through the shell, so its read set stayed permanently
empty and every edit to an existing file failed. In run
01KYD4360GN6SED4BYEVGYP4XT all 28 `edit_file` calls failed, 25 of them
on the guard. The agent read `package.json` with `sed` and `cat`,
hex-dumped it trying to diagnose the rejections, then routed around the
guard with `sed -i`, which the guard never covered. It prevented no
blind write; it converted content-anchored edits into an unreviewed
in-place shell rewrite.
Neither Codex nor Kimi Code enforces read-before-write at runtime.
Codex's `apply_patch` `Add File` overwrites an existing path silently;
Kimi Code's `Write` has no check at all. Both rely on the exact-match
requirement in their edit tools, which is stronger proof of inspection
than a read set, plus per-write approval.
Tool descriptions and the Kimi prompt keep telling the model to read
before editing -- that guidance matches Kimi Code's own `edit.md` and
still prevents `old_string not found` -- but no longer claim the
workspace refuses unread writes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codex drives the GPT-5.6 models with a much narrower tool set than the
other OpenAI models: a shell, `apply_patch`, and `update_plan`. It has no
file-read, file-write, grep, glob, or fetch tool at all -- reading and
searching go through the shell, and every write goes through
`apply_patch`. Offering 5.6 fabro's extra tools advertises affordances its
instructions never mention, so this adds a profile that registers only
what Codex does.
The profile is selected per model via `agent_profile = "gpt56"` on the six
5.6 rows (three each on `openai` and `openrouter`), following the existing
Kimi-over-a-gateway pattern. Every other model on those providers keeps
its provider default, with no code branch and no version sniffing.
- `ToolVocabulary::Codex` renames `shell` to `shell_command`; a strum
alias keeps `from_any_name` resolving it to `NativeTool::Shell`, so
permissions, categories, and telemetry still key on the canonical name.
- `shell_command` gains `workdir`, passed to the `cwd` argument
`execute_shell_command` already accepted, with Codex's "always set
`workdir`, do not `cd`" guidance.
- `prompts/gpt56.md.j2` is adapted from Codex's 5.6 `base_instructions`,
which are byte-identical across Sol, Terra, and Luna. A header comment
records provenance and the departures fabro's harness forces.
This is an alignment-only pass: it matches Codex's tool contract while
keeping direct tool calls. Codex actually drives 5.6 in code mode, with a
single `exec` tool taking JavaScript and every other tool reached through
a `tools` object inside a V8 isolate. That is deliberately out of scope.
Luna's `multi_agent_version: v1` (vs v2 on Sol and Terra) is also out of
scope. It only changes the sub-agent tool set, which fabro registers from
the caller rather than the profile, and fabro's current set matches
neither version exactly.
Two server cancel-timing tests are adjusted. `gpt-5.6-sol` is the
`openai` provider's default model, so runs that name no model now build a
3-tool profile instead of an 8-tool one and reach their first stage
sooner. `full_http_lifecycle_cancel` asserted `status.kind == "blocked"`
at the instant of cancel, which the worker is free to change the moment it
is signaled; it now accepts either live state, matching the tolerance its
own comment already documents for `pending_control`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stage-summary preamble rendered per-stage token usage for every
completed LLM stage: "Model: kimi-k3, 92.6k tokens in / 41.1k out" at
compact fidelity and "Tokens: N in / N out" at summary:high. Agents read
that as their own remaining budget.
In run 01KYCM3EG4KMCVRDYNV93PZWBV an implementation stage stopped after 2
of 9 units, reasoning "We have around 100k tokens, but time constraints
are an issue" and recording the rest as halted "within the available
execution window". The 92.6k it saw was the preceding plan stage's
billing telemetry, the only token quantity anywhere in its context. It
had used 11% of a 1,050,000-token window and 0.8% of a 24h stage timeout,
and no harness limit was near.
These counts have no task value to the agent: they describe a different
model's usage on an earlier stage, they are stale by one stage, and
nothing in the preamble distinguishes them from a budget. Keep the model
id and files touched, which carry provenance the agent can act on.
Both tests that asserted the counts now assert their absence, so the
regression is caught rather than re-snapshotted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extend the Daytona Bash probe to cover the streaming toolbox-session transport in addition to the direct process exec. The two build different requests, so passing one is not evidence for the other: the `exec` regression fixed in #636 left every streaming command stalling until its timeout while the lifecycle probe reported a healthy sandbox. The session probe reuses the streaming path's own command construction and completion wait, so a transport that suppresses Daytona's exit-code bookkeeping fails at the lifecycle boundary with a remediation that names the wrapper-shell contract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run the streaming Bash wrapper as a child of Daytona's session shell so the provider can resume its bookkeeping and persist the command exit code. Add a regression test that exercises the sourced-command contract and preserves a nonzero exit status.
