checkpoint

⚒️ Generated with [Fabro](https://fabro.sh)
This commit is contained in:
Fabro 2026-05-23 16:26:00 -04:00
parent c69a3ac640
commit 30e3b024b7
7 changed files with 538 additions and 9 deletions

245
run.json

File diff suppressed because one or more lines are too long

View file

@ -0,0 +1 @@
blob://sha256/12ae32cb1ec02d01eda3581b127c1fee3b0dc53572ed6baf239721a03d82e126

View file

@ -0,0 +1,8 @@
{
"output": "blob://sha256/12ae32cb1ec02d01eda3581b127c1fee3b0dc53572ed6baf239721a03d82e126",
"exit_code": 0,
"duration_ms": 148439,
"termination": "exited",
"output_bytes": 0,
"live_streaming": false
}

View file

@ -0,0 +1,6 @@
{
"outcome": "succeeded",
"notes": "Script completed: cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1",
"failure_reason": null,
"timestamp": "2026-05-23T20:00:57.004958Z"
}

View file

@ -0,0 +1,254 @@
Goal: # Output Schema Validation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add `output_schema` validation for agent and prompt nodes, with context-preserving repair turns when structured output does not validate.
**Architecture:** Introduce a small structured-output layer in `fabro-workflow` that resolves node-level schema declarations, extracts JSON output, validates it, and produces either routing side effects or a parsed custom output context update. Agent and prompt execution must perform schema repair inside the active LLM conversation instead of using the workflow executor retry path.
**Tech Stack:** Rust, Graphviz workflow attrs, `serde_json`, workspace `jsonschema`, existing `fabro-llm::ResponseFormat`, Fabro agent sessions, `cargo nextest`.
---
## Public Interface
Workflow authors can opt in on agent and prompt nodes:
```dot
review [
shape=tab,
output_schema="routing",
output_retries=2
]
audit [
shape=tab,
output_schema="@schemas/audit-result.schema.json",
output_retries=2
]
```
- `output_schema="routing"` uses Fabro's built-in routing directive schema.
- `output_schema="@path/to/schema.json"` loads a JSON Schema file through existing workflow file-reference rules.
- `output_retries` controls corrective turns inside the same node execution. Default: `2`. `0` means validate once and fail without a repair turn.
- Schema failures are terminal node failures after `output_retries` is exhausted. They are not `retry_requested` outcomes and do not consume `max_retries`.
- `backend="acp"` with `output_schema` is unsupported in v1 and returns a clear validation error.
## Implementation Tasks
### Task 1: Node Attributes And File Reference Resolution
**Files:**
- Modify: `lib/crates/fabro-types/src/graph.rs`
- Modify: `lib/crates/fabro-workflow/src/static_reference.rs`
- Modify: `lib/crates/fabro-workflow/src/transforms/file_inlining.rs`
- Test: existing unit tests in those files
- [ ] Add `Node::output_schema(&self) -> Option<&str>` next to other agent/prompt attrs.
- [ ] Add `Node::output_retries(&self) -> i64` returning `self.int_attr("output_retries").unwrap_or(2).max(0)`.
- [ ] Teach static reference validation that node attr `output_schema` values starting with `@` are file inline references.
- [ ] Extend file inlining so `output_schema="@schemas/foo.json"` is replaced with the schema file contents before execution, while `output_schema="routing"` stays unchanged.
- [ ] Add tests for absent attrs, default retries, zero retries, file inlining, and unresolved schema reference diagnostics.
Run:
```bash
cargo nextest run -p fabro-types -p fabro-workflow graph:: file_inlining static_reference
```
Expected: targeted tests pass.
### Task 2: Structured Output Module
**Files:**
- Create: `lib/crates/fabro-workflow/src/handler/structured_output.rs`
- Modify: `lib/crates/fabro-workflow/src/handler/mod.rs`
- Modify: `lib/crates/fabro-workflow/Cargo.toml`
- Test: unit tests in `structured_output.rs`
- [ ] Add `jsonschema.workspace = true` to `fabro-workflow` dependencies.
- [ ] Define `OutputSchemaKind` with `Routing` and `JsonSchema { schema: serde_json::Value }`.
- [ ] Parse `node.output_schema()` into `None`, `Routing`, or custom JSON Schema. Treat literal `routing` as the only built-in keyword.
- [ ] Add a built-in routing schema requiring an object with at least one recognized field: `preferred_next_label`, `outcome`, `failure_reason`, `suggested_next_ids`, or `context_updates`.
- [ ] Reuse balanced-object scanning semantics for response text: validate the last JSON object that is relevant to the selected schema.
- [ ] Return a structured validation result containing the parsed JSON object, concise error messages, and enough information to build a repair prompt.
- [ ] Add tests for valid routing JSON, missing routing fields, wrong routing field types, valid custom schema, invalid custom schema, invalid JSON, and no JSON object.
Run:
```bash
cargo nextest run -p fabro-workflow structured_output
```
Expected: structured-output unit tests pass.
### Task 3: Routing Extraction Compatibility
**Files:**
- Modify: `lib/crates/fabro-workflow/src/handler/agent.rs`
- Test: existing agent handler unit tests
- [ ] Keep the loose default unchanged when `output_schema` is absent.
- [ ] Move current `STATUS_FIELDS`, balanced JSON scanning, and routing-field application behind reusable functions in `structured_output.rs` or call the new module from `agent.rs`.
- [ ] For `output_schema="routing"`, require schema-valid routing JSON and surface validation failures for repair instead of silently ignoring bad candidates.
- [ ] Preserve existing routing fallback priority for agent nodes: response text first, then `status.json`, then last file touched.
- [ ] Keep prompt-node routing behavior response-only unless later tasks explicitly add prompt `status.json` support.
Run:
```bash
cargo nextest run -p fabro-workflow handler::agent
```
Expected: existing loose routing tests still pass, plus new strict routing tests pass.
### Task 4: Prompt Node Same-Context Repair
**Files:**
- Modify: `lib/crates/fabro-workflow/src/handler/llm/api.rs`
- Modify: `lib/crates/fabro-workflow/src/handler/prompt.rs`
- Test: prompt/API backend tests in those files
- [ ] In `AgentApiBackend::one_shot`, keep `messages` mutable across attempts.
- [ ] When a prompt node has a custom JSON Schema, set `response_format=JsonSchema` on the initial and repair LLM requests. For `routing`, use `JsonObject` or no provider-native schema if provider behavior would conflict with Fabro's routing extraction.
- [ ] After each LLM response, validate according to `output_schema`.
- [ ] On validation failure with repair attempts remaining, append `Message::assistant(response.text())`, then append a corrective `Message::user(repair_message)`, and call `client.complete` again with the same messages.
- [ ] On success, return the validated response text and aggregate usage across all attempts.
- [ ] On exhaustion, return a terminal failed outcome with failure reason `output schema validation failed after N repair attempt(s)`.
- [ ] Update `PromptHandler` so validated custom output is added to `context_updates["output.{node_id}"]`; routing output still updates outcome routing fields.
Run:
```bash
cargo nextest run -p fabro-workflow handler::prompt handler::llm::api
```
Expected: prompt repair keeps previous assistant output in the message list and succeeds after a corrective response.
### Task 5: Agent Node Same-Session Repair
**Files:**
- Modify: `lib/crates/fabro-workflow/src/handler/llm/api.rs`
- Modify: `lib/crates/fabro-workflow/src/handler/agent.rs`
- Test: agent/API backend tests in those files
- [ ] In `AgentApiBackend::run`, validate the final assistant response before releasing, closing, or caching the session.
- [ ] On validation failure with repair attempts remaining, call `session.process_input(repair_message)` on the same `Session`.
- [ ] Recompute the final assistant response after each repair turn from `session.history()`.
- [ ] Aggregate usage across all new assistant turns, including repair turns, without double-counting reused session history.
- [ ] Do not set provider-native `response_format` for agent sessions in v1, because agent sessions may need normal tool-use messages before final output.
- [ ] Return terminal failure after exhaustion; do not return a retryable backend error and do not request workflow node retry.
- [ ] Update `AgentHandler` to apply validated routing/custom output to the final `Outcome`.
Run:
```bash
cargo nextest run -p fabro-workflow handler::agent handler::llm::api
```
Expected: agent repair sends a second `process_input` to the same session and final validated output drives outcome/context updates.
### Task 6: ACP Guardrail
**Files:**
- Modify: `lib/crates/fabro-workflow/src/handler/llm/acp.rs`
- Test: ACP backend tests in that file
- [ ] At the start of `AgentAcpBackend::run`, reject nodes where `node.output_schema().is_some()`.
- [ ] Use a clear error message: `output_schema is not supported with backend="acp" in this release`.
- [ ] Add a test proving the ACP backend does not launch a process when `output_schema` is present.
Run:
```bash
cargo nextest run -p fabro-workflow handler::llm::acp
```
Expected: ACP guardrail test passes.
### Task 7: Docs
**Files:**
- Modify: `docs/public/agents/outputs.mdx`
- Modify: `docs/public/reference/dot-language.mdx`
- [ ] Document `output_schema="routing"` and `output_schema="@schema.json"` under routing/structured outputs.
- [ ] Document same-context repair behavior explicitly: Fabro sends validation feedback to the same agent/prompt context before failing.
- [ ] Document `output_retries`, default `2`, and distinction from `max_retries`.
- [ ] Document v1 scope: agent/prompt nodes only; ACP unsupported; custom schema output stored at `output.{node_id}`.
Run:
```bash
rg -n "output_schema|output_retries|output\\." docs/public/agents/outputs.mdx docs/public/reference/dot-language.mdx
```
Expected: docs mention the new attrs and storage behavior.
### Task 8: Full Verification
**Files:**
- No new files beyond prior tasks
- [ ] Run focused workflow tests:
```bash
cargo nextest run -p fabro-workflow
```
- [ ] Run formatting check:
```bash
cargo +nightly-2026-04-14 fmt --check --all
```
- [ ] Run clippy:
```bash
cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings
```
- [ ] If snapshots change, inspect before accepting:
```bash
cargo insta pending-snapshots
```
Only run `cargo insta accept` after verifying every pending snapshot is expected.
## Acceptance Criteria
- Existing workflows without `output_schema` behave exactly as before.
- `output_schema="routing"` prevents malformed/missing routing JSON from silently falling through to normal edge selection.
- Invalid structured output results in a corrective LLM turn in the same context window.
- Prompt repair preserves previous assistant output in the message list.
- Agent repair preserves the same live session and does not re-run the node from scratch.
- Exhausted output repair attempts produce a clear terminal failure.
- Custom schema output is available to downstream nodes at `output.{node_id}`.
- Docs clearly distinguish `output_retries` from `max_retries`.
## Assumptions
- `output_retries=2` is the default.
- Custom schema validation targets the final JSON object in the response text.
- `status.json` fallback remains routing-specific.
- Provider-native response schema is used for prompt nodes only where it is safe.
- ACP support can be added later after there is a guaranteed context-preserving repair mechanism.
## Completed stages
- **toolchain**: succeeded
- Script: `command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1`
- Output:
```
cargo 1.95.0 (f2d3ce0bd 2026-03-21)
```
- **preflight_compile**: succeeded
- Script: `cargo check -q --workspace 2>&1`
- Output: (empty)
- **preflight_lint**: succeeded
- Script: `cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1`
- Output: (empty)
Read the plan file referenced in the goal and implement every step. Make all the code changes described in the plan. Use red/green TDD.

View file

@ -0,0 +1,6 @@
{
"mode": "agent",
"provider": "openai",
"model": "gpt-5.5",
"reasoning_effort": "xhigh"
}

View file

@ -0,0 +1,27 @@
Implemented output schema validation end-to-end.
Summary:
- Added `output_schema` / `output_retries` node attrs.
- Added `output_schema` static/file reference handling and inlining.
- Added `handler::structured_output` for:
- routing/custom schema parsing
- balanced JSON extraction
- JSON Schema validation
- repair prompts
- routing/custom context application
- Integrated strict routing/custom output into agent and prompt handlers.
- Added same-context repair for:
- prompt API calls via mutable message history
- agent API sessions via the same live `Session`
- Added ACP guardrail:
- `output_schema is not supported with backend="acp" in this release`
- test verifies no process launch
- Added custom output storage at `output.{node_id}`.
- Updated docs for routing/custom schemas, `output_retries`, repair behavior, ACP limitation.
Validation run:
- `cargo nextest run -p fabro-types -p fabro-workflow graph:: file_inlining static_reference` ✅
- `cargo nextest run -p fabro-workflow` ✅
- `cargo +nightly-2026-04-14 fmt --check --all` ✅
- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings` ✅
- `cargo insta pending-snapshots` unavailable in this environment (`cargo-insta` not installed); checked for `**/*.snap.new` and found none.