mirror of
https://github.com/fabro-sh/fabro.git
synced 2026-10-10 03:30:59 +00:00
parent
8dc421d824
commit
bf2fec0e7a
7 changed files with 639 additions and 14 deletions
334
run.json
334
run.json
File diff suppressed because one or more lines are too long
1
stages/008-verify@1/output.log
Normal file
1
stages/008-verify@1/output.log
Normal file
|
|
@ -0,0 +1 @@
|
|||
blob://sha256/2f966243140b2bc52abcd68a19b2845a58da45f51166baeee8d741c3d91b52de
|
||||
8
stages/008-verify@1/script_timing.json
Normal file
8
stages/008-verify@1/script_timing.json
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"output": "blob://sha256/2f966243140b2bc52abcd68a19b2845a58da45f51166baeee8d741c3d91b52de",
|
||||
"exit_code": 128,
|
||||
"duration_ms": 1427,
|
||||
"termination": "exited",
|
||||
"output_bytes": 502,
|
||||
"live_streaming": true
|
||||
}
|
||||
6
stages/008-verify@1/status.json
Normal file
6
stages/008-verify@1/status.json
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"outcome": "failed",
|
||||
"notes": null,
|
||||
"failure_reason": "Script failed with exit code: 128\n\n## output\nFrom https://github.com/fabro-sh/fabro\n * branch main -> FETCH_HEAD\n dbe3e3966..eda9e8855 main -> origin/main\nCommitter identity unknown\n\n*** Please tell me who you are.\n\nRun\n\n git config --global user.email \"you@example.com\"\n git config --global user.name \"Your Name\"\n\nto set your account's default identity.\nOmit --global to set the identity only in this repository.\n\nfatal: unable to auto-detect email address (got 'root@7bf578ba-8e1b-4e46-b584-1efed7b5eb9c.(none)')\n",
|
||||
"timestamp": "2026-05-23T20:46:10.757319Z"
|
||||
}
|
||||
287
stages/009-fixup@1/prompt.md
Normal file
287
stages/009-fixup@1/prompt.md
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
Goal: # Output Schema Validation Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add `output_schema` validation for agent and prompt nodes, with context-preserving repair turns when structured output does not validate.
|
||||
|
||||
**Architecture:** Introduce a small structured-output layer in `fabro-workflow` that resolves node-level schema declarations, extracts JSON output, validates it, and produces either routing side effects or a parsed custom output context update. Agent and prompt execution must perform schema repair inside the active LLM conversation instead of using the workflow executor retry path.
|
||||
|
||||
**Tech Stack:** Rust, Graphviz workflow attrs, `serde_json`, workspace `jsonschema`, existing `fabro-llm::ResponseFormat`, Fabro agent sessions, `cargo nextest`.
|
||||
|
||||
---
|
||||
|
||||
## Public Interface
|
||||
|
||||
Workflow authors can opt in on agent and prompt nodes:
|
||||
|
||||
```dot
|
||||
review [
|
||||
shape=tab,
|
||||
output_schema="routing",
|
||||
output_retries=2
|
||||
]
|
||||
|
||||
audit [
|
||||
shape=tab,
|
||||
output_schema="@schemas/audit-result.schema.json",
|
||||
output_retries=2
|
||||
]
|
||||
```
|
||||
|
||||
- `output_schema="routing"` uses Fabro's built-in routing directive schema.
|
||||
- `output_schema="@path/to/schema.json"` loads a JSON Schema file through existing workflow file-reference rules.
|
||||
- `output_retries` controls corrective turns inside the same node execution. Default: `2`. `0` means validate once and fail without a repair turn.
|
||||
- Schema failures are terminal node failures after `output_retries` is exhausted. They are not `retry_requested` outcomes and do not consume `max_retries`.
|
||||
- `backend="acp"` with `output_schema` is unsupported in v1 and returns a clear validation error.
|
||||
|
||||
## Implementation Tasks
|
||||
|
||||
### Task 1: Node Attributes And File Reference Resolution
|
||||
|
||||
**Files:**
|
||||
- Modify: `lib/crates/fabro-types/src/graph.rs`
|
||||
- Modify: `lib/crates/fabro-workflow/src/static_reference.rs`
|
||||
- Modify: `lib/crates/fabro-workflow/src/transforms/file_inlining.rs`
|
||||
- Test: existing unit tests in those files
|
||||
|
||||
- [ ] Add `Node::output_schema(&self) -> Option<&str>` next to other agent/prompt attrs.
|
||||
- [ ] Add `Node::output_retries(&self) -> i64` returning `self.int_attr("output_retries").unwrap_or(2).max(0)`.
|
||||
- [ ] Teach static reference validation that node attr `output_schema` values starting with `@` are file inline references.
|
||||
- [ ] Extend file inlining so `output_schema="@schemas/foo.json"` is replaced with the schema file contents before execution, while `output_schema="routing"` stays unchanged.
|
||||
- [ ] Add tests for absent attrs, default retries, zero retries, file inlining, and unresolved schema reference diagnostics.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cargo nextest run -p fabro-types -p fabro-workflow graph:: file_inlining static_reference
|
||||
```
|
||||
|
||||
Expected: targeted tests pass.
|
||||
|
||||
### Task 2: Structured Output Module
|
||||
|
||||
**Files:**
|
||||
- Create: `lib/crates/fabro-workflow/src/handler/structured_output.rs`
|
||||
- Modify: `lib/crates/fabro-workflow/src/handler/mod.rs`
|
||||
- Modify: `lib/crates/fabro-workflow/Cargo.toml`
|
||||
- Test: unit tests in `structured_output.rs`
|
||||
|
||||
- [ ] Add `jsonschema.workspace = true` to `fabro-workflow` dependencies.
|
||||
- [ ] Define `OutputSchemaKind` with `Routing` and `JsonSchema { schema: serde_json::Value }`.
|
||||
- [ ] Parse `node.output_schema()` into `None`, `Routing`, or custom JSON Schema. Treat literal `routing` as the only built-in keyword.
|
||||
- [ ] Add a built-in routing schema requiring an object with at least one recognized field: `preferred_next_label`, `outcome`, `failure_reason`, `suggested_next_ids`, or `context_updates`.
|
||||
- [ ] Reuse balanced-object scanning semantics for response text: validate the last JSON object that is relevant to the selected schema.
|
||||
- [ ] Return a structured validation result containing the parsed JSON object, concise error messages, and enough information to build a repair prompt.
|
||||
- [ ] Add tests for valid routing JSON, missing routing fields, wrong routing field types, valid custom schema, invalid custom schema, invalid JSON, and no JSON object.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cargo nextest run -p fabro-workflow structured_output
|
||||
```
|
||||
|
||||
Expected: structured-output unit tests pass.
|
||||
|
||||
### Task 3: Routing Extraction Compatibility
|
||||
|
||||
**Files:**
|
||||
- Modify: `lib/crates/fabro-workflow/src/handler/agent.rs`
|
||||
- Test: existing agent handler unit tests
|
||||
|
||||
- [ ] Keep the loose default unchanged when `output_schema` is absent.
|
||||
- [ ] Move current `STATUS_FIELDS`, balanced JSON scanning, and routing-field application behind reusable functions in `structured_output.rs` or call the new module from `agent.rs`.
|
||||
- [ ] For `output_schema="routing"`, require schema-valid routing JSON and surface validation failures for repair instead of silently ignoring bad candidates.
|
||||
- [ ] Preserve existing routing fallback priority for agent nodes: response text first, then `status.json`, then last file touched.
|
||||
- [ ] Keep prompt-node routing behavior response-only unless later tasks explicitly add prompt `status.json` support.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cargo nextest run -p fabro-workflow handler::agent
|
||||
```
|
||||
|
||||
Expected: existing loose routing tests still pass, plus new strict routing tests pass.
|
||||
|
||||
### Task 4: Prompt Node Same-Context Repair
|
||||
|
||||
**Files:**
|
||||
- Modify: `lib/crates/fabro-workflow/src/handler/llm/api.rs`
|
||||
- Modify: `lib/crates/fabro-workflow/src/handler/prompt.rs`
|
||||
- Test: prompt/API backend tests in those files
|
||||
|
||||
- [ ] In `AgentApiBackend::one_shot`, keep `messages` mutable across attempts.
|
||||
- [ ] When a prompt node has a custom JSON Schema, set `response_format=JsonSchema` on the initial and repair LLM requests. For `routing`, use `JsonObject` or no provider-native schema if provider behavior would conflict with Fabro's routing extraction.
|
||||
- [ ] After each LLM response, validate according to `output_schema`.
|
||||
- [ ] On validation failure with repair attempts remaining, append `Message::assistant(response.text())`, then append a corrective `Message::user(repair_message)`, and call `client.complete` again with the same messages.
|
||||
- [ ] On success, return the validated response text and aggregate usage across all attempts.
|
||||
- [ ] On exhaustion, return a terminal failed outcome with failure reason `output schema validation failed after N repair attempt(s)`.
|
||||
- [ ] Update `PromptHandler` so validated custom output is added to `context_updates["output.{node_id}"]`; routing output still updates outcome routing fields.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cargo nextest run -p fabro-workflow handler::prompt handler::llm::api
|
||||
```
|
||||
|
||||
Expected: prompt repair keeps previous assistant output in the message list and succeeds after a corrective response.
|
||||
|
||||
### Task 5: Agent Node Same-Session Repair
|
||||
|
||||
**Files:**
|
||||
- Modify: `lib/crates/fabro-workflow/src/handler/llm/api.rs`
|
||||
- Modify: `lib/crates/fabro-workflow/src/handler/agent.rs`
|
||||
- Test: agent/API backend tests in those files
|
||||
|
||||
- [ ] In `AgentApiBackend::run`, validate the final assistant response before releasing, closing, or caching the session.
|
||||
- [ ] On validation failure with repair attempts remaining, call `session.process_input(repair_message)` on the same `Session`.
|
||||
- [ ] Recompute the final assistant response after each repair turn from `session.history()`.
|
||||
- [ ] Aggregate usage across all new assistant turns, including repair turns, without double-counting reused session history.
|
||||
- [ ] Do not set provider-native `response_format` for agent sessions in v1, because agent sessions may need normal tool-use messages before final output.
|
||||
- [ ] Return terminal failure after exhaustion; do not return a retryable backend error and do not request workflow node retry.
|
||||
- [ ] Update `AgentHandler` to apply validated routing/custom output to the final `Outcome`.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cargo nextest run -p fabro-workflow handler::agent handler::llm::api
|
||||
```
|
||||
|
||||
Expected: agent repair sends a second `process_input` to the same session and final validated output drives outcome/context updates.
|
||||
|
||||
### Task 6: ACP Guardrail
|
||||
|
||||
**Files:**
|
||||
- Modify: `lib/crates/fabro-workflow/src/handler/llm/acp.rs`
|
||||
- Test: ACP backend tests in that file
|
||||
|
||||
- [ ] At the start of `AgentAcpBackend::run`, reject nodes where `node.output_schema().is_some()`.
|
||||
- [ ] Use a clear error message: `output_schema is not supported with backend="acp" in this release`.
|
||||
- [ ] Add a test proving the ACP backend does not launch a process when `output_schema` is present.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cargo nextest run -p fabro-workflow handler::llm::acp
|
||||
```
|
||||
|
||||
Expected: ACP guardrail test passes.
|
||||
|
||||
### Task 7: Docs
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/public/agents/outputs.mdx`
|
||||
- Modify: `docs/public/reference/dot-language.mdx`
|
||||
|
||||
- [ ] Document `output_schema="routing"` and `output_schema="@schema.json"` under routing/structured outputs.
|
||||
- [ ] Document same-context repair behavior explicitly: Fabro sends validation feedback to the same agent/prompt context before failing.
|
||||
- [ ] Document `output_retries`, default `2`, and distinction from `max_retries`.
|
||||
- [ ] Document v1 scope: agent/prompt nodes only; ACP unsupported; custom schema output stored at `output.{node_id}`.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
rg -n "output_schema|output_retries|output\\." docs/public/agents/outputs.mdx docs/public/reference/dot-language.mdx
|
||||
```
|
||||
|
||||
Expected: docs mention the new attrs and storage behavior.
|
||||
|
||||
### Task 8: Full Verification
|
||||
|
||||
**Files:**
|
||||
- No new files beyond prior tasks
|
||||
|
||||
- [ ] Run focused workflow tests:
|
||||
|
||||
```bash
|
||||
cargo nextest run -p fabro-workflow
|
||||
```
|
||||
|
||||
- [ ] Run formatting check:
|
||||
|
||||
```bash
|
||||
cargo +nightly-2026-04-14 fmt --check --all
|
||||
```
|
||||
|
||||
- [ ] Run clippy:
|
||||
|
||||
```bash
|
||||
cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings
|
||||
```
|
||||
|
||||
- [ ] If snapshots change, inspect before accepting:
|
||||
|
||||
```bash
|
||||
cargo insta pending-snapshots
|
||||
```
|
||||
|
||||
Only run `cargo insta accept` after verifying every pending snapshot is expected.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Existing workflows without `output_schema` behave exactly as before.
|
||||
- `output_schema="routing"` prevents malformed/missing routing JSON from silently falling through to normal edge selection.
|
||||
- Invalid structured output results in a corrective LLM turn in the same context window.
|
||||
- Prompt repair preserves previous assistant output in the message list.
|
||||
- Agent repair preserves the same live session and does not re-run the node from scratch.
|
||||
- Exhausted output repair attempts produce a clear terminal failure.
|
||||
- Custom schema output is available to downstream nodes at `output.{node_id}`.
|
||||
- Docs clearly distinguish `output_retries` from `max_retries`.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- `output_retries=2` is the default.
|
||||
- Custom schema validation targets the final JSON object in the response text.
|
||||
- `status.json` fallback remains routing-specific.
|
||||
- Provider-native response schema is used for prompt nodes only where it is safe.
|
||||
- ACP support can be added later after there is a guaranteed context-preserving repair mechanism.
|
||||
|
||||
|
||||
## Completed stages
|
||||
- **toolchain**: succeeded
|
||||
- Script: `command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1`
|
||||
- Output:
|
||||
```
|
||||
cargo 1.95.0 (f2d3ce0bd 2026-03-21)
|
||||
```
|
||||
- **preflight_compile**: succeeded
|
||||
- Script: `cargo check -q --workspace 2>&1`
|
||||
- Output: (empty)
|
||||
- **preflight_lint**: succeeded
|
||||
- Script: `cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1`
|
||||
- Output: (empty)
|
||||
- **implement**: succeeded
|
||||
- Model: gpt-5.5, 329.8k tokens in / 54.8k out
|
||||
- Files: /home/daytona/workspace/fabro/lib/crates/fabro-workflow/src/handler/structured_output.rs
|
||||
- **simplify_opus**: succeeded
|
||||
- Model: claude-opus-4-7, 141.5k tokens in / 35.9k out
|
||||
- Files: /home/daytona/workspace/fabro/lib/crates/fabro-workflow/src/handler/agent.rs, /home/daytona/workspace/fabro/lib/crates/fabro-workflow/src/handler/llm/api.rs, /home/daytona/workspace/fabro/lib/crates/fabro-workflow/src/handler/structured_output.rs
|
||||
- **simplify_gpt**: succeeded
|
||||
- Model: gpt-5.5, 116.0k tokens in / 6.3k out
|
||||
- **verify**: failed
|
||||
- Script: `git fetch origin main 2>&1 && git merge --no-edit --no-stat origin/main 2>&1 && cargo +nightly-2026-04-14 fmt --all 2>&1 && cargo dev docs refresh 2>&1 && cargo +nightly-2026-04-14 fmt --check --all 2>&1 && ! rg -n 'AuthMode::Disabled|RunAuthMethod|RunSubjectProvenance|\bActorRef\b|\bActorKind\b|AuthenticatedSubject|AuthenticatedService|AuthorizeRunScoped|AuthorizeRunBlob|AuthorizeStageArtifact|AuthorizeCommandLog|auth_method\s*==\s*"disabled"' lib/crates apps lib/packages docs/public/api-reference/fabro-api.yaml 2>&1 && cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --workspace --status-level slow --profile ci 2>&1 && cargo dev docs check 2>&1 && bun install --frozen-lockfile 2>&1 && (cd apps/fabro-web && bun run typecheck) 2>&1 && (cd apps/fabro-web && bun run test) 2>&1 && (cd lib/packages/fabro-api-client && bun run typecheck) 2>&1 && cargo dev build -- -p fabro-cli --release 2>&1`
|
||||
- Output:
|
||||
```
|
||||
From https://github.com/fabro-sh/fabro
|
||||
* branch main -> FETCH_HEAD
|
||||
dbe3e3966..eda9e8855 main -> origin/main
|
||||
Committer identity unknown
|
||||
|
||||
*** Please tell me who you are.
|
||||
|
||||
Run
|
||||
|
||||
git config --global user.email "you@example.com"
|
||||
git config --global user.name "Your Name"
|
||||
|
||||
to set your account's default identity.
|
||||
Omit --global to set the identity only in this repository.
|
||||
|
||||
fatal: unable to auto-detect email address (got 'root@7bf578ba-8e1b-4e46-b584-1efed7b5eb9c.(none)')
|
||||
```
|
||||
|
||||
## Context
|
||||
- failure_class: deterministic
|
||||
- failure_signature: verify|deterministic|script failed with exit code: <n> ## output from https://github.com/fabro-sh/fabro * branch main -> fetch_head <hex>..<hex> main -> origin/main committer identity unknown *** please tell me who you are. run git config --global user.email "y
|
||||
|
||||
|
||||
The verify step failed. Read the build output from context and fix all format, clippy, Rust test, docs, TypeScript typecheck/test, and build failures.
|
||||
5
stages/009-fixup@1/provider_used.json
Normal file
5
stages/009-fixup@1/provider_used.json
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
{
|
||||
"mode": "agent",
|
||||
"provider": "anthropic",
|
||||
"model": "claude-opus-4-7"
|
||||
}
|
||||
12
stages/009-fixup@1/response.md
Normal file
12
stages/009-fixup@1/response.md
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
All verify steps pass:
|
||||
- Merge with origin/main: clean
|
||||
- Format check: clean
|
||||
- Docs refresh: up-to-date
|
||||
- Banned auth patterns: none
|
||||
- Clippy: clean
|
||||
- Tests: 6273 passed, 181 skipped
|
||||
- Docs check: up-to-date
|
||||
- bun install / typecheck (web + api-client) / web tests: all pass
|
||||
- Release build (`-p fabro-cli`): success
|
||||
|
||||
The verify failure root cause was missing local git identity for the merge commit. I configured `user.email`/`user.name` and re-ran every verify step end-to-end.
|
||||
Loading…
Add table
Reference in a new issue