Commit graph

3107 commits

Author SHA1 Message Date
Bryan Helmkamp
3d8e17aedf Write progress.ndjson and live.json to logs dir during pipeline runs
Every PipelineEvent is now logged as a JSON envelope with timestamp,
run_id, and event fields — matching the Kilroy reference implementation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:50:19 -05:00
Bryan Helmkamp
fa828dd8b1 Increase connect timeout to 30s, disable request/stream_read timeouts
LLM API calls should not have HTTP-level request or stream-read timeouts.
These are better controlled at the application level via TimeoutConfig
(total/per_step). The connect timeout is increased from 10s to 30s since
network conditions vary.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:46:50 -05:00
Bryan Helmkamp
9665fad133 Revert clippy config to defaults, remove all pedantic/nursery/cargo lint suppressions
Removed the workspace-level clippy lint config that enabled all, pedantic, nursery,
and cargo lint groups. Removed all #[allow(clippy::...)] annotations that were only
needed to suppress those extra lints, and fixed the few default clippy warnings that
were uncovered.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:42:28 -05:00
Bryan Helmkamp
f166bb4959 Replace text-scanning with report_outcome tool for routing
The previous approach scanned LLM response text for JSON containing
routing fields (extract_status_fields), which was fragile and violated
the spec's abstraction boundary. Replace with two clean mechanisms:

- Backend registers a report_outcome tool that the LLM calls to declare
  routing decisions (status, preferred_next_label, context_updates, etc.)
- Handler auto-generates a routing preamble listing available edge labels,
  appended to the prompt only when 2+ labeled unconditional edges exist

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:29:25 -05:00
Bryan Helmkamp
7222da1230 Store response.<node_id> in context after each LLM node
Downstream nodes in multi-model pipelines reference response.<node_id>
context keys in their prompts. The codergen handler was only storing
last_response (truncated) and last_stage, so these references resolved
to nothing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:12:29 -05:00
Bryan Helmkamp
28cdefec67 Add token usage and cost tracking to pipeline runs
Surface LLM token consumption and dollar cost at every verbosity level:
default mode appends per-stage tokens/cost, verbose modes include it in
event summary/detail, and the Pipeline Result section shows a total.
Cost is computed from the catalog pricing for Anthropic models; providers
without pricing (OpenAI, Gemini) show token counts only. Dry runs with
zero tokens omit the cost line entirely.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:01:17 -05:00
Bryan Helmkamp
46d4659633 Fix OpenAI error field, add Anthropic provider options pass-through, and improve parity tests
- Use `status: "incomplete"` instead of `is_error` for OpenAI tool results (fixes rejection)
- Add merge_provider_options to forward unknown anthropic provider options to API body
- Derive Clone on Client to enable subagent session factory
- Enable error_recovery scenario for all providers now that OpenAI is fixed
- Improve subagent_spawn test to actually exercise spawn/wait/read workflow
- Adjust multi-turn cache test temperature to 0.5

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 21:21:45 -05:00
Bryan Helmkamp
b4068b6364 Add missing integration tests for multi-turn caching, cross-provider parity, and attractor E2E
Test 1 (llm crate): Multi-turn cache verification runs 6 conversation turns
with a large system prompt (~5460 tokens) and verifies cache_read_tokens on
the final turn. Anthropic threshold 0.5, OpenAI/Gemini 0.0 (automatic
caching not guaranteed).

Test 2 (agent crate): Cross-provider parity matrix with 15 scenarios
(file CRUD, shell, grep/glob, editing, steering, reasoning effort, loop
detection, error recovery, etc.) across Anthropic, OpenAI, and Gemini.
41 total tests. Some scenarios excluded for OpenAI due to gpt-4o-mini
limitations (no reasoning.effort, is_error rejection, weak editing).

Test 3 (attractor crate): E2E pipeline with real LLM using AgentBackend,
AutoApproveInterviewer, and default_registry. Verifies pipeline success,
artifact files, goal gate outcomes, and checkpoint state.

All tests are #[ignore] and require API keys to run.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:24:34 -05:00
Bryan Helmkamp
aa9b05a552 Add timing output to default (non-verbose) pipeline runs
Show per-stage completion/failure timing and total pipeline duration
in the result block, even without -v flag. Uses a new
format_duration_human helper for human-readable durations (ms/s/m s).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:16:12 -05:00
Bryan Helmkamp
2a2a610654 Replace abort_flag with CancellationToken for abort-aware process cancellation
Thread CancellationToken into tool executors and exec_command so that
running processes are killed (SIGTERM -> 2s -> SIGKILL) when abort fires,
rather than only checking the flag between LLM calls. Key changes:

- ToolExecutor type gains CancellationToken parameter
- ExecutionEnvironment::exec_command gains cancel_token param
- LocalExecutionEnvironment uses tokio::select! (completion vs timeout
  vs cancellation) with extracted sigterm_then_kill helper
- DockerExecutionEnvironment uses same select! pattern
- Session replaces Arc<AtomicBool> with CancellationToken, passes
  child_token() per tool call
- Shell tool forwards cancel token to exec_command
- CLI SIGINT handler calls cancel_token.cancel()

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:10:10 -05:00
Bryan Helmkamp
028353c82e Replace manual CLI prompts with dialoguer for arrow-key navigation
Interactive TTY sessions now use dialoguer widgets (Select, MultiSelect,
Confirm, Input) instead of raw eprintln/read_line. Non-TTY input falls
back to the existing line-based reader. Suppresses redundant "Stage
started" inform message for wait.human nodes since the prompt itself
serves as notification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:58:18 -05:00
Bryan Helmkamp
7e7e8a9088 Update spec-dod-multimodel.dot for Rust: fix paths, build commands, and language references
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:50:10 -05:00
Bryan Helmkamp
3dfa0f9044 Add shape selectors, LLM stream timeouts, MultiSelect questions, and test coverage
- Stylesheet: add bare-word Shape selector (specificity between Universal and Class)
- LLM: apply per_step timeout to connection and total timeout to stream (Section 4.7)
- Interviewer: add MultiSelect question type alongside MultipleChoice
- Session: move SessionStart/SessionEnd to initialize()/close(), deduplicate close logic
- Engine: return Ok(fail outcome) instead of error when goal gate unsatisfied with no retry_target
- Docker: mark Docker-dependent tests with #[ignore]
- Validation: add extensive unit test coverage for all rule types
- Integration: update tests to match engine/fidelity changes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:43:05 -05:00
Bryan Helmkamp
a3a7f9fb5c Add --no-dotenv flag to ullm binary and use it in tests instead of env_clear()
Prevents tests from loading .env and making real API calls without
manipulating environment state.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:24:03 -05:00
Bryan Helmkamp
a3605ec0d4 Speed up slow tests: drop SSE broadcast on completion, switch to rustls-tls, replace hardcoded sleeps with poll loops, and prevent real API calls in ullm tests
- Drop event_tx from ManagedPipeline when pipeline completes/cancels/fails so
  SSE streams end promptly instead of blocking until timeout (3.5s → 0.02s)
- Switch reqwest from native-tls to rustls-tls to avoid 500ms macOS cert store
  load per process (0.67s → 0.005s per OpenAI adapter test)
- Replace hardcoded sleep(500ms)/sleep(200ms)/sleep(100ms) in server and
  integration tests with 10ms poll loops (0.2-0.5s → 0.02-0.03s each)
- Add env_clear() to ullm prompt tests to prevent .env from triggering real
  Anthropic API calls (0.45s → 0.15s)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:19:09 -05:00
Bryan Helmkamp
61eb72f038 Update spec-dod fix prompt to run cargo test before and after changes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:34:41 -05:00
Bryan Helmkamp
36e37e5458 Print logs directory at pipeline start in verbose mode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:33:28 -05:00
Bryan Helmkamp
480364c2b0 Add flow-control data to STAGE_COMPLETED event
Add status, preferred_label, and suggested_next_ids fields to the
StageCompleted event so pipeline flow decisions are visible in CLI
output and the web UI, making it easier to debug why a pipeline
took a particular path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:32:42 -05:00
Bryan Helmkamp
95a4ab24d5 Thread EngineServices through Handler::execute() to eliminate circular dependency
ParallelHandler and SubPipelineHandler needed Arc<HandlerRegistry> at
construction time but also lived inside the registry, creating a circular
dependency. The previous fix special-cased ParallelHandler as a separate
field on PipelineEngine with a resolve_handler() override.

Instead, add an EngineServices struct (registry + emitter) passed through
Handler::execute(). ParallelHandler and SubPipelineHandler become unit
structs that get what they need at execution time. No special-casing,
both register normally in default_registry() like every other handler.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:31:55 -05:00
Bryan Helmkamp
74a6b219cc Fix spec: FAIL does not trigger retries, only RETRY does
The test criteria incorrectly said FAIL outcomes should be retried.
The pseudocode (Section 3.5) and the C reference implementation both
return immediately on FAIL. Only RETRY status triggers the retry loop.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:19:10 -05:00
Bryan Helmkamp
9f9b461971 Restore exactly-one terminal node validation per spec
The spec says "exactly one" exit node in the shape table (line 184),
exit handler docs (line 648), and test criteria (line 1834). The lint
rule table (line 1437) says "at least one" but is the minority. The
previous commit incorrectly reverted this — the terminal node change
was not causing the test failures (all three were from retry-on-Fail).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:12:45 -05:00
Bryan Helmkamp
6406fccd19 Fix regressions from ceaa9bf: restore Fail as non-retryable and allow multiple terminal nodes
Revert two incorrect behavioral changes introduced in ceaa9bf:

1. StageStatus::Fail must return immediately, not retry. Fail is a
   deliberate routing outcome (e.g. to take a "fail" edge). Retrying it
   caused conditional_branching and manager_loop tests to hang.

2. Pipelines can legitimately have multiple terminal nodes. The
   "exactly one" constraint broke branching_loop_back_on_failure.
   Restore "at least one" validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:09:36 -05:00
Bryan Helmkamp
06480db417 Register parallel handler to prevent dry-run hang on parallel pipelines
The parallel handler was never registered in default_registry() due to a
circular dependency (ParallelHandler::new needs Arc<HandlerRegistry>).
Break the cycle by storing ParallelHandler as a separate field on
PipelineEngine, created after Arc-wrapping the registry and emitter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:04:16 -05:00
Bryan Helmkamp
ceaa9bf560 Improve session abort handling, emit tool output deltas, and tighten validation
- Break out of streaming loop on abort and drop the stream before emitting
  SessionEnd to properly cancel the HTTP connection
- Emit ToolCallOutputDelta events for tool call results in both sequential
  and parallel execution paths
- Retry on StageStatus::Fail in addition to Retry in pipeline engine
- Set preferred_label on WaitHumanHandler choice outcomes
- Enforce exactly one terminal node in pipeline validation
- Downgrade unreachable node diagnostic from Error to Warning
- Update context window test to match 1M token limit

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:39:04 -05:00
Bryan Helmkamp
834b7f6160 Display relative paths in CLI tool call output
Strip the CWD prefix from string arguments in format_tool_args so tool
call output shows relative paths instead of absolute ones.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:53:19 -05:00
Bryan Helmkamp
9ddb325390 Clarify spec-dod-multimodel prompts to respond with JSON inline instead of writing files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:46:03 -05:00
Bryan Helmkamp
6071bb0aa7 Handle signature_delta streaming event for Anthropic thinking blocks
The Anthropic streaming API sends thinking block signatures via a
signature_delta event, not in content_block_start or content_block_stop.
The parser was ignoring this event type, falling back to the empty
placeholder signature from content_block_start, causing "Invalid
signature in thinking block" errors on multi-turn conversations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:39:42 -05:00
Bryan Helmkamp
9342b5be76 Preserve thinking block signatures for Anthropic multi-turn conversations
The Anthropic API requires a `signature` field on thinking blocks when
they are sent back in conversation history. Previously, thinking blocks
were stored as plain text in Turn::Assistant.reasoning, losing the
signature. Now Thinking/RedactedThinking content parts are preserved
in provider_parts (which retains signatures), and the lossy
reconstruction path is skipped when provider_parts already contains
thinking blocks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:37:12 -05:00
Bryan Helmkamp
6fee3d82f1 Parse routing directives from LLM response text in CodergenHandler
When CodergenHandler receives CodergenResult::Text, it now extracts
preferred_next_label, suggested_next_ids, and context_updates from
the last JSON object in the response. This enables edge selection
via condition matching and preferred label instead of always falling
through to unconditional edges.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:31:35 -05:00
Bryan Helmkamp
242e82fffd Use Claude Opus 4.6 with 1M context window as default Anthropic model
Switch default model from claude-sonnet-4-5 to claude-opus-4-6 across
run and serve commands. Enable the 1M token context window via the
context-1m-2025-08-07 beta header for opus-4-6 requests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:23:45 -05:00
Bryan Helmkamp
4f31a84ee6 Skip OpenAI function calls with empty names to avoid model-internal items
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:07:23 -05:00
Bryan Helmkamp
e52b19d1ce Clarify spec-dod prompts to respond with JSON inline instead of writing files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:05:30 -05:00
Bryan Helmkamp
f4f58078de Update default Gemini model to gemini-3.1-pro-preview and print model name on startup
Updates the default Gemini model from gemini-3-pro-preview to
gemini-3.1-pro-preview across agent, attractor, and ullm CLIs.
Adds "Using model:" output to stderr in both agent and ullm CLIs
so users can see which model is being used.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:05:16 -05:00
Bryan Helmkamp
366643d6ef Add attractor-web React frontend for pipeline dashboard
Bun-based React app with pipeline dashboard UI including event log,
graph view, context/checkpoint panels, question panel, and status bar.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:47:03 -05:00
Bryan Helmkamp
b4f65e2555 Increase stream_read timeout from 30s to 120s
Prevents premature timeouts on slower streaming responses.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:46:32 -05:00
Bryan Helmkamp
e64c0a3328 Switch default model to claude-opus-4-6 and use adaptive thinking
Updates the default Anthropic model from claude-sonnet-4-5 to
claude-opus-4-6 and replaces beta headers with adaptive thinking
provider options for 4.6 models.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:46:31 -05:00
Bryan Helmkamp
379e90c11d Add --docker flag to attractor run and serve commands
When --docker is passed, agent tools execute inside a Docker container
via DockerExecutionEnvironment instead of running on the host. The host
working directory is bind-mounted into the container at /workspace.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:42:10 -05:00
Bryan Helmkamp
ffa92cd23d Add GET /pipelines endpoint and pipeline list UI
Adds a list endpoint that returns all in-memory pipelines with their
status, and a "Recent Pipelines" section in the start form that polls
every 3s so users can navigate to any pipeline started since server launch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:39:28 -05:00
Bryan Helmkamp
e0f33e072e Fix Dockerfile.agent to use apt ripgrep for multi-arch support
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:31:19 -05:00
Bryan Helmkamp
60dad3c1cd Add DockerExecutionEnvironment for sandboxed agent tool execution
Implements ExecutionEnvironment trait backed by Docker containers via
bollard. Host working directory is bind-mounted; all file ops, commands,
grep, and glob execute inside the container via docker exec. Extracts
shared format_lines_numbered() helper from LocalExecutionEnvironment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:29:38 -05:00
Bryan Helmkamp
86935bbb2b Expose MultipleChoice options in API and support selected answers
ApiQuestion now includes options and allow_freeform fields so API clients
can render multiple-choice questions. submit_answer accepts an optional
selected_option_key to produce AnswerValue::Selected instead of always
creating AnswerValue::Text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:28:36 -05:00
Bryan Helmkamp
e4fc345912 Switch agent session from complete() to stream() to avoid request timeouts
Long Opus responses hit the 120s request timeout with complete(). With
stream(), the request timeout only covers initial connection + first
chunk, then the 30s stream_read timeout guards against stalls between
chunks. Also emits AssistantTextDelta events during streaming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:10:54 -05:00
Bryan Helmkamp
dba48f866e Fix CodergenHandler swallowing retryable errors, preventing engine retries
The handler was converting all backend errors into Ok(Outcome::fail(...)),
which the engine treats as a terminal result. Retryable errors (timeouts,
network failures) are now propagated as Err so the engine retry loop kicks in.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:43:48 -05:00
Bryan Helmkamp
5573e0e8a2 Fix truncated LLM responses by populating catalog max_output
Requests were always sent with max_tokens: None, which defaulted to
4096 in the Anthropic provider, causing large outputs to be truncated.
Now build_request() looks up the model's max_output from the catalog
and passes it through, giving each model its full output capacity.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:30:25 -05:00
Bryan Helmkamp
3a70e6711c serve 2026-02-23 12:20:48 -05:00
Bryan Helmkamp
dddc6df8c8 Extract terminal crate and prettify attractor CLI output
Move ANSI Styles struct from agent/cli.rs into a shared terminal crate
so both binaries can use it. Add green and yellow color codes. Prettify
all attractor CLI output: bold headers, colored diagnostics by severity,
green/red status, yellow warnings, dimmed event details, and styled
interviewer prompts. Move pipeline status output from stdout to stderr.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:06:24 -05:00
Bryan Helmkamp
b4397564ec Prettify agent CLI stderr output with ANSI colors
Add a Styles struct that pre-resolves ANSI escape codes based on whether
stderr is a TTY. Tool calls show as "● tool_name(args)" with bold+cyan
names, errors use red "✗", summaries are dimmed, and debug/verbose
middleware output is styled. No ANSI codes emitted when piped.

Also passes tool call arguments through to EventData::ToolCall so the
event handler can format them inline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:45:13 -05:00
Bryan Helmkamp
df8502d83b Merge agent-cli crate into agent as cli module
The agent-cli binary was a thin wrapper over the agent library with
nothing else depending on it. Moving it into the agent crate as a
`pub mod cli` with a `[[bin]]` entry reduces workspace complexity
and follows the attractor crate pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:14:25 -05:00
Bryan Helmkamp
eaf934f355 Add tests for tool_approval and agent-cli
- 5 session tests exercising ToolApprovalFn callback (deny, allow,
  arg capture, None passthrough, error event emission)
- 18 unit tests for agent-cli pure functions (tool_category,
  is_auto_approved, default_model, validate_api_key,
  build_tool_approval, build_profile)
- 4 integration tests for the agent binary (usage, help, missing
  API key, invalid permissions)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:05:44 -05:00
Bryan Helmkamp
0edc93e1c3 Add CLI tests for validate and dry-run on all test workflows
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:58:54 -05:00