Commit graph

16 commits

Author SHA1 Message Date
Bryan Helmkamp
3d8e17aedf Write progress.ndjson and live.json to logs dir during pipeline runs
Every PipelineEvent is now logged as a JSON envelope with timestamp,
run_id, and event fields — matching the Kilroy reference implementation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:50:19 -05:00
Bryan Helmkamp
9665fad133 Revert clippy config to defaults, remove all pedantic/nursery/cargo lint suppressions
Removed the workspace-level clippy lint config that enabled all, pedantic, nursery,
and cargo lint groups. Removed all #[allow(clippy::...)] annotations that were only
needed to suppress those extra lints, and fixed the few default clippy warnings that
were uncovered.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:42:28 -05:00
Bryan Helmkamp
28cdefec67 Add token usage and cost tracking to pipeline runs
Surface LLM token consumption and dollar cost at every verbosity level:
default mode appends per-stage tokens/cost, verbose modes include it in
event summary/detail, and the Pipeline Result section shows a total.
Cost is computed from the catalog pricing for Anthropic models; providers
without pricing (OpenAI, Gemini) show token counts only. Dry runs with
zero tokens omit the cost line entirely.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:01:17 -05:00
Bryan Helmkamp
b4068b6364 Add missing integration tests for multi-turn caching, cross-provider parity, and attractor E2E
Test 1 (llm crate): Multi-turn cache verification runs 6 conversation turns
with a large system prompt (~5460 tokens) and verifies cache_read_tokens on
the final turn. Anthropic threshold 0.5, OpenAI/Gemini 0.0 (automatic
caching not guaranteed).

Test 2 (agent crate): Cross-provider parity matrix with 15 scenarios
(file CRUD, shell, grep/glob, editing, steering, reasoning effort, loop
detection, error recovery, etc.) across Anthropic, OpenAI, and Gemini.
41 total tests. Some scenarios excluded for OpenAI due to gpt-4o-mini
limitations (no reasoning.effort, is_error rejection, weak editing).

Test 3 (attractor crate): E2E pipeline with real LLM using AgentBackend,
AutoApproveInterviewer, and default_registry. Verifies pipeline success,
artifact files, goal gate outcomes, and checkpoint state.

All tests are #[ignore] and require API keys to run.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:24:34 -05:00
Bryan Helmkamp
3dfa0f9044 Add shape selectors, LLM stream timeouts, MultiSelect questions, and test coverage
- Stylesheet: add bare-word Shape selector (specificity between Universal and Class)
- LLM: apply per_step timeout to connection and total timeout to stream (Section 4.7)
- Interviewer: add MultiSelect question type alongside MultipleChoice
- Session: move SessionStart/SessionEnd to initialize()/close(), deduplicate close logic
- Engine: return Ok(fail outcome) instead of error when goal gate unsatisfied with no retry_target
- Docker: mark Docker-dependent tests with #[ignore]
- Validation: add extensive unit test coverage for all rule types
- Integration: update tests to match engine/fidelity changes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:43:05 -05:00
Bryan Helmkamp
a3605ec0d4 Speed up slow tests: drop SSE broadcast on completion, switch to rustls-tls, replace hardcoded sleeps with poll loops, and prevent real API calls in ullm tests
- Drop event_tx from ManagedPipeline when pipeline completes/cancels/fails so
  SSE streams end promptly instead of blocking until timeout (3.5s → 0.02s)
- Switch reqwest from native-tls to rustls-tls to avoid 500ms macOS cert store
  load per process (0.67s → 0.005s per OpenAI adapter test)
- Replace hardcoded sleep(500ms)/sleep(200ms)/sleep(100ms) in server and
  integration tests with 10ms poll loops (0.2-0.5s → 0.02-0.03s each)
- Add env_clear() to ullm prompt tests to prevent .env from triggering real
  Anthropic API calls (0.45s → 0.15s)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:19:09 -05:00
Bryan Helmkamp
95a4ab24d5 Thread EngineServices through Handler::execute() to eliminate circular dependency
ParallelHandler and SubPipelineHandler needed Arc<HandlerRegistry> at
construction time but also lived inside the registry, creating a circular
dependency. The previous fix special-cased ParallelHandler as a separate
field on PipelineEngine with a resolve_handler() override.

Instead, add an EngineServices struct (registry + emitter) passed through
Handler::execute(). ParallelHandler and SubPipelineHandler become unit
structs that get what they need at execution time. No special-casing,
both register normally in default_registry() like every other handler.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:31:55 -05:00
Bryan Helmkamp
3a70e6711c serve 2026-02-23 12:20:48 -05:00
Bryan Helmkamp
dddc6df8c8 Extract terminal crate and prettify attractor CLI output
Move ANSI Styles struct from agent/cli.rs into a shared terminal crate
so both binaries can use it. Add green and yellow color codes. Prettify
all attractor CLI output: bold headers, colored diagnostics by severity,
green/red status, yellow warnings, dimmed event details, and styled
interviewer prompts. Move pipeline status output from stdout to stderr.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:06:24 -05:00
Bryan Helmkamp
0edc93e1c3 Add CLI tests for validate and dry-run on all test workflows
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 10:58:54 -05:00
Bryan Helmkamp
4bb2de5a49 Rename unified-llm crate to llm
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 09:22:14 -05:00
Bryan Helmkamp
a917a8d615 Implement 6 spec gaps: preamble synthesis, thread_id plumbing, cancellation, recording replay, retry presets, inform()
- Gap 1: Real preamble synthesis per fidelity mode (truncate, compact,
  summary:low/medium/high) in new preamble.rs module, replacing placeholder
- Gap 2: Pass thread_id to CodergenBackend.run() so backends can reuse
  LLM sessions across nodes sharing the same thread
- Gap 7: Engine cancellation via AtomicBool token checked between nodes,
  wired to server cancel endpoint, new Cancelled error variant
- Gap 8: RecordingInterviewer serialization (to_json/from_json, file I/O)
  and new ReplayInterviewer for replaying recorded Q&A sessions
- Gap 12: Preset retry policies selectable from DOT via retry_policy attr
  (none, standard, aggressive, linear, patient)
- Gap 15: Engine calls Interviewer.inform() at pipeline start, stage
  start, and stage complete lifecycle points

37 new tests (450 unit + 49 integration, all passing).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 08:23:27 -05:00
Bryan Helmkamp
08d76b93de Add 8 integration tests: HTTP lifecycle, SSE events, sub-pipeline, manager loop, graph merge, real LLM
- Change server registry_factory to accept Arc<dyn Interviewer> so WaitHumanHandler
  shares the same WebInterviewer as the REST API endpoints
- Add full HTTP lifecycle tests: approve-and-complete flow + cancel flow
- Add SSE event stream content parsing test with frame-level verification
- Add sub-pipeline E2E test through the engine with context propagation
- Add manager loop E2E test with SimulatingChildObserver
- Add graph merge E2E test verifying module prefixing and execution ordering
- Add 3 real LLM tests (#[ignore]) using claude-haiku via AnthropicAdapter
- Add dotenvy and http-body-util dev dependencies

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 13:07:26 -04:00
Bryan Helmkamp
c15b532bcf Implement three missing spec features: SubPipelineHandler, GraphMergeTransform, HTTP Server
Add SubPipelineHandler (handler/sub_pipeline.rs) that inline-executes a parsed
sub-graph within the same engine, reading DOT source from node attributes and
propagating context diffs back to the parent pipeline.

Add GraphMergeTransform (transform.rs) that merges nodes and edges from secondary
graphs into a primary graph with namespace-prefixed IDs to avoid collisions.

Add WebInterviewer (interviewer/web.rs) backed by oneshot channels for async
question/answer flow, and HTTP server (server.rs) with 8 axum endpoints behind
a "server" feature flag for pipeline management and human-in-the-loop via web.

Fix tempfile dev-dependency usage in server production code by using std::env::temp_dir.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 12:07:44 -04:00
Bryan Helmkamp
a2f9e87b7c Close spec compliance gaps: retry context, thread resolution, fan-in logging
Fix 3 confirmed gaps from spec compliance review (85 items, 91.8% aligned):

- Write internal.retry_count.<node_id> to PipelineContext after retries
  so handlers and conditions can access retry counts (spec 5.1)
- Add graph-level default_thread (step 3) to 5-step thread ID resolution,
  pass graph param to resolve_thread_id (spec 5.4)
- Write prompt.md/response.md in fan_in LLM evaluation path (spec 5.6)

Also includes pre-existing improvements: checkpoint stores node_outcomes
and next_node_id for correct resume, engine timeout enforcement,
auto_status support, fidelity degradation on resume, preamble injection,
is_retryable error classification, stylesheet specificity correction,
full stylesheet parse validation, direction_valid lint rule, pre-hook
returns Skipped not Fail, fan-in score-based sorting and all-fail
detection, manager_loop child autostart and steer cooldown.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 10:49:21 -04:00
Bryan Helmkamp
34836200e5 Implement attractor crate: DOT-based pipeline runner with full spec compliance
Adds the attractor crate implementing all 11 sections of the attractor spec:
- DOT parser (lexer, grammar, semantic analysis) for strict DOT subset
- Pipeline execution engine with edge selection, goal gates, retry logic,
  failure routing, checkpoint save/resume, and loop_restart
- 9 node handlers: start, exit, codergen, wait_human, conditional, parallel
  (concurrent with join/error policies), fan_in (with LLM eval), tool, manager_loop
- State management: PipelineContext, Outcome, Artifact store, fidelity resolution
- Human-in-the-loop: Interviewer trait with auto_approve, callback, queue,
  recording, and console implementations, plus timeout enforcement
- Validation: 14 built-in lint rules with custom rule registration API
- Model stylesheet with universal/shape/class/ID selectors and specificity
- Transforms: variable expansion, stylesheet application, preamble; plus
  PipelineBuilder with register_transform and prepare_pipeline
- Condition expression language with =, !=, bare-key truthiness, && combinator
- Event system with all 16 event types emitted by engine and handlers
- Tool call hooks (pre/post) for CodergenHandler
- Run directory with manifest.json and per-node status.json

370 tests (354 unit + 16 integration) covering all spec sections.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 12:17:32 -04:00