# Cartography scout report: tests and evaluations

Assessed revision: `2bcf94fed8a9b429f18d9196fa824711d6f4cb0a` (`2bcf94fed`)

Scope: all 180 tracked files under `test/**` and `evals/**`. Cargo workspace
manifests, test consumers, repository documentation sources, and implementation
entry points were consulted only as boundary evidence and are not included in
this scope's counts.

Applicable instructions read: `AGENTS.md`, `CONTRIBUTING.md`, and
`docs/internal/testing-strategy.md` (`CLAUDE.md` is a symlink to `AGENTS.md`).

## Boundary decisions

- `twin-openai` and `twin-github` are separate components. Each is a distinct
  Cargo workspace member with its own protocol surface, router, state model,
  lifecycle, fixtures, and consumers. Their common use as local fake services
  is not enough to combine OpenAI scenario/stream behavior with GitHub
  repository/authentication behavior.
- The checked-in workflow fixtures outside `test/docs/**` are proposed as a
  shared `workflow-test-corpus` component. They are all user-facing workflow,
  configuration, prompt, and template inputs, and they are intentionally
  consumed across CLI, workflow, graph-language, rendering, and validation
  tests. Keeping them together avoids assigning shared compatibility data to
  one arbitrary production consumer.
- `test/docs/**` is proposed as a separate
  `documentation-workflow-tests` component. It has its own extraction and
  multi-phase runner entry points and owns a documentation-derived but curated
  executable corpus. The tracked fixtures are test source: the checklist
  records extracted, assembled, and adapted cases, and `run_tests.sh` executes
  them directly. They are therefore assigned rather than excluded as generated
  output.
- The SWE-bench tooling is a distinct evaluation component. It owns a
  generation, grading, monitoring, environment-generation, and result-recording
  workflow that is independent of the normal Cargo test lifecycle.
- `evals/swe-bench/scoreboard/**` is not executable evaluation source. The
  evaluation README calls it a Git-tracked permanent record, and
  `record_results.py` writes every tracked file shape beneath it. Those 16
  recorded outputs are proposed as a global exclusion.
- The two distribution shell tests and the benchmark-analysis SQL do not form a
  coherent component together. They are recommended additions to existing
  components, described after the component proposals.

## Proposed components

### `twin-openai` — OpenAI protocol twin

- **File count:** 35 (28 Rust, 5 Markdown, 1 Cargo manifest, 1 `.gitignore`)
- **Purpose:** Provides a deterministic OpenAI-compatible HTTP service for
  black-box and protocol-contract tests, including scripted successes,
  failures, streaming, request inspection, and live shape comparison.
- **Globs:** `test/twin/openai/**`
- **Exclude globs:** `[]`
- **Entry points:** `test/twin/openai/src/main.rs:main`,
  `test/twin/openai/src/lib.rs:build_app`,
  `test/twin/openai/src/lib.rs:build_app_with_config`,
  `test/twin/openai/src/app.rs:router`
- **Owns:** server bind/configuration lifecycle; `/v1/responses` and
  `/v1/chat/completions` request/response contracts; bearer-token namespaces;
  FIFO scenario queues; deterministic response IDs; normalized request logs;
  SSE construction and transport-failure behavior; admin reset/scenario APIs;
  debug UI and snapshots; local and opt-in live contract suites.
- **Depends on candidates:** `fabro-http`, `fabro-static`
- **Evidence:**
  - `Cargo.toml` — lists `test/twin/openai` as a workspace member and exposes
    `twin-openai` as a workspace dependency.
  - `test/twin/openai/Cargo.toml` — declares a non-published library/binary
    package described as a fake OpenAI-compatible server.
  - `test/twin/openai/src/app.rs:router` and
    `test/twin/openai/src/openai/mod.rs:router` — compose the health, OpenAI,
    admin, and debug HTTP surfaces.
  - `test/twin/openai/src/state.rs:AppState` — owns namespaced response
    counters, scenario queues, and request logs.
  - `test/twin/openai/src/engine/scenario.rs:ScenarioScript` — defines scripted
    success, application-error, delay, partial/malformed stream, and hang
    behavior.
  - `test/twin/openai/tests/common/mod.rs:spawn_server` and the eight sibling
    contract suites — exercise the service as a protocol boundary; the ignored
    `live_openai_contract.rs` compares supported protocol shapes with the live
    API.
  - `lib/foundation/fabro-test/Cargo.toml` and
    `lib/foundation/fabro-test/src/lib.rs:twin_openai` — show the shared
    integration-test harness consuming this package as an in-process service.

### `twin-github` — GitHub protocol twin

- **File count:** 20 (17 Rust, 2 PEM fixtures, 1 Cargo manifest)
- **Purpose:** Provides an in-process fake GitHub service with seeded mutable
  state and temporary Git repositories for black-box GitHub App, OAuth, API,
  GraphQL, and smart-HTTP tests.
- **Globs:** `test/twin/github/**`
- **Exclude globs:** `[]`
- **Entry points:** `test/twin/github/src/server.rs:TestServer::start`,
  `test/twin/github/src/server.rs:build_router`,
  `test/twin/github/src/state.rs:AppState`,
  `test/twin/github/src/fixtures.rs:FixtureState::into_app_state`
- **Owns:** ephemeral listener and shutdown lifecycle; temporary bare Git
  repositories; fake apps, installations, repositories, branches, pull
  requests, releases, projects, comments, webhook configuration, manifest
  conversions, access tokens, OAuth codes/tokens/users; GitHub authentication
  checks; bundled test RSA key pair.
- **Depends on candidates:** `fabro-http`
- **Evidence:**
  - `Cargo.toml` — lists `test/twin/github` independently as a workspace member
    and workspace dependency.
  - `test/twin/github/Cargo.toml` — declares a non-published library package
    described as a fake GitHub API server.
  - `test/twin/github/src/handlers/mod.rs:build_router` — registers the GitHub
    App, installation, branch, pull-request, manifest, OAuth, user, release,
    GraphQL, and Git smart-HTTP routes.
  - `test/twin/github/src/state.rs:AppState` — owns the central seeded and
    mutable GitHub-domain state.
  - `test/twin/github/src/server.rs:TestServer::start` — initializes temporary
    Git repositories, binds an ephemeral listener, and controls graceful
    shutdown.
  - `test/twin/github/src/fixtures.rs:FixtureState` and
    `test/twin/github/src/testdata/*.pem` — define reusable seeded service data
    and the owned authentication fixtures.
  - `lib/foundation/fabro-test/src/lib.rs:TwinGitHub` and
    `lib/apps/fabro-cli/tests/it/support/auth_harness.rs` — show this twin
    serving the CLI/server authentication integration boundary.

### `workflow-test-corpus` — Shared workflow compatibility fixtures

- **File count:** 42
  - 8 root `test/*.fabro` workflows
  - 14 `test/attractor/*.dot` compatibility graphs
  - 3 `test/dot-compatibility/*.fabro` graphs
  - 17 templating/configuration files under the four templated fixture trees
- **Purpose:** Supplies reusable user-facing workflow, compatibility,
  configuration, prompt, partial, and template inputs to cross-crate parser,
  validator, renderer, workflow, and CLI tests.
- **Globs:** `test/*.fabro`, `test/attractor/**`,
  `test/dot-compatibility/**`, `test/templated_inputs/**`,
  `test/templated_unbound_imported/**`,
  `test/templated_unbound_partial/**`, `test/templates/**`
- **Exclude globs:** `[]`
- **Entry points:** `test/simple.fabro`,
  `test/attractor/simple_example.dot`,
  `test/dot-compatibility/acp-agent-chain.fabro`,
  `test/templates/static_dependencies/workflow.fabro`,
  `test/templates/sibling_partial/workflow.fabro`
- **Owns:** representative valid and invalid workflow shapes; branching,
  conditions, parallelism, styles, and legacy syntax cases; Attractor DOT
  compatibility graphs; shared DOT parse/render/validation cases; template
  input, import, include, sibling-partial, static-dependency, and
  missing-dependency fixture trees.
- **Depends on candidates:** `fabro-cli`, `fabro-graphviz`, `fabro-template`,
  `fabro-test`, `fabro-validate`, `fabro-workflow`
- **Evidence:**
  - `docs/internal/testing-strategy.md` — explicitly recognizes checked-in
    user-facing workflows, configs, prompts, and repository contents as shared
    fixtures.
  - `lib/foundation/fabro-test/src/lib.rs:TestContext::install_fixture` —
    resolves named inputs from the repository `test/` directory for isolated
    CLI tests.
  - `lib/apps/fabro-cli/tests/it/cmd/validate.rs` and
    `lib/apps/fabro-cli/tests/it/workflow/dry_run_examples.rs` — consume the
    root workflows and all templating fixture trees as black-box CLI inputs.
  - `lib/components/fabro-workflow/tests/it/attractor_compat.rs` — enumerates
    and parses every graph in `test/attractor/**`.
  - `lib/components/fabro-graphviz/src/render.rs:dot_compatibility_fixtures`
    and
    `lib/components/fabro-validate/src/lib.rs:dot_compatibility_fixtures` —
    independently enumerate the same three `test/dot-compatibility/**` inputs,
    establishing that corpus as shared rather than crate-local.

### `documentation-workflow-tests` — Documentation workflow conformance

- **File count:** 55 (40 Fabro workflows, 7 shell files, 5 Markdown files, 2
  run TOML files, 1 Python extractor)
- **Purpose:** Extracts, curates, validates, preflights, and executes workflow
  examples and companion files derived from Fabro documentation.
- **Globs:** `test/docs/**`
- **Exclude globs:** `[]`
- **Entry points:** `test/docs/run_tests.sh`,
  `test/docs/extract_dots.py:main`, `test/docs/CHECKLIST.md`
- **Owns:** documentation-example corpus layout; prompt and script stubs;
  variable-bearing run configurations; extraction naming and stub generation;
  validate/preflight/dry-run/live phase selection; parallel execution and
  temporary result/run directories; the documented corpus checklist.
- **Depends on candidates:** `fabro-cli`, `fabro-workflow`, the final
  documentation-site component
- **Evidence:**
  - `test/docs/run_tests.sh:run_one` — discovers all 40 tracked `*.fabro`
    examples and invokes the built `fabro` binary in validate, preflight,
    dry-run, model-specific, or full execution modes.
  - `test/docs/extract_dots.py:main` — reads documentation Markdown, extracts
    complete DOT graphs, and creates companion prompt stubs and run
    configurations under `test/docs`.
  - `test/docs/CHECKLIST.md` — documents the 40-example corpus, distinguishes
    extracted and assembled cases, records companion-file needs, and provides
    the runner commands.
  - `.claude/skills/docs/SKILL.md` — instructs documentation changes containing
    full DOT graphs to run `./test/docs/run_tests.sh validate`, tying this
    harness to the documentation change lifecycle.

### `swe-bench-evaluation` — SWE-bench evaluation workflow

- **File count:** 9 (6 Python scripts, 1 Fabro workflow, 1 requirements file, 1
  README)
- **Purpose:** Generates Fabro patches for SWE-bench Lite instances, grades
  them through Daytona or the official harness, monitors runs, and records
  normalized result summaries.
- **Globs:** `evals/swe-bench/*.py`, `evals/swe-bench/*.fabro`,
  `evals/swe-bench/*.txt`, `evals/swe-bench/README.md`
- **Exclude globs:** `[]` (the sibling scoreboard is a global exclusion)
- **Entry points:** `evals/swe-bench/run_eval.py:main`,
  `evals/swe-bench/evaluate_daytona.py:main`,
  `evals/swe-bench/evaluate.py:main`,
  `evals/swe-bench/record_results.py:main`,
  `evals/swe-bench/status.py:main`,
  `evals/swe-bench/gen_dockerfile.py:main`
- **Owns:** SWE-bench Lite dataset selection; per-instance goal/workflow/TOML
  generation; Daytona snapshot and sandbox specifications; Fabro subprocess
  orchestration and timeout cleanup; patch extraction; official and
  Daytona-based grading; progress summaries; scoreboard record schema and
  leaderboard regeneration.
- **Depends on candidates:** `fabro-cli`, `fabro-sandbox`,
  `fabro-workflow`
- **External dependencies:** Hugging Face `datasets`, the `swebench` harness,
  Daytona, and optionally Docker through the official harness.
- **Evidence:**
  - `evals/swe-bench/README.md` — defines the three-stage generate, evaluate,
    and record lifecycle, the two grading backends, and raw-versus-recorded
    result locations.
  - `evals/swe-bench/run_eval.py:run_instance` — creates per-instance Fabro
    workflows/configs, invokes `fabro run`, and extracts produced patches.
  - `evals/swe-bench/evaluate_daytona.py` — creates grading workflows and
    executes held-out tests in Daytona snapshots.
  - `evals/swe-bench/evaluate.py:main` — exposes the alternative official
    Docker-backed `swebench.harness.run_evaluation` path.
  - `evals/swe-bench/gen_dockerfile.py:generate_dockerfile` — translates
    SWE-bench repository/version specs into reusable sandbox images.
  - `evals/swe-bench/record_results.py:main` and
    `regenerate_leaderboard` — define and write the tracked scoreboard record
    formats.

## Recommended additions to existing components

These files are assigned in the coverage accounting but do not justify new
components:

| File | Recommended component | Reason |
| --- | --- | --- |
| `test/bin/install_test.sh` | documentation/web scout's marketing-site component | It is a black-box shell contract test whose sole product target is `apps/marketing/public/install.sh`; it owns a fake `gh` executable and temporary install home only for that test. |
| `test/bin/release_test.sh` | `fabro-build-tooling` | It is an executable release-mode shell contract and changes with the repository release-automation lifecycle. |
| `test/analysis/bench-tests-diff.sql` | `fabro-build-tooling` | Its documented inputs are the two CSVs produced by `cargo dev bench-tests`, whose implementation is `lib/foundation/fabro-dev/src/commands/bench_tests.rs`. |

## Global exclusion

### Recorded SWE-bench scoreboards

- **Globs:** `evals/swe-bench/scoreboard/**`
- **Tracked files:** 16 (1 leaderboard JSON plus 5 run directories containing
  one `README.md`, one `meta.json`, and one `instances.jsonl` each)
- **Reason:** committed evaluation records generated by
  `evals/swe-bench/record_results.py`, not executable evaluation source.
- **Evidence:** `evals/swe-bench/README.md` calls the directory a Git-tracked
  permanent record; `record_results.py` writes `instances.jsonl`, `meta.json`,
  each run `README.md`, and regenerates `leaderboard.json`.

Raw `evals/swe-bench/results/**` data is also described as generated output,
but it is not tracked at the assessed revision and therefore is not part of
the 180-file inventory.

No `test/docs/**` files are excluded. Although the extractor derives some
files from documentation, the tracked corpus includes assembled/adapted
executable fixtures and companion stubs/configuration, and the runner consumes
those files as test inputs.

## Coverage

| Assignment | Tracked files |
| --- | ---: |
| `twin-openai` | 35 |
| `twin-github` | 20 |
| `workflow-test-corpus` | 42 |
| `documentation-workflow-tests` | 55 |
| `swe-bench-evaluation` | 9 |
| Recommended addition to marketing-site component | 1 |
| Recommended additions to `fabro-build-tooling` | 2 |
| Global exclusion: SWE-bench scoreboards | 16 |
| **Scoped inventory** | **180** |

- **Assigned:** 164 (161 in proposed test/evaluation components and 3 additions
  to existing components)
- **Excluded:** 16
- **Unmapped:** 0
- **Overlap:** 0
- **Accounting check:** `164 + 16 + 0 = 180`
- **Unmapped files:** `[]`

## External boundary evidence consulted

These files are outside the scoped inventory and are neither assigned nor
counted as unmapped:

- `Cargo.toml` — workspace membership and workspace dependency declarations for
  both twin services.
- `lib/foundation/fabro-test/Cargo.toml` and
  `lib/foundation/fabro-test/src/lib.rs` — shared fixture installation and twin
  service consumption.
- `lib/apps/fabro-cli/tests/it/**` — black-box workflow fixture and twin-GitHub
  consumers.
- `lib/components/fabro-workflow/tests/it/attractor_compat.rs` — Attractor
  corpus consumer.
- `lib/components/fabro-graphviz/src/render.rs` and
  `lib/components/fabro-validate/src/lib.rs` — shared DOT compatibility corpus
  consumers.
- `docs/internal/testing-strategy.md` — repository test-layer and fixture
  ownership policy.
- `.claude/skills/docs/SKILL.md` — documentation test-runner invocation policy.
- `apps/marketing/public/install.sh` — install shell-test target.
- `lib/foundation/fabro-dev/src/commands/bench_tests.rs` — benchmark CSV
  producer consumed by the analysis SQL.

## Genuine boundary questions

1. Should `workflow-test-corpus` remain a distinct 42-file shared data
   component, as proposed, or should reconciliation distribute its three
   subcorpora to `fabro-cli` (25 general/template fixtures),
   `fabro-workflow` (14 Attractor fixtures), and `fabro-graphviz` (3 shared DOT
   compatibility fixtures)? The cross-crate consumers support a shared
   boundary, while the production behaviors they exercise support attachment.
2. Should `documentation-workflow-tests` remain a separate executable harness,
   or should its 55 files be included in the documentation-site component?
   Its runner and phase lifecycle support separation; its source derivation and
   documentation-change trigger support inclusion with documentation.
3. Should `test/bin/release_test.sh` be assigned to `fabro-build-tooling` as a
   release-lifecycle contract, or remain separately unmapped until the final
   map determines which current release entry point owns that shell contract?