GitNexus/TESTING.md

# Testing — GitNexus

How we structure tests and which commands to run locally and in CI.

## Packages

| Package        | Path            | Runner     | Notes                      |
| -------------- | --------------- | ---------- | -------------------------- |
| CLI + MCP core | `gitnexus/`     | Vitest     | Primary test surface in CI |
| Web UI         | `gitnexus-web/` | Vitest     | Unit/component tests       |
| Web UI E2E     | `gitnexus-web/` | Playwright | Run when changing UI flows |

## Test lanes

### `gitnexus/` commands

From `gitnexus/`:

| Command                       | What it runs                                       | When to use                           |
| ----------------------------- | -------------------------------------------------- | ------------------------------------- |
| `npm test`                    | Full suite (all 3 vitest projects)                 | Before opening a PR                   |
| `npm run typecheck:tests`     | TypeScript checks for source, tests, and helpers   | Before opening a PR                   |
| `npm run test:unit`           | Unit tests only (`test/unit/`)                     | Tight development loop                |
| `npm run test:integration`    | Integration tests (`test/integration/`)            | After changing pipelines, DB, workers |
| `npm run test:coverage`       | Full suite + v8 coverage with thresholds           | Checking coverage impact              |
| `npm run test:parity`         | Scope-resolution parity for all migrated languages | After changing resolver or scope code |
| `npm run test:cross-platform` | Platform-sensitive subset only                     | Debugging a Windows/macOS issue       |
| `npm run test:watch`          | Vitest in watch mode                               | Active development                    |
| `npm run test:dupes`          | Duplicate-test scan of lines changed vs `main`     | Before adding tests or benchmarks     |

Vitest transpiles TypeScript without checking types. Run `npm run typecheck:tests`
in `gitnexus/` to check test code and helpers against the production types. This
uses `tsc --noEmit -p tsconfig.test.json`; parser input files under `test/fixtures/`
are excluded because they are sample source code, not part of the test program.

### `gitnexus-web/` commands

From `gitnexus-web/`:

| Command                 | What it runs                  | When to use                                                         |
| ----------------------- | ----------------------------- | ------------------------------------------------------------------- |
| `npm test`              | Unit/component tests (vitest) | After changing web code                                             |
| `npm run test:coverage` | Unit tests + coverage         | Checking coverage impact                                            |
| `npm run test:e2e`      | Playwright browser tests      | After changing UI flows (requires `gitnexus serve` + `npm run dev`) |

### Before opening a PR

```bash
# gitnexus-shared/dist must exist first. `npm install` / `npm run build` in
# gitnexus/ compiles it via parent `lib/tsc.js` (do not npm ci gitnexus-shared).
cd gitnexus && npx tsc --noEmit && npm run typecheck:tests && npm test
cd ../gitnexus-web && npx tsc -b --noEmit && npm test
```

## Pre-commit hook

A husky pre-commit hook (`.husky/pre-commit`) runs automatically on every `git commit`:

1. **Formatting** — `lint-staged` runs prettier on staged files
2. **`gitnexus-web/` files staged** → `tsc -b --noEmit`
3. **`gitnexus/` files staged** → `tsc --noEmit`

Tests do **not** run in the pre-commit hook — they run in CI (`ci-tests.yml`) only.

Skip with `git commit --no-verify` (use sparingly).

## Vitest projects

`gitnexus/vitest.config.ts` defines three projects for safety isolation:

| Project   | Files                                              | Parallelism | Purpose                                             |
| --------- | -------------------------------------------------- | ----------- | --------------------------------------------------- |
| `lbug-db` | Native LadybugDB integration tests (explicit list) | Sequential  | Prevents file-lock conflicts from native mmap addon |
| `cli-e2e` | `skills-e2e.test.ts`                               | Sequential  | CLI process spawning requires serial execution      |
| `default` | Everything else                                    | Parallel    | Fast execution for pure logic and parser tests      |

When adding a new test that uses native LadybugDB (`@ladybugdb/core`), add it to the `lbug-db` project's explicit include list and the `default` project's exclude list.

## Test categories

- **Unit** — Pure logic, parsers, graph/query helpers; fast; no network.
- **Integration** — Real combinations (filesystem, MCP wiring, larger pipelines) as already organized under `gitnexus/test/integration`.
- **Resolver / parity** — Language-specific call-resolution tests in `test/integration/resolvers/`.
- **E2E (web)** — Critical user paths only; prefer `data-testid` attributes for stable selectors. Tests run against real backend (`gitnexus serve`) and Vite dev server.

## Scope-resolution tests

Every language resolves calls and inheritance through the scope-resolution pipeline — the legacy call-resolution DAG and the per-language `REGISTRY_PRIMARY_<LANG>` flag were removed in RING4-1 (#942). Each language's resolver test lives at `test/integration/resolvers/<slug>.test.ts` and runs once, on the single scope-resolution path, as part of the normal `tests` job (`vitest test/**/*.test.ts`).

Adding a language: register its `ScopeResolver` in `scope-resolution/pipeline/registry.ts` (`SCOPE_RESOLVERS`) and add the resolver test file — no workflow or config edit needed.

## Cross-platform testing

Windows and macOS CI runs only the platform-sensitive test subset (~50 files out of 373). The full suite runs on Ubuntu.

The subset is defined in `gitnexus/scripts/cross-platform-tests.ts` and includes:

- **Platform-specific logic** — tests with `process.platform` guards, path.sep behavior, EPERM/EBUSY error classification
- **Native LadybugDB** — all `lbug-*` integration tests (N-API addon with known platform-varying behavior)
- **Process spawning / CLI** — tests using real `child_process.spawn`, shell quoting, CLI invocations
- **Worker threads** — tests spawning real `worker_threads`
- **Native addon loading** — tree-sitter grammar loading smoke tests
- **Filesystem behavior** — CRLF handling, directory walking, symlinks

When adding a platform-sensitive test, add it to the appropriate section in `scripts/cross-platform-tests.ts`.

### Automatic shard balancing

CI uses `vitest-shard-balancer` to distribute test files by measured duration.
Timing history lives in GitHub Actions storage; adding a test does not require
editing a weight table. Windows, macOS, and Ubuntu coverage each have their own
profile because native startup and coverage costs differ.

The `shard-plan` job restores history once and uploads `shard-timing-input`.
Every shard and report merge downloads that same snapshot. Retries reuse it,
so a newly published history cannot change the partition halfway through a run.
Fork approval can increment the attempt number before any job executes. Attempt
2 initializes a snapshot only when GitHub confirms attempt 1 required approval
and ran zero jobs. Later retries always require the frozen snapshot.

If the snapshot cannot be restored, the retry fails. If it was never uploaded,
deleted, or expired, trigger a fresh workflow run with a new commit; rerunning
the old run must not replace its timing input.
With no usable history, the sequencer spreads sequential/native work using
generic estimates; new files in an established profile use the package's median.

The merge reporter includes setup, collection, and test execution costs and
smooths new measurements with previous ones. It rejects missing, duplicate, or
unexpected modules and unsuccessful runs, and retains prior costs for skipped
modules. Native blobs are merged on their originating OS. Only main-branch push
runs that pass the execution audit publish the updated history to the Actions
cache. Cache expiry affects balance, not test discovery or completeness.

For a local reproduction, download the run's `shard-timing-input` artifact into
`gitnexus/.vitest/timings/` and set `GITNEXUS_SHARD_PROFILE` to `windows-latest`,
`macos-latest`, or `coverage` (the default). Then use the same `--shard=i/n`
argument with `npm run test:cross-platform -- --shard=i/n` or `npm test -- --shard=i/n`.
Keep `.vitest/` untracked. Bump the workflow's timing-cache schema key when a
runner/measurement change makes the stored profiles incompatible.

### Confirming no tests are orphaned

Every test file matches one of the three vitest projects. To verify:

```bash
cd gitnexus
npx vitest list 2>/dev/null | wc -l  # should match total test count
```

To check the cross-platform list is up to date, run `npm run test:cross-platform` — it fails fast if any listed file is missing.

## CI integration

GitHub Actions (`.github/workflows/ci.yml`) orchestrate:

| Workflow              | Jobs                                                              | Purpose                                                               |
| --------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------- |
| `ci-quality.yml`      | format, lint, typecheck, typecheck-web, workflow-convention       | Code quality gates                                                    |
| `ci-tests.yml`        | ubuntu/coverage, cross-platform (Win/Mac), packaged-install-smoke | Full suite + coverage on Ubuntu; platform-sensitive subset on Win/Mac |
| `ci-scope-parity.yml` | discover, parity                                                  | Scope-resolution parity for all migrated languages                    |
| `ci-e2e.yml`          | e2e (chromium)                                                    | Playwright E2E, gated on `gitnexus-web/**` changes                    |

The `CI Gate` job in `ci.yml` requires the quality and test workflows to pass.
The browser E2E workflow must pass or be skipped because no web files changed.

Branch protection also requires six platform check names from the former
three-shard matrix. These names remain as aggregate gates: all native shards
and the `every test executed` audit must succeed before any of them passes.
Failed, cancelled, skipped, or missing dependency results fail these gates.
The actual native tests run in the current Windows/macOS shard matrix.

The `typecheck` job runs both the production compiler check and
`npm run typecheck:tests`. A type error in either check fails the job and the CI gate.

### Complete execution, including platform and benchmark tests

The required `every test executed` job reconciles execution receipts from Ubuntu
coverage, every Windows/macOS shard, the serial benchmark run, and the real Python
workflow preflight. It requires a recorded pass for every collected test. A skip
on Linux is satisfied only by a pass of that exact test in another required job.
Test identities include the file, suite/title and source location; ambiguous
parameterized cases must have unique titles. Missing receipts, missing test files,
unhandled runner errors, failed hooks, failed assertions, and tests with no pass
all fail the gate. Web tests are checked separately with the same rules.

The locked pytest suite and Linux/Windows containment jobs also upload JUnit
receipts. The same gate checks every Python test file was collected and every
case passed in at least one job, while preserving failures from any job. The
Linux containment job supplies Bubblewrap, the pinned CLI, and built Vitest
dependencies for tests that cannot run in the basic Python job. Python results
are included in the combined PR report.

The PR report shows the reconciled result as **Unverified**. Zero means every test
has execution evidence; individual OS logs still show tests that require another
OS as skipped. Failed executions remain failures even if another job passes.

### Duplicate-test scan (advisory)

The `Duplicate test scan` job in `ci.yml` runs `npm run test:dupes` on the lines a PR
changes, and the PR report lists what it finds under **Possible duplicate tests**. It
never fails a build. It reports:

| Kind                 | Meaning                                                                     |
| -------------------- | --------------------------------------------------------------------------- |
| `same-test`          | Same body and same setup as an existing test                                |
| `assertions-pinned`  | Every assertion is a literal `fn(...)` → matcher another test already makes |
| `same-fixture-setup` | A `before*` hook builds a fixture another describe already builds           |
| `copied-helper`      | A top-level helper identical to one in another file or `test/helpers`       |
| `orphan-bench-gate`  | A bench script with `--check` that no workflow or npm script runs           |

"Same setup" includes enclosing hooks, describe-level statements, wrapper arguments,
loop heads, `each` tables, referenced file-local declarations and resolved imports, so
identical bodies that run against different fixtures are not reported. Mark an
intentional repeat with `// dupe-ok: <reason>` on the reported line or the line above.
Locally, `npm run test:dupes` compares against `origin/main`; pass `--base <ref>` to
change that, or `--all` to scan every test. Python tests are not scanned.

`npm run test:benchmarks` discovers all tests gated by `GITNEXUS_BENCH` and runs
them serially. Keep timing measurements out of parallel coverage workers. The
`eval-tests` job installs the locked Python dependencies and runs the workflow
preflight Vitest tests as well as pytest; those tests must exercise real Python
validation, never a stubbed success.

## Mutation testing tooling

`gitnexus/tools/mutation/` holds a Stryker test runner for Vitest 5 (`vitest5`) and the tools
that validate it. The runner supports Vitest `5.0.x` only and refuses any other version. Its
validated scope is the pure-JS category: 251 unit test files (4,667 tests) that load no native
code, workers or child processes and read no fixtures. The gates stay out of the main suite
(only the tools' unit tests in `test/unit/mutation/` run there), and nothing ships in the npm
package.

Every run works in a prepared copy outside the checkout (`--out`/`--work` must lie outside it).
The copy blanks each `as` type assertion in the mutated files with spaces of equal length,
because Stryker never mutates code under one, and relocates `gitnexus-shared/src` into the
package. From `gitnexus/` (each tool documents its options in its file header):

| Command                                | What it does                                                          |
| -------------------------------------- | --------------------------------------------------------------------- |
| `npm run test:mutation:run`            | One Stryker run over a scope file with the `vitest5` runner           |
| `npm run test:mutation:classify`       | Lists the pure-JS unit test files                                     |
| `npm run test:mutation:fidelity`       | Gate 1: the unmutated run matches plain Vitest; records the envelope  |
| `npm run test:mutation:conformance`    | Gate 2: pinned outcomes of the conformance fixture project            |
| `npm run test:mutation:oracle`         | Gate 3: compares runner runs with the frozen oracle manifests         |
| `npm run test:mutation:oracle:freeze`  | Freezes (or re-freezes) one oracle manifest                           |
| `npm run test:mutation:oracle:command` | The oracle's own Stryker run, with Stryker's `command` runner         |
| `npm run test:mutation:controls`       | Gate 4: planted positive controls and determinism across two profiles |
| `npm run test:mutation:bench`          | Oracle cost benchmark                                                 |
| `npm run test:mutation:prepare`        | Builds a prepared copy on its own                                     |

Frozen artifacts live in `gitnexus/test/fixtures/`: `mutation-conformance/` (the gate 2
fixture project and `expected.json`), `mutation-oracle/` (the gate 1 envelope and one gate 3
manifest per oracle scope: `shared`, `schema-constants`, `middleware`) and
`mutation-controls/planted.json` (gate 4).

Re-freeze a manifest when gate 3 reports "re-freeze required", that is after a change to a
scope's tests or their imports, the Vitest configs, the prepare tool, or the Stryker, Vitest or
Node major version. Use the `refreeze` command recorded in the manifest, with a new seed.
Re-select the controls when gate 4 reports "re-select required". After a Vitest, Stryker or Node
upgrade, re-run all four gates. Refresh `expected.json` from the conformance `observed.json`
only for an intended change.

`.github/workflows/mutation-runner-validation.yml` runs gate 2 on PRs that touch the tooling,
its fixtures, `gitnexus/package.json`, the lockfile or `gitnexus/vitest.config.ts`. A manual
dispatch also runs gates 1, 3 and 4 and the benchmark, or re-freezes oracle manifests sharded
by scope and uploads them for a maintainer to commit. The workflow is advisory.

### Test Scenario Map

`gitnexus/tools/scenario-map/` builds a report on the eligible pure-JS tests: tests that repeat
a scenario, tests that kill no evaluated mutant, and per-file gaps. It runs Layers 0–3 of the
design (`docs/plans/2026-10-23-test-scenario-map-design.md`) on the `vitest5` runner. It only
reports. It never edits, deletes or folds a test.

From `gitnexus/`, run `npm run test:scenario-map -- <stage> --work <dir>`. The work directory
must lie outside the checkout. Every stage reads and writes JSON there, so a local `all` run and
CI produce the same report. `value-flow`, `all` and `controls` need a built `dist/`
(`npm run build`). The CLI header (`tools/scenario-map/scenario-map.ts`) documents every flag
and file.

| Stage        | What it does                                                         | Writes (under `--work`)                                                                    |
| ------------ | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `plan`       | Eligibility, Layers 0–2, the run manifest, candidates, shard plan    | `plan.json`, `scenario-run-manifest.json`, `layer0.json`, `layer2.json`, `coverage/`       |
| `value-flow` | PDG index, static value-flow check, trace run, final shard plan      | `value-flow.json`                                                                          |
| `shards`     | Prints the final shard count (the CI matrix size)                    | Nothing                                                                                    |
| `shard`      | `--index <i> --of <n>`: Layer 3 runs, row deletion, isolation pass   | `observations/`, `batches/`, `isolation/`, `row-deletion-shard-<i>.json`, `shard-<i>.json` |
| `merge`      | Merges every result, computes verdicts, compares with `--baseline`   | `scenario-map-report.json`, `scenario-map-report.md`, `issue-body.md`                      |
| `all`        | `plan` (one shard), `value-flow`, `shard` 0 of 1 and `merge`         | All of the above                                                                           |
| `controls`   | `all` over the controls fixture, then compares every expected result | All of the above                                                                           |

Exit code 0 means complete results. Exit code 1 means the stage finished with incomplete results,
or a controls verdict differs. Exit code 2 is a usage or setup error.

The controls fixture is `gitnexus/test/fixtures/mutation-scenario-map/`: one pair per
`*.case.ts` file (16 pairs). It holds the pure-JS adversarial pairs of design §8, an
order-dependent pair, an object-key row and positive controls. `expected.json` was written from
the design before any pipeline run. Never regenerate it from observed output: a mismatch is a
finding. The fixture has its own Vitest config and tsconfig, and no repo project collects it.
The `scenario-map-controls` job in `.github/workflows/mutation-runner-validation.yml` runs
`controls` on the same PRs as gate 2.

**Supported matchers** (`SUPPORTED_MATCHERS` in `tools/scenario-map/facet/checks.ts`), after
`expect(…)` or `expect.soft(…)` and an optional `.not`, `.resolves` or `.rejects`:
`toBe`, `toEqual`, `toStrictEqual`, `toMatchObject`, `toBeTruthy`, `toBeFalsy`, `toBeNull`,
`toBeUndefined`, `toBeDefined`, `toBeNaN`, `toBeNullable`, `toBeTypeOf`, `toBeInstanceOf`,
`toBeGreaterThan`, `toBeGreaterThanOrEqual`, `toBeLessThan`, `toBeLessThanOrEqual`,
`toBeCloseTo`, `toBeOneOf`, `toContain`, `toContainEqual`, `toHaveLength`, `toHaveProperty`,
`toMatch`, `toThrow`, `toThrowError`, `toHaveBeenCalled`, `toHaveBeenCalledOnce`,
`toHaveBeenCalledTimes`, `toHaveBeenCalledWith`, `toHaveBeenCalledExactlyOnceWith`,
`toHaveBeenLastCalledWith`, `toHaveBeenNthCalledWith`, `toHaveBeenCalledBefore`,
`toHaveBeenCalledAfter`, `toBeCalled`, `toBeCalledTimes`, `toBeCalledWith`, `toHaveReturned`,
`toHaveReturnedTimes`, `toHaveReturnedWith`, `toHaveLastReturnedWith`, `toHaveNthReturnedWith`,
`toReturn`, `toReturnTimes`, `toReturnWith`, `toHaveResolved`, `toHaveResolvedTimes`,
`toHaveResolvedWith`, `toHaveLastResolvedWith`, `toHaveNthResolvedWith`, and the snapshot
matchers `toMatchSnapshot`, `toMatchInlineSnapshot`, `toMatchFileSnapshot`,
`toThrowErrorMatchingSnapshot`, `toThrowErrorMatchingInlineSnapshot`. `expect.assertions(n)` and
`expect.hasAssertions()` count as checks. Any other form is unsupported and blocks strong
evidence: `assert.*`, chai-style chains, custom matchers, `toSatisfy`, and an `expect` inside a
loop, a conditional, a catch clause, a nested function or a helper.

**Wrapped built-ins** of the trace run (`BASE_WRAP_LIST` in
`tools/scenario-map/value-flow/builtins.ts`), by group:

- `math`: every `Math` function except `random`.
- `string`: `String.prototype` `toUpperCase`, `toLowerCase`, `toLocaleUpperCase`,
  `toLocaleLowerCase`, `normalize`, `localeCompare`, `trim`, `trimStart`, `trimEnd`, `padStart`,
  `padEnd`, `slice`, `substring`, `substr`, `replace`, `replaceAll`, `split`, `at`, `charAt`,
  `charCodeAt`, `codePointAt`, `startsWith`, `endsWith`, `includes`, `indexOf`, `lastIndexOf`,
  `repeat`.
- `regexp`: `RegExp.prototype.test` and `exec`; `String.prototype.match`, `matchAll` and
  `search`.
- `number`: `parseInt`, `parseFloat`, the `Number` constructor, and `Number.prototype.toFixed`,
  `toPrecision`, `toExponential`, `toString`, `toLocaleString`.
- `json`: `JSON.parse`, `JSON.stringify`.
- `intl`: the `Intl` constructors `Collator`, `DateTimeFormat`, `NumberFormat`, `PluralRules`,
  `RelativeTimeFormat`, `ListFormat`, `Segmenter`, `DisplayNames`; `Collator.prototype.compare`,
  `DateTimeFormat.prototype.format` and `NumberFormat.prototype.format`; and the methods
  `formatToParts`, `formatRange`, `formatRangeToParts`, `select`, `selectRange`, `segment` and
  `of`.
- `date`: the `Date` constructor, `Date.UTC`, `Date.parse`, every `Date.prototype` getter and
  setter, and its formatters.
- `sort`: `Array.prototype.sort` and `toSorted`, recorded with the receiver and the comparator.

The trace also wraps the opaque callees that the static value-flow check finds. Storage and
table reads stay unwrapped: array `push` and `unshift`, and every method of `Map`, `Set`,
`WeakMap` and `WeakSet`.

`.github/workflows/test-scenario-map.yml` runs the pipeline every Monday and on manual dispatch
(inputs `seed` and `shards`). Its jobs are `plan`, `value-flow`, `shards`, a `shard` matrix,
`merge` and `upsert-issue`. Every job is advisory (`continue-on-error`). Only a complete run from
the default branch uploads the `scenario-map-baseline` artifact and replaces the body of the
tracking issue (label `test-scenario-map`), unless the issue already records a newer run. A
partial run adds a comment and leaves the body alone. The workflow never closes the issue.

Mark a human keep with `// dupe-ok: <reason>` on either test of a pair, on the test call's line
or the line above. The report then lists the pair as kept, with its evidence.

## Regression testing

Re-run the full relevant suite when:

- Prompt or agent-behavior documentation changes (if tests encode behavior)
- Model or embedding-related code paths change
- Graph schema, query contracts, or MCP tool shapes change
- Dependencies with parsing or runtime impact upgrade

## User acceptance / beta (optional)

For staged releases or UI betas: deploy to a staging environment, collect structured feedback, watch errors and latency, then iterate before a wider release.